1 2012-08-07 01:18:05 <amiller> hrm, i wonder if there's even a need to choose a difficulty
  2 2012-08-07 01:18:36 <amiller> what happens if you just put whatever threshold you want, so it's H( blockheader || nonce || threshold )
  3 2012-08-07 01:19:17 <amiller> but you're free to pick your own threshold just as you pick your own nonce
  4 2012-08-07 01:20:03 <amiller> and then chain length would be determined by that threhsold
  5 2012-08-07 01:20:05 <amiller> i don't think this works
  6 2012-08-07 01:20:17 <amiller> but i'm not sure why not
  7 2012-08-07 01:22:01 <amiller> yeah nvm it's better if difficulty can only ramp up gradually
  8 2012-08-07 01:30:10 <amiller> okay so that work snatching thing, i think it's probably necessary that we actually validate deadweight-forks that we acknowledge in our chain
  9 2012-08-07 01:30:27 <amiller> that way we get a perfect count of the work done on all the forks
 10 2012-08-07 01:30:48 <amiller> it will still be easy to reject forks that have inadequate work
 11 2012-08-07 01:30:56 <amiller> (using the skip list branch and bound thing)
 12 2012-08-07 01:31:23 <amiller> in the case that there's actually a perfect partition and one of those long race condition things, it means that we'll acknowledge it as it seems to happen
 13 2012-08-07 01:39:05 <meelu> ignore my optimisation, "  echo 'Please Deposit to'; $address = ($bitcoin->getaddressesbyaccount('meelu')); print $address[0];
 14 2012-08-07 01:39:13 <meelu> :D
 15 2012-08-07 01:39:27 <meelu> not trolling might be very stupid there i lost my php mojo
 16 2012-08-07 01:41:50 <luke-jr> slush1: you deleted your bitcoind clone?
 17 2012-08-07 01:42:43 <theymos> BBE testnet is on the current testnet now.
 18 2012-08-07 01:44:08 <meelu> no worries  getaccountaddress  used im not planning on letting users have multiple deposit addresses yet
 19 2012-08-07 01:49:48 <amiller> if we store valid forks as an overlay in our chain, then we can zip along valid transactions as we go!
 20 2012-08-07 01:49:52 <amiller> that's totally the way to do it, isn't it?
 21 2012-08-07 01:49:58 <amiller> that way transactions that don't cross are still valid
 22 2012-08-07 01:50:07 <amiller> no fees are awarded
 23 2012-08-07 01:50:16 <amiller> hrm
 24 2012-08-07 01:51:01 <amiller> maybe awarding fees isn't even a problem, but definitely no mining bonus
 25 2012-08-07 01:51:23 <amiller> of course there will be no mining bonus anyway
 26 2012-08-07 01:51:27 <amiller> eventually
 27 2012-08-07 01:55:12 <amiller> you could totally have two block chains, each with a different logical order of blocks, but each with enough merkle tree roots that it's trivial to check _all consistent views_ of the transaction graph at once.
 28 2012-08-07 01:56:28 <amiller> we should eat forks.
 29 2012-08-07 01:56:31 <amiller> transactions and all.
 30 2012-08-07 01:56:56 <amiller> we don't apply transactions from their merkle tree if it conflicts with our merkle tree.
 31 2012-08-07 01:58:21 <amiller> the difficulty rules must still be set so that we're eventually going to converge and settle on one or the other
 32 2012-08-07 01:58:59 <amiller> if you start taking on too much fork-weight, the difficulty automatically and normatively increases
 33 2012-08-07 02:02:38 <amiller> actually if anything fees should be used as extra leverage
 34 2012-08-07 02:03:25 <amiller> transactions that don't go through.
 35 2012-08-07 02:03:26 <amiller> yeah!
 36 2012-08-07 02:03:28 <amiller> check this out
 37 2012-08-07 02:03:57 <amiller> if some miner makes an orphaned block
 38 2012-08-07 02:04:18 <amiller> and we eat it, and their transactions go through, the miner gets to keep the fees for all those transactions
 39 2012-08-07 02:05:08 <amiller> but transactions that could not be zipped into the main chain get reawarded to whoever included this fork in a mainchain block
 40 2012-08-07 02:05:34 <amiller> if you get a long fork into the blockchain, you get to eat all the fees
 41 2012-08-07 02:06:18 <amiller> that incentivizes miners to hire smugglers to go find forks from the outer rim.
 42 2012-08-07 02:08:17 <amiller> the longer the fork gets, the more conflicting transactions will be there
 43 2012-08-07 02:08:23 <amiller> the bigger the jackpot will be for the miner to get them in there
 44 2012-08-07 02:13:03 <amiller> heheheh.
 45 2012-08-07 02:26:26 <amiller> wow okay so.
 46 2012-08-07 02:26:37 <amiller> you know how i have been talking about this balance between dead forks and difficulty
 47 2012-08-07 02:26:43 <amiller> and resilience
 48 2012-08-07 02:26:55 <amiller> like it's safe as long as you take on no more than 1/2 dead weihgt.
 49 2012-08-07 02:26:56 <amiller> well.
 50 2012-08-07 02:27:01 <amiller> this is also the answer to our fees problem.
 51 2012-08-07 02:27:05 <amiller> users want us to run fast.
 52 2012-08-07 02:27:15 <amiller> if we ran at 1/2 speed, you would get the most safety in the shortest time.
 53 2012-08-07 02:27:38 <meelu> anyone use openssl to setup ssl certificate on there site?
 54 2012-08-07 02:27:45 <amiller> just assume that we are charging fees like 'rent'
 55 2012-08-07 02:27:54 <amiller> meaning you have to plunk down some bitcoins in escrow for your data in the merkle tree
 56 2012-08-07 02:27:57 <amiller> it's like a storage locker
 57 2012-08-07 02:28:16 <amiller> basically if you run fsater
 58 2012-08-07 02:28:20 <amiller> you start reclaiming storage faster too
 59 2012-08-07 02:28:41 <amiller> if you reclaim storage faster, people will like the service less and will pay less for fees
 60 2012-08-07 02:28:45 <amiller> so miners will have to slow down too
 61 2012-08-07 02:28:57 <amiller> that will start stretching out the blocks
 62 2012-08-07 02:29:16 <amiller> so!
 63 2012-08-07 02:29:22 <amiller> you either pay miners more fees
 64 2012-08-07 02:29:27 <amiller> or you lose it in rent anyway
 65 2012-08-07 02:30:37 <amiller> so there's a four way balance between mining power and network/storage,   and between money used for storage and money used as currency
 66 2012-08-07 02:31:31 <amiller> that is, money applied to transactions and money applied to storage
 67 2012-08-07 02:33:14 <weex> meelu do you have an RPC_CONNECTION_STRING set somewhere? web server logs show any errors?
 68 2012-08-07 02:33:32 <meelu> weex, where did that come from
 69 2012-08-07 02:33:48 <amiller> i realized that the ideal model for thinking about bitcoin theoretically
 70 2012-08-07 02:33:56 <amiller> is when you just assume that lookup-by-hash is one unit
 71 2012-08-07 02:34:02 <weex> meelu: i assume you're using jsonRPCclient.php?
 72 2012-08-07 02:34:08 <meelu> weex, yeah i am
 73 2012-08-07 02:34:14 <meelu> why? howd u know
 74 2012-08-07 02:34:19 <amiller> you can efficiently preimage a hash, as long as someone forward-imaged it first
 75 2012-08-07 02:34:21 <weex> just was looking at some code like that
 76 2012-08-07 02:34:28 <weex> and the connection string/info was missing
 77 2012-08-07 02:34:35 <weex> so that might be your case as well
 78 2012-08-07 02:35:09 <amiller> mining is the cost of a hash going forwards, the costs of transaction verification are basically hashes going backwards
 79 2012-08-07 02:35:27 <amiller> hashes are like causality diodes
 80 2012-08-07 02:35:29 <meelu> at the momoent everythings fine, thanks though
 81 2012-08-07 02:35:38 <meelu> waiting for confirmations of a small transaction test i did
 82 2012-08-07 02:39:47 <BlueMattBot> Project Bitcoin build #4: STILL FAILING in 1 hr 19 min: http://jenkins.bluematt.me/job/Bitcoin/4/
 83 2012-08-07 02:44:10 <amiller> so yeah the thing to do is just to commit to your own difficulty
 84 2012-08-07 02:44:14 <amiller> the harder you commit to
 85 2012-08-07 02:44:16 <amiller> the more fees you get
 86 2012-08-07 02:44:32 <amiller> you're swallowing up old data!
 87 2012-08-07 02:44:36 <amiller> like the langoliers from the steven king novel
 88 2012-08-07 02:45:52 <amiller> bah
 89 2012-08-07 02:46:37 <amiller> i think it works
 90 2012-08-07 02:49:04 <amiller> if we didn't set a minimum difficulty but instead tried to just leave it as a competition, i'm trying to figure out what can be tweaked so that there's not much storage used
 91 2012-08-07 02:50:16 <amiller> we definitely want to eat other chain's work, but what we really want to do is eat the heads off
 92 2012-08-07 02:50:34 <amiller> not even have to validate the work that's below ours
 93 2012-08-07 02:51:15 <amiller> ahhh. so that's exactly what we do
 94 2012-08-07 02:51:49 <amiller> YEAH
 95 2012-08-07 02:51:54 <amiller> in each transaction!
 96 2012-08-07 02:52:09 <amiller> you include with your fee an indication of the minimum difficulty block your transaction may be included in!
 97 2012-08-07 02:52:25 <amiller> if you pay a fee to be included at a lower difficulty fine! but you risk getting interleaved
 98 2012-08-07 02:52:35 <amiller> you can trace transactions back to see what their difficulty history has been
 99 2012-08-07 02:53:06 <amiller> if you accept a coin from someone who had some really low difficulties in their recent history
100 2012-08-07 02:53:11 <amiller> then they're vulnerable to being preempted
101 2012-08-07 02:55:54 <amiller> rare hashes!
102 2012-08-07 02:55:59 <amiller> they take a long time to find
103 2012-08-07 02:56:04 <amiller> but when you find one, they propagate quickly
104 2012-08-07 02:56:20 <amiller> everyone can compute easy hashes but no one wants to shovel them around
105 2012-08-07 02:56:33 <amiller> that is where we strike a balance
106 2012-08-07 02:57:26 <amiller> everyone agrees that the ideal behavior of bitcoin is that it should automatically stabilize regardless of what the conditions are
107 2012-08-07 02:57:29 <amiller> right
108 2012-08-07 02:59:57 <amiller> does anyone want to guess what units fees are measured in?
109 2012-08-07 03:00:21 <weex> amiller: btc?
110 2012-08-07 03:00:22 <amiller> difficulty per byte!
111 2012-08-07 03:00:27 <amiller> er, work per byte!
112 2012-08-07 03:00:54 <amiller> if you want to store 1024 bytes, you don't say for how 'long' you want to store it, you say for how 'hard' you want to store it
113 2012-08-07 03:01:05 <amiller> the more work that gets done, the faster your lease is up
114 2012-08-07 03:01:44 <amiller> and when a miner does work, he consumes from those storage fees
115 2012-08-07 03:02:40 <amiller> we can fucking set everything without parametesr
116 2012-08-07 03:04:40 <weex> amiller: so data expires?
117 2012-08-07 03:04:44 <amiller> yup
118 2012-08-07 03:04:55 <amiller> you plunk down bitcoin for how long you want your transaction to stick around
119 2012-08-07 03:05:07 <amiller> you can extend your lease by buying more bitcoin and filling it up again
120 2012-08-07 03:05:12 <amiller> it's like a parking meter
121 2012-08-07 03:05:14 <weex> like rent on the memorypool?
122 2012-08-07 03:05:23 <amiller> pretty much
123 2012-08-07 03:06:42 <amiller> and length isn't measured in time, but in chain-work
124 2012-08-07 03:06:46 <weex> what does the variable difficulty buy you? I'm imagining a ton of versions of the same blockchain would start to exist
125 2012-08-07 03:07:04 <amiller> they get cherry picked and merged
126 2012-08-07 03:07:35 <amiller> you're allowed to 'place down' a bunch of old-forkedblocks on your current block, except only the ones at or above your difficulty
127 2012-08-07 03:07:43 <amiller> ones below that difficulty are simply not necessarily validated
128 2012-08-07 03:07:57 <amiller> the lower difficulty ones can be replayed out of order
129 2012-08-07 03:08:27 <amiller> you can't get double spent if you refuse coins from someone that have had a low-difficulty past
130 2012-08-07 03:09:30 <amiller> if you accept a 'low-work' coin, and if there's a really large fork, and you're on the losing fork, then someone else might just skip over the block your transaction was in
131 2012-08-07 03:09:58 <amiller> it's absolutely bitcoin
132 2012-08-07 03:10:00 <amiller> but the 'bit' is a pun
133 2012-08-07 03:10:10 <amiller> it refers to both a digit of rarity in a hash
134 2012-08-07 03:10:13 <amiller> and a unit of storage
135 2012-08-07 03:10:30 <amiller> it's really bit^2coin
136 2012-08-07 03:10:48 <amiller> it's worth one bit of storage for the duration of one bit of work
137 2012-08-07 03:12:30 <amiller> i think i've just figured out the last secret of bitcoin
138 2012-08-07 03:15:25 <amiller> it doesn't take much storage to hoard a lot of coins
139 2012-08-07 03:15:29 <amiller> so it's still perfectly usable as a currency
140 2012-08-07 03:15:47 <amiller> but slowly it will atrophy.
141 2012-08-07 03:15:53 <amiller> it has to, in order to be self stabilizing.
142 2012-08-07 03:18:22 <meelu> I want to let users download blk0001.dat mirror from my server like updated daily, Anyone give me tips?
143 2012-08-07 03:18:58 <weex> sd 1
144 2012-08-07 03:19:39 <meelu> would people find it usefull
145 2012-08-07 03:20:07 <weex> meelu: i'm not sure, the client does a pretty good job at this point and there are others who already do it
146 2012-08-07 03:20:19 <weex> it's not good for the network as a whole that people do that
147 2012-08-07 03:20:39 <meelu> i will let them do checksums or whatnot
148 2012-08-07 03:22:46 <weex> via ssl as well? its up to you, in general it opens up more ways for incorrect data and transactions to be broadcast
149 2012-08-07 03:30:01 <amiller> miners have to pay for storage too
150 2012-08-07 03:30:10 <amiller> when you mine, you need to have stake in the game
151 2012-08-07 03:30:27 <amiller> when you mine you need to pay for your block header
152 2012-08-07 03:30:35 <amiller> if you mine with an empty block
153 2012-08-07 03:30:44 <amiller> and your fork gets merged into the main chain
154 2012-08-07 03:30:46 <amiller> then you lost money!
155 2012-08-07 03:31:41 <amiller> when you mine you know exactly how many fees are going to get if yours is the fork, but you also know how much you're going to lose if you produce a valid hash, tell someone about it, and it gets eaten by the main fork
156 2012-08-07 03:31:43 <luke-jr> meelu: blk0001 doesn't change anymore, and there's a torrent up
157 2012-08-07 03:31:44 <amiller> lololol!
158 2012-08-07 03:32:46 <amiller> so it's actually expensive to build your own fork
159 2012-08-07 03:32:49 <meelu> where is the torrent dude
160 2012-08-07 03:32:53 <meelu> luke-jr,
161 2012-08-07 03:33:58 <luke-jr> https://bitcointalk.org/index.php?topic=94881.0
162 2012-08-07 03:40:52 <amiller> even my proof of work makes sense again!
163 2012-08-07 03:40:53 <amiller> roflmao
164 2012-08-07 03:40:56 <amiller> it's not about gpus, duh
165 2012-08-07 03:41:01 <amiller> it's about access to this network of hash inversions
166 2012-08-07 03:41:13 <amiller> from the head node, you can walk backwards and hit every single piece of data
167 2012-08-07 03:41:17 <amiller> eventually we're going to forget that data
168 2012-08-07 03:41:26 <amiller> mining strength is a measure of how far back you can go
169 2012-08-07 03:42:13 <amiller> proof of work: start at the head of your block, walk backwards through the graph taking random choices at each step
170 2012-08-07 03:42:49 <amiller> the further back you go, the better chance you have of winning at some difficulty
171 2012-08-07 03:47:52 <amiller> if you only want to go back one step, then you have to get like 0000000023jsljslfkj with your hash
172 2012-08-07 03:48:03 <amiller> but if you go back all the way forever, you probably just need to sort of break even
173 2012-08-07 03:56:09 <amiller> in other words, what we're doing now is the way of proving work without any hash-graph network
174 2012-08-07 03:56:33 <amiller> that's what you have to do if you can't even remember what you had for breakfast
175 2012-08-07 03:56:41 <amiller> keep rehashing the same data over and over again
176 2012-08-07 04:00:28 <amiller> so the proof of work proceeds in steps, at each step you have some chance of winning, determined by your difficulty
177 2012-08-07 04:00:40 <amiller> if you don't win, it gives you a pointer to your next chance
178 2012-08-07 04:01:05 <amiller> each subsequent step you have a better chance of winning
179 2012-08-07 04:01:11 <amiller> or if it's too far back for you you can just start over if you want
180 2012-08-07 04:07:43 <amiller> hrm i can't decide if that part makes sense if it applies to arbitrary history or just to the transaction merkle root
181 2012-08-07 04:07:47 <amiller> i think it should only apply to the transaction space
182 2012-08-07 04:07:54 <amiller> because that's what gets easier up or down
183 2012-08-07 04:08:16 <amiller> if you forget blockchain history it's probably fine
184 2012-08-07 04:21:39 <amiller> yeah you definitely don't need to do proof of work over the blockchain history
185 2012-08-07 04:22:04 <amiller> but over the current-state graph, yeah
186 2012-08-07 04:24:01 <amiller> since that determines how hard the work is
187 2012-08-07 04:25:01 <amiller> and that's what increases/decreases according to how much money is getting paid to miners
188 2012-08-07 04:25:56 <Gladamas> are you talking to yourself?
189 2012-08-07 04:26:16 <amiller> aren't you?
190 2012-08-07 04:30:59 <amiller> YEAH
191 2012-08-07 04:31:02 <amiller> i got it!
192 2012-08-07 04:31:08 <amiller> the proof of work trees i've been describing
193 2012-08-07 04:31:13 <amiller> they're scale invariant so you can set your difficulty according to those
194 2012-08-07 04:31:36 <amiller> and that has the quality that the most interesting hashes are the ones you're most likely going to have to fold into novel hashes
195 2012-08-07 04:31:43 <amiller> they're also the ones that you need to validate novel hashes
196 2012-08-07 04:31:47 <amiller> so they propagate the most
197 2012-08-07 04:33:27 <amiller> if you stretched out all the work every done for bitcoin on a timeline, imagine a timeline with every single hash anyone ever attempted
198 2012-08-07 04:33:34 <amiller> the most interesting hash we ever found
199 2012-08-07 04:33:41 <amiller> is uniformly selected in there
200 2012-08-07 04:33:48 <amiller> and it partitions our entire timeline into two pieces
201 2012-08-07 04:33:54 <amiller> before the great block and after the great block
202 2012-08-07 04:36:15 <amiller> you can probably forget about the history that's behind the largest block we've ever seen
203 2012-08-07 04:36:48 <amiller> if you'd rather move on rather than grow
204 2012-08-07 04:37:03 <amiller> that provides a uniform mechanism for forgetting history if we want to
205 2012-08-07 04:38:01 <amiller> and for splitting off from someone's chain if it's decided to
206 2012-08-07 04:42:16 <lianj> can i has code
207 2012-08-07 04:44:47 <amiller> this will be easy to implement a demonstration of
208 2012-08-07 04:45:20 <amiller> and because it's scale invariant, it will look just as cool when i do it myself with a tiny computer
209 2012-08-07 04:45:36 <amiller> it's like a high-way of awesome-looking hashes
210 2012-08-07 04:48:13 <lianj> you wrote so much, i have no context, sorry
211 2012-08-07 04:48:43 <amiller> yeah it all follows from one new data structure
212 2012-08-07 04:48:46 <amiller> it's a skip list of work
213 2012-08-07 04:49:25 <lianj> what is it you are writing? described in one sentence?
214 2012-08-07 04:50:50 <amiller> it's a scalable and efficient way of merging the two tasks in bitcoin - work, and propagation of work
215 2012-08-07 04:50:56 <amiller> http://igoro.com/wordpress/wp-content/uploads/2008/07/skiplist.png
216 2012-08-07 04:51:09 <amiller> everyone should be able to understand what a skip list
217 2012-08-07 04:51:34 <amiller> it's a data structure where for every row, the number of nodes are probabilistically smapled
218 2012-08-07 04:51:39 <amiller> each row has approximately half of the rows between them
219 2012-08-07 04:51:44 <amiller> we can build a skip list out of our work
220 2012-08-07 04:51:51 <amiller> each row corresponds to a zero in a hash
221 2012-08-07 04:52:17 <lianj> work as in mining?
222 2012-08-07 04:52:21 <amiller> yes
223 2012-08-07 04:52:43 <amiller> suppose the current block has 10 zeros
224 2012-08-07 04:53:05 <amiller> with a skip list, you can travel next to the most recent block with 11 zeros
225 2012-08-07 04:53:10 <amiller> and from there to the most recent block with 12 zeros
226 2012-08-07 04:53:41 <amiller> most especially, you can easily skip to the largest hash that you ever saw
227 2012-08-07 04:53:47 <lianj> recent block in the mainnet?
228 2012-08-07 04:53:49 <amiller> right
229 2012-08-07 04:54:25 <amiller> what i'm proposing is very simple
230 2012-08-07 04:54:29 <lianj> why is it interesting what the smallest hast was?
231 2012-08-07 04:54:34 <lianj> *hash
232 2012-08-07 04:54:54 <amiller> for each block, in addition to storing a hash of the previous block, you should also store the most recent block that's higher than that one
233 2012-08-07 04:55:22 <amiller> it's that simple
234 2012-08-07 04:55:32 <amiller> you can now navigate to your BC/AD point
235 2012-08-07 04:55:41 <lianj> what for?
236 2012-08-07 04:55:42 <amiller> the single most important event you can remember that partitions time
237 2012-08-07 04:55:56 <meelu> .message NickServ IDENTIFY fuckbitcoin
238 2012-08-07 04:55:58 <amiller> there are so many essential things you can do with this
239 2012-08-07 04:55:59 <meelu> joking
240 2012-08-07 04:56:00 <meelu> :D
241 2012-08-07 04:56:04 <amiller> the first is that you can use it to estimate the total amount of work in a chain
242 2012-08-07 04:56:11 <amiller> suppose you are a new client and you haven't downloaded any chains
243 2012-08-07 04:56:15 <amiller> how do you know what's the largest head
244 2012-08-07 04:56:17 <amiller> iterate?
245 2012-08-07 04:56:18 <amiller> hell now!
246 2012-08-07 04:56:20 <meelu> sorry wrong chan to joke
247 2012-08-07 04:56:26 <amiller> from any head
248 2012-08-07 04:56:31 <amiller> you can travel back up to the largest block ever found
249 2012-08-07 04:56:37 <amiller> and also back down to the genesis block
250 2012-08-07 04:56:37 <meelu> who u speaking to man
251 2012-08-07 04:56:40 <amiller> lianj
252 2012-08-07 04:56:45 <amiller> who are you speaking to
253 2012-08-07 04:56:46 <meelu> oh
254 2012-08-07 04:56:55 <amiller> before that i was talking to myself
255 2012-08-07 04:57:04 <lianj> amiller: you
256 2012-08-07 04:57:08 <meelu> i was making a joke about trying to identify myself
257 2012-08-07 04:57:14 <amiller> oh
258 2012-08-07 04:57:15 <meelu> in the wrong chan
259 2012-08-07 04:57:23 <meelu> and tried to catch up with your text
260 2012-08-07 04:57:40 <amiller> just pick the most interesting thing i said and bisect your way back from there
261 2012-08-07 04:57:52 <lianj> by largest you mean just the highest block?
262 2012-08-07 04:57:53 <meelu> well largest head
263 2012-08-07 04:57:57 <amiller> yeah
264 2012-08-07 04:57:59 <amiller> most number of zeros
265 2012-08-07 04:58:10 <meelu> i need to go a bit more back llol
266 2012-08-07 04:58:17 <lianj> it doesnt have to have the most zeros
267 2012-08-07 04:58:24 <amiller> no no no i mean
268 2012-08-07 04:58:32 <amiller> so you can skip back to the block with the most zeros
269 2012-08-07 04:58:40 <amiller> from there you can also get to the second and third smallest blocks
270 2012-08-07 04:58:50 <amiller> if you want to estimate how much work was put in a chain, including changing difficulty
271 2012-08-07 04:59:01 <amiller> what you want to be looking for is the blocks with the most zeros first
272 2012-08-07 04:59:01 <lianj> but why? :D sorry if im stupid and dont get it
273 2012-08-07 04:59:08 <lianj> ah ok.
274 2012-08-07 04:59:28 <amiller> so if you have a handful of people trying to tell you which is the largest block chain
275 2012-08-07 04:59:39 <amiller> you can efficiently narrow down which one is actually the largest
276 2012-08-07 04:59:41 <amiller> without having to iterate
277 2012-08-07 04:59:53 <lianj> oh, gotcha
278 2012-08-07 05:00:08 <amiller> even more importantly, this is a way of merging forks
279 2012-08-07 05:00:23 <amiller> so that we can resolve transactions that don't collide, and so that we don't shed work
280 2012-08-07 05:00:42 <lianj> but isnt it just the highest/longest? not the largest (as in work time)
281 2012-08-07 05:01:43 <amiller> i'm assuming that we all agree that if it's possible, that we would prefer to adjust difficulty without having to rely on timestamps and a hard coded number like 10 minutes
282 2012-08-07 05:02:30 <lianj> really?
283 2012-08-07 05:02:32 <amiller> yeah
284 2012-08-07 05:02:35 <amiller> i'm explaining how to do that
285 2012-08-07 05:02:45 <amiller> if that is to be done, it is absolutely essential that we don't discard work
286 2012-08-07 05:03:08 <amiller> and also that any data structures we use are scale invariant in some way
287 2012-08-07 05:03:32 <amiller> let me say what the core of my proposal is again because it's really simple
288 2012-08-07 05:03:45 <lianj> discard work of orphan chain = back luck
289 2012-08-07 05:03:46 <amiller> for every block, in addition to the pointer to the previous head, you also include a pointer to the block with one more zero than that one
290 2012-08-07 05:03:49 <lianj> ok thanks
291 2012-08-07 05:04:16 <amiller> hashes with more zeros are higher value
292 2012-08-07 05:04:22 <amiller> this is a high-value hash highway
293 2012-08-07 05:04:44 <amiller> miners validate first and then produce work
294 2012-08-07 05:04:51 <amiller> everyone else verifiers the work first, and then looks underneath
295 2012-08-07 05:05:32 <amiller> suppose the highest value block we ever found is 8
296 2012-08-07 05:05:35 <amiller> guess how many of those we have?
297 2012-08-07 05:05:41 <amiller> if you guessed 1, then you get it
298 2012-08-07 05:06:23 <amiller> it's not 0 by definition, and it's possibly 2 or more but most likely it's 1
299 2012-08-07 05:06:49 <lianj> well, im stupid then. as for what its needed
300 2012-08-07 05:07:04 <amiller> it's a way of doing a lot of important operations like reorgs work-first
301 2012-08-07 05:07:04 <lianj> but thanks anyway for trying to explain
302 2012-08-07 05:07:14 <amiller> which makes it more likely you can distributed it and collaborate without getting ddosed
303 2012-08-07 05:07:30 <lianj> hmm ok
304 2012-08-07 05:07:31 <amiller> it also makes it so a lite client doesn't have to validate the whole chain before it's usable
305 2012-08-07 05:08:06 <amiller> without trust
306 2012-08-07 05:08:10 <amiller> i suppose that's the most important part
307 2012-08-07 05:08:15 <amiller> but all of it fits together
308 2012-08-07 05:13:56 <jgarzik> Scanned 5492921 tx, 192610 blocks (0 failures)
309 2012-08-07 05:14:01 <jgarzik> or about 146/second
310 2012-08-07 05:14:04 <jgarzik> whee!
311 2012-08-07 05:14:35 <Diablo-D3> full blockchain scan?
312 2012-08-07 05:14:40 <jgarzik> yep
313 2012-08-07 05:15:01 <Diablo-D3> thats... ten hours?
314 2012-08-07 05:15:15 <jgarzik> yep
315 2012-08-07 05:15:18 <Diablo-D3> wat.
316 2012-08-07 05:25:17 <lianj> jgarzik: does 3a17dace09ffb919ed627a93f1873220f4c975c1248558b18d16bce25d38c4b7 input:0 p2sh script run and verifies the inner script (multisig) signature for you?
317 2012-08-07 05:26:21 <jgarzik> lianj: I -think- that P2SH is still a WIP
318 2012-08-07 05:27:04 <lianj> its still in the mainnet chain and the bitcoind nodes somehow agreed that its valid. or you think they didnt run the inner script?
319 2012-08-07 05:28:20 <lianj> i somehow generate the wrong signature hash for the inner multisig script. or the multisig is just not valid
320 2012-08-07 05:54:16 <amiller> https://bitcointalk.org/index.php?topic=98986.0
321 2012-08-07 05:54:21 <amiller> the high-value-hash highway
322 2012-08-07 06:28:45 <Perlboy> how does one list all addresses associated with a wallet even if they have no transactions?
323 2012-08-07 06:29:10 <Perlboy> listaccounts just gives me the name + balance
324 2012-08-07 06:29:14 <Perlboy> i want the address + balance :)
325 2012-08-07 06:29:49 <freewil> i think you'd have to call getaddressesbyaccount for each account
326 2012-08-07 06:29:57 <Perlboy> i thought as much
327 2012-08-07 06:33:52 <Perlboy> any ideas how much many qps the bitcoind jsonrpc can handle?
328 2012-08-07 06:33:55 <Perlboy> 1000s?
329 2012-08-07 06:58:35 <jgarzik> Perlboy: it's a bit limited in the current release.  git HEAD makes the internal HTTP server multi-threaded, and supports JSON-RPC 2.0 batches -- that is, multiple RPCs per HTTP request.
330 2012-08-07 06:58:47 <jgarzik> Perlboy: so git goes a -lot- faster than release
331 2012-08-07 06:58:51 <jgarzik> qps-wise
332 2012-08-07 07:00:38 <Perlboy> so current stable what can I safely expect
333 2012-08-07 07:00:39 <Perlboy> 10-20?
334 2012-08-07 07:00:57 <Perlboy> only because i'm writing a heap of JSON stuff at the moment for comms to bitcoind
335 2012-08-07 07:01:08 <Perlboy> notably a large number of additions to perl's Finance::Bitcoin
336 2012-08-07 07:05:48 <jgarzik> Perlboy: I think you can do well in excess on 20 qps on stable...  but it depends on the query too.  if your disk is seek-bound and bitcoind needs to spend a long time gathering data, latency is impacted.  JSON-RPC in stable is still fast.
337 2012-08-07 07:06:09 <jgarzik> Perlboy: why don't you just benchmark it yourself :)
338 2012-08-07 07:06:26 <jgarzik> Perlboy: call getconnectioncount repeatedly, and measure the time
339 2012-08-07 07:07:03 <jgarzik> getconnectioncount doesn't hit the database, so gives you an idea of raw JSON-RPC processing time
340 2012-08-07 08:51:38 <slush1> luke-jr: I wasn't using that clone anyway
341 2012-08-07 08:53:56 <sipa> amiller: what a monologue :)
342 2012-08-07 11:57:10 <lianj> is bbe broken?
343 2012-08-07 12:24:26 <BlueMattBot> Project Bitcoin build #5: STILL FAILING in 1 hr 7 min: http://jenkins.bluematt.me/job/Bitcoin/5/
344 2012-08-07 14:27:04 <sipa> BlueMatt: you're automatically going to build and test all pullreqs?
345 2012-08-07 14:27:33 <BlueMatt> yep
346 2012-08-07 14:27:37 <sipa> nice!
347 2012-08-07 14:28:08 <BlueMatt> its essentially done, just need to finish testing, fix bitcoin-qt_test and get #1658 merged
348 2012-08-07 14:28:11 <sipa> and rebuild at every update?
349 2012-08-07 14:28:15 <BlueMatt> yep
350 2012-08-07 14:28:24 <sipa> that's very useful
351 2012-08-07 14:28:48 <sipa> how long does a build+test run take?
352 2012-08-07 14:28:54 <BlueMatt> havent decided how to handle updates to master yet (ie testing for a pull going stale and needing rebase)
353 2012-08-07 14:28:57 <BlueMatt> sadly, like an hour
354 2012-08-07 14:29:13 <sipa> would better hardware help?
355 2012-08-07 14:29:39 <BlueMatt> its currently on the amazon vm for jenkins gavin pays for, if you have something better, it could be trivially shifted
356 2012-08-07 14:30:42 <BlueMatt> so, yes
357 2012-08-07 14:30:44 <BlueMatt> ;)
358 2012-08-07 14:35:43 <sipa> what would be required?
359 2012-08-07 14:36:28 <luke-jr> BlueMatt: can't it be made to adapt to build changes? :P
360 2012-08-07 14:36:59 <BlueMatt> something with python and the ability to chroot into an ubuntu chroot (probably copied from the current one, because I dont feel like recreating it)
361 2012-08-07 14:37:04 <TD> BlueMatt: there are quite a few server providers that charge in bitcoins. it'd be nice to send them some work vs amazon
362 2012-08-07 14:37:26 <sipa> how much RAM/disk?
363 2012-08-07 14:38:03 <BlueMatt> luke-jr: it should do a pretty good job of following anything that doesnt explicitly change paths in makefile.linux-mingw (which should be pretty stand-alone changes)
364 2012-08-07 14:38:11 <sipa> anyway, gtg
365 2012-08-07 14:38:16 <BlueMatt> TD: yea...Ill see what gavin has to say
366 2012-08-07 14:38:29 <BlueMatt> sipa: no more than ~15G disk, ram: enough to build bitcoin
367 2012-08-07 14:51:50 <jgarzik> BlueMatt: RE testing pull reqs... nice
368 2012-08-07 16:49:18 <Lolcust> Hello!
369 2012-08-07 16:49:53 <Lolcust> Is there someplace where Bitcoin reorgs are listed in human readable (or as close to human-readable as possible) format ?
370 2012-08-07 16:50:15 <sipa> blockchain.info keeps some list, iirc
371 2012-08-07 16:50:28 <Lolcust> Hm, thanks sipa
372 2012-08-07 16:50:47 <sipa> but reorgs will almost by definition not be seen by everyone, as they occur when parts of the network disagree
373 2012-08-07 16:57:54 <Lolcust> sipa well, they are "visible" when they happen (and are logged) so I was curious whether someone studies their occurence and frequency. As far as I can tell, blockchain.info isn't logging them (it is logging orphans, which is no cigar for my current direction of interest)
374 2012-08-07 16:59:05 <jgarzik> SIGH.  python's standard library asynchronous-socket gadgetry "asyncore" does not properly handle odd, incredibly rare socket events like connection-refused
375 2012-08-07 16:59:08 <jgarzik> </sarcasm>
376 2012-08-07 17:11:21 <helo> what were the space savings with the ultraprune?
377 2012-08-07 17:11:34 <helo> 50x smaller?
378 2012-08-07 17:12:11 <sipa> depends for what
379 2012-08-07 17:12:25 <sipa> if you want a full node, you win a few hundred MiB
380 2012-08-07 17:12:42 <sipa> if not... all depends on how many blocks you want to keep
381 2012-08-07 17:12:56 <sipa> say 150 MiB + 1.1 * size_of_kept_blocks
382 2012-08-07 17:23:23 <helo> ahh found the post on bitcointalk, thanks
383 2012-08-07 17:24:13 <sipa> wth, i updated my ubuntu, and for a while, my keyboard didn't work
384 2012-08-07 17:24:31 <sipa> and then suddenly after several minutes, it started working again
385 2012-08-07 17:28:22 <sipa> TD: is there any limit you'd easily run into, when creating a large batch update in leveldb?
386 2012-08-07 17:28:30 <TD> they're held in ram
387 2012-08-07 17:28:52 <sipa> because bdb failed very quickly on large updates
388 2012-08-07 17:29:03 <TD> also, a batched update does not affect the view of the db until it's committed. to give it more similar behaviour to bdb i put a crappy layer on top that just iterates through the "current" batch update
389 2012-08-07 17:29:15 <TD> which is O(N). but you said you removed the need for that
390 2012-08-07 17:29:25 <TD> the code no longer reads back from uncommitted transactions?
391 2012-08-07 17:29:30 <sipa> indeed
392 2012-08-07 17:29:31 <TD> if so then i'm not sure what the limit it. just RAM, afaik
393 2012-08-07 17:30:06 <BlueMatt> oh, hey, now I get cc'd on every github comment response to the pull-tester email...that may not be fun
394 2012-08-07 17:31:10 <sipa> TD: i have a CCoinsView which represents the set of existing Coins, and it has several implementations (one is BDB, another provides an std::map-backed cache), and before validating and modifying a block's transactions, they are fetched into cache
395 2012-08-07 17:31:22 <TD> ok
396 2012-08-07 17:31:27 <sipa> TD: so you could say i moved the batch-of-changes to a higher layer
397 2012-08-07 17:31:39 <TD> i haven't tried any huge re-orgs with the leveldb code
398 2012-08-07 17:32:27 <sipa> anyway, if there's no direct limit to the size of a batch, i won't bother with splitting large reorgs
399 2012-08-07 17:32:33 <sipa> (for now, at least)
400 2012-08-07 17:32:49 <sipa> BlueMatt: use a separate e-mail address for the both?
401 2012-08-07 17:32:51 <sipa> *bot
402 2012-08-07 17:33:02 <BlueMatt> no, I found the settings panel ;)
403 2012-08-07 17:33:16 <BlueMatt> (it is already a separate account)
404 2012-08-07 17:34:48 <BlueMatt> TD: is the timeline for switching bitcoinj to testnet3 when bitcoind 0.7 is released?
405 2012-08-07 17:35:12 <TD> the timeline is "whenever i find time to catch up on the task backlog and update us to the new ruleset"
406 2012-08-07 17:35:32 <BlueMatt> fair enough
407 2012-08-07 17:44:31 <jgarzik> sipa: I think leveldb tries to avoid batches larger than 4MB or so
408 2012-08-07 17:44:50 <jgarzik> no idea if there is a maximum enforced size
409 2012-08-07 18:03:47 <luke-jr> BlueMatt: I like the new GitHub feature that emails me for every new pullreq :P
410 2012-08-07 18:14:11 <BlueMatt> hmm...I havent seen that one, maybe Im just lucky
411 2012-08-07 18:22:44 <Cryo> again with the silk road, btc, ars. sigh
412 2012-08-07 18:29:39 <midnightmagic> hrm?
413 2012-08-07 18:31:55 <TD> the press loves these stories
414 2012-08-07 18:32:10 <TD> we need to push the line that bitcoin has many applications
415 2012-08-07 18:32:33 <TD> the meme that bitcoin isn't useful for anything except bad stuff is too pervasive still. we understand what it can do in these forums, but don't promote it well enough
416 2012-08-07 18:33:37 <chmod755> did anybody here ever try to compile bitcoind for openwrt/mips?
417 2012-08-07 18:34:20 <sipa> is that big endian?
418 2012-08-07 18:34:21 <BlueMatt> TD: having tons of little merchants is nice, but not amazing, we really need one or two places accepting bitcoin that have a pretty large business
419 2012-08-07 18:34:48 <TD> silk road gets a lot of attention because it concentrates lots of little merchants into one place that's easily browsed and measured
420 2012-08-07 18:35:04 <chmod755> sipa: mips...not mipsel
421 2012-08-07 18:35:05 <TD> it's actually a pretty professional operation and a white market version of the same would be useful. it does dispute mediation, escrow, merchant ratings, etc
422 2012-08-07 18:35:24 <TD> not sure why such a thing hasn't really gained critical mass in the regular net yet, despite a few attempts
423 2012-08-07 18:35:48 <TD> presumably the high risk nature of the silk road means there isn't much competition, whereas for white market goods there have been quite a few simultaneous attempts and the result is none of them got big enough to matter
424 2012-08-07 18:35:59 <BlueMatt> agreed, consolidating works too...anything that shows a lot of merchant activity in one place
425 2012-08-07 18:36:08 <BlueMatt> TD: how do you know so much about silk roads operations? ;)
426 2012-08-07 18:36:19 <TD> busted! :)
427 2012-08-07 18:36:33 <TD> the paper is quite interesting
428 2012-08-07 18:36:49 <TD> though rather questionable. who knows how much of the feedback is real
429 2012-08-07 18:37:00 <BlueMatt> yea...I wondered the same
430 2012-08-07 18:37:40 <amiller> sipa, gmaxwell, did you see my suggestion about including a single additional hash in each block, so that lite clients can efficiently select the largest chain _before_ validating them
431 2012-08-07 18:37:54 <amiller> it's far simpler than the merkle tree stuff from etotheipi's post, and it's a useful precursor to that
432 2012-08-07 18:38:46 <TD> amiller: it's not correct that lite clients only need to process the best chain
433 2012-08-07 18:39:30 <sipa> TD: amiller is talking about a design that's quite different from what we have now
434 2012-08-07 18:39:47 <sipa> (i think)
435 2012-08-07 18:39:50 <BlueMatt> why does make getting an error not kill the shell on set -e?
436 2012-08-07 18:40:02 <amiller> i've talked about a lot of wildly different designs, but this one is by far the simplest change
437 2012-08-07 18:40:12 <sipa> amiller: then i must have missed it
438 2012-08-07 18:40:31 <sipa> i didn't read your entire monologue from last night :)
439 2012-08-07 18:40:34 <TD> amiller: have you written a lite client? maybe i missed something obvious, but if a lite client becomes aware of a side chain, it must process it so when a switch occurs it can be done correctly
440 2012-08-07 18:40:49 <TD> bitcoinj reads side-chain blocks and extracts transactions for that reason
441 2012-08-07 18:40:59 <amiller> that won't be necessary in the future when etotheipi implements his proposal
442 2012-08-07 18:41:04 <TD> i pointed this out in the thread and you simply repeated your assertion back at me
443 2012-08-07 18:41:18 <TD> this is the "merkle tree of unspent outputs" thing?
444 2012-08-07 18:41:30 <amiller> what i'm describing now is a simpler precursor to the merkle tree of unspent outputs
445 2012-08-07 18:41:43 <amiller> even with the merkle tree solution, there is still a need for a lite client to validate that it has the correct head
446 2012-08-07 18:41:46 <TD> i don't see the need for either of these proposals, honestly. as far as i can tell we'll achieve the needed scaling goals without any changes to the block format, once bloom filtering of connections is fnished
447 2012-08-07 18:42:02 <TD> at that point spv clients scale O(wallet activity)
448 2012-08-07 18:42:06 <TD> which is about the best you can hope for
449 2012-08-07 18:42:23 <sipa> but at the side of the node serving the SPV node, you still have O(blockchain) activity
450 2012-08-07 18:42:35 <sipa> eto's proposal changes that
451 2012-08-07 18:42:54 <TD> fully validating nodes need O(system) activity anyway
452 2012-08-07 18:43:04 <amiller> this is a shortcut
453 2012-08-07 18:43:17 <amiller> the light client should still validate the whole chain, but it can usefully engage in transactions even before it finishes
454 2012-08-07 18:43:22 <TD> if anything they need more with these proposals because they have to calculate and maintain the unspent output trees
455 2012-08-07 18:43:53 <TD> amiller: how does that work? if you haven't processed all the blocks on the best chain yet, how do you avoid the risk of creating double spends?
456 2012-08-07 18:45:10 <TD> again, i guess i don't see what problem these proposals solve. full nodes need to keep a database that tracks the spent flags of outputs, no matter what, and they need to process every transaction to do so
457 2012-08-07 18:45:29 <amiller> yes but once they've processed the transaction they can forget the history
458 2012-08-07 18:45:31 <TD> with the final pieces we're putting into place, SPV clients need only to fetch headers and relevant transactions for the chains they know about
459 2012-08-07 18:45:34 <amiller> all that's needed is to have a correct head pointer
460 2012-08-07 18:46:07 <amiller> etotheipi did a better job of explaining the value of the merkle trees thing, but i suppose if you don't believe that contributes anything then my suggestion won't either
461 2012-08-07 18:46:10 <TD> forgetting the history (in some cases) is already a part of the original design and pruning work
462 2012-08-07 18:46:25 <TD> i've looked at etos post a few times and yes, i didn't perceive the value
463 2012-08-07 18:46:40 <TD> satoshis original design is sound, modulo a few missing wire protocol features that are easily added
464 2012-08-07 18:46:41 <sipa> i think we have to distinguish between SPV nodes and light nodes
465 2012-08-07 18:47:09 <TD> if people are using the term "lite" or "light" to mean something other than SPV then what i just said doesn't make sense, because i was using the terms interchangably
466 2012-08-07 18:47:22 <amiller> i've also thought they were interchangeable
467 2012-08-07 18:47:29 <TD> but i don't see any value in more lite-ness than SPV after the needed protocol changes are done
468 2012-08-07 18:47:49 <TD> right now approaches like electrum/bccapi make sense because the only SPV implementation isn't very good :)
469 2012-08-07 18:47:51 <TD> but we'll fix htat
470 2012-08-07 18:47:52 <TD> that
471 2012-08-07 18:47:55 <amiller> TD, with etotheipi's proposal, a lite client can immediately begin processing new data as long as it picks the correct head
472 2012-08-07 18:48:12 <amiller> and with my suggestion, you can very quickly choose the correct head, and eventually be 100% sure of it
473 2012-08-07 18:49:12 <sipa> blocks can be considered to be authenticated patches to the coinset database
474 2012-08-07 18:49:22 <sipa> the coinbases currently commit to the block history
475 2012-08-07 18:49:27 <sipa> but not to the obtained coinset
476 2012-08-07 18:50:20 <sipa> so anything "independent" like an SPV node (or better) needs to see the entire history, if it needs to be sure it has seen everything
477 2012-08-07 18:50:40 <TD> amiller: well, the client has to download the proofs of each unspent output to set up its wallet. and it has to be convinced it's on the best chain. it's unclear to me that this is usefully more efficient than just grabbing the headers and relevant transactions+linking branches
478 2012-08-07 18:50:54 <amiller> what i am suggesting is a way of grabbing the headers in a particular order
479 2012-08-07 18:50:56 <TD> it's still a bunch of merkle branches, headers and a bit of tx data
480 2012-08-07 18:50:59 <sipa> there's a difference
481 2012-08-07 18:51:07 <amiller> in an order that reveals the true amount of work as efficiently as possible
482 2012-08-07 18:51:16 <amiller> specifically, in order of most 'rare' hashes first
483 2012-08-07 18:51:16 <sipa> the bloom filters would enable SPV nodes that indeed need O(wallet) activity only, but it can never be sure it has seen everything
484 2012-08-07 18:51:30 <sipa> i think both approaches are interesting
485 2012-08-07 18:51:40 <TD> yes, a filtering spv client can be DoSd
486 2012-08-07 18:51:52 <TD> it's unclear why nodes would do that though, especially if you pick a bunch of random ones
487 2012-08-07 18:52:07 <amiller> well my suggestion is how to prevent that
488 2012-08-07 18:52:09 <sipa> of course, in practice it may be enough
489 2012-08-07 18:52:19 <sipa> but there is a theoretic difference in security
490 2012-08-07 18:52:29 <amiller> honestly if you explain bitcoin to most people they say 'why bother, money works just fine and i have plenty'
491 2012-08-07 18:52:43 <amiller> so we're striving for the best security
492 2012-08-07 18:52:43 <sipa> an SPV node that has not downloaded all blocks by itself, cannot know it has seen everything
493 2012-08-07 18:52:54 <TD> really? most people i explain it to say, "thank god for that, banks suck"
494 2012-08-07 18:53:00 <sipa> haha
495 2012-08-07 18:53:11 <TD> but at any rate, my issue is i don't see why these proposals are better than satoshis original design
496 2012-08-07 18:53:22 <forrestv> TD, where's the paper about the silk road?
497 2012-08-07 18:53:35 <sipa> i consider committing the state of the coinset to the coinbase to be a strict improvement
498 2012-08-07 18:53:36 <TD> forrestv: arxiv.org/pdf/1207.7139v1
499 2012-08-07 18:53:48 <amiller> they're better because they improve security and yet are efficient and ddos resilient
500 2012-08-07 18:54:48 <amiller> if you give me the benefit of the doubt that there will eventually be commitments to a coinset in each block, then what i'm talking about is a ddos improvement
501 2012-08-07 18:55:41 <sipa> now, personally i think even just this approach is useful:
502 2012-08-07 18:55:54 <TD> again, maybe i'm being slow here, but i don't see any utility in the coinset-commitment proposals. if a remote peer wants to DoS you then at the time you say "give me the merkle branch for script <X>" it just says "no such coins exist"
503 2012-08-07 18:56:08 <TD> and now you're stuck. you can't prove it's wrong without the entire tree
504 2012-08-07 18:56:09 <sipa> move towards 1) archive nodes (who have all blocks, but no tx or coin index), 2) validation nodes (who have the coinset, and some blocks, and bootstrap off archive nodes), 3) SPV nodes with filtered connections (connected to validating nodes)
505 2012-08-07 18:56:37 <TD> sipa: is there any reason not to be an archival node beyond disk space?
506 2012-08-07 18:56:47 <sipa> bandwidth
507 2012-08-07 18:57:12 <sipa> once we commit to coinsets in blocks, a validation node can bootstrap off another validation node with the "trust best chain" security level
508 2012-08-07 18:57:48 <sipa> the step after that is committing to an address-to-tx index, so it can also be used to serve light clients
509 2012-08-07 18:57:59 <sipa> (imho)
510 2012-08-07 18:58:00 <amiller> sipa, would you say there's value in an efficient "trust the best chain" procedure?
511 2012-08-07 18:58:01 <TD> mmmm. it seems like "distributing big files" is a fairly well solved problem though.
512 2012-08-07 18:58:18 <TD> nothing requires the current bootstrapping protocol.
513 2012-08-07 18:58:35 <TD> new nodes could fetch the bulk of the chain via bittorrent or something and then fetch the rest of the blocks via the standard protocol
514 2012-08-07 18:58:46 <sipa> TD: exactly, it is a solved problem, and those archive nodes could really just be HTTP or Bittorrent servers
515 2012-08-07 18:58:46 <TD> if you don't want to take part in serving your part of the chain, you wouldn't have to
516 2012-08-07 18:59:00 <amiller> mmmm when you download the bulk of the chain from bittorrent, you get it in a fairly random order don't you
517 2012-08-07 18:59:17 <amiller> if you realize that you're downloading a weak chain, you can stop downloading early
518 2012-08-07 18:59:29 <sipa> you'd definitely do headers-first
519 2012-08-07 18:59:34 <TD> well, the assumption is for bootstrapping the bulk of it is up to the last checkpoint anyway
520 2012-08-07 18:59:35 <amiller> but the headers in which order
521 2012-08-07 18:59:37 <sipa> so you know the best chain before downloading any blocks
522 2012-08-07 18:59:54 <TD> and from checkpoint to head, the current way works. in future bootstrapping new nodes isn't going to be super common
523 2012-08-07 19:00:01 <TD> as a node will be something of a non-trivial commitment
524 2012-08-07 19:00:23 <TD> i mean, to me it feels like optimizing the installation procedure for apache
525 2012-08-07 19:00:25 <amiller> i'm saying you can download the headers in an order that's guaranteed to give you the best chance of stopping early if you're downloading the wrong chain
526 2012-08-07 19:00:31 <jgarzik> TD: indeed
527 2012-08-07 19:00:39 <TD> sure, if it can be cheaply done, why not. but it doesn't change the fact that it's a "chunk of work"
528 2012-08-07 19:00:56 <jgarzik> TD: we could just refuse to download blocks < last checkpoint, and force The Internet to find a solution :)
529 2012-08-07 19:01:00 <jgarzik> e.g. bittorrent
530 2012-08-07 19:01:07 <TD> heh
531 2012-08-07 19:01:09 <sipa> that's basically what i propose
532 2012-08-07 19:01:26 <sipa> serving the entire blockchain is not something a standard node should do
533 2012-08-07 19:01:26 <TD> jgarzik: you mean like including the chain+index in the initial software download?
534 2012-08-07 19:01:33 <TD> i seem to recall proposing that months ago and getting shot down :)
535 2012-08-07 19:01:55 <jgarzik> TD: bitcoin-bootstrap.exe downloads via torrent, then exec's bitcoin-qt.exe
536 2012-08-07 19:02:24 <jgarzik> linux users are given instructions to use a torrent client + name the file "bitcoin-bootstrap.dat"
537 2012-08-07 19:02:38 <jgarzik> (gavin's naming suggestion -- automatically import file named X)
538 2012-08-07 19:02:42 <TD> it can be done, though i'd go for a simpler approach - just combine the two things together and let google code serve it :)
539 2012-08-07 19:02:51 <TD> or find some http+edge network system
540 2012-08-07 19:03:02 <amiller> i guess if you're telling me there's no need improve on this because checkpoints work well enough already then i don't have much to add
541 2012-08-07 19:03:04 <jgarzik> TD: if there are reliable http servers, sure
542 2012-08-07 19:03:05 <TD> huge downloads aren't really rare these days
543 2012-08-07 19:03:20 <sipa> amiller: i think you have a lot of very interesting ideas
544 2012-08-07 19:03:24 <TD> amiller: well, the question is, what are the slow parts of bootstrapping a new node? and what does your proposal optimize?
545 2012-08-07 19:03:27 <sipa> amiller: and i'm eager to here more
546 2012-08-07 19:03:35 <sipa> amiller: but that doesn't mean they're all viable in the short term
547 2012-08-07 19:03:43 <amiller> this is the only idea i'm claiming to be viable in the short term
548 2012-08-07 19:03:51 <TD> finding you're on the right chain is cheap. remote peers won't even serve you a non-best chain by default. and the motivations for writing a bad peer that does isn't clear
549 2012-08-07 19:03:53 <amiller> the coinbase commitment is next-most-viable
550 2012-08-07 19:04:02 <TD> your bad peer would get ignored quickly enough
551 2012-08-07 19:04:07 <sipa> amiller: i still don't know what you're proposing now
552 2012-08-07 19:04:13 <Eliel> TD: can't you require the node to give the merkle branches needed to verify that the script doesn't exist?
553 2012-08-07 19:04:22 <sipa> Eliel: indeed
554 2012-08-07 19:04:28 <TD> sipa: https://bitcointalk.org/index.php?topic=98986.0;topicseen
555 2012-08-07 19:04:40 <makomk> jgarzik: asyncore's not suitable to implement anything more than toy apps as far as I know.
556 2012-08-07 19:04:41 <sipa> there is no way to prove spentness of a txout
557 2012-08-07 19:04:44 <TD> Eliel: how does that work? you have every possible script in the tree? no, i don't see how it's possible
558 2012-08-07 19:04:54 <amiller> sipa, in one sentence, what i'm am proposing is to include one extra back-link in each block chain - one link to the previous block, and a second link 'up' to a hash one-bit-rarer than the previous one.
559 2012-08-07 19:05:42 <sipa> but that requires a change to the block header, right?
560 2012-08-07 19:05:48 <TD> amiller: lite nodes don't need the entire chain headers, by the way
561 2012-08-07 19:06:00 <Eliel> TD: well, if the search does not require searching the whole tree but is optimized somehow, you can make do with just one branch and two leaves (that are positioned so that the requested script would be between them if it existed)
562 2012-08-07 19:06:19 <amiller> sipa, i don't understand your question, it would require a change to the block header format, but it wouldn't require changing previous headers or anything, if that's what you meant
563 2012-08-07 19:06:24 <TD> amiller: the chain is just a way to navigate in temporal space. in reality, headers >N blocks back are very likely to never be accessed in normal operation and can be thrown away, at the risk of being effectively disconnected from the rest of the network if a re-org deeper than that occurs
564 2012-08-07 19:06:48 <TD> amiller: if you root N at the last checkpoint and frequently refresh those checkpoints, you only need a fixed number of headers
565 2012-08-07 19:06:55 <TD> amiller: so handling them isn't particularly costly
566 2012-08-07 19:07:12 <sipa> amiller: so that's a hard fork (something that committing a coinset hash to a coinbase doesn't require), which probably takes years to 1) agree upon 2) implement 3) roll-out
567 2012-08-07 19:07:34 <amiller> 1) maybe 2) it's exceedingly simple
568 2012-08-07 19:07:46 <sipa> amiller: it still takes years :)
569 2012-08-07 19:07:47 <amiller> also it could be implemented just as easily in any alt chain
570 2012-08-07 19:07:49 <BlueMatt> TD: I know its got a while before you look at it, but I went ahead and pushed my bitcoinj fullscripts branch
571 2012-08-07 19:08:01 <TD> Eliel: ah, i see. i guess that could work, yes
572 2012-08-07 19:08:12 <TD> Eliel: it could be quite expensive to sort every script like that
573 2012-08-07 19:08:18 <TD> BlueMatt: cool!
574 2012-08-07 19:09:01 <forrestv> why would it take so long to roll out? we've implemented changes that break mining before pretty quickly (BIPs 16 and 30)
575 2012-08-07 19:09:12 <TD> does anyone know how many unique addresses there are in the current block chain?
576 2012-08-07 19:09:19 <sipa> forrestv: those are not backward-incompatible changes
577 2012-08-07 19:09:34 <amiller> this could easily be backward compatible
578 2012-08-07 19:09:36 <TD> forrestv: only miners had to upgrade for those. hard-forking changes require everyone to upgrade whether they mine or not
579 2012-08-07 19:09:39 <sipa> forrestv: so they only require a miner supermajority, and not a full network upgrade (== everyone)
580 2012-08-07 19:09:48 <forrestv> why does this require a full network upgrade?
581 2012-08-07 19:10:01 <forrestv> stick it in the coinbase q:
582 2012-08-07 19:10:02 <sipa> if it changes the header, it means a full network upgrade
583 2012-08-07 19:10:13 <sipa> if you put it in the coinbase, sure, that's something else
584 2012-08-07 19:10:29 <sipa> but that does mean you don't get to see the upward-pointers before downloading the block
585 2012-08-07 19:10:39 <sipa> (unless we add a special getheaderpluscoinbase P2P command)
586 2012-08-07 19:10:42 <amiller> can you download just the coinbase?
587 2012-08-07 19:10:48 <forrestv> with the amount of stuff we're considering requiring in the coinbase, it might make sense to only have a hash of some extensible structure in the coinbase
588 2012-08-07 19:11:36 <amiller> but yes i'm recommending a change to the header
589 2012-08-07 19:11:59 <sipa> in that case, i don't think it's worth the effort
590 2012-08-07 19:12:14 <sipa> (though it may be considered if at some point we plan a breaking change anyway)
591 2012-08-07 19:12:53 <weex> would anyone like to admit having a hand in http://en.wikipedia.org/wiki/Bitcoin?
592 2012-08-07 19:13:13 <amiller> sure, perhaps it will be the breaking change that also includes the unspent-coins commitment
593 2012-08-07 19:13:13 <TD> even then, i still don't really buy it. but whatever.
594 2012-08-07 19:13:16 <sipa> weex: i never touched it
595 2012-08-07 19:13:46 <amiller> i'm trying to get feedback on how to explain this solution, but i'm not sure where my explanation is missing its mark
596 2012-08-07 19:14:05 <TD> having actually written a lite client (has anyone else?), if an UOMT appeared tomorrow in blocks and was being checked/built by all the miners, i wouldn't prioritize using it
597 2012-08-07 19:14:06 <amiller> somewhere in between checkpoints-already-work-fine and changes-are-too-hard-anyway
598 2012-08-07 19:14:06 <weex> sipa: smart, i'm going to spend some time on it but taking it slow
599 2012-08-07 19:14:31 <TD> because it isn't clear to me, how it would really help the user experience, and it'd require a lot of new code that would replace existing code which already works
600 2012-08-07 19:15:12 <amiller> it's only going to get harder to make important changes later, i don't see the harm in striving now to build an efficiently scalable system
601 2012-08-07 19:15:15 <sipa> amiller: the theoretical advantage is nice; finding the best tip in O(log n) instead of O(n) (i assume, i didn't read the proposal entirely)
602 2012-08-07 19:15:20 <TD> syncing the chain, at least on mobile, is already barely UX impacting and we're missing simple optimizations
603 2012-08-07 19:15:28 <sipa> amiller: but downloading headers has a very low constant factor
604 2012-08-07 19:15:50 <amiller> it's a linear factor if you have many chains to choose from
605 2012-08-07 19:16:24 <TD> but you don't
606 2012-08-07 19:16:43 <sipa> you always just ask for the best chain, and download all headers from there to the point you know
607 2012-08-07 19:16:59 <amiller> right but potentially that is still linear in the amount of time it's been since you can remember
608 2012-08-07 19:17:07 <sipa> yes, it is linear
609 2012-08-07 19:17:21 <amiller> so this solves exactly that problem but in much better than linear time
610 2012-08-07 19:17:24 <sipa> but with a very small constant factor (80 bytes transfer + storage) per block
611 2012-08-07 19:17:31 <sipa> yes, i agree with that
612 2012-08-07 19:17:40 <amiller> ok that's all i suppose
613 2012-08-07 19:19:33 <sipa> is your research actually about bitcoin?
614 2012-08-07 19:19:44 <TD> i still don't get it. you don't really download the headers, even with a lite client. you download headers+[relevant] transactions
615 2012-08-07 19:19:48 <amiller> TD, tomorrow's security problems are not predicted from todays UX problems
616 2012-08-07 19:19:51 <TD> today that means you download full blocks
617 2012-08-07 19:20:02 <TD> tomorrow it means headers+filtered transactions and their branches
618 2012-08-07 19:20:21 <TD> but you still need the transaction data from each block, or some indication that there are no relevant transactions in that header
619 2012-08-07 19:20:23 <sipa> TD: and with amiller's idea, it could be some headers+filtered transactions
620 2012-08-07 19:20:38 <sipa> but i agree the practical advantage is really small
621 2012-08-07 19:20:42 <TD> you need the header to verify the branch
622 2012-08-07 19:20:51 <sipa> with his proposal, you wouldn't
623 2012-08-07 19:21:10 <TD> if a remote node gives me a header that contains a backwards link to a block 10 blocks back
624 2012-08-07 19:21:13 <midnightmagic> amiller: oh hey is this that state snapshot idea you were talking about like a year ago with me?
625 2012-08-07 19:21:18 <TD> and there is a relevant transaction 5 blocks back
626 2012-08-07 19:21:34 <TD> i don't see how being able to efficiently know which block is 10 blocks back helps me. i need the merkle root of the block 5 blocks back
627 2012-08-07 19:21:42 <amiller> midnightmagic, it builds on that. and i'm pretty much assigning credit to the state snapshot thing to etotheipi now who has the clearest proposal for it
628 2012-08-07 19:21:44 <TD> so the remote node has to send me that anyway
629 2012-08-07 19:21:47 <sipa> TD: this is about finding the best header to ask for
630 2012-08-07 19:22:02 <midnightmagic> amiller: I love that you're still thinking about it. :) awesome
631 2012-08-07 19:22:04 <sipa> TD: there could be a "give me all filtered transactions up to block X"
632 2012-08-07 19:22:11 <sipa> but it does indeed seem pointless
633 2012-08-07 19:22:15 <TD> you need the headers for every block containing a filtered transaction
634 2012-08-07 19:22:19 <sipa> as we are already trusting the peer to give us everything
635 2012-08-07 19:22:24 <amiller> TD, i am interested in solving fundamental problems for the scalability of bitcoin thousands of years into the future, not just for the next few months when no one ever falls behind more than ten blocks
636 2012-08-07 19:22:26 <TD> and a header that doesn't connect to other headers you have, is worthless. it can be fake.
637 2012-08-07 19:22:37 <amiller> solving those problems now is important even earlier, because it will build more trust that the protocol is correct and sound
638 2012-08-07 19:22:50 <TD> amiller: bitcoin already scales just fine with the much simpler changes that are already being implemented. i have full trust that the existing protocol is correct and sound. that's why i'm pushing back.
639 2012-08-07 19:23:01 <TD> the 10 blocks was just an example to illustrate that you need the headers
640 2012-08-07 19:23:07 <TD> which at any rate are a trivial amount of data.
641 2012-08-07 19:23:10 <TD> a few megs per year
642 2012-08-07 19:23:15 <sipa> indeed
643 2012-08-07 19:23:47 <sipa> i think i see the theoretical advantage, but i think the improvement will ever outweigh other costs that need to be paid anyway
644 2012-08-07 19:24:01 <TD> i'm not sure i even see a theoretical advantage. when i try to figure out how i'd use it in bitcoinj, i hit a wall
645 2012-08-07 19:24:02 <midnightmagic> :-)
646 2012-08-07 19:24:32 <TD> i don't understand what code changes i'd make because i still need an unbroken chain of headers, of all chains the client is aware of, with the relevant transactions in each
647 2012-08-07 19:25:27 <sipa> anyway, amiller, am i right to assume this: if i'd ask for all transactions that affect me, the peer would send me (the blockchain tip header, any block header of a block with a tx in i need, all block headers in the tree formed by the previous 2 upwards until genesis)
648 2012-08-07 19:25:49 <sipa> + the filtered transactions and their merkle paths to their containing block
649 2012-08-07 19:26:12 <amiller> sipa, with etotheipis proposal you would not need to ask for 'trasnactions' that affect you, you would ask for unspent-coins that affect you, and you ask for those using just the current head
650 2012-08-07 19:26:28 <sipa> amiller: right, agree
651 2012-08-07 19:26:39 <amiller> sipa, if you do not know the current head, then you say 'give me the largest head and the supporting proof data in order!'
652 2012-08-07 19:26:56 <amiller> then your bitcoinj client says at the bottom 'estimated work in chain: a bajliion megahashes.  likelihood: 67%'
653 2012-08-07 19:27:03 <amiller> and then that lieklihood increases very rapidly
654 2012-08-07 19:27:09 <amiller> rapidly at first, slowing down once it reaches 99.99% and so on
655 2012-08-07 19:27:18 <sipa> right
656 2012-08-07 19:27:26 <amiller> if you eventually get to 100%, then you've in fact validated the entire chain from front to back like a full node
657 2012-08-07 19:27:36 <sipa> its usefulness only becomes apparent in combination with a coinset commitment
658 2012-08-07 19:27:42 <amiller> thats correct
659 2012-08-07 19:27:47 <sipa> as that means you don't need intermediate block headers
660 2012-08-07 19:30:17 <TD> ok
661 2012-08-07 19:30:53 <midnightmagic> sipa: when you timed your ultraprune node during the initial download and found it took 22 minutes, were you using -loadblock= or -connect= ?
662 2012-08-07 19:31:02 <sipa> midnightmagic: loadblock
663 2012-08-07 19:33:32 <sipa> midnightmagic: it's 7 minutes now (if i disable sigchecking), by the way
664 2012-08-07 19:34:08 <amiller> thank you for your discussion and feedback sipa, you too TD
665 2012-08-07 19:34:41 <sipa> amiller: i'll definitely want to talk to you at the london conference
666 2012-08-07 19:35:17 <amiller> awesome, i have no idea who else is going
667 2012-08-07 19:35:18 <TD> yes, looking forward to meeting you! it'll be cool
668 2012-08-07 19:35:22 <amiller> both of you! cool
669 2012-08-07 19:35:35 <TD> don't take my feedback the wrong way. we need more people thinking about these core design issues
670 2012-08-07 19:35:48 <amiller> no worries
671 2012-08-07 19:35:51 <TD> in this case i happen to think the current plan will scale well. but there are many other aspects of bitcoin that can use attention
672 2012-08-07 19:37:46 <midnightmagic> sipa: from bitcoin/ultraprune?!  it's definitely not 7 minutes for me..! :-)
673 2012-08-07 19:38:47 <midnightmagic> sipa: around block 4207 it's taking upwards of about 0.7s per block. is there some trick I'm missing? what's the bitcoind command-line you're using?
674 2012-08-07 19:39:18 <BlueMatt> I believe he disabled script checking for the 7 minute one
675 2012-08-07 19:39:41 <BlueMatt> or maybe just sig verification
676 2012-08-07 19:39:42 <sipa> midnightmagic: in my current branch on github there are two test commits on top
677 2012-08-07 19:39:50 <midnightmagic> sipa: yep got those.
678 2012-08-07 19:40:03 <sipa> one calculates a checksum for the entire coinset at every block, which is insanely expensive
679 2012-08-07 19:40:08 <sipa> just remove that commit
680 2012-08-07 19:40:11 <midnightmagic> oh okay
681 2012-08-07 19:40:27 <sipa> and the one before keeps the coinset in a std::map instead of on disk
682 2012-08-07 19:40:39 <sipa> so you need to wipe your directory every time :)
683 2012-08-07 19:40:45 <sipa> so remove that one too
684 2012-08-07 19:41:01 <midnightmagic> that would explain why I had to wipe the directory every time
685 2012-08-07 19:42:54 <midnightmagic> woops there we go.
686 2012-08-07 19:43:33 <midnightmagic> i wonder what happens if I -checklevel=0
687 2012-08-07 19:50:54 <sipa> midnightmagic: ./bitcoind -logtimestamps -datadir=/home/pw/.bitcoin-tmpfs -loadblock=blk0001-mainnet.dat
688 2012-08-07 19:50:57 <sipa> is what i use
689 2012-08-07 19:51:10 <sipa> (for now; the 7 minutes was not in a tmpfs)
690 2012-08-07 19:51:46 <midnightmagic> thanks
691 2012-08-07 19:53:42 <midnightmagic> sipa: lots of reorgs?
692 2012-08-07 19:54:11 <sipa> oh, it always says reorg
693 2012-08-07 19:54:17 <sipa> every block connection is a reorg
694 2012-08-07 19:54:24 <midnightmagic> it's doing them 20 at a time.
695 2012-08-07 19:54:34 <sipa> yes
696 2012-08-07 19:54:35 <midnightmagic> k
697 2012-08-07 19:55:09 <midnightmagic> oh yeah that's nice and quick even up to 160k
698 2012-08-07 19:55:17 <BlueMatt> arg...we have a ton of issues in the test suites on master...
699 2012-08-07 19:56:56 <midnightmagic> sipa: how difficult would it be do you think in terms of difficulty to built in a manual-pruning mechanism? i want to be able to prune tx I know are unspendable (like the grafitti stuff)
700 2012-08-07 19:57:46 <midnightmagic> given list-of-tx, prune them regardless of spend-status (or perhaps just if they are end-of-line unspent so breaks can't be created)
701 2012-08-07 19:57:46 <sipa> midnightmagic: i suppose you could have a script or so that gets executed for each block, and replies which tx's are not to be stored
702 2012-08-07 19:58:03 <sipa> ultraprune is not really tx-pruning, it is block-pruning
703 2012-08-07 19:58:13 <sipa> it can only remove entire old blocks
704 2012-08-07 19:58:24 <sipa> (but they can still have spendable tx's)
705 2012-08-07 19:58:38 <sipa> however, you could prevent storing in the first place i guess
706 2012-08-07 19:58:41 <midnightmagic> this is not the stemming embryonically-envisioned in satoshi's paper?
707 2012-08-07 19:58:46 <sipa> no
708 2012-08-07 19:58:48 <midnightmagic> doh
709 2012-08-07 19:58:54 <sipa> imho, it's better ;)
710 2012-08-07 19:59:40 <sipa> it keeps 1) a block database with all block, or only recent blocks
711 2012-08-07 19:59:49 <sipa> and 2) a separate set of unspent txouts
712 2012-08-07 20:00:04 <midnightmagic> so, this is removing blocks which have no unspent tx in them, and reconnecting via some kind of new edge?
713 2012-08-07 20:00:17 <sipa> no
714 2012-08-07 20:00:27 <midnightmagic> sorry I wrote that and clicked enter before you began explaining.
715 2012-08-07 20:00:32 <sipa> it only needs the blocks for serving them, reorganising, and rescanning
716 2012-08-07 20:00:41 <sipa> for all the rest, the txout set suffices
717 2012-08-07 20:01:06 <sipa> so if you give up the ability to serve/rescan/reorg old blocks, there is no need to keep them
718 2012-08-07 20:01:08 <midnightmagic> tx verification just checks (and potentially updates) the txout set then?
719 2012-08-07 20:01:12 <sipa> yes
720 2012-08-07 20:01:52 <midnightmagic> is it possible to verify the state of the txout set in terms of transition? (so I can detect corruption for example)?
721 2012-08-07 20:02:12 <sipa> not really
722 2012-08-07 20:02:13 <midnightmagic> or am I trusting that my on-disk txout set is Just Accurate.
723 2012-08-07 20:02:19 <sipa> that
724 2012-08-07 20:02:21 <midnightmagic> k
725 2012-08-07 20:02:28 <sipa> there are some tests possible
726 2012-08-07 20:02:30 <sipa> but not too many
727 2012-08-07 20:02:42 <sipa> however, the txout set is very small (120 MB or so)
728 2012-08-07 20:02:59 <midnightmagic> that surprises me, but I don't know why it would.
729 2012-08-07 20:03:20 <sipa> why it would what?
730 2012-08-07 20:03:30 <sipa> there are only 1.5-2 million unspent txout
731 2012-08-07 20:03:32 <midnightmagic> surprise me that the txout set seems so small.
732 2012-08-07 20:04:20 <midnightmagic> this txout set could be used to do a sort of slow satoshi-style when-i'm-not-busy tx stemming of old blocks too..
733 2012-08-07 20:04:54 <sipa> ?
734 2012-08-07 20:05:20 <sipa> once you prune the actual blocks, you can't serve them, reorg before them, or rescan them
735 2012-08-07 20:05:27 <sipa> so they become useless whatsoever
736 2012-08-07 20:05:30 <midnightmagic> if this txout set is so small, and the block headers are so small, would a traditional tx stemming also yield (after a db checkpoint+restore when complete) a much smaller set of blk* files? I thought the last time gmaxwell said something about tx stemming, it wouldn't really result in a very much smaller set of stored data.
737 2012-08-07 20:06:01 <sipa> well, i suppose you can still rescan non-spent txs
738 2012-08-07 20:06:03 <midnightmagic> not totally useless: you can still do a tx verification and detect corruption..?
739 2012-08-07 20:06:13 <sipa> ?
740 2012-08-07 20:06:36 <sipa> so say you have a current unspent-txout-set, and pruned blk* files
741 2012-08-07 20:06:40 <sipa> what do you want to do?
742 2012-08-07 20:06:52 <sipa> imho, those blk* files are completely useless
743 2012-08-07 20:07:28 <midnightmagic> hrm..
744 2012-08-07 20:08:24 <TD> amiller: here's a problem i have thought a lot about but can't find a solution too (that doesn't involve a hard forking change)
745 2012-08-07 20:08:37 <TD> amiller: today SPV clients cannot verify block coinbases, which means they cannot verify the inflation schedule is being followed
746 2012-08-07 20:08:56 <TD> if many users or merchants ended up on SPV clients, this would open the door to a miner conspiracy in order to change the inflation rules
747 2012-08-07 20:09:10 <TD> predictable inflation is one of the most fundamental rules of bitcoin. it is one of the foundational agreements that binds us all together
748 2012-08-07 20:09:12 <midnightmagic> what's stored with the txout set? can you cryptographically verify after all that your txout set has been corrupted if someone tries to spend one that has become corrupted? do we have the merkle for that tx?
749 2012-08-07 20:09:15 <TD> so if it was violated, that would be very bad
750 2012-08-07 20:09:25 <TD> unfortunately checking coinbase values means having a copy of every transaction, afaict
751 2012-08-07 20:09:34 <sipa> midnightmagic: the txout set coins txouts; nothing more
752 2012-08-07 20:09:42 <amiller> TD, not necessarily
753 2012-08-07 20:09:46 <sipa> midnightmagic: s/coins/contains/
754 2012-08-07 20:09:48 <amiller> aren't transactions already stored in blocks according to merkle trees
755 2012-08-07 20:10:00 <amiller> so it's not a list of the whole set of transactions
756 2012-08-07 20:10:03 <midnightmagic> sipa: so either i find theunspent, or not, and if not, I assume it's invalid rather than checking up to the block header. correct?
757 2012-08-07 20:10:04 <TD> yes. but transactions don't specify their input values, only output values. it's a very terse protocol
758 2012-08-07 20:10:41 <TD> coinbase txn value is inflation + fees
759 2012-08-07 20:10:50 <midnightmagic> sipa: I really really like how fast this -loadblock is btw.  so awesome.
760 2012-08-07 20:10:51 <TD> you can't know the fees without knowing every transaction in the block and all dependent transactions
761 2012-08-07 20:11:28 <sipa> you need the set of unspent txouts, to verify that, imho
762 2012-08-07 20:11:35 <BlueMatt> TD: if you did the commit-all-unspent-outputs-to-merkle-tree change, then you would be able to check it with nothing more than that set, which could be gotten from some 3rd-party?
763 2012-08-07 20:11:52 <BlueMatt> makes you only half-SPV, but still
764 2012-08-07 20:12:02 <BlueMatt> do it for 1/100 blocks, np
765 2012-08-07 20:12:16 <TD> yeah i was thinking of random spot checks
766 2012-08-07 20:12:40 <TD> if every SPV client does random spot checks and broadcasts messages saying "this block is bogus and here's a proof" ...... then possibly you can prevent such an attack
767 2012-08-07 20:12:59 <TD> a spot check doesn't technically need all the transactions. you can ask the remote nodes to provide the connected transactions + merkle branches on demand
768 2012-08-07 20:13:09 <BlueMatt> ye
769 2012-08-07 20:13:10 <sipa> well, right now, most of the generation is from the subsidy, and not from fees
770 2012-08-07 20:13:20 <sipa> wait, never mind
771 2012-08-07 20:13:21 <TD> it's a crapton of data to download, possibly, but if you only do it rarely ...
772 2012-08-07 20:13:22 <amiller> the subsidy rules aren't any harder to check than those transactions
773 2012-08-07 20:13:33 <TD> right, the inflation is the easy part to check
774 2012-08-07 20:13:35 <midnightmagic> i don't need to verify the coinbase generation of a txout, just that spending a tx is spending one from the unspent tx's merkle, verifiable up to the block hash. or am I misapprehending the utility of that?
775 2012-08-07 20:13:37 <TD> it's just a formula
776 2012-08-07 20:14:21 <TD> one interesting question is whether the work can be sharded amongst co-operating SPV clients
777 2012-08-07 20:14:33 <TD> i'm not sure. need to ponder it more
778 2012-08-07 20:14:38 <amiller> TD, i'm positive it can
779 2012-08-07 20:15:26 <amiller> the property that's interesting about hash graphs is that every step requires only local information
780 2012-08-07 20:15:39 <amiller> i like to imagine that the gossip network is a sort of magical hash inversion cloud
781 2012-08-07 20:15:43 <midnightmagic> sipa: Here's my concern, and why I'm peppering with questions. I experience a LOT of db* corruption. Like a lot. Probably once a week, due to unsupported patches, or crashed machines, or unstable hardware. I'm just curious how I would verify and/or rebuild my txout set cheaply in the event I trashed it because of one of the many, weird problems I tend to encounter. :)
782 2012-08-07 20:15:46 <TD> one question is whether a miner conspiracy could just agree to create a kind of "second coinbase" that grants themselves free money, possibly connected to a transaction that was already spent. so maybe you do need the unspent outputs
783 2012-08-07 20:15:50 <amiller> you give it a hash, it tells you what's on the other side - as long as someone has computed that hash before and propagated it
784 2012-08-07 20:15:59 <TD> but then ..... you can't rely on miners to give you that. because the attack under consideration is a miner conspiracy
785 2012-08-07 20:16:02 <sipa> midnightmagic: you use -loadblock :)
786 2012-08-07 20:16:05 <amiller> everything necessary for validation can be done using only a single piece of information and queries to that network
787 2012-08-07 20:16:05 <midnightmagic> lol
788 2012-08-07 20:16:10 <amiller> so that's why it's easy to shard
789 2012-08-07 20:16:19 <midnightmagic> well. okay.
790 2012-08-07 20:16:29 <TD> midnightmagic: would you be willing to test the leveldb branch?
791 2012-08-07 20:16:35 <sipa> midnightmagic: but i hope leveldb also helps to reduce or random database corruptions
792 2012-08-07 20:16:45 <TD> midnightmagic: i'd really like to know if leveldb is more robust than bdb. if you really hit problems once a week, we can find out soon enough
793 2012-08-07 20:16:46 <midnightmagic> TD: yeah I'd love to.
794 2012-08-07 20:16:49 <sipa> (i have never in my life seen a corrupted database, by the way)
795 2012-08-07 20:16:57 <sipa> well, happing on my own machine
796 2012-08-07 20:17:10 <midnightmagic> sipa: I have, they happen all the time to me it's horrible.
797 2012-08-07 20:17:13 <BlueMatt> TD: would have to keep coinbases around (or just block->coinbase hash map) and then require each tx link back to generation if you dont want to store the info...