1 2015-04-24 00:00:36 <CodeShark> I just wish it had access to more context information - like txout value or block height
  2 2015-04-24 00:01:40 <CodeShark> or other scripts in the transaction
  3 2015-04-24 00:08:37 <CodeShark> adding these things would vastly increase the power of it
  4 2015-04-24 00:13:31 <CodeShark> I was thinking another significant advantage of going to only p2sh is that we can use the same script hash as an outpoint index, making address indexing unnecessary. The only trick is to make them unique by adding a nonce
  5 2015-04-24 00:18:34 <CodeShark> the idea of using the tx hash to identify an outpoint is sorta silly, if you think about it, given that it's the txouts that need indexing when doing validation
  6 2015-04-24 00:21:16 <CodeShark> hash160 produces 96 bits less than sha256...so we still have 96 bits left over for a nonce as well as an extra 32-bit integer for outpoint index (which is really irrelevant for validation - we should just always sort the inputs and outputs canonically)
  7 2015-04-24 00:21:31 <gmaxwell> "adding a nonce" leaves you with either the potential of replay (either intentional, or due to a screwup because keeping state is really hard) or maintaining an unprunable counter.
  8 2015-04-24 00:21:41 <CodeShark> yeah, that's the only trick, as I said :)
  9 2015-04-24 00:23:50 <brendafdez> I was wondering what would be required to compile bitcoind on a rooted Android device? I tried googling it but found surprisingly few results about this.
 10 2015-04-24 00:23:57 <gmaxwell> In any case, the current design is intentional; and inherently moots replay. hash160 is not really good for cases where collisions must be avoided; it's fine when only second preimage security matters.
 11 2015-04-24 00:24:42 <CodeShark> or we could just enforce a one-time use policy for all txout scripts :)
 12 2015-04-24 00:25:17 <CodeShark> except we then need to also track spent txouts when validating
 13 2015-04-24 00:26:05 <CodeShark> or perhaps the rule could be only one open transaction at a time to a particular script
 14 2015-04-24 00:26:12 <CodeShark> once it gets closed out it can be reused again
 15 2015-04-24 00:26:38 <CodeShark> this would allow, for example, using the same script as an input for change
 16 2015-04-24 00:27:36 <CodeShark> and we could use double sha256 rather than hash160
 17 2015-04-24 00:28:04 <CodeShark> at least for indexing the txout
 18 2015-04-24 00:32:56 <CodeShark> an added benefit of this approach (albeit a bit hackish) is the ability to revoke compromised keys by sending tiny amounts to them and keeping the transactions open.
 19 2015-04-24 00:33:34 <CodeShark> well, not exactly...I suppose if they are really compromised someone could spend them...
 20 2015-04-24 00:34:02 <CodeShark> that's another feature that I'd really, really love to have, though - the ability to revoke compromised keys
 21 2015-04-24 00:34:16 <CodeShark> effectively making any transaction sending to the revoked script invalid
 22 2015-04-24 00:34:50 <brendafdez> I mean I know that bitcoind doesn't run on Android yet, but I was curious bc I see it runs on a Raspberry pi or similar hardware on linux, so I'd like to know if having it on android is a possibility or there's some major reason that would prevent it.
 23 2015-04-24 00:37:00 <CodeShark> although the need for key revocation would be less of an issue if payment scripts were actually renegotiated for each transaction...but requiring renegotiation for each transaction might be problematic for certain applications where a device needs to be able to generate transactions without connecting to the network, i.e.
 24 2015-04-24 00:37:31 <gmaxwell> brendafdez: android uses the linux kernel but it is a LONG way from a normal linux system. If someone did the work to get the dependencies all built, it should work there.
 25 2015-04-24 00:38:02 <CodeShark> and even with something like the payment protocol you need a way to revoke certificates
 26 2015-04-24 00:38:23 <CodeShark> which is something that's totally broken anyhow
 27 2015-04-24 00:39:33 <brendafdez> hmm, i see
 28 2015-04-24 00:41:40 <CodeShark> anyhow, adding OP codes to the script to grab the block height and other scripts in the transaction would be glorious :)
 29 2015-04-24 00:42:21 <CodeShark> I've already got a bunch of security applications for that
 30 2015-04-24 00:43:08 <gmaxwell> height is not reorg safe; so it's not a 'free choice'
 31 2015-04-24 00:43:46 <CodeShark> what are the implications?
 32 2015-04-24 00:43:54 <CodeShark> what's the worse that could happen?
 33 2015-04-24 00:44:28 <CodeShark> *worst
 34 2015-04-24 00:45:14 <CodeShark> is that exploitable, gmaxwell?
 35 2015-04-24 00:45:20 <gmaxwell> Complicates reasoning about the safty of accepting transactions;  basically in bitcoin a transaction can never become invalid without- at worst- someone intentionally doublespending it.  If script is no longer a pure function of a transaction then external things can invalidate otherwise perfetly honest transactions; so you can get weird things like reorgs forever destroying coins.
 36 2015-04-24 00:45:44 <CodeShark> but wouldn't that be the fault of whoever wrote the script?
 37 2015-04-24 00:46:44 <CodeShark> I see your point...but then again, you probably shouldn't be accepting coins that might get reorged for anything substantial from people you don't know :)
 38 2015-04-24 00:46:44 <gmaxwell> Sure, but it 'taints' all descendant coins. This is part of the reason that coinbase outputs are not spendable for 100 blocks; to avoid the non-fungibility from the differential safty.
 39 2015-04-24 00:47:01 <gmaxwell> Not just from people you don't know, you really want to go N steps back. :(
 40 2015-04-24 00:47:36 <gmaxwell> basically coins that have non-pure functions in their 'recent' (by blockchain height) path are not fully fungible with other coins.
 41 2015-04-24 00:48:52 <gmaxwell> Not the end of the world or anything, and perhaps you could propagate a counter "last blockheight in this coins history where it was affected by an unsafe script"
 42 2015-04-24 00:49:30 <CodeShark> right - I think these issues can be mitigated...and the extra power afforded by this is worth the complication
 43 2015-04-24 00:49:33 <gmaxwell> But yet another thing that seems like a capricious limitation that actually sidesteps a bunch of complexity.
 44 2015-04-24 00:54:29 <CodeShark> what about the ability to access other scripts in the transaction? that doesn't affect fungibility
 45 2015-04-24 00:55:39 <gmaxwell> Yea, I think the ability to access other scripts is fine. Anything where the script is a pure function of the transaction is 'fine' from that perspective (though, watch out for malleability surprises. :) )
 46 2015-04-24 01:09:16 <CodeShark> I actually just thought of a use case where p2sh does not suffice - say you want to make it such that a scriptPubKey expires after a certain block height
 47 2015-04-24 01:09:36 <CodeShark> with p2sh you would not be able to check this condition until it gets spent
 48 2015-04-24 01:10:44 <gmaxwell> so?
 49 2015-04-24 01:11:02 <CodeShark> the point would be to disallow confirming the transaction in the first place
 50 2015-04-24 01:11:24 <CodeShark> once you're passed that block height, that script is no good anymore
 51 2015-04-24 01:11:34 <gmaxwell> Sure and you can't spend it then.
 52 2015-04-24 01:11:54 <gmaxwell> nothing can stop people from burning money; generate a random pubkey, etc.
 53 2015-04-24 01:12:15 <CodeShark> but the transaction would still be valid...which would defeat the use case I'm thinking...namely, preventing people from accidentally sending to an old address you'd like to revoke
 54 2015-04-24 01:13:18 <gmaxwell> The network can't ever prevent people from intentionally burning money. If you're not talking about intentionally burning money; then you can accomplish what you want without a blockchain normative feature, more or less.
 55 2015-04-24 01:13:20 <CodeShark> by encoding the block height itself into the address it means there's no need to index revoked keys
 56 2015-04-24 01:19:03 <CodeShark> having to perpetually keep these keys and monitor the network for transactions is operational overhead...and security sensitive overhead
 57 2015-04-24 01:19:54 <gmaxwell> So you encode a lifetime into the address. And if the user hacks the address to bypass it, all they've done is intentionally burned funds.
 58 2015-04-24 01:20:08 <CodeShark> exactly
 59 2015-04-24 01:21:11 <CodeShark> in that case it's no longer on you
 60 2015-04-24 01:24:57 <CodeShark> there's a way to even prevent what you say...but at a significant cost...namely, requiring the address issuer to include signature(s). but this is probably not worth it
 61 2015-04-24 01:25:49 <CodeShark> in other words, you need to prove that whoever issued the address has a way to spend it
 62 2015-04-24 01:26:37 <CodeShark> probably not worth it :)
 63 2015-04-24 01:27:09 <gmaxwell> no even that doesn't prevent burning coins intentionally. :)
 64 2015-04-24 01:27:26 <CodeShark> it would also make BIP32 pubkeys essentially useless
 65 2015-04-24 01:27:32 <gmaxwell> (I mean, I can trivially generate signatures for pubkeys where no one knows the private key.
 66 2015-04-24 01:27:35 <gmaxwell> )
 67 2015-04-24 01:27:58 <CodeShark> heh
 68 2015-04-24 01:28:33 <gmaxwell> Anyways, my point above was that was just some address encoding and UI thing; doesn't change the blockchain at all.
 69 2015-04-24 01:28:46 <CodeShark> right
 70 2015-04-24 01:30:27 <CodeShark> supporting key revocation at a fundamental level is very complicated and expensive
 71 2015-04-24 01:31:51 <CodeShark> but supporting it in the UI is probably good enough for a good number of use cases
 72 2015-04-24 01:32:51 <CodeShark> I should say supporting key revocation is not necessarily that complicated...but it is expensive
 73 2015-04-24 01:33:07 <CodeShark> having to keep all those revoked keys in fast memory when doing validation
 74 2015-04-24 01:33:54 <CodeShark> I suppose you could use a bloom filter for stage 1
 75 2015-04-24 01:34:01 <CodeShark> and only verify if you get a false positive
 76 2015-04-24 01:36:04 <CodeShark> and even then, the OP_CHECKSIG operation only happens when it gets spent - so it doesn't prevent people from sending to an unspendable output
 77 2015-04-24 01:36:55 <CodeShark> ok, maybe it is a little complicated :p
 78 2015-04-24 04:24:07 <super3> created a simple pull to clean up the format in /depends
 79 2015-04-24 04:24:08 <super3> https://github.com/bitcoin/bitcoin/pull/6045
 80 2015-04-24 04:24:29 <super3> could someone review it and make sure i didn't make any mistakes?
 81 2015-04-24 05:02:26 <kevinmauro> How to Build a Brain-to-Brain Internet: Proof-of-Humanity Blockchains as a Foundation for Brain-to-Brain Security
 82 2015-04-24 05:03:06 <kevinmauro> Looking for some feedback on the first white paper. let me know if you'd be willing to provide some aggressive feedback. braininternetproject@gmail.com kevin.mauro@duke.edu <--- email me for a listserv/ more information
 83 2015-04-24 05:24:03 <Luke-Jr> kevinmauro: sounds off-topic
 84 2015-04-24 05:28:10 <Luke-Jr> CodeShark: anyhow, the dead idea gmaxwell wanted me to point you at the other day: https://gist.github.com/luke-jr/10d9812760f85cd3ee49
 85 2015-04-24 05:28:32 <CodeShark> heh
 86 2015-04-24 05:30:02 <CodeShark> yeah, it didn't sound like a particularly good idea...no offense ;)
 87 2015-04-24 05:30:30 <CodeShark> if we want an extra nonce we should just add an extra header field
 88 2015-04-24 05:31:33 <CodeShark> in fact, we should just replace the merkle root with an arbitrary hash value - and then use a secondary header
 89 2015-04-24 05:32:11 <Luke-Jr> that's essentially my present view
 90 2015-04-24 05:32:30 <gmaxwell> CodeShark: ... lol; an extra header field; and invalidate hundreds of millions of dollars of hardware lol.
 91 2015-04-24 05:32:48 <CodeShark> gmaxwell: if we replace the merkle root with a secondary header hash it won't break any of that
 92 2015-04-24 05:32:52 <Luke-Jr> O.o
 93 2015-04-24 05:33:29 <CodeShark> stick the merkle root and the second nonce into the secondary header
 94 2015-04-24 05:33:41 <Luke-Jr> right, I was O.o-ing at gmaxwell leaving
 95 2015-04-24 05:33:42 <CodeShark> and we can stop sticking crap into coinbase once and for all :p
 96 2015-04-24 05:34:03 <CodeShark> oh right, he left - lol
 97 2015-04-24 05:34:53 <CodeShark> my view is that the transaction merkle root is WAY too data format specific to be including in the main header
 98 2015-04-24 05:35:14 <CodeShark> it affords very little flexibility
 99 2015-04-24 05:35:18 <CodeShark> it's not needed for PoW
100 2015-04-24 05:35:50 <CodeShark> make it an arbitrary hash value and now you have a general PoW mechanism that can be applied to any blockchain storing any data in any format
101 2015-04-24 05:36:59 <CodeShark> anyhow, why did gmaxwell leave?
102 2015-04-24 05:38:25 <kadoban> Maybe he's practicing his dramatic exits.
103 2015-04-24 05:41:07 <Luke-Jr> someone can ask him in PM or #bitcoin, it's too meta for here
104 2015-04-24 05:51:49 <CodeShark> I'll give him the benefit of the doubt and assume he tripped over his power cord...until provided with additional info
105 2015-04-24 05:54:28 <ZjP> (it was a part not a disconnect)
106 2015-04-24 05:56:07 <CodeShark> oh, hmm
107 2015-04-24 06:25:48 <phantomcircuit> CodeShark, there's no point to mucking about like that
108 2015-04-24 06:26:02 <CodeShark> ?
109 2015-04-24 06:26:14 <phantomcircuit> you might as well just define an entirely new format which is 80 bytes with a 32bit nonce at the end
110 2015-04-24 06:26:57 <CodeShark> the point is we should not be committing to something that requires deep knowledge of internal structures in block headers
111 2015-04-24 06:27:12 <CodeShark> for maximal flexibility without changing header size
112 2015-04-24 06:27:33 <CodeShark> and the merkle tree structure requires deep knowledge of internal data structures that are irrelevant for PoW
113 2015-04-24 06:29:13 <CodeShark> I don't really have any qualms with the other data fields in the header (well, except for perhaps the fact that the nonce is too tiny)
114 2015-04-24 06:35:39 <CodeShark> if you need to convince yourself of just how ugly it is to use this approach, take a look at namecoin's merge mining implementation :p
115 2015-04-24 06:45:09 <phantomcircuit> CodeShark, to be fair namecoins merged mining implementation is the worst way to do it without being obviously wrong
116 2015-04-24 06:45:21 <phantomcircuit> which is impressive in it's own special way
117 2015-04-24 06:45:27 <CodeShark> true - lol
118 2015-04-24 06:48:17 <CodeShark> as if requiring knowledge of one merkle tree structure wasn't bad enough - they had to go ahead and nest in another
119 2015-04-24 06:54:18 <sipa> CodeShark: changing the header format is just not feasible, no matter how awesome it is
120 2015-04-24 06:54:48 <CodeShark> I hate that way of thinking - let's brainstorm and figure out a way to make it feasible
121 2015-04-24 06:55:08 <CodeShark> no major technological breakthrough was ever done with that attitude :p
122 2015-04-24 06:55:55 <CodeShark> we need a mechanism to support hardforking with minimal disruption
123 2015-04-24 06:55:58 <sipa> cfields_: i have not thought about how to deal with cobtexts in libconsensus, feel free to suggest something
124 2015-04-24 06:56:18 <CodeShark> I don't have an answer to the issue of how we construct such a mechanism...only some ideas...but we need it
125 2015-04-24 06:56:29 <CodeShark> sooner or later we WILL need to hardfork
126 2015-04-24 06:56:40 <CodeShark> and when that happens we have a chance to fix a bunch of stuff
127 2015-04-24 06:56:41 <sipa> phantomcircuit: still not 100% no
128 2015-04-24 06:59:41 <sipa> CodeShark: this is not just a hardfork
129 2015-04-24 06:59:53 <CodeShark> no?
130 2015-04-24 07:00:08 <CodeShark> I mean, sure, it requires applications to migrate
131 2015-04-24 07:00:08 <sipa> CodeShark: it's a change every piece of software and hardware on the planet
132 2015-04-24 07:00:14 <CodeShark> it does not require changes to hardware
133 2015-04-24 07:00:26 <sipa> chabging the header? yes it does
134 2015-04-24 07:00:49 <CodeShark> no, replacing the merkle root with an arbitrary 256 bits that are irrelevant to the PoW algorithm doesn't require any hardware changes
135 2015-04-24 07:01:14 <sipa> from the PoW algorithm, the merkle root is already arbitrary 256 bits
136 2015-04-24 07:01:22 <CodeShark> right - exactly
137 2015-04-24 07:01:31 <sipa> changing it to arbitrary 256 bits whatsoever breaks spv security
138 2015-04-24 07:01:48 <sipa> i know you want extra commitments
139 2015-04-24 07:01:51 <sipa> we all do
140 2015-04-24 07:01:53 <CodeShark> SPV is crap right now - using partial merkle trees from block headers is a stupid way to do it
141 2015-04-24 07:02:03 <CodeShark> I've actually had to implement it on the wallet end
142 2015-04-24 07:02:26 <sipa> that's how compact proofs over merkle trees will always look like
143 2015-04-24 07:02:30 <sipa> sorry
144 2015-04-24 07:02:39 <sipa> can you please stop bitching
145 2015-04-24 07:02:40 <CodeShark> both ripple and ethereum are using radix trees that do not require rebalancing
146 2015-04-24 07:02:52 <sipa> and think about viable solutions?
147 2015-04-24 07:03:02 <CodeShark> means you can commit to a state and also provide short proofs
148 2015-04-24 07:03:18 <sipa> yes, the current utxo commitments idea also uses that
149 2015-04-24 07:03:29 <CodeShark> right - that's what we should be committing to
150 2015-04-24 07:03:40 <sipa> i do not think so
151 2015-04-24 07:03:50 <sipa> but i'm willing to be convinced otherwise
152 2015-04-24 07:04:08 <sipa> (as opposed to chabging header formats, which is just a no go)
153 2015-04-24 07:04:43 <sipa> you can do extra commitments as a soft fork
154 2015-04-24 07:04:53 <sipa> yes, it'd a bit uglier
155 2015-04-24 07:05:01 <sipa> it's also totally feasible
156 2015-04-24 07:06:23 <sipa> you're always talking about some vague problem here, and then come up with the least viable solution to it, yelling "we MUST do this!"
157 2015-04-24 07:06:32 <sipa> sorry but that's fud
158 2015-04-24 07:06:54 <sipa> it may hurt your developers feelings to use a non-optimally clean solution
159 2015-04-24 07:07:27 <sipa> but we're also talking about millions of dollars invested in an existing, very hard to change, ecosystem
160 2015-04-24 07:07:47 <CodeShark> that's not my fault that people invested so much into stuff that's so hard to change
161 2015-04-24 07:07:54 <CodeShark> why should it stop me from inventing new stuff?
162 2015-04-24 07:08:33 <CodeShark> I think too much effort is being put into avoiding hardforks and app migrations rather than just accepting we'll need to do it sooner or later
163 2015-04-24 07:09:15 <sipa> i think a hardfork (requiring full nodes to upgrade) at some point is feasible
164 2015-04-24 07:09:31 <sipa> requiring mining hardware or wallets to change is not
165 2015-04-24 07:09:57 <CodeShark> it will have to be sooner or later because the current SPV is shit
166 2015-04-24 07:10:06 <CodeShark> it's going to just lead to more and more centralized validation services
167 2015-04-24 07:10:15 <CodeShark> completely defeating the purpose of bitcoin
168 2015-04-24 07:10:47 <sipa> i think that is inevitable
169 2015-04-24 07:10:53 <CodeShark> SPV is not trivial to implement, doesn't provide a solid security model, and is still slow and inefficient
170 2015-04-24 07:11:05 <sipa> and has nothing to do with what you are suggesting
171 2015-04-24 07:11:12 <sipa> utxo commitments would help spv
172 2015-04-24 07:11:14 <CodeShark> it has everything to do with wallets having to be reimplemented
173 2015-04-24 07:11:27 <CodeShark> at least the hardest part of the wallet
174 2015-04-24 07:11:38 <CodeShark> perhaps the buttons and dialog boxes don't need to be reimplemented p
175 2015-04-24 07:11:42 <CodeShark> but the network stack does
176 2015-04-24 07:11:48 <sipa> but can be done without any of the things you suggest
177 2015-04-24 07:12:11 <sipa> yes, if i would create bitcoin from scratch, i would take that into account
178 2015-04-24 07:12:36 <sipa> everyone is aware of the complexity involved - myself, trust me, included
179 2015-04-24 07:12:49 <sipa> but that's just the nature of the problem
180 2015-04-24 07:13:34 <sipa> i do not believe for a second that an "optimal" solution using utxo commitments would ve significantly easier to implement correctly and efficiently
181 2015-04-24 07:13:41 <sipa> quite the opposite
182 2015-04-24 07:14:18 <CodeShark> but that's more than compensated by the fact that sync is orders of magnitude faster...and it supports a viable security model
183 2015-04-24 07:14:43 <sipa> the security model is identical: trust miner majority
184 2015-04-24 07:15:15 <CodeShark> no, right now the SPV implementations require connecting to multiple noncolluding peers to make sure none of their transactions have been withheld
185 2015-04-24 07:15:26 <sipa> yes, agree
186 2015-04-24 07:15:42 <sipa> that is shitty, and something the current model does not cover
187 2015-04-24 07:15:54 <sipa> and improving that is very worthwhile
188 2015-04-24 07:16:36 <sipa> but utxo commitmemts also have huge costs - in terms of complexity for full nodes and efficiently of utxo updates
189 2015-04-24 07:16:50 <sipa> which _also_ have very direct centralization effects
190 2015-04-24 07:16:58 <sipa> i'm not saying you're wrong here
191 2015-04-24 07:17:11 <sipa> just that there is a lot to consider
192 2015-04-24 07:17:23 <CodeShark> more complexity, yes - but not sure about it being too costly in terms of computing resources
193 2015-04-24 07:17:55 <sipa> at least the utxo set size would be twice as jarge (as every node needs the haah of its children cached)
194 2015-04-24 07:18:06 <sipa> and every update needs to update all parent's nodes
195 2015-04-24 07:18:52 <sipa> valudation is currently dominated by latency and signature checking
196 2015-04-24 07:19:33 <sipa> but utxo commitments (especially if we want larger blocks) likely change that
197 2015-04-24 07:20:02 <sipa> i have not seen a sufficiently optimized implementation to judge it correctly
198 2015-04-24 07:20:35 <sipa> plus, there are worries that this effectively forces every full node to maintain the utxo set size in fast memory
199 2015-04-24 07:21:02 <sipa> while there are competing models (with different tradeoffs) that do not do this
200 2015-04-24 07:21:08 <CodeShark> such as?
201 2015-04-24 07:21:21 <sipa> stxo commitments being one
202 2015-04-24 07:21:39 <sipa> all txouts are appended to a single insertion-ordered tree
203 2015-04-24 07:21:47 <sipa> full nodes only maintain the root
204 2015-04-24 07:22:20 <sipa> the txouts in it have a boolean saying whether they are spendable or not
205 2015-04-24 07:22:51 <sipa> trznsactions carry a proof that guaranteea the inputs they are using exists, and how to update their spent bit
206 2015-04-24 07:23:33 <sipa> larger bandwidth, harder for wallets (you need sort of an external - but trustless - indexing service to build the proofs)
207 2015-04-24 07:24:04 <sipa> but much lower resources for full nodes, both in storage, in io, and in cpu
208 2015-04-24 07:24:14 <CodeShark> in theory, we're talking an O(log n) factor, no?
209 2015-04-24 07:24:19 <CodeShark> for any of these approaches
210 2015-04-24 07:24:22 <sipa> this is an idea by petertodd here
211 2015-04-24 07:24:29 <sipa> for what?
212 2015-04-24 07:24:47 <sipa> stxo can be done with o(1) memory for full nodes
213 2015-04-24 07:25:12 <sipa> but o(log n) extra data in transactions
214 2015-04-24 07:26:15 <CodeShark> right - so it's o(log n) somewhere or another - chances are we'll have to upgrade our crypto to something more costly to verify before this becomes a problem, I think
215 2015-04-24 07:26:26 <CodeShark> :p
216 2015-04-24 07:26:29 <CodeShark> well, maybe not
217 2015-04-24 07:26:33 <sipa> the utxo set size is growing quickly
218 2015-04-24 07:26:45 <sipa> and there is no incentive to minimize its growth
219 2015-04-24 07:27:17 <CodeShark> right - such incentives are possible, but they require economic changes
220 2015-04-24 07:27:45 <sipa> if it grows so large that it does not fit in ram anymore reasonably on a bunch of important nodes, block relay speed can be affectes very quickly suddenly
221 2015-04-24 07:28:02 <sipa> that's what i like about stxo, that it makes all that irrelevant
222 2015-04-24 07:28:06 <CodeShark> but the blockchain itself grows at least linearly...and that's assuming a constant transaction volume
223 2015-04-24 07:28:16 <CodeShark> so adding a log n factor seems practically negligible
224 2015-04-24 07:28:39 <sipa> on the other hand, stxo is a very differrent model than what we have now
225 2015-04-24 07:28:54 <sipa> and it will be very nontrivial to migrate to it
226 2015-04-24 07:29:12 <sipa> (though it's a soft fork, so not everyone needs to upgrade at once)
227 2015-04-24 07:29:34 <sipa> well it depends on what the factor is applied to
228 2015-04-24 07:30:17 <CodeShark> I guess it comes down to a tradeoff between storage and bandwidth
229 2015-04-24 07:30:24 <sipa> if utxo set access becomes _the_ bottleneck, and everything else vanishes compares to it, stxo is totally worth it, and utxo commitmemts would be horrible
230 2015-04-24 07:30:34 <sipa> indeed
231 2015-04-24 07:30:47 <sipa> and storage and bandwidth for whom
232 2015-04-24 07:31:46 <sipa> anyway, you're very welcome to help think about these things, and come up with solutions
233 2015-04-24 07:32:05 <sipa> please.just don't say that people are not aware of this, or not thinking about it
234 2015-04-24 07:32:12 <CodeShark> I never said that
235 2015-04-24 07:32:20 <sipa> you do give that impression
236 2015-04-24 07:32:26 <sipa> apologies if i was wrong
237 2015-04-24 07:34:25 <CodeShark> I keep saying that most of these issues are very familiar to the core devs...but very little is actually done because the migration issue is very hard
238 2015-04-24 07:35:25 <sipa> yes
239 2015-04-24 07:35:43 <sipa> but bitching that they must change won't help
240 2015-04-24 07:36:17 <sipa> writing an efficient utxo commitment implememtation, and simulating and benchmarking it, would
241 2015-04-24 07:36:34 <sipa> or coming up with something else
242 2015-04-24 07:36:45 <CodeShark> I don't see what I'm doing as bitching - I've implemented many of these things or worked on other projects that have implemented them...but whenever I bring them up in here, I generally get this "that's impossible" attitude and that's what pisses me off
243 2015-04-24 07:37:09 <sipa> you suggest them in ways that are impossible, yes
244 2015-04-24 07:37:34 <sipa> not the ideas themselves, but how you seem to imply the only way is a headers format change
245 2015-04-24 07:37:38 <sipa> come on, please
246 2015-04-24 07:37:44 <CodeShark> I never said that either
247 2015-04-24 07:37:52 <sipa> then again, apologies
248 2015-04-24 07:38:15 <CodeShark> my only claim regarding header format is that it could be much, much, much cleaner - but it's not necessary to support these other changes
249 2015-04-24 07:38:22 <sipa> sorry, i know you just want to improve the ecosystem
250 2015-04-24 07:38:24 <sipa> ok, good
251 2015-04-24 07:38:26 <sipa> i agree
252 2015-04-24 07:39:02 <CodeShark> the frustration is mostly due to seeing other devs have to spend excessive time working around issues we know are ugly
253 2015-04-24 07:39:15 <CodeShark> rather than spending that time working on inventing new ideas
254 2015-04-24 07:41:46 <CodeShark> of course it's possible to work around the block header stuff - but it means much fewer developers will even attempt anything - and those that do are likely to have to spend a lot more time fixing bugs...which slows down the evolution of the ecosystem considerably
255 2015-04-24 07:42:33 <CodeShark> it also makes merging different dev branches much, much, much harder
256 2015-04-24 07:42:49 <CodeShark> and far more error prone
257 2015-04-24 07:43:16 <CodeShark> it's not about whether it's a technical limitation in terms of whether we can actually implement it in theory - it's about that it significantly weighs down our development process
258 2015-04-24 07:45:37 <CodeShark> anyhow, the block header issue and the txo commitment issue can be treated independently
259 2015-04-24 07:45:53 <jonasschnelli> uh. Whats up with http://bitcoinstats.com? cdecker?
260 2015-04-24 07:46:27 <sipa> CodeShark: i think it is an inherent difficulty in consensus systems that this type of problem appears
261 2015-04-24 07:46:56 <sipa> CodeShark: and sure, things can be better or can be worse
262 2015-04-24 07:47:14 <CodeShark> yes - I think we need to study this problem in greater depth and consider it in the design and the development process
263 2015-04-24 07:47:28 <sipa> CodeShark: but changing consensus stuff, and requiring existing software to change, may also piss off developers
264 2015-04-24 07:47:35 <Luke-Jr> 
ACTION notes code reuse (libraries) would make changes like this easier

265 2015-04-24 07:48:21 <CodeShark> yes, indeed, sipa - but we could build in deprecation mechanisms and make the migration a part of the deployment process rather than pretending we'll never need to migrate
266 2015-04-24 07:49:12 <CodeShark> it would be absolutely unprecedented if the first implementation of an idea turned out to never need tweaking for all eternity
267 2015-04-24 07:49:12 <sipa> i do have some philosophical difficulties with that, but it may be inevitable
268 2015-04-24 07:49:44 <sipa> as in: who gets to decide what hardforking changes we make :)
269 2015-04-24 07:49:54 <CodeShark> right - but I think that's inevitable
270 2015-04-24 07:49:58 <sipa> that's more than technicalities - it becomes political
271 2015-04-24 07:50:10 <CodeShark> we'll have to make hardforks sooner or later - not all will require deprecating apps
272 2015-04-24 07:50:11 <sipa> people have different interesrs
273 2015-04-24 07:50:17 <CodeShark> and yes, it does become political
274 2015-04-24 07:50:33 <CodeShark> but I think that's inevitable - and not just because of new technical discoveries
275 2015-04-24 07:50:42 <jonasschnelli> Bip32 concept question (maybe repeating): my implementation now stored the masterkey seed and the ext pubkey of the internal (m/44/0'/0'/0) and external (m/44'/0'/0'/1) keychain. The int/ext pubkeys are used to create keys. When a privkey is required, i loop through the chain starting with the masterkey by creating with the stored seed. Does this make sense?
276 2015-04-24 07:50:52 <CodeShark> the incentives model might need tweaking to be better economically aligned with the real world
277 2015-04-24 07:51:00 <CodeShark> and that's a REALLY hard problem
278 2015-04-24 07:51:03 <sipa> CodeShark: yes
279 2015-04-24 07:51:04 <CodeShark> but it's inevitable
280 2015-04-24 07:51:08 <sipa> maybe
281 2015-04-24 07:52:57 <CodeShark> anyhow, I propose we start with relatively simple technical problems that do not require significant economic changes (but do require changes in consensus rules) and try to find a mechanism for deploying them
282 2015-04-24 07:53:36 <CodeShark> and changes that do not force existing apps to migrate immediately...but that can provide a migration path that's more gradual
283 2015-04-24 07:53:41 <sipa> find me a non-political hard fork with widespread support, and easy implementation
284 2015-04-24 07:54:32 <sipa> there has been one such hardfork before... the may 15 2013 one, where the 0.7 bdb database limitation was removed
285 2015-04-24 07:55:01 <sipa> and i was appalled by how little people complained
286 2015-04-24 07:55:27 <sipa> how nobody cared that the developers forced an upgrade of the rules of the system
287 2015-04-24 07:56:09 <CodeShark> yeah, that was handled less than ideally :p
288 2015-04-24 07:56:35 <sipa> i do believe that a political one will be different though
289 2015-04-24 08:00:31 <CodeShark> the max block size change might be an up-and-coming political one
290 2015-04-24 08:01:29 <CodeShark> which is technically very simple to change but presents considerable challenges in achieving community consensus and ensuring deployment causes the minimum amount of disruption
291 2015-04-24 08:01:40 <sipa> yes
292 2015-04-24 08:04:42 <CodeShark> for better or worse the core devs exercise considerable influence here - if enough core devs were to all make a public statement that it was beneficial I think most of the community would go along with it. of course, philosophically this is a little problematic because it implies a certain degree of centralization
293 2015-04-24 08:05:25 <CodeShark> but we don't really have a "fair" voting system for this kind of a thing
294 2015-04-24 08:06:19 <CodeShark> the only intrinsic metrics for vote "weight" are hashing power and stake
295 2015-04-24 08:07:08 <CodeShark> and any other sort of vote requires a process external to the protocol entirely
296 2015-04-24 08:08:20 <CodeShark> which we do informally on github and here already...but usually over issues that do not imply significant economic changes
297 2015-04-24 08:09:06 <CodeShark> the core dev voting process is very reputation-based
298 2015-04-24 08:09:11 <CodeShark> certainly not trustless
299 2015-04-24 08:13:53 <CodeShark> I'm not sure that this type of technology can really be fully decentralized from inception - it seems to require some early stages where there's a large degree of centralization until it can be bootstrapped
300 2015-04-24 08:14:42 <CodeShark> the tricky thing here is making sure the incentives are such that power is relinquished
301 2015-04-24 08:15:04 <CodeShark> once people get too powerful they don't easily give it up
302 2015-04-24 08:16:04 <CodeShark> and in any case, there can't really be a political system that's perfectly fair
303 2015-04-24 08:16:32 <CodeShark> so it's certainly troubling from a crypto purist perspective
304 2015-04-24 08:19:01 <CodeShark>  making everything that's already been invested into this network seem practically negligible.
305 2015-04-24 08:19:01 <CodeShark> my main concern is this: sure, there's been millions invested into one particular protocol and network...but it's still far from reaching global mainstream adoption...it's still tiny compared to other stuff. others are working on new networks and protocols that don't carry as much baggage...with the benefit of hindsight and new theoretical breakthrough discoveries. eventually, one might be able to reach scale effectively,
306 2015-04-24 08:19:44 <CodeShark> so unless we consider how we'll migrate before we have this kind of competition, I fear the fact that all this has been invested into this network will become moot anyhow
307 2015-04-24 08:20:31 <CodeShark> first mover advantage is drastically overrated - macs came before windows pcs, yahoo came before google, myspace came before facebook, etc.. etc...
308 2015-04-24 08:20:43 <CodeShark> we're overestimating the network effect
309 2015-04-24 09:07:18 <jtimon> coryfields_ around? on #6051, thought about declaring CheckBlock in consensus.h withot moving it to blockverify yet?
310 2015-04-24 09:11:34 <jtimon> CodeShark came before pcs and they they changed their architecture from RISC to the widely use intel x86 CISC
311 2015-04-24 09:11:54 <jtimon> if something better comes along, there's no reason why bitcoin can't adopt that
312 2015-04-24 09:14:11 <CodeShark> jtimon: we can't even make minor structural changes without breaking everything right now
313 2015-04-24 09:16:03 <sipa> we definitely can't make minor structural changes that aren't implemented, don't have widespread support, and make unknown tradeoffs.
314 2015-04-24 09:31:56 <timothy> why new bitcoin-core with pruning can't be used with wallet?
315 2015-04-24 09:32:18 <jonasschnelli> timothy: it could, it's just not implemented
316 2015-04-24 09:32:24 <timothy> oh ok
317 2015-04-24 09:33:19 <timothy> will it be implemented for next release?
318 2015-04-24 09:35:01 <sipa> timothy: if someone implements it
319 2015-04-24 09:35:08 <sipa> :)
320 2015-04-24 09:35:53 <timothy> :P
321 2015-04-24 09:36:27 <jonasschnelli> didn't had a look at it but it cannot be that complex.
322 2015-04-24 09:36:52 <sipa> just need to make sure rescanning triggers appropriate warnings etc
323 2015-04-24 09:37:24 <jonasschnelli> right. Rescanning in pruned mode can be exhausting...
324 2015-04-24 09:37:40 <sipa> "exhausting" ?
325 2015-04-24 09:37:44 <sipa> it's just not possible...
326 2015-04-24 09:38:04 <sipa> (or you'd need to implement logic to go re-fetch missing blocks, of course)
327 2015-04-24 09:39:08 <HostFat> so pruning currently isn't a feature for the common user
328 2015-04-24 09:39:22 <jonasschnelli> Hmm.. so what if someone adds a key older then last available block before pruning? Error?
329 2015-04-24 09:39:28 <sipa> jonasschnelli: yes
330 2015-04-24 09:39:40 <sipa> HostFat: bitcoin core on itself already isn't, unfortunately
331 2015-04-24 09:39:52 <jonasschnelli> I thought restarting getting the blockchain....
332 2015-04-24 09:40:02 <sipa> jonasschnelli: sure, feel free to implement that
333 2015-04-24 09:40:33 <jonasschnelli> but yes. Maybe start with a error if key-age < oldest-available-block
334 2015-04-24 09:40:39 <sipa> jonasschnelli: i'd really like an abstraction where client modules can request full block data asynchronously
335 2015-04-24 09:40:51 <sipa> jonasschnelli: and then have both the wallet and the full node validation as optional modules :)
336 2015-04-24 09:42:14 <jonasschnelli> sipa: Right. That is also my long-term goal... but right now i don't see that this will succeed once.
337 2015-04-24 09:42:34 <sipa> sure, everything in small steps
338 2015-04-24 09:42:46 <jonasschnelli> simple modularization steps get no attention. :)
339 2015-04-24 09:43:03 <sipa> jonasschnelli: what do i need to look at?
340 2015-04-24 09:43:12 <sipa> sorry, there have been too many pr for me to follow
341 2015-04-24 09:43:44 <jonasschnelli> yeah. To many PRs at the moment.
342 2015-04-24 09:43:54 <jonasschnelli> Things like that: https://github.com/bitcoin/bitcoin/pull/5990 would go into this direction
343 2015-04-24 09:44:00 <jonasschnelli> or that: https://github.com/bitcoin/bitcoin/pull/5994
344 2015-04-24 09:44:08 <sipa> if i see a dozen PRs by the same person, i tend to assume they're not really ready for review
345 2015-04-24 09:44:24 <jonasschnelli> and this: https://github.com/bitcoin/bitcoin/pull/6006
346 2015-04-24 09:44:47 <jonasschnelli> mainly it's about decoupling wallet, miner and a flexible rpc table.
347 2015-04-24 09:45:38 <jonasschnelli> These are the basic steps for going into the direction of a full-node-daemon who use module to validate, etc.
348 2015-04-24 09:45:45 <jonasschnelli> *modules
349 2015-04-24 09:46:25 <jtimon> mhmm, I have 11 opened PRs and just because I closed all the dependent ones with the exception of #6051 (I had more before)
350 2015-04-24 09:46:58 <jtimon> I know I wasn't organizing them the best way, but having more open PR doesn't mean they're not ready
351 2015-04-24 09:47:30 <jonasschnelli> 
ACTION hands over jtimon the badge for most PR activity. :)

352 2015-04-24 09:47:47 <Luke-Jr> there's no reason the common user can't use Bitcoin Core ..
353 2015-04-24 09:47:47 <sipa> jtimon: yeah, i know - this is my problem, not yours
354 2015-04-24 09:48:03 <jtimon> often means that many of them are very small and are easy to review
355 2015-04-24 09:48:05 <sipa> jtimon: i just tend to see "oh, lots of activity... let's wait until it settles down"
356 2015-04-24 09:48:16 <sipa> also... not knowing which one to look at first
357 2015-04-24 09:48:38 <sipa> again, not really a criticism on you, just trying to explain my own actions
358 2015-04-24 09:50:05 <jtimon> that's why I think "dependent" PRs that cannot be merged but list their dependencies (other PRs that can be merged now) is itneresting, I saw jonasschnelli doing that and I thought that can help people to "know what to loook at" or "see the big picture"
359 2015-04-24 09:50:26 <sipa> i just see 11 PRs :)
360 2015-04-24 09:50:33 <jtimon> unfortunately the moveonly stuff made #5946  unreadable so I had to close it
361 2015-04-24 09:51:41 <jtimon> sipa I believe I've closed like 10 PRs in the last 2 days, I think before that I had 16 or something like that
362 2015-04-24 09:52:00 <jonasschnelli> jtimon: Agreed. Moveonly, refactoring and modularization PRs are normally boring because most people can't see a short-term benefit.
363 2015-04-24 09:52:22 <jonasschnelli> bringing in a big PR like a new wallet will raise complains about unreviewablility
364 2015-04-24 09:52:32 <jtimon> yeah, and if you mix it with other changes then they can't see anything
365 2015-04-24 09:52:46 <sipa> anyway, feel free to poke me for a review if you feel blocked
366 2015-04-24 09:52:55 <jonasschnelli> sure. Thanks sipa.
367 2015-04-24 09:53:26 <jtimon> sipa well, I've been waiting for #5696 for long, there's an outdated ack from you I think
368 2015-04-24 09:53:54 <jtimon> #5669 and #5975 are quite blocking too
369 2015-04-24 09:54:06 <jonasschnelli> Insted of twiddeling ones thumbs, i decided to just continue on top of some unmerged PR. Tradeoff: lots of rebaseing and amend readding of stuff.
370 2015-04-24 09:54:18 <jtimon> I mean, #5669 not that much, we can just include main from the consensus stuff temporarily
371 2015-04-24 09:55:04 <jtimon> but in combination with the moveonly will allow a big include cleanup
372 2015-04-24 09:56:37 <jtimon> so I guess #5696 and #5669 are my biggest priorities
373 2015-04-24 09:58:43 <jtimon> and although it will need rebase after those are merged, the sooner I know there's nothing ugly in #6051  the better
374 2015-04-24 10:01:14 <jtimon> oops I had another dependent one open
375 2015-04-24 10:08:21 <morcos> jtimon: i know you've been getting it from all sides, but i do also agree its a bit difficult to figure out how to help with all the PR's especially when they change
376 2015-04-24 10:08:53 <morcos> perhaps one thing that would be useful is a separate design doc or something, so we could see a description of the high level vision
377 2015-04-24 10:09:20 <morcos> this could link to a branch or dependent PR that showed the bleeding edge of code progress