1 2014-04-03 00:08:32 <petertodd> gmaxwell: balance based systems have clear and deterministic behavior under reorganizations too you know, provided they have the concept of transactions. Of course, whether or not the clear and deterministic behavior happens to be useful or safe is another matter...
  2 2014-04-03 00:09:55 <gmaxwell> only determinstic once they're in the chain though, the unspent transactions are more non-determinstic is mostly what I was thinking about. I'm not quite sure how to clearly express the difference, though I know you understand it too.
  3 2014-04-03 00:11:33 <petertodd> yeah, that's perfectly fair to say
  4 2014-04-03 00:14:57 <petertodd> though note that with regard to unconfirmed transactions, if you have miner validation that doesn't allow a address balance to go negative, then unconfirmed balance adjusting transactions are no big deal - the exact order is non-deterministic, but the outcome is still sane
  5 2014-04-03 00:18:56 <petertodd> also, people often forget the deeper issue that unless you consume globally unique transaction outputs it's non-trivial to have a secure balance-based transation mechanism - you have to ensure a signature authorizing a spend can only be used once
  6 2014-04-03 00:25:27 <kazcw> what about storing a nSpendCount with the balance and signing it as part of the TxIn?
  7 2014-04-03 00:28:03 <gavinandresen> 
ACTION ponders the implications of letting the miners choose the order of a maze of dependent unconfirmed transactions in a balance-based system….  probably either a DoS attack lurking there or some weird, surprising "your transaction won't confirm unless you increase your balance" behavior.

  8 2014-04-03 00:30:07 <gmaxwell> gavinandresen: one problem that is really annyoing is say you want to reissue a transaction in order to revise it or pay a greater fee... in a plain balance based system there appears to be no safe way to do this (except if you always spend all your coins for a given balance..., or if it has some other mechenism).
  9 2014-04-03 00:31:01 <gmaxwell> petertodd: yea consider a 'simplistic' anti-replay mechnism that tried to just not allow an identical spend to be repeated, and how that would have interacted with DSA malleability. 0_o
 10 2014-04-03 00:31:28 <kazcw> gmaxwell: wouldn't the nSpendCount nonce solve that? reuse the nonce to get highlander-semantics
 11 2014-04-03 00:31:39 <kazcw> ("there can be only one")
 12 2014-04-03 00:32:40 <SoftwareMechanic> What's the difference between a "public key x" and a "public key y", and a pubkey (as shown by bitcoind) ?
 13 2014-04-03 00:32:53 <kazcw> an unconfirmed spend would hold up any later spends, which is just like unconifirmed change
 14 2014-04-03 00:33:00 <SoftwareMechanic> I'm confused as to how the pubkey is generated, cause I was thinking it would be 128 bytes
 15 2014-04-03 00:33:07 <gmaxwell> kazcw: perhaps, as I said "additional mechenisms"  but I suspect that when you extend that logically to the most flexible form (e.g. being able to pick and choose a mix of conflicts) you actually end up with something very much like bitcoin.
 16 2014-04-03 00:33:12 <maaku> SoftwareMechanic: a public key is a elliptic curve point, which has an x and y coordinate
 17 2014-04-03 00:33:16 <petertodd> gmaxwell: great example! systems like mastercoin can only get away with it because they're sitting on top of an underlying transaction-based system
 18 2014-04-03 00:33:40 <SoftwareMechanic> @maaku, ya I've got that part.  But how do you go from that to a "pubkey"
 19 2014-04-03 00:33:57 <maaku> SoftwareMechanic: that's all a pubkey is
 20 2014-04-03 00:33:59 <SoftwareMechanic> I guess I'm not really sure what the output of bitcoindd i showing me
 21 2014-04-03 00:34:00 <petertodd> gmaxwell: transactions also save you from having to have some huge "spent tx" list - sequential transactions can be easily pruned
 22 2014-04-03 00:34:31 <SoftwareMechanic> @maaku That's what I though, but if you run bitcoind, the pubkey shown is only 64 bytes long, not 128
 23 2014-04-03 00:34:53 <SoftwareMechanic> It's like it's only showing half of the public key
 24 2014-04-03 00:35:00 <kazcw> SoftwareMechanic: the y of a bitcoin pubkey can be derived from the x, so that's actually what it is
 25 2014-04-03 00:35:15 <SoftwareMechanic> aahh, interesting
 26 2014-04-03 00:35:31 <SoftwareMechanic> since the curve contants are well known
 27 2014-04-03 00:36:32 <gmaxwell> kazcw: well technically you need one additional bit of data to derrive y from x. :)
 28 2014-04-03 00:36:55 <kazcw> SoftwareMechanic: also, compressed format is 32 bytes, uncompressed is 64. Twice that would be the number of hexadecimal chars, if that's what you're looking at
 29 2014-04-03 00:36:56 <gavinandresen> SoftwareMechanic: public keys should be 33 or 65 bytes long (one byte, then 32 bytes for x OR one byte, then 32 for x and 32 for y).  I think.  eight times 32 == 256… yes, that's right.
 30 2014-04-03 00:37:36 <SoftwareMechanic> ah, ok, I was off by one on my editor column count
 31 2014-04-03 00:38:02 <gmaxwell> that additional prefix byte will encode that extra bit if only the x-corrid is provided.
 32 2014-04-03 00:38:47 <SoftwareMechanic> man, this stuff goes deep
 33 2014-04-03 00:39:11 <gavinandresen> … and new versions of bitcoin core always generate compressed (33-byte) keys, so unless you're using an old wallet that's what you'll see
 34 2014-04-03 00:40:24 <SoftwareMechanic> @gavinandresen So what I'm seeing out of 'bitcoind validateaddress' would be the 2 33-byte contatenated keys is what I think I'm hearing.
 35 2014-04-03 00:43:14 <gavinandresen> SoftwareMechanic: umm…. if you are validating a plain-old, begins-with-1 bitcoin address, then you'll get either 33 bytes (0x01 and then 256-bit-x-public-key-coordinate, if I recall correctly) or 65 bytes (0x03/0x04 and then 2 256-bit coordinates)
 36 2014-04-03 00:43:44 <gavinandresen> I don't remember what validateaddress does with p2sh multisig addresses (that begin with '3'), I'd have to look.
 37 2014-04-03 00:44:12 <SoftwareMechanic> This is for the plain single
 38 2014-04-03 00:45:10 <SoftwareMechanic> Looks like it's the 65 byte form, although it's prepended by a 0x02, presumably because it's a testnet address
 39 2014-04-03 00:46:29 <SoftwareMechanic> for multisig it gives a 'hex' string, which I'm presuming is the p2sh script contents
 40 2014-04-03 00:48:16 <kazcw> SoftwareMechanic: is the pubkey 65 bytes, or 65 hex characters?
 41 2014-04-03 00:49:13 <SoftwareMechanic> kazcw: yes, I am an idiot, hex characters
 42 2014-04-03 00:49:32 <SoftwareMechanic> I've looked at way to much hex today
 43 2014-04-03 00:51:53 <Zarutian> SoftwareMechanic: 0x43555253455321
 44 2014-04-03 00:52:16 <SoftwareMechanic> Your not helping
 45 2014-04-03 00:52:18 <SoftwareMechanic> :)
 46 2014-04-03 01:30:00 <BCB> what happens to transactions in the mempool after they are included in a block?
 47 2014-04-03 01:31:55 <phantomcircuit> BCB, they are removed from the mempool
 48 2014-04-03 01:36:44 <BCB> phantomcircuit, ic
 49 2014-04-03 02:35:23 <arge> hi
 50 2014-04-03 02:35:40 <arge> good night
 51 2014-04-03 03:42:51 <tlrobinson> does anyone keep an archive of orophaned blocks?
 52 2014-04-03 03:43:02 <tlrobinson> orphaned
 53 2014-04-03 03:43:21 <vetch> orphaned blocks aren't written to disk.
 54 2014-04-03 03:43:29 <Luke-Jr> blocks aren't orphaned, either.
 55 2014-04-03 03:43:48 <Luke-Jr> they may arrive orphan, but never become orphan ;)
 56 2014-04-03 03:44:18 <Luke-Jr> presumably tlrobinson means stale/rejected blocks
 57 2014-04-03 03:44:44 <tlrobinson> errr yeah
 58 2014-04-03 04:01:33 <joelsbeard> Anyone know a resource on creating your own keys for an address?
 59 2014-04-03 04:01:47 <joelsbeard> Out of curiosity not necesity
 60 2014-04-03 04:03:07 <kazcw> What do you mean? You can create an address from a public key, which you create from a private key. You cannot create a public or private key from an address.
 61 2014-04-03 04:03:42 <joelsbeard> Yeah, I would like to create a private / public key pair which I would then use for a wallet address
 62 2014-04-03 04:05:43 <kazcw> there are several steps involved - generating the keypair, compressing the public key (optional/recommended), hashing that key, base58check-encoding the hash
 63 2014-04-03 04:07:43 <joelsbeard> What would you use to create the keys? Is that something I would do with openssl? Do people create their own keys with something they wrote themselves?
 64 2014-04-03 04:12:31 <Luke-Jr> joelsbeard: usually they use a bitcoin wallet
 65 2014-04-03 04:12:40 <Luke-Jr> keys are not something users mess with
 66 2014-04-03 04:14:05 <joelsbeard> Yeah, I would like to as an exercise only
 67 2014-04-03 04:14:18 <Luke-Jr> https://github.com/sipa/secp256k1 is a library
 68 2014-04-03 04:16:58 <shazow> joelsbeard: also this is a nice high-level library for messing around with things like generating your own wallets/keys: https://github.com/onenameio/coinkit
 69 2014-04-03 04:18:21 <joelsbeard> Nice, thanks Luke-Jr, and shazow
 70 2014-04-03 04:22:57 <dgenr8> What would you say defines the bitcoin protocol.  The behavior of the reference client, or something else?
 71 2014-04-03 04:25:40 <shazow> dgenr8: (I'm probably not the best person to answer this, but) I suspect it's something between the specification in the public documentation and the actual implementation as held by the majority of the computing power in the network.
 72 2014-04-03 04:28:49 <shazow> probably analogous to asking the same question about any other protocol like TCP or HTTP (though some are adhered more closely to the spec than others)
 73 2014-04-03 04:30:54 <maaku> dgenr8: the behavior of the reference client
 74 2014-04-03 04:31:57 <maaku> anything else written about the bitcoin protocol is not normative
 75 2014-04-03 04:32:21 <maaku> (e.g. look at BIP 42)
 76 2014-04-03 04:33:14 <dgenr8> or at least not complete, since ref client has behavior not specified elsewhere
 77 2014-04-03 04:33:53 <vetch> there's even cases where the comments in the bitcoin source are different to how the code behaves.
 78 2014-04-03 04:34:22 <dgenr8> you should fix those comments :)
 79 2014-04-03 04:36:55 <dgenr8> So, for example: reference client appears to have no way to include a transaction in a block, unless it is in the mempool.
 80 2014-04-03 04:37:22 <dgenr8> And there is no way to get a transaction into the mempool, unless is passes IsStandard
 81 2014-04-03 04:37:24 <maaku> dgenr8: it will be a very long time until the bitcoin consensus code is fully understood, again read BIP 42 if you haven't already (bitcoin it turns out, does not have a 21 million cap)
 82 2014-04-03 04:38:17 <maaku> dgenr8: that's not consensus code :)
 83 2014-04-03 04:38:28 <dgenr8> meaning?
 84 2014-04-03 04:38:28 <maaku> IsStandard is not bitcoin protocol
 85 2014-04-03 04:38:45 <dgenr8> It is in the reference client
 86 2014-04-03 04:38:54 <maaku> there is a subset of the bitcoin code which controls whether a block and its transactions are valid or not
 87 2014-04-03 04:39:11 <maaku> *that* is consensus code, aka the bitcoin protocol
 88 2014-04-03 04:39:30 <dgenr8> Yes I was just revieweing CreateBlock.  It looks for transactions in the mempool
 89 2014-04-03 04:39:39 <maaku> for the other stuff, it doesn't matter
 90 2014-04-03 04:39:53 <maaku> you could have a client with no concept of a mempool, and that'd be fine
 91 2014-04-03 04:40:54 <dgenr8> But could you have a client that includes transactions in a block, which were not either created by it, or relayed to it via the bitcoin network
 92 2014-04-03 04:41:10 <maaku> dgenr8: sure, of course
 93 2014-04-03 04:41:17 <maaku> push a transaction to Eligius and that's exactly what happens
 94 2014-04-03 04:41:19 <dgenr8> ... something which is not possible with the reference client
 95 2014-04-03 04:42:02 <dgenr8> So what Eligius does defines the standard then
 96 2014-04-03 04:42:12 <maaku> no, no no
 97 2014-04-03 04:42:26 <maaku> the bitcoin protocol is a *subset* of the reference client
 98 2014-04-03 04:42:40 <maaku> Eligius has to comply with that subset 100%, or its blocks will be orphaned
 99 2014-04-03 04:43:19 <maaku> but Luke-Jr is free to do whatever he wants with the other aspects (e.g. relay policy) for his Eligius pool
100 2014-04-03 04:43:39 <dgenr8> This conversation seems to indicate that what Eligius does is a superset of what the reference client does
101 2014-04-03 04:44:16 <maaku> dgenr8: no, a superset of the consensus code of the reference client, which is itself a subset of what the reference client does
102 2014-04-03 04:44:25 <maaku> does that make sense?
103 2014-04-03 04:44:49 <dgenr8> So CreateBlock is not part of the consensus code?
104 2014-04-03 04:44:56 <maaku> if you drew a graph it'd look like a venn diagram
105 2014-04-03 04:45:40 <maaku> there are a few things CreateBlock does that are consensus-critical, e.g. difficulty adjustments
106 2014-04-03 04:45:48 <maaku> but the rest (e.g. transaction selection) is not
107 2014-04-03 04:45:54 <Luke-Jr> CreateBlock is not part of the consensus code. at all.
108 2014-04-03 04:45:58 <Luke-Jr> AcceptBlock is.
109 2014-04-03 04:46:12 <aynstein> there is a way to generate a block on demand on testnet right?
110 2014-04-03 04:46:20 <Luke-Jr> aynstein: almost
111 2014-04-03 04:46:25 <dgenr8> How does one determine what is part of consensus code?
112 2014-04-03 04:46:27 <maaku> Luke-Jr: well CreateBlock must create something that AcceptBlock accepts ;)
113 2014-04-03 04:46:41 <maaku> dgenr8: by logic
114 2014-04-03 04:47:00 <aynstein> Luke-Jr: can you give me a starting point to learn about it?
115 2014-04-03 04:47:01 <Luke-Jr> dgenr8: if the reference software rejects it as invalid, you've violated consensus code. :P
116 2014-04-03 04:47:15 <Luke-Jr> dgenr8: so for example, a theoretical bug in LevelDB could be consensus code.
117 2014-04-03 04:47:39 <Luke-Jr> aynstein: the rule is "if block time is > 20 minutes past the previous block time, it is allowed to PoW at bdiff 1"
118 2014-04-03 04:47:52 <aynstein> ahhh
119 2014-04-03 04:49:46 <aynstein> i thought i remembered some sort of gen feature in bitcoind... besides the (testnet) diff 1 reset, nothing comes to mind?
120 2014-04-03 04:49:59 <dgenr8> Luke-Jr: by invalid, are you referring to a particular class of error message, the result of a certain function or set of functions, or ...
121 2014-04-03 04:50:20 <Luke-Jr> dgenr8: I mean "invalid".
122 2014-04-03 04:50:24 <maaku> dgenr8: AcceptBlock() returns false
123 2014-04-03 04:50:45 <Luke-Jr> aynstein: that's regtest mode, not testnet
124 2014-04-03 04:51:18 <dgenr8> Luke-Jr: tautology
125 2014-04-03 04:52:00 <maaku> dgenr8: how?
126 2014-04-03 04:52:01 <Luke-Jr> dgenr8: don't ask obvious/dumb questions then
127 2014-04-03 04:52:24 <maaku> dgenr8: grep "INVALID" main.cpp
128 2014-04-03 04:53:24 <maaku> but take note that those explicit tests don't define what is valid or not
129 2014-04-03 04:54:15 <maaku> if some weird bug in a dependent library causes bitcoind's validation thread to crash with an unhandled exception, whatever triggers that exception makes the block invalid
130 2014-04-03 04:54:28 <maaku> and that's a consensus rule we didn't know about
131 2014-04-03 05:01:54 <dgenr8> The reference client is complete, ie capable of running the entire network from generation to wallets, by itself
132 2014-04-03 05:02:24 <dgenr8> It is often said that bitcoin is open source.  So tell me...
133 2014-04-03 05:05:35 <dgenr8> If I wanted to modify the source to check that a transaction which has never been seen on the network, spends the same inputs as one that has
134 2014-04-03 05:05:52 <dgenr8> the public transaction has priority
135 2014-04-03 05:06:15 <dgenr8> It's a rhetorical question becuase with the reference client alone, it already is not possible
136 2014-04-03 05:06:33 <kazcw> that doesn't make any sense
137 2014-04-03 05:07:39 <dgenr8> It doesn't?  I mean priority for inclusion in a new block.
138 2014-04-03 05:14:59 <Luke-Jr> dgenr8: if you didn't see it, you won't include it..
139 2014-04-03 05:15:30 <dgenr8> Indeed.
140 2014-04-03 05:16:13 <Luke-Jr> so, there is no modification needed
141 2014-04-03 05:17:17 <dgenr8> So you would not be opposed to such a rule.  Provided, of course, that the public transaction was seen.
142 2014-04-03 05:18:27 <maaku> dgenr8: I'm not sure what you're asking
143 2014-04-03 05:18:36 <emowataji> how would it get the transaction if its never been seen on the network?
144 2014-04-03 05:18:54 <dgenr8> Excellent question
145 2014-04-03 05:19:02 <Belxjander> maaku: I think it is verification of existing transactions against cached-for-inclusion?
146 2014-04-03 05:19:44 <maaku> Belxjander: existing meaning in the chain? then the conflicting transaction is not in the mempool
147 2014-04-03 05:20:05 <maaku> dgenr8: the reference client ignores transactions double-spending inputs of transactions already in the mempool
148 2014-04-03 05:20:13 <dgenr8> There are a few ways.  Email, carrier pigeon, websockets...
149 2014-04-03 05:20:47 <maaku> dgenr8: then you know which one you received first, and that one is the one the reference client uses
150 2014-04-03 05:21:13 <maaku> for block creation that is
151 2014-04-03 05:22:31 <dgenr8> maaku:  Yes.  I think that behavior gains additional importance as 0-conf transactions become more common.
152 2014-04-03 05:23:41 <maaku> dgenr8: 0-conf transactions will not and should not become more common
153 2014-04-03 05:23:45 <maaku> they cannot be secured
154 2014-04-03 05:23:59 <maaku> reacting to zeroconf transactions is a money-losing mistake
155 2014-04-03 05:24:24 <dgenr8> Yet there is a lot of activity around the payment protocol - for buying coffee, etc.
156 2014-04-03 05:28:13 <maaku> dgenr8: because the effort and risk of double-spending an in-person, small-value PoS transaction is very high
157 2014-04-03 05:28:28 <maaku> but you could do it, and do it easily with the right connections
158 2014-04-03 05:30:00 <maaku> i doubt zero-conf transactions for high value electronics items will be so commonplace
159 2014-04-03 05:30:28 <Luke-Jr> maaku: meh, it can be. just get ID
160 2014-04-03 05:30:55 <Luke-Jr> and sue if you're double-spetn
161 2014-04-03 05:31:24 <dgenr8> Somebody will create a kit to spend to themselves in a non-standard transaction carrier pigeoned to miner of their choice, if they haven't already
162 2014-04-03 05:32:55 <Luke-Jr> dgenr8: so? disputing a credit card isn't much easier
163 2014-04-03 05:32:59 <Luke-Jr> harder*
164 2014-04-03 05:34:42 <dgenr8> Ain't nobody to call when you're double-spent out of a latte
165 2014-04-03 05:35:50 <Luke-Jr> dgenr8: nor when your latte charge is disputed
166 2014-04-03 05:36:10 <dgenr8> But seriously, I think businesses know this instinctively already, and it is holding back adoption.
167 2014-04-03 05:37:29 <Luke-Jr> dgenr8: are you trolling?
168 2014-04-03 05:38:30 <dgenr8> Only for myself.
169 2014-04-03 05:49:14 <[-krypto-]> we are staring a news channel, #AltNews you guys are welcome to join and to give updates on your projects
170 2014-04-03 07:51:58 <Diablo-D3> http://www.reuters.com/article/2014/04/01/us-bitcoin-mtgox-karpeles-idUSBREA3021920140401
171 2014-04-03 11:47:55 <olalonde> question here
172 2014-04-03 11:48:16 <olalonde> what exactly does bitcoind store in the block files? does it store the raw blocks? does it order them in some way?
173 2014-04-03 11:54:17 <vetch> olalonde: just raw blocks concatenated and an index of where each block is on the disk. the order isn't deterministic and isn't the same across instances.
174 2014-04-03 11:55:14 <olalonde> vetch: ok thanks. what about orphan blocks / blocks that are not part of the main chain... are they ever deleted?
175 2014-04-03 11:57:03 <vetch> olalonde: side chains remain untouched. orphans aren't ever written to disk as they aren't part of the chain by definition.
176 2014-04-03 11:59:14 <olalonde> vetch: ok thanks. so in theory if you downloaded the blockchain from scratch you'd save some disk space right? or are sidechains downloaded as well?
177 2014-04-03 11:59:18 <wumpus>  you can execute bitcoind with -printblocktree to show the tree of stored blocks (note, broken in 0.9, but fixed in master)
178 2014-04-03 11:59:35 <olalonde> cool
179 2014-04-03 12:00:21 <vetch> olalonde: probably if you were comparing a very old and very new node there might be a difference but I don't imagine it would be huge. early blocks are fairly small.
180 2014-04-03 12:00:21 <wumpus> olalonde: there is a linearize.py script in the repository that can be used to create a linear bootstrap file, you could then delete the blocks and re-bootstrap from that
181 2014-04-03 12:00:44 <olalonde> wumpus: cool
182 2014-04-03 12:00:50 <olalonde> right
183 2014-04-03 12:00:53 <wumpus> then again you'll save only very little, no large side-chains exist
184 2014-04-03 12:01:14 <wumpus> (at least, I've never seen them...)
185 2014-04-03 12:01:15 <olalonde> right.. I'm just trying to confirm I understood correctly.. not asking a practical question :)
186 2014-04-03 12:02:41 <vetch> compressing the blocks on disk would probably afford you more space if you could do it transparently to the node. they seem to compress surprisingly well.
187 2014-04-03 12:03:22 <vetch> they're basically never accessed so read latency isn't a massive issue.
188 2014-04-03 12:04:23 <wumpus> blocks are read on a per-block granularity, so you need to make sure that you compress per block, compressing the entire file would not be useful
189 2014-04-03 12:05:34 <wumpus> (ie, most compression methods don't allow for seeking in the middle of a file, so the compression blocks would have to align with bitcoin blocks)
190 2014-04-03 12:05:36 <vetch> presumably most of the savings are coming from duplicate public keys, having the compression dictionary spanning files would have the most impact in that case.
191 2014-04-03 12:06:08 <wumpus> so, as concluded many times, probably not worth it for the 15-20% you'd save
192 2014-04-03 12:06:19 <vetch> probably not worth the developer time to work it out. disk space is cheap c
193 2014-04-03 12:07:13 <vetch> still could be fun for distributing the bootstrap files.
194 2014-04-03 12:07:47 <wumpus> nothing prevents you from compressing the bootstrap file already
195 2014-04-03 12:08:36 <vetch> exactly.
196 2014-04-03 12:08:43 <wumpus> I remember someone was even working on making it possible to read a compressed bootstrap file directly in bitcoind, but a pull never materialized
197 2014-04-03 12:09:10 <wumpus> in the end, no one cares about disk space enough to spend enough work on it :)
198 2014-04-03 12:10:30 <vetch> probably lots more important things to work on in the client
199 2014-04-03 12:11:10 <wumpus> you could say that
200 2014-04-03 12:22:50 <olalonde> question regarding SPV nodes
201 2014-04-03 12:29:09 <olalonde> a SPV client when it first connects to the network for the first time would need to download all block headers?
202 2014-04-03 12:29:42 <olalonde> assuming its wallet can contain "used" keys
203 2014-04-03 12:30:38 <wumpus> in principle, yes
204 2014-04-03 12:32:01 <wumpus> although bitcoinj recently added a way to add checkpoints, so that if you trust the checkpoints, block headers only have to be downloaded starting from the last checkpoint before the birthday of the first key
205 2014-04-03 12:32:09 <olalonde> I'm trying to understand why electrum uses trusted servers but also claims to implement SPV. what are the trusted servers used for?
206 2014-04-03 12:32:30 <olalonde> ok
207 2014-04-03 12:32:58 <olalonde> bitcoinj is a SPV client?
208 2014-04-03 12:33:15 <wumpus> I'm not sure whether any of the wallets use that functionality, in general the block headers are considered small enough to not bother
209 2014-04-03 12:33:37 <wumpus> sort of, it's a library that can be used to make a SPV client
210 2014-04-03 12:33:53 <olalonde> ok
211 2014-04-03 12:34:28 <wumpus> I'm surprised that you don't know it, it's the most well known implementation of bitcoin node functionality apart from bitcoind itself
212 2014-04-03 12:35:16 <olalonde> I knew it was a java bitcoin client / library
213 2014-04-03 12:35:43 <olalonde> wasn't sure about the SPV part :P
214 2014-04-03 12:37:52 <kinlo> olalonde: the "trusted" part is that you need a "trusted" server so they don't hide transactions from you
215 2014-04-03 12:38:21 <kinlo> there are a few things a "trusted" server in electrum can do
216 2014-04-03 12:38:40 <kinlo> you just have the block headers, you have no idea which transactions are inside those blocks
217 2014-04-03 12:38:58 <kinlo> so you need to trust that the server, if you ask for all transactions for a certain address, will reply with all transactions.
218 2014-04-03 12:39:34 <kinlo> you also need to trust the server that they will know your balance, as you do tell the server which addresses to verify
219 2014-04-03 12:39:37 <olalonde> doesn't the merkle root in block header sguarantee that you are not given fake transactions though?
220 2014-04-03 12:39:55 <kinlo> olalonde: yes, but it does not guarantee you are giving all transactions
221 2014-04-03 12:39:59 <hearn> wumpus: actually most “good” wallets that use bitcoinj do use checkpoints
222 2014-04-03 12:40:20 <hearn> wumpus: though it’s invisible to users. it reduces initial data usage and speeds up the time at which point block confirmations start appearing
223 2014-04-03 12:40:25 <hearn> wumpus: for instance mutibit and android wallet both do
224 2014-04-03 12:40:40 <kinlo> olalonde: so the server could just reply you with already spent txid's.  Or just hide one large one
225 2014-04-03 12:40:46 <hearn> olalonde: over time the Electrum servers have become less trusted indeed
226 2014-04-03 12:41:02 <kinlo> olalonde: the impact is minimal, but some form of trust must be there
227 2014-04-03 12:41:03 <hearn> olalonde: eventually i anticipate that the difference between Electrum servers and P2P nodes will become smaller and smaller, and one day vanish
228 2014-04-03 12:41:14 <olalonde> I'm not trying to bash on electrum.. I just wonder what the SPV model lacks that the electrum servers make up for
229 2014-04-03 12:41:55 <kinlo> olalonde: the clients still need to learn all their transactions.  electrum just queries a central database - one of the stratum servers it is using
230 2014-04-03 12:41:55 <olalonde> hearn: ok I see
231 2014-04-03 12:42:02 <hearn> mostly, Electrum servers provide “instant query” support. at least that’s the theory
232 2014-04-03 12:42:13 <hearn> so you don’t have to scan the block chain looking for transactions.
233 2014-04-03 12:42:18 <kinlo> multibit just downloads all blocks (with a filter to reduce bandwith) and just doesn't record anything but it's own transactions
234 2014-04-03 12:42:26 <kinlo> so electrum is much faster
235 2014-04-03 12:42:43 <olalonde> I thought the bitcoin protocol allowed to query for individual transactions
236 2014-04-03 12:42:44 <hearn> “real” SPV clients like bitcoinj provide the remote server with a Bloom filter that can match false positives (by design). so in theory it can be more private, although for various reasons bcj doesn’t actually garble the filters much today
237 2014-04-03 12:43:00 <hearn> electrum clients provide the server an ungarbled list of addresses which are looked up in a db
238 2014-04-03 12:43:15 <hearn> so electrum servers are a lot more expensive to run than bitcoin nodes, because calculating the databases is expensive
239 2014-04-03 12:43:56 <vetch> probably once the database exists it's not too expensive to query it, much like a real bitcoin node.
240 2014-04-03 12:45:28 <olalonde> interesting stuff
241 2014-04-03 12:45:52 <vetch> the bloom filters requested by bitcoinj seem to be fairly expensive for the node to process. it's probably not all that scalable in the long term.
242 2014-04-03 12:46:26 <hearn> right, at some point bloom filtering will stop scaling
243 2014-04-03 12:46:38 <hearn> as every user has to be tested against every transaction
244 2014-04-03 12:47:17 <hearn> there are ways to optimise the current implementation, though, which wasn’t really written for max speed
245 2014-04-03 12:48:11 <vetch> hopefully further optimisations don't allow 1/0 again.
246 2014-04-03 12:48:30 <hearn> indeed
247 2014-04-03 12:48:54 <hearn> ideally we would catch UNIX signals and turn them into exceptions, like as is done on Windows
248 2014-04-03 12:49:35 <olalonde> there's something I'm not sure I understand about SPV clients
249 2014-04-03 12:50:25 <hearn> what’s that
250 2014-04-03 12:50:28 <olalonde> it wouldn't be possible to run 2 different SPV clients that have the same wallet right? the SPV client must know about all transactions it created right?
251 2014-04-03 12:50:48 <hearn> of course you can
252 2014-04-03 12:50:56 <hearn> (and people do)
253 2014-04-03 12:51:04 <hearn> they can see their own transactions in the block chain
254 2014-04-03 12:51:06 <olalonde> hmm then I misunderstand something :P
255 2014-04-03 12:51:30 <olalonde> hearn: right.. but isn't the point of a SPV that it doesn't need to download all the blockchain?
256 2014-04-03 12:51:32 <hearn> reading the bloom filtering BIP might help
257 2014-04-03 12:51:41 <hearn> SPV clients don’t download the whole chain, just parts of it
258 2014-04-03 12:51:56 <hearn> they download transactions that match a filter, the block headers, and a merkle branch (mathematical proof) tying them together
259 2014-04-03 12:52:05 <olalonde> so... how does it know it didn't miss a transaction that might be of importance to its wallet?
260 2014-04-03 12:52:18 <olalonde> ok , I will read that BIP I guess
261 2014-04-03 12:52:56 <hearn> it doesn't
262 2014-04-03 12:53:08 <hearn> in theory a remote node could lie-by-omission
263 2014-04-03 12:53:20 <hearn> however you have thousands of nodes to pick from, and SPV clients choose more or less randomly
264 2014-04-03 12:53:26 <hearn> so there isn’t much to be gained from such an attack
265 2014-04-03 12:53:35 <olalonde> ah ok
266 2014-04-03 12:53:51 <olalonde> makes sense
267 2014-04-03 12:57:05 <olalonde> so before BIP 37... SPV clients were just like full nodes except they didn't save all the blocks locally.. only transactions that mattered
268 2014-04-03 12:57:44 <olalonde> after BIP 37.. they can download only the transactions that are relevant to them
269 2014-04-03 12:59:20 <vetch> did they even exist before bloom filtering?
270 2014-04-03 12:59:50 <olalonde> Android "Bitcoin Wallet" apparently
271 2014-04-03 13:00:46 <vetch> if you take it in that sense than the Bitcoin Core client is SPV too. if no peers want to download blocks from you, the block files can "safely" be deleted.
272 2014-04-03 13:01:01 <wumpus> right, without bloom filtering SPV clients would still have to fetch and process the entire blocks
273 2014-04-03 13:01:06 <vetch> you'll crash out as soon as they do, so it's clearly a stupid thing to do.
274 2014-04-03 13:01:45 <wumpus> vetch: you'd still have to add headers-first logic
275 2014-04-03 13:01:58 <olalonde> right
276 2014-04-03 13:02:28 <hearn> vetch: they did yes
277 2014-04-03 13:02:37 <hearn> vetch: matt and I did bloom filtering to make them faster
278 2014-04-03 13:02:39 <wumpus> vetch: and wallet rescans would have to be changed to re-fetch the blocks from peers
279 2014-04-03 13:03:03 <hearn> there are some other differences
280 2014-04-03 13:03:03 <vetch> hearn: interesting, thanks for clearing that up.
281 2014-04-03 13:03:09 <hearn> bitcoinj does not store all headers in RAM like Core does
282 2014-04-03 13:03:14 <hearn> (because they wouldn’t fit)
283 2014-04-03 13:03:24 <olalonde> the worse possible attack on SPV client would be to prevent it from seeing some transactions. not sure how this could be exploited for profit... any ideas?
284 2014-04-03 13:03:40 <gribble> 22.4275970459
285 2014-04-03 13:03:40 <sipa> ;;calc [blocks]*80/1048576
286 2014-04-03 13:04:27 <vetch> olalonde: could make some money revealing people's identity I suppose if you were maliciously inclined.
287 2014-04-03 13:04:36 <olalonde> right
288 2014-04-03 13:04:58 <hearn> identity?
289 2014-04-03 13:05:14 <vetch> olalonde: bloom filters can be set up to return false positives in an attempt to avoid it though.
290 2014-04-03 13:05:22 <wumpus> olalonde: you could also lie to them about unconfirmed transactions, if they show them, as they have no way of verifying whether their inputs are unspent
291 2014-04-03 13:05:58 <olalonde> right
292 2014-04-03 13:06:00 <vetch> hearn: user x wanted to know about a and b, therefore they are the same person. could be damaging in some very edge cases.
293 2014-04-03 13:06:11 <hearn> ah, you mean linkage of addresses
294 2014-04-03 13:06:16 <hearn> that’s what the garbling is for
295 2014-04-03 13:06:19 <hearn> (if we used it)
296 2014-04-03 13:06:20 <sipa> olalonde: and servers can hide transactions
297 2014-04-03 13:07:00 <olalonde> sipa: yes.. my question was how could this attack be exploited for profit? I can't think of a scenario but there is probably one
298 2014-04-03 13:07:13 <vetch> hearn: presumably with enough time and pervasive enough coverage of nodes you could build proof of garbage (assuming it is random each time). is that a reasonable assumption?
299 2014-04-03 13:07:59 <hearn> right, if you can correlate a bunch of filters together you can try to strip out noise. it’s not a perfect way to preserve privacy by any means.
300 2014-04-03 13:08:04 <olalonde> wumpus: right.. but SPV clients use block depth as a proxy to confirm transactions right? seems reasonable
301 2014-04-03 13:08:04 <vetch> hearn: I'm working under the assumption that you could eliminate the garbage by seeing which information is similar between requested filters.
302 2014-04-03 13:08:06 <hearn> more like, “it was easy, so we did it"
303 2014-04-03 13:08:20 <vetch> hearn: understood.
304 2014-04-03 13:08:24 <hearn> vetch: you can try to ensure filters remain consistent from a nodes perspective over time
305 2014-04-03 13:08:34 <hearn> but again, there are limits to what we do to ensure privacy with filtering today
306 2014-04-03 13:09:38 <vetch> hearn: its a useful tradeoff. considerably better privacy than blockchain.info and cloudflare, surely.
307 2014-04-03 13:09:46 <hearn> sure
308 2014-04-03 13:13:04 <hearn> long term i guess we’ll go to some fancy PIR protocol, if we can afford it
309 2014-04-03 13:13:14 <sipa> PIR?
310 2014-04-03 13:13:30 <hearn> private information retrieval. the field of cryptography that deals with oblivious databases and things
311 2014-04-03 13:13:34 <sipa> ic
312 2014-04-03 13:13:43 <hearn> alternatively, we might move to a world where wallets mostly don’t learn about transactions via the block chain
313 2014-04-03 13:14:07 <hearn> and then scanning would become less important
314 2014-04-03 13:14:15 <hearn> however that’d require really huge adoption of bip70
315 2014-04-03 13:15:45 <wumpus> but even then, wouldn't they still want to know whether an transaction was confirmed?
316 2014-04-03 13:16:01 <sipa> wumpus: they'd still run a full node to query for validity, i guess
317 2014-04-03 13:16:22 <sipa> all you need is a maintained utxo set
318 2014-04-03 13:17:29 <wumpus> sipa: yes in the case of a trusted full node a lot more is possible
319 2014-04-03 13:19:03 <hearn> wumpus: for transactions that were sent to them, they just need a merkle branch, which the sender could provide
320 2014-04-03 13:19:28 <hearn> for wanting to know when an unconfirmed transaction confirms, you can still do Bloom filtering, but all you’re matching against is tx hashes at that point, which is super fast
321 2014-04-03 13:22:00 <olalonde> after broadcasting a transaction , couldn't the SPV node just download full blocks until the transaction is confirmed?
322 2014-04-03 13:22:29 <wumpus> why?
323 2014-04-03 13:23:09 <olalonde> assuming privacy is a concern
324 2014-04-03 13:23:18 <hearn> you can’t easily do that on some devices
325 2014-04-03 13:23:23 <hearn> it takes up a lot of bandwidth even today
326 2014-04-03 13:23:36 <hearn> but like i said, you can just set a bloom filter that looks for the pending transactions you have in your wallet
327 2014-04-03 13:23:40 <hearn> and nothing else
328 2014-04-03 13:24:03 <hearn> unless you handed out addresses outside of BIP70, of course, in which case you need to look for payments to them. which in practice all wallets do
329 2014-04-03 13:25:15 <olalonde> right
330 2014-04-03 13:36:44 <arjen-jonathan> Anyone here who is willing to answer a question about CCoinViews?
331 2014-04-03 13:37:45 <arjen-jonathan> Specifically: I'm using GetOutputFor on a CCoinsViewCache to get the output from an entry in vxin. But it triggers an assertion on bitcoind startup.
332 2014-04-03 13:47:47 <arjen-jonathan> Anyone here who is willing to answer a question about CCoinViews?
333 2014-04-03 14:07:58 <wumpus> what assertion?
334 2014-04-03 14:12:32 <sipa> arjen-jonathan: it only works for existing outputs
335 2014-04-03 14:13:40 <sipa> arjen-jonathan: i.e., not yet spent
336 2014-04-03 14:22:26 <arjen-jonathan> sipa: How would I get outputs that ARE spent then? Do I need need to get a view on the mempool?
337 2014-04-03 14:25:50 <sipa> arjen-jonathan: outputs that are spent don't exist in the set of unspent outputs (which is what CCoinsView is)
338 2014-04-03 14:27:03 <sipa> arjen-jonathan: what do you want to accomplish?
339 2014-04-03 14:28:16 <arjen-jonathan> sipa: Aah, I misread the inline documentation.
340 2014-04-03 14:28:58 <arjen-jonathan> sipa: I have a transaction and I'd like to get the outputs that are referenced as the inputs.
341 2014-04-03 14:29:12 <sipa> arjen-jonathan: you need a full transaction index for that
342 2014-04-03 14:29:20 <sipa> arjen-jonathan: and it's not very efficient
343 2014-04-03 14:29:50 <arjen-jonathan> sipa: Makes sense :)
344 2014-04-03 14:30:41 <arjen-jonathan> sipa: But if a coin is unspent, I can get it using CCoinsViewCache getOutputFor?
345 2014-04-03 14:30:49 <sipa> yes
346 2014-04-03 14:30:59 <sipa> well, cache is just a cache
347 2014-04-03 14:31:05 <sipa> the question is unspent where
348 2014-04-03 14:31:18 <sipa> pcoinsTip will give you the UTXO set for the current active block chain
349 2014-04-03 14:31:36 <arjen-jonathan> Ofcourse
350 2014-04-03 14:31:39 <sipa> there's a ccoinsview for the mempool too, which adds unspent transactions on top
351 2014-04-03 14:32:04 <arjen-jonathan> Thanks, that's all I needed to know for now.
352 2014-04-03 14:51:37 <gmaxwell> sipa: time to rescale your hashrate charts again.
353 2014-04-03 14:51:49 <sipa> grrr!
354 2014-04-03 14:52:30 <sipa> updating
355 2014-04-03 15:01:11 <hearn> amazing
356 2014-04-03 15:01:20 <hearn> mining is still speeding up even with the falling price
357 2014-04-03 15:01:22 <HaltingState> sipa, does generate privatekey always succeed? for any 32 byte input?
358 2014-04-03 15:01:26 <hearn> i guess there are still so many efficiencies left to go
359 2014-04-03 15:02:25 <gmaxwell> hearn: there is a backlog of already purchased hardware in the mfgr pipeline. :)
360 2014-04-03 15:03:19 <hearn> sigh
361 2014-04-03 15:03:24 <hearn> network traffic is at the same level it was a year ago though
362 2014-04-03 15:03:30 <gmaxwell> HaltingState: to keep the key space uniform the secret key should be smaller than the order of the curve; though the difference from uniform is so small it probably doesn't really matter.
363 2014-04-03 15:03:31 <hearn> we haven’t really increased usage of bitcoin very much in the past 12 months
364 2014-04-03 15:03:43 <hearn> perhaps the composition has changed
365 2014-04-03 15:03:49 <gmaxwell> hearn: I think usage has increased a ton; a year ago the network was flooded by fake usage.
366 2014-04-03 15:03:54 <hearn> ah yes you’re right
367 2014-04-03 15:03:55 <hearn> https://blockchain.info/charts/n-transactions-excluding-popular
368 2014-04-03 15:03:59 <hearn> that’s such a useful graph
369 2014-04-03 15:04:21 <hearn> so usage roughly doubled
370 2014-04-03 15:04:24 <hearn> not enough >:)
371 2014-04-03 15:04:41 <HaltingState> gmaxwell, but in sipa's library; will generate private key always succeed?: or can it fail for some 32 byte inputs? someone using my library is asking me because of something with deterministic wallet gen
372 2014-04-03 15:04:59 <hearn> 
ACTION needs a break

373 2014-04-03 15:05:03 <hearn> -> walk
374 2014-04-03 15:05:37 <HaltingState> they are sha256 seed value and then passing in the 32 bytes as input and wondering if it can fail to generate private key; i dont think it can, because its just raising the base point to that power and its a "int" technically and not point on curve
375 2014-04-03 15:05:54 <HaltingState> but sipa's library might reject some inputs and that is what i am wondering about
376 2014-04-03 15:06:02 <gmaxwell> I think you should read the code.
377 2014-04-03 15:07:22 <HaltingState> if the probability of failure is 1 in 2^64 or something absurdly low, that is OK
378 2014-04-03 15:07:44 <HaltingState> but if its significant, i need to know; it says it can return 0 for "invalid secret"
379 2014-04-03 15:08:14 <HaltingState> ok; it cant return 0
380 2014-04-03 15:08:15 <gmaxwell> HaltingState: your question is offtopic for this channel (not bitcoin related); but I'd be happy if your usage caused you to go do some code review of code we care about.
381 2014-04-03 15:08:44 <HaltingState> gmaxwell, its bitcoin related because bitcoin will be using this library and its bitcoin deterministic wallet related
382 2014-04-03 15:09:21 <gmaxwell> HaltingState: BIP32 is well specified in this case and requires you to advance to the next key if the value is too large.
383 2014-04-03 15:11:40 <HaltingState> gmaxwell, the question was related to "what happens if privatekey generation fails for some seed input", but it says "Return 1" and never uses the failure branch, so its OK/never happens
384 2014-04-03 15:28:49 <netg_> 11
385 2014-04-03 15:29:15 <antephialtic> "Why do libertarians like functional programming? Because it's stateless."
386 2014-04-03 15:29:56 <danielpbarron> :>
387 2014-04-03 15:37:56 <ThomasV> why is OP_CHECKMULTISIG sensitive to signature order?
388 2014-04-03 15:38:57 <gmaxwell> ThomasV: because otherwise it would need to do N^2 work to validate a signature.
389 2014-04-03 15:40:41 <ThomasV> gmaxwell: makes sense. perhaps it should be mentioned on https://github.com/bitcoin/bips/blob/master/bip-0011.mediawiki
390 2014-04-03 16:40:14 <hearn> sipa: what do you think about a getutxo p2p message. ie, you send a COutPoint and get back a CTxOut
391 2014-04-03 16:42:07 <petertodd> you should get back a CTransaction so it's secure
392 2014-04-03 16:47:21 <maaku> petertodd: we have to develop with pruning in mind
393 2014-04-03 16:49:37 <gmaxwell> hearn: sounds like something that would be super prone to abuse.
394 2014-04-03 16:49:52 <r0sc0e> sorry, i had a disconnect
395 2014-04-03 16:49:54 <hearn> petertodd: you can’t do that
396 2014-04-03 16:49:56 <gmaxwell> e.g. Checkout my awesome image downloader utility that uses the bitcoin network as a DHT.
397 2014-04-03 16:50:01 <hearn> petertodd: without a full index. that’s why i said, getutxo
398 2014-04-03 16:50:07 <hearn> for what i want, i don’t need full transactions anyway
399 2014-04-03 16:50:10 <r0sc0e> can anyone tell me how to fix my problem with blockchain?
400 2014-04-03 16:50:30 <gmaxwell> hearn: petertodd's point is that nodes can just give you garbage data and you probably cannot tell.
401 2014-04-03 16:50:33 <gmaxwell> r0sc0e: wrong channel.
402 2014-04-03 16:51:28 <hearn> it’s sufficient to run a scriptSig in an unbroadcast transaction that you have
403 2014-04-03 16:51:32 <r0sc0e> okay sorry
404 2014-04-03 16:51:40 <sipa> hearn, gmaxwell, petertodd: yeah, i really don't like data that cannot be authenticated to the p2p protocol
405 2014-04-03 16:51:50 <gmaxwell> hearn: ah, I see what you want there.
406 2014-04-03 16:51:51 <sipa> with a committed utxo set, that is different
407 2014-04-03 16:52:11 <hearn> *shrug* authenticated in what way? if you take the UTXO and run it against a tx scriptSig, that authenticates it
408 2014-04-03 16:52:28 <hearn> for the purposes of checking, is this tx likely to be spendable without actually broadcasting it and thus committing it
409 2014-04-03 16:52:58 <gmaxwell> hearn: if your goal is to tell if an unconfirmed transaction is valid or not, I can just give you a bad one, then also give you a bogus utxo that it passes, but it's still not valid.
410 2014-04-03 16:53:16 <hearn> the source of the tx in this case is not the p2p network
411 2014-04-03 16:53:24 <petertodd> maaku: I know that, which is why I said it as to suggest it's not a good idea!
412 2014-04-03 16:54:02 <hearn> and i am willing to make some weak assumptions about the trustworthyness of nodes in this case.
413 2014-04-03 16:54:02 <petertodd> maaku: more concretely, you could do a model where what's actually returned is a block height, and then you get the block to get the actual utxo (it's just a hint then)
414 2014-04-03 16:54:47 <petertodd> hearn: sigh, you really want to give me an excuse to rent some EC2 capacity?
415 2014-04-03 16:54:50 <hearn> the alternative is either for my app to calculate the UTXO set itself (expensive + exposes me to full validation bugs in bitcoinj), or query a Core node for the UTXO set it already has, in which case I could do it via RPC but then my app ends up “requiring” a trusted Core node just because of the message format
416 2014-04-03 16:55:05 <hearn> petertodd: i’d be delighted if you wasted your money!
417 2014-04-03 16:55:16 <gmaxwell> well you're still subject to full validation bugs when checking the transaction.
418 2014-04-03 16:55:33 <gmaxwell> hearn: why not just assume it's valid and realy it and see what happens?
419 2014-04-03 16:55:38 <hearn> yes, indeed, but i can constrain the format of the tx i’m handling in this particular instance. whereas i cannot constrain what ends up in blocks
420 2014-04-03 16:56:36 <hearn> because the tx is not fully valid in this case. i’m playing with SIGHASH_ANYONECANPAY inputs. sigh. do i really need to post full design docs for such basic questions? i would much rather define a new p2p message and constrain it to localhost if necessary, than add a JSON-RPC, because the first form can be more easily exposed to other users in a decentralised case
421 2014-04-03 16:56:59 <hearn> if a node operator desires to do that, for instance. as the alternatives all push people towards more decentralisation
422 2014-04-03 16:57:02 <gmaxwell> It's generally really rude to constrain what coins a third party is spending... and if it's not a third party; why not just trust it?  In any case, the unauthenticated data concern is I think secondary to the problem of turning the bitcoin utxo set into a cheap DHT file trading service.
423 2014-04-03 16:57:04 <maaku> hearn: this is something that will come with UTXO hash commitments
424 2014-04-03 16:57:26 <petertodd> gmaxwell: good to bring that up
425 2014-04-03 16:57:28 <gmaxwell> maaku: will it? thats not entirely clear to me.
426 2014-04-03 16:57:59 <hearn> gmaxwell: people can and do already put custom data into the UTXO set. that should not stop us building useful features for financial applications.
427 2014-04-03 16:58:11 <hearn> maaku: not willing to wait for those: much larger, riskier and more controversial change than just adding a new p2p message
428 2014-04-03 16:58:45 <maaku> hearn: except the message is useless without a committed hash root as reference...
429 2014-04-03 16:58:50 <gmaxwell> hearn: And yes, you actually do have to justify your application when you want to change the bitcoin p2p protocol, esp do so in a way that requires nodes to distribute unauthenticated data, esp with potential efficient abusive uses which are far more obvious than financial ones.
430 2014-04-03 16:59:13 <gmaxwell> The questions being asked aren't unreasonable.
431 2014-04-03 16:59:14 <petertodd> gmaxwell: indeed, if you claim such non-financial uses are an attack, then don't enable cheaper attacks
432 2014-04-03 16:59:46 <gmaxwell> hearn: people do store things today, but right now they aren't very efficient.
433 2014-04-03 16:59:51 <petertodd> gmaxwell: I certainly don't think making the utxo set into a awsome decentralized database is a good idea, and I've probably explored those usecases more than anyone else here
434 2014-04-03 17:00:04 <gmaxwell> petertodd: maybe its unavoidable.
435 2014-04-03 17:00:08 <maaku> petertodd gmaxwell: i would prefer we adopt the phrase non-payment data, since many of these applications are financial...
436 2014-04-03 17:00:15 <hearn> you can easily store files in OP_RETURN outputs and grab them with Bloom filters, today. and that requires filtering entire blocks. this is hardly worse.
437 2014-04-03 17:01:06 <petertodd> gmaxwell: well, e.g. with UTXO expiration w/ TXO fallbacks the worst effects of that usage are restricted
438 2014-04-03 17:01:20 <petertodd> maaku: +1
439 2014-04-03 17:01:24 <gmaxwell> hearn: you have a point, in particular it only increases the key size somewhat (e.g. you have to add a filterspec and a height)
440 2014-04-03 17:02:13 <petertodd> hearn: well that's part of why I proposed prefix filters, so that peopel doing that "grab it via bloom" wouldn't put so much load on the network
441 2014-04-03 17:02:17 <gmaxwell> (right now that is held back by keeping op_return data compariable to the key size via IsStandard(), but the same applies to a free lookup call)
442 2014-04-03 17:02:38 <petertodd> gmaxwell: nah, the way op_return was done is actually worse there - it's cheaper to use other data encoding methods
443 2014-04-03 17:02:57 <gmaxwell> petertodd: they're all pretty limited in size.
444 2014-04-03 17:03:27 <hearn> yes, it may make sense to switch to some other kind of filtering type in future.
445 2014-04-03 17:04:03 <hearn> although i suspect the bulk of the load created by bloom filtering is actually loading and parsing the block into memory
446 2014-04-03 17:04:04 <gmaxwell> hearn: Okay, I mostly withdraw my DHT app concern. Still left with unauthenticated data potentially incentivizing sybil attacks. :(
447 2014-04-03 17:04:46 <gmaxwell> 
ACTION puts his bet on most of the load actually being in the memory allocator while it seralizes and deseralizes things. 

448 2014-04-03 17:05:18 <hearn> for a node that runs with -txindex the merkle branch could be returned too, but then we’d have to convince sipa that a service bit to find such nodes is a good idea :)
449 2014-04-03 17:09:01 <giustoXricordarv>  raid della regina elisabetta 2 a roma
450 2014-04-03 17:15:43 <petertodd> gmaxwell: it's not how limited they are, it's the cost per byte
451 2014-04-03 17:16:27 <petertodd> hearn: it'd be useful to have the infrastructure so that such a service bit could actually be found on the network
452 2014-04-03 17:16:53 <hearn> service bits are included in addr broadcasts and sent as part of getaddr, so that’s not a problem
453 2014-04-03 17:17:13 <hearn> at most, we’d need an extension to getaddr to let a peer ask for nodes matching a bitmask
454 2014-04-03 17:17:18 <hearn> in case there aren’t enough to be in the default return set
455 2014-04-03 17:17:37 <petertodd> hearn: yes, fails for a low % of nodes adopting it though, want to be able to actually find said nodes beter than we can now
456 2014-04-03 17:17:41 <gmaxwell> petertodd: Right mostly I've been thinking in terms of the ratio of key bytes plus overhead bytes to good data bytes.
457 2014-04-03 17:18:30 <petertodd> gmaxwell: exactly - right now P2SH redeemScripts w/ multisig are cheapest
458 2014-04-03 17:19:24 <gmaxwell> hm. and you point out that an unauthenticated utxo query would make utxo storage cheaper. it's a point.
459 2014-04-03 17:20:12 <gmaxwell> (My thinking above was 'okay you can already do this now reasonably cheaply by adding a height and filterspec to your key'.. but indeed it would not be nice to make the utxo the cheaper place to store data)
460 2014-04-03 17:20:13 <hearn> cheapest in what sense?
461 2014-04-03 17:20:26 <petertodd> hearn: cost per byte published
462 2014-04-03 17:20:36 <hearn> P2SH results in larger transactions overall so the fees should be higher
463 2014-04-03 17:21:28 <petertodd> you want to publish 10KB of data in the bitcoin blockchain: with the current IsStandard() rules, what's the cheapest way to do that?
464 2014-04-03 17:22:29 <gmaxwell> hearn: the p2sh increase is pretty small, and it moves data into the scriptsig instead of the utxo set which— all things equal— is desirable.
465 2014-04-03 17:22:56 <hearn> i thought we’re talking about abusive users here, who don’t care about the desirability of such things but rather just whatever results in lowest fees?
466 2014-04-03 17:24:48 <petertodd> yes, and from there perspective, the question is what's the cheapest way to publish their data? remember that you have to include destroyed coins due to the IsDust() rule
467 2014-04-03 17:25:17 <petertodd> also, remember that >3 pubkeys is IsStandard() in a redeemScript, but not a scriptPubKey
468 2014-04-03 17:26:32 <hearn> wait, IsStandard changed for P2SH scripts vs regular scripts?
469 2014-04-03 17:26:35 <hearn> when did that happen?
470 2014-04-03 17:27:12 <hearn> what’s the limit in a redeemScript then?
471 2014-04-03 17:27:48 <petertodd> it's always been like that, the m <= 3 test isn't applied in Solver()
472 2014-04-03 17:28:00 <petertodd> the limit is <=500 byte scriptSig
473 2014-04-03 17:28:27 <petertodd> see https://github.com/bitcoin/bitcoin/pull/3843
474 2014-04-03 17:28:46 <ribasushi> is there a test suite that is "fuzzing" the solver?
475 2014-04-03 17:29:08 <ribasushi> i.e. if I were to write my own validator - is there some prior art of "seemingly ok, but actually invalid" txns?
476 2014-04-03 17:30:28 <petertodd> ribasushi: pull-reqs accepted; it's a good idea
477 2014-04-03 17:31:07 <hearn> ribasushi: there are JSON files that contain hand written tests
478 2014-04-03 17:31:19 <hearn> ribasushi: they are not likely to be comprehensive. i’m not even sure what the coverage is on them
479 2014-04-03 17:31:45 <petertodd> gmaxwell: incidentally, something I noticed is that some of the data-encoding methods make backtracking and re-requesting blocks via bloom filters very attractive - at a huge cost to the network in terms of load of course
480 2014-04-03 17:32:22 <hearn> petertodd: btw i’m not sure bitcoinj does generate non-canonical CHECKMULTISIGs, i think that reading of the code was wrong. but i have to go back and dig through those pull requests again - they changed quite a few times.
481 2014-04-03 17:32:24 <ribasushi> aye
482 2014-04-03 17:32:53 <petertodd> hearn: well if anyone is using bitcoinj for CHECKMULTISIG it must not, because the only non-canonical CHECKMULTISIG I've seen are ones I generated myself
483 2014-04-03 17:33:02 <gmaxwell> petertodd: things that require extra traffic to fetch are also less good for the abuser.
484 2014-04-03 17:33:11 <hearn> yeah right now i don’t think anyone is running micropayment channels in production, and that’s all it’s used for at the moment
485 2014-04-03 17:33:29 <hearn> i wrote a nice server and GUI based file downloading client that used payment channels, but never quite finished it off