1 2015-11-14 03:13:05 <kanzure> jgarzik: it seems that if we are not quick enough to reject an email from someone, the next time they send an email the moderation action applies to both emails at the same time...
  2 2015-11-14 03:13:17 <jgarzik> odd
  3 2015-11-14 04:11:54 <jcorgan> i think if you click on the numbered link you can apply mod actions individually
  4 2015-11-14 04:19:43 <kanzure> oh okay. also, i thought i was in the other channel when i said that- oops.
  5 2015-11-14 04:19:50 <kanzure> (namely, #bitcoin)
  6 2015-11-14 06:26:19 <BlueMatt> jtoomim: "putting validating nodes between spv miners and spv clients" <--- you seem to be suffering from some misunderstanding of security model here
  7 2015-11-14 06:26:40 <BlueMatt> this introduces explicit trust on the part of spv clients to the specific, random, often malicious node they're connected to
  8 2015-11-14 06:27:07 <BlueMatt> sure, if its a node you do actually trust (ie many large merchants operate this way - bitcoin core nodes on the network and their own spv infrastructure that connects only to their own bitcoin core instances) fine
  9 2015-11-14 06:27:15 <BlueMatt> but this is almost never the case for the average spv client
 10 2015-11-14 06:27:32 <BlueMatt> and you do NOT want to have to trust the first node you happen to connect to on the network or Bad Things Happen (TM)
 11 2015-11-14 08:13:20 <b-itcoinssg> can anyone point me a utxo set parser? preferably in nodejs, but C++ is ok too.
 12 2015-11-14 08:40:16 <Luke-Jr> b-itcoinssg: that.. doesn't really make any sense
 13 2015-11-14 08:40:31 <Luke-Jr> the UTXO set doesn't have a well-defined format (yet)
 14 2015-11-14 08:41:16 <b-itcoinssg> K
 15 2015-11-14 08:41:17 <wumpus> right - the format of the UTXO set at this point doesn't matter because blocks don't commit to it
 16 2015-11-14 08:41:45 <wumpus> implementations can use any storage format this is efficient for them
 17 2015-11-14 08:42:14 <wumpus> s/this/that
 18 2015-11-14 08:46:58 <b-itcoinssg> thanks, since I have your attention, I had a question about merkle tree construction, I am wondering why old leafed tree is resolved by duplicating it rather than treating it as a separate branch
 19 2015-11-14 08:47:09 <b-itcoinssg> here is a link to an example https://github.com/chainpoint/chainpoint/issues/8#issuecomment-136762999
 20 2015-11-14 08:48:09 <Luke-Jr> b-itcoinssg: because Satoshi didn't design his merkle trees very well; it's a defect we maintain only because changing the protocol is difficult
 21 2015-11-14 08:48:26 <b-itcoinssg> ok
 22 2015-11-14 08:48:39 <gmaxwell> making it different in size makes the computation irregular, e.g. harder to vectorize.  Though that it also doesn't commit to the member count is unfortunate and results in a second preimage.
 23 2015-11-14 08:49:18 <Luke-Jr> gmaxwell: well, you could always pad with zeros too
 24 2015-11-14 08:51:01 <wumpus> ah yes, he should have added a distinguisher between leaf nodes and intermediate nodes
 25 2015-11-14 08:51:42 <wumpus> do not learn merkle trees from satoshi's implementation - we have an explicit comment about that at the appropriate place in the source code :)
 26 2015-11-14 08:51:47 <Luke-Jr> that too, but it wouldn't fix the problems with duplicating the hash I think
 27 2015-11-14 08:52:53 <wumpus> it would IIRC.. intermediate node: <00> <hash1> <hash2>  leaf node: <01> <datahash>   .. no longer possible to swap them
 28 2015-11-14 08:52:56 <gmaxwell> Luke-Jr: if there was a tag to code interior vs leaf it could also code the number of children, and then it doesn't matter if you copy or use zeros.
 29 2015-11-14 08:52:59 <wumpus> or maybe you're talking about a differnet issue
 30 2015-11-14 08:53:22 <gmaxwell> wumpus: there are two different second preimage problems. He's referring to the other one.
 31 2015-11-14 08:54:47 <wumpus> okay
 32 2015-11-14 08:55:11 <Luke-Jr> speaking of which, did we decide to do anything about the more recently discovered one?
 33 2015-11-14 08:55:34 <Luke-Jr> I forget :/
 34 2015-11-14 08:56:38 <gmaxwell> I wrote a patch but was waiting for figures on the simulation.
 35 2015-11-14 08:56:48 <gmaxwell> fell off my radar, pushing it back on.
 36 2015-11-14 08:59:59 <Luke-Jr> gmaxwell: IIRC we concluded that it was very difficult and had not already occurred accidentally
 37 2015-11-14 09:03:29 <b-itcoinssg> n'way, thanks guys, super helpful as usual, Luke-Jr & Gmaxwell: are you guys on twitter?
 38 2015-11-14 09:05:02 <b-itcoinssg> hehe
 39 2015-11-14 09:07:57 <wumpus> ideally there would be some document or book we can refer people to that want to learn merkle trees the right way
 40 2015-11-14 09:08:12 <wumpus> it's a pretty common question
 41 2015-11-14 09:08:23 <gmaxwell> I refer people to the certificate transparncy RFC.
 42 2015-11-14 09:08:50 <Luke-Jr> someone could write up a merkletree hardfork BIP that never leaves Draft status :P
 43 2015-11-14 09:09:05 <Luke-Jr> (a wiki article may be a better option)
 44 2015-11-14 09:11:13 <wumpus> bleh, the wikipedia article (not very useful in itself for an introduction) has a patent as reference :s
 45 2015-11-14 09:11:41 <Luke-Jr> I meant Bitcoin Wiki :p
 46 2015-11-14 09:12:14 <gmaxwell> wumpus: it's the best kind of patent: an expired one.
 47 2015-11-14 09:12:31 <gmaxwell> (one filed a few days before I was born...)
 48 2015-11-14 09:12:48 <Luke-Jr> lol
 49 2015-11-14 09:41:16 <wumpus> hah! that's good to hear
 50 2015-11-14 10:04:27 <jtoomim> BlueMatt: in theory, it takes only one validation from a trusted party for the probability of an invalid block to be zero. In practice, "trust" is a context-dependent metric that varies between 0 and 1 that reflects the probability that one party is trying to screw the other party over.
 51 2015-11-14 10:05:01 <jtoomim> "trust" increases in contexts in which it is difficult to steal without getting caught
 52 2015-11-14 10:05:43 <jtoomim> if you completely trust a node, then you could take what it says about block validity to be true without further questioning, but that is rarely the case and not the scenario we should be planning for
 53 2015-11-14 10:06:27 <jtoomim> however, we may be able to set up the network in such a way that it is difficult for a node to deceive without getting caught
 54 2015-11-14 10:07:26 <jtoomim> one way is to have each spv node communicate with multiple full nodes and to cross-reference their responses
 55 2015-11-14 10:08:01 <gmaxwell> jtoomim: we have a mechrnism for that, thats what the proof of work in bitcoin solves; iff participants are validating. (/ you wait for several confirmations instead of just a few)
 56 2015-11-14 10:08:06 <jtoomim> another way to build trust is to ask a bunch of innocuous questions mixed in with a few questions that matter
 57 2015-11-14 10:08:48 <jtoomim> how do you prove that the participants are validating?
 58 2015-11-14 10:08:52 <gmaxwell> jtoomim: there is no way to know you're even talking to seperate nodes, eclipse attacks are very easy in p2p networks in general; and software that has done a lot more to stop them than bitcoin are still fairly easily exploited. :(
 59 2015-11-14 10:08:58 <jtoomim> there is only the assumption there that they are
 60 2015-11-14 10:09:12 <gmaxwell> jtoomim: it's a security assumption of the bitcoin system.
 61 2015-11-14 10:09:33 <jtoomim> is it actually defended by anything?
 62 2015-11-14 10:10:17 <jtoomim> what's to stop an attacker from participating in the SPV mining network by posing as a medium-small miner, grabbing invalid blocks when they happen, and using them for their own benefit?
 63 2015-11-14 10:10:18 <gmaxwell> No, it's an assumption; like the other things in level 0 bitcoin assumptions, there is an "economic incentives" argument, but they're not especially strong.
 64 2015-11-14 10:11:11 <jtoomim> if we permit relay of unverified blocks, we make it a little easier to get invalid blocks with valid PoW for attacks like these, but practically not too much easier
 65 2015-11-14 10:11:19 <gmaxwell> jtoomim: bitcoin.pdf assumes no such thing exists as a "spv mining network", because it assumes that the cost of validating is negligible.
 66 2015-11-14 10:11:22 <jtoomim> it's not really much different from connecting to matt's relay network
 67 2015-11-14 10:11:59 <jtoomim> in practice, the cost of validation is small, but not quite negligible
 68 2015-11-14 10:12:00 <gmaxwell> jtoomim: I know, I pointed that out. :) the verificationless mining is it's own problem; I don't think it's a reason to not agressively forward.
 69 2015-11-14 10:12:42 <jtoomim> so my opinion is that we should stop fighting it and try to figure out how to do it as safely as possible
 70 2015-11-14 10:12:47 <gmaxwell> jtoomim: there are many complications, including the presence of dumb issues and "unknown sources of delay"; and mining with verification is the most expident, fully general "solution" to a hundred different potential sources of delay.
 71 2015-11-14 10:13:10 <gmaxwell> And it's one that you can deploy with only a very limited understanding of the bitcoin system (it's actually easier the less you know)
 72 2015-11-14 10:14:01 <jtoomim> (p.s. i had a bit of ethanol earlier, so i'm not as sharp as i could be, sorry)
 73 2015-11-14 10:14:47 <jtoomim> sorry, i'm not sure i know what kind of delay you're talking about
 74 2015-11-14 10:14:59 <gmaxwell> jtoomim: Making it safe requires no SPV users accepting small numbers of confirmations. Attempting to magically solve the sybil problems "somehow" is to discard decades of failed attempts to invent p2p ecash, which absent the POW solution were all failures. :)
 75 2015-11-14 10:15:00 <jtoomim> do you mean propagation delay, or delay in the certainty of a transaction confirmation?
 76 2015-11-14 10:16:14 <jtoomim> i don't see SPV transactions as needing to be 100% safe in order to be useful
 77 2015-11-14 10:16:28 <jtoomim> I see there being a different level of acceptable risk for different types of transactions
 78 2015-11-14 10:16:32 <gmaxwell> jtoomim: I mean anything that would result in you getting orphaned. There are MANY places in the pipeline from network to mining chip and back again that can add delays. And mining off other pools' stratum fragments or headers is a 50 line of code bypass for 3/4th of them that doesn't require you to understand or optimize much of anything.
 79 2015-11-14 10:17:07 <gmaxwell> jtoomim: I never said that, but that kind of probablistic reasoning doesn't work too usefully against intentional attacks, at least much less than it does for random failures.
 80 2015-11-14 10:17:21 <jtoomim> speaking from a business perspective, that is. a merchant can look at their losses due to attacks, and choose the level of verification that causes the amount of inconvenience ot their customers (and consequently lost revenues) that does not exceed their fraud expoure
 81 2015-11-14 10:17:40 <jtoomim> in terms of the game theory, it's a multi-round game
 82 2015-11-14 10:17:56 <jtoomim> letting the attackers win gives them an incentive to try again
 83 2015-11-14 10:18:02 <gmaxwell> Say your webpage has a billion pixels on it, but due to a bug 1 pixel sends 0.01 BTC from your wallet to the clicker.  Now, 1 in a billion, that sounds like pretty good odds in your favor. And perhaps for six months you'll be fine, but then someone finds the pixel and you're bankrupt in an hour.
 84 2015-11-14 10:18:18 <jtoomim> but it also gives the defenders a chance to change their security policies to adapt to the environment
 85 2015-11-14 10:18:26 <jtoomim> sorta like a market for inconvenience
 86 2015-11-14 10:18:39 <gmaxwell> Attackers are adaptive, they are not random. And as a result security percentages have a nasty habbit of being very two sided.. 100% or 0%, especially when the attacker has many degress of freedom.
 87 2015-11-14 10:18:41 <jtoomim> hopefully, lightning will solve the problem completely
 88 2015-11-14 10:19:21 <jtoomim> but... eh, i should stop rambling
 89 2015-11-14 10:19:23 <jtoomim> too much ethanol
 90 2015-11-14 10:19:31 <jtoomim> sorry, i should have just gone to sleep instead of wasting your time
 91 2015-11-14 10:19:42 <gmaxwell> I know of bitcoin merchants that had adopted the "I don't need 100%" and had a pile of hurestics and API calls that they used to protect themselves from zero conf attacks... and then came home one night after dinner and found themselves $100k poorer. :(
 92 2015-11-14 10:20:11 <jtoomim> definitely, merchants of digital goods should be very wary of 0-conf
 93 2015-11-14 10:20:13 <gmaxwell> So, yea, sometimes you can skirt the middle ground effectively; but its quite a gamble, if you estimate wrong the consequences could be big. :-/
 94 2015-11-14 10:21:37 <gmaxwell> And for people working on the software, we don't know how everyone will use it-- worse, we know the users can't be expected to full understand it (esp since we don't) ... and we have to resist moral hazards around optimizations that dump risk on the people who may be least equipt to deal with them. No magic solutions.
 95 2015-11-14 10:23:35 <jtoomim> so the specific concern that we're worried about here with unverified forwarding is that an invalid block will be produced by a mistake, and then forwarded to an attacker who is just waiting for such a block, and has an attack already laid out that uses that block, yes?
 96 2015-11-14 10:23:56 <jtoomim> so perhaps they've set a trap
 97 2015-11-14 10:24:08 <jtoomim> they try to continuously double-spend transactions
 98 2015-11-14 10:24:19 <jtoomim> by sending one transaction to one miner, and a conflicting transaction to another miner
 99 2015-11-14 10:24:21 <gmaxwell> I'm not concerned about unverified forwarding, it has to be tagged, but thats no big deal.
100 2015-11-14 10:24:58 <jtoomim> waiting for one to mine one in an invalid block, which they then hear and report incorrectly to a valid block to an SPV wallet in order to doublespend against that wallet?
101 2015-11-14 10:25:48 <gmaxwell> jtoomim: though, no the invalid block attack pattern is that you rent a modest amount of hashpower, and after some number of days you use it to mine a single block that spends 21 million bitcoins you don't have to _every_ bitcoin service that accepts 1/2/3 conf and will let you round trip the funds right out.
102 2015-11-14 10:26:24 <gmaxwell> Since you're making an invalid block you have no limit to the number of coins you spend, so just use infinity coins and attack as many things as you can fit outputs in the block.
103 2015-11-14 10:26:29 <jtoomim> the issue i see about unverified block forwarding is that it makes it easy to falsely pass it off as a verified block to an unwitting SPV wallet. I don't see it as a big issue. I don't currently see any other issues.
104 2015-11-14 10:27:20 <gmaxwell> Now, maybe no one extends your block and you are successful only with the 1 conf things,  though due to verification free mining you have not terrible odds at also picking up 2 and 3, even if non-verifying miners have 30 second timeouts.
105 2015-11-14 10:27:25 <jtoomim> gmaxwell unverified block forwarding doesn't help the sacrificing-PoW-for-attack-tx attack any as far as i can see
106 2015-11-14 10:29:00 <gmaxwell> jtoomim: I've told you like a bazillion times that I don't think it matters here.  Matt mostly doesn't like it because verification is something like 1% of the total delay budget, so spending time on that seems like a waste to him, esp since maybe it makes verificationless mining more dangerous. (the latter point, I meh at)
107 2015-11-14 10:29:34 <jtoomim> eh, my data so far aren't very good, but it sorta looks like verification is more than 1%
108 2015-11-14 10:29:47 <jtoomim> i should have better data by december 7th though
109 2015-11-14 10:30:13 <gmaxwell> I'm only continuing to talk about the verificationless mining because you keep suggesting that it can be made safe, and I think (and simulation results suggest) that the strategies for making it "safe" don't really.
110 2015-11-14 10:30:19 <jtoomim> also, verification serializes network transmission, which is a bigger deal
111 2015-11-14 10:30:31 <jtoomim> you can't upload anything of a block until you've downloaded the whole thing and verified it
112 2015-11-14 10:30:53 <jtoomim> if you abandon the verification, then you can upload the first packet as soon as you've downloaded the first packet
113 2015-11-14 10:30:54 <gmaxwell> jtoomim: if you're talking about your testing, the testnet setup you have is quite unrepresentative.
114 2015-11-14 10:31:07 <jtoomim> that makes the time constant on the exponential growth function much much smaller
115 2015-11-14 10:31:29 <gmaxwell> No you can't. if you forward without at least checking POW then anyone can knock the network to its kneeds trivially by just sending noise.
116 2015-11-14 10:31:45 <jtoomim> gmaxwell of course it is, but it may still be informative, and it's not entirely clear in which way it's unrepresentative
117 2015-11-14 10:32:04 <jtoomim> no relay network, lots of misconfigured machines, fewer low-bandwidth machines
118 2015-11-14 10:32:31 <jtoomim> and the big kicker is that with the testnet we have a lot of peers that refuse to forward blocks because they think that >1MB is invalid
119 2015-11-14 10:32:53 <gmaxwell> jtoomim: Not clear to you, but that just means its more misleading.  The main reasons its not represenative is signature caching, and no one on testnet is running the relay network protocol.
120 2015-11-14 10:32:59 <jtoomim> but we can still collect the time between getdata complete and updatetip complete
121 2015-11-14 10:34:38 <gmaxwell> The relay network protocol converts most 1MB blocks to just a hand full of kilobytes.  (http://bitcoinrelaynetwork.org/stats.html second graph, log scale)
122 2015-11-14 10:34:52 <jtoomim> yes, i've looked at that
123 2015-11-14 10:35:12 <jtoomim> except it looks like matt updated the stats
124 2015-11-14 10:35:20 <jtoomim> last time i looked, the top graphs were unreadably noisy
125 2015-11-14 10:35:47 <gmaxwell> it just had lots of data, he purged old data after a protocol update which mostly fixed the spam attacks knocking down the hitrate.
126 2015-11-14 10:36:00 <gmaxwell> (you can download the raw data)
127 2015-11-14 10:36:24 <jtoomim> the relay network doesn't work with noncooperative miners who keep transactions secret, so this testing will be slightly more informative on adversarial conditions than cooperative conditions
128 2015-11-14 10:37:17 <gmaxwell> well not actually adversarial though, it's testnet after all. :) Actually adversarial conditions involve sending transactions fast where you want them fast; and slow or not at all otherwise. :)
129 2015-11-14 10:37:27 <gmaxwell> But sure, I don't think it's worthless.
130 2015-11-14 10:38:02 <jtoomim> well, pure selfish mining is a different thing
131 2015-11-14 10:38:16 <gmaxwell> Sorry, I'm a little jaded that for three years academic papers have been using numbers about bitcoin network transmission speed from a paper by cdecker that was horribly outdated when he published it and quite litterally 100x slower than the current software. :)
132 2015-11-14 10:38:24 <jtoomim> volume-enhanced selfish mining is another
133 2015-11-14 10:38:38 <gmaxwell> There are many more things than "selfish mining" in the world. :)
134 2015-11-14 10:38:47 <jtoomim> yeah
135 2015-11-14 10:39:00 <jtoomim> hey, did you read the blocktorrent proposal i posted on the list about a month ago?
136 2015-11-14 10:39:29 <jtoomim> oh btw, i have not yet seen any evidence of slow-start-like behavior on our tcp connections
137 2015-11-14 10:39:37 <gmaxwell> I did, there was an older proposal like it. That approach has a lot of overhead.
138 2015-11-14 10:39:45 <jtoomim> doesn't mean it isn't there, as my temporal resolution was poor
139 2015-11-14 10:40:28 <jtoomim> and the combination with weak blocks?
140 2015-11-14 10:40:57 <gmaxwell> jtoomim: then you must not have any non-low-latency links?  Matt's says he's lucky to get 2mbit for actual timings to a single peer on relay network hub numbers.
141 2015-11-14 10:41:01 <jtoomim> encoding a block as a set of merkle nodes of various levels, possibly with a few insertions and deletions, could be done in advance by the miner
142 2015-11-14 10:41:30 <jtoomim> well matt is sending small messages
143 2015-11-14 10:41:47 <jtoomim> where latency is the most important part of total transit time
144 2015-11-14 10:42:07 <jtoomim> total transit = base latency + size/bandwidth, first order approximation
145 2015-11-14 10:42:31 <jtoomim> we've got a lot of VPSs in our mix, which means very high bandwidth
146 2015-11-14 10:42:38 <jtoomim> 100 Mbps to 1 Gbps
147 2015-11-14 10:42:53 <jtoomim> so does matt
148 2015-11-14 10:43:19 <jtoomim> so the 70-150 ms base latency is the main determinant for him, and a moderate factor for us
149 2015-11-14 10:43:46 <gmaxwell> jtoomim: I didn't see you had a combination of weak blocks.  But in any case, normally 95%+ of the data is already transmitted. sending all that data again, and then O(N) more data for hash tree fragments, is a losing proposition.
150 2015-11-14 10:45:06 <gmaxwell> jtoomim: I don't think thats the right way of thinking about that, fairness in the network falls off rapidly with increased delays; yes, a larger amount transmitted gets proportionally less hit from ramping effects, but thats because its taking enormously longer.
151 2015-11-14 10:45:49 <jtoomim> O(N) more data for hash tree fragments? I don't see that as being true
152 2015-11-14 10:46:16 <jtoomim> probably sqrt(n)
153 2015-11-14 10:46:43 <jtoomim> or n/16, perhaps
154 2015-11-14 10:46:59 <gmaxwell> jtoomim: the total interior nodes in a hashtree is equal to the size of the tree.  There is just a constant factor in there because transactions are variable sizes. :)
155 2015-11-14 10:47:15 <gmaxwell> Yea, sure, a constant.
156 2015-11-14 10:47:20 <jtoomim> you can compute the interior nodes from the leaves
157 2015-11-14 10:47:30 <jtoomim> you can choose which layers of the tree to transmit based on the total size of the block
158 2015-11-14 10:47:49 <jtoomim> or based on what layers you have seen recently in other miners' weak blocks
159 2015-11-14 10:48:20 <gmaxwell> Yes, and then you get a constant reduction based on how large your groups are.
160 2015-11-14 10:48:55 <jtoomim> and if you have that group in the cache already, then you can avoid downloading those hashes and those transactions
161 2015-11-14 10:49:29 <jtoomim> or if the sender knows what merkle nodes you have, then they can encode the actual merkle nodes using those nodes plus a diff
162 2015-11-14 10:50:06 <gmaxwell> jtoomim: ugh. then doing that you get weird edge effects where you can only copy in dyadic chunks. This would require very constrained block construction to get any sharing at all
163 2015-11-14 10:50:34 <jtoomim> not necessarily
164 2015-11-14 10:50:38 <jtoomim> i was worried about that too
165 2015-11-14 10:50:54 <jtoomim> but you can encode the block as a list of merkle nodes of different levels
166 2015-11-14 10:50:59 <gmaxwell> This is needlessly complex, it's quite possible to efficiently use already known knoweldge without any constraints on block construction.
167 2015-11-14 10:51:13 <jtoomim> and just reconstruct the base list, and then go back to the actual merkel tree for the block
168 2015-11-14 10:51:38 <jtoomim> basically you can just use merkle nodes as a dictionary of transactions
169 2015-11-14 10:52:03 <jtoomim> you lose the nice property of being able to identify that a hash is part of the root in some ways, but we don't need that
170 2015-11-14 10:52:23 <jtoomim> mostly what we're doing is just using the merkle hashes as a ready handle for a bundle of txs
171 2015-11-14 10:52:55 <jtoomim> kind of like a dictionary based compression algorithm
172 2015-11-14 10:53:04 <gmaxwell> Looking up your post, you said that the IBLT stuff requires strict ordering; that isn't so-- though I can see why thinking that would send you down the hashtree parcel path.  Thats what gavin wrote up, and it simplifies things; but it's not something that was part of my earlier designs in this space. https://en.bitcoin.it/wiki/User:Gmaxwell/block_network_coding
173 2015-11-14 10:55:10 <gmaxwell> in any case, the point I was making about your dictionary is that unless the block is constructed in a very predictable ordering, you need an exponentially sized dictionary to map out all the possible ways of mixing and matching.
174 2015-11-14 10:55:37 <gmaxwell> even if you don't even try to prove membership under a block header anymore.
175 2015-11-14 10:55:48 <jtoomim> yes, you don't want to encode every possible arrangement
176 2015-11-14 10:57:25 <jtoomim> it's meant to be an approach that works quite well with typical blocks, and devloves to being a little worse than an approach that just sends the block as a bunch of raw bytes in adversarial conditions with 0% cache hit rates
177 2015-11-14 10:58:09 <jtoomim> except that it allows for better distribution of bandwidth among peers
178 2015-11-14 10:59:40 <jtoomim> yeah, i'm not sure how important proving membership is before download has been initiated if the encoding hits the recipient's cache
179 2015-11-14 11:00:03 <gmaxwell> The consequence of not proving membership means you cannot relay until you have all the data.
180 2015-11-14 11:00:40 <gmaxwell> (else someone tells they have a 'block' and feed you megabytes of linenoise which the whole network spams itself with)
181 2015-11-14 11:00:45 <jtoomim> you can either miss the cache with a provable encoding, or you can more often hit the cache using an encoding that can only be proven after the cache hit (and maybe a few adjacent ones) has been made
182 2015-11-14 11:01:38 <jtoomim> you don't need all of the data, just a section that has mod 2^n borders
183 2015-11-14 11:02:10 <gmaxwell> and then the overhead of the membership proofs for that part.
184 2015-11-14 11:02:11 <jtoomim> so you can authoritatively forward subsets of what you have downloaded. complicated
185 2015-11-14 11:02:19 <gmaxwell> Yea.
186 2015-11-14 11:03:23 <gmaxwell> jtoomim: though there are I think much simpler and more effective schemes for typical blocks; e.g. remember the last N txids you communicated with a peer, then send the difference in the feerate sorted ID list and any missing transactions. This is what the relay network protocol does, though it doesn't even bother with the differential encoding of the indexes, because they're so small.
187 2015-11-14 11:03:34 <jtoomim> maybe just sending the block with the most efficient coding and banning peers that send invalid block encodings will likely be sufficient
188 2015-11-14 11:03:46 <jtoomim> or maybe sending encodings for a provable node
189 2015-11-14 11:04:04 <jtoomim> and allowing a little bit of overlap
190 2015-11-14 11:04:14 <gmaxwell> jtoomim: banning is fine, but you cannot couple banning with blind forwarding. (else someone sends in junk then the whole network bans each other. :) )
191 2015-11-14 11:05:27 <gmaxwell> And also as I mentioned IBLT doesn't need fixed ordering. It's just if you imagine the ordering is fixed you don't have to communicate the ordering. But if the ordering is pretty predictable (as you assume) then sending the ordering doesn't take much space.
192 2015-11-14 11:05:37 <jtoomim> yes, i get that
193 2015-11-14 11:05:49 <jtoomim> well, i get that now
194 2015-11-14 11:06:50 <gmaxwell> After the iblt process you have the set of txn but don't know what order, you could feerate order them, then send the set of edits from that. :)  the worst case size is log2(n!) but anything that would have a hope of working with your hashing grouping would be pretty small with a simple run-length encoded edit list.
195 2015-11-14 11:06:58 <jtoomim> i think there are a lot of optimizations that could be done with the blocktorrent method to compress the data
196 2015-11-14 11:07:24 <jtoomim> but without them, i think it would use resources a lot more effectively than what we have now, and would not be much more complicated
197 2015-11-14 11:07:56 <jtoomim> the optimizations are what add the most complexity
198 2015-11-14 11:08:44 <gmaxwell> Well if you think it wouldn't be complicated I'd suggest implementing it.  The relay network protocol sounds much simpler than what you suggest, and yet it gets most of the gains of fancier schemes; so I can't argue your point that simple schemes can be much better than the dumbest possible thing (the bitcoin p2p handling of blocks)
199 2015-11-14 11:09:01 <gmaxwell> (er I can't argue with)
200 2015-11-14 11:09:39 <jtoomim> it can be implemented as a recursive algorithm fairly easily with a minimal number of p2p messages added or extended
201 2015-11-14 11:09:58 <jtoomim> i'd like to implement it. time is a finite resource.
202 2015-11-14 11:10:07 <gmaxwell> but if simple is the goal, the block relay protocol basically ticks that box. Hard to be simpler.  If the goal is to be nearly as efficient as possible; the coding-theory based schemes are asymtopically optimal.
203 2015-11-14 11:11:18 <gmaxwell> maybe the middle is also useful, dunno. I'm doubtful anyone will implement it if you won't: Most people don't actually care about working on any of this, when it comes down to actually doing it.  Alas. :)
204 2015-11-14 11:11:39 <jtoomim> the goal is to be computationally simple while making good use of bandwidth and engaging a large number of nodes' bandwidth quickly
205 2015-11-14 11:12:10 <jtoomim> to reduce the time needed to get everybody a copy of the block to be close to the time needed to get one person a copy of the block
206 2015-11-14 11:12:41 <jtoomim> to be nearly O(1) on the number of network nodes
207 2015-11-14 11:12:51 <gmaxwell> well on the en.bitcoin.it page above I give a scheme which basically does that network wide while using most of the available cross-sectional bandwidth.
208 2015-11-14 11:12:58 <gmaxwell> I turns out to end up cpu limited. :(
209 2015-11-14 11:13:03 <gmaxwell> er s/I/it/
210 2015-11-14 11:18:51 <gmaxwell> jtoomim: your scheme with your optimizations and relaxiations starts reminding me of spinal codes, btw. http://nms.csail.mit.edu/spinal/
211 2015-11-14 16:16:13 <jtoomim> is there a way to set multiple debug= settings at the same time without doing all of them (e.g. net and bench)?
212 2015-11-14 16:18:42 <sipa_> yes
213 2015-11-14 16:18:52 <sipa_> -debug=x -debug=y
214 2015-11-14 16:19:39 <jtoomim> and that works in bitcoin.conf too?
215 2015-11-14 16:30:30 <sipa_> jtoomim: yes
216 2015-11-14 16:30:49 <jtoomim> great, thanks sipa
217 2015-11-14 17:47:20 <fredrin> wow, many people, such silence.
218 2015-11-14 17:48:27 <sipa> such idle
219 2015-11-14 17:50:10 <fredrin> true
220 2015-11-14 17:50:25 <fredrin> what are people doing on a saturday?
221 2015-11-14 17:50:38 <fredrin> Whatching terra news
222 2015-11-14 17:58:02 <andre_pt> hy, i m from portugal
223 2015-11-14 17:58:35 <andre_pt> where can i found more info and inscription?
224 2015-11-14 17:59:35 <andre_pt> i would like to work with translations and any kund of back office
225 2015-11-14 18:02:41 <fredrin> aaaaand he's gone
226 2015-11-14 18:02:54 <jl2012> hi sipa, would you like to fix OP_IF/NOTIF malleability as part of BIP62, or let it be a seperate BIP?
227 2015-11-14 18:02:58 <jl2012> https://github.com/jl2012/bips/blob/master/opifmalleability.mediawiki
228 2015-11-14 18:07:24 <sipa> jl2012: i personally plan on retiring bip62, and work on more inclusive solutions like normalized txids or segregated witness instead
229 2015-11-14 18:07:48 <sipa> bip62 is a rathole... adding new types of transaction may always require more rules
230 2015-11-14 18:17:46 <jl2012> sipa: it makes sense
231 2015-11-14 18:32:17 <mbruder> http://btcforparis.blogspot.gr/
232 2015-11-14 18:34:51 <jcorgan> i think it's about that time we try to populate bitcoin-discuss and shift the macro-goals and blocksize from -dev to -discuss
233 2015-11-14 21:32:14 <jcorgan> or not, it seems
234 2015-11-14 23:10:10 <nan`> anyone here do ethereum dev? #ethereum-dev is always dead
235 2015-11-14 23:27:16 <midnightmagic> Hrm..
236 2015-11-14 23:28:21 <midnightmagic> It looks like I get the same gitian sigs as 0.10.3 when I build 0.10.3rc2.