1 2016-01-03 02:26:33 <rusty> kanzure, btcdrak: hmm, so Marco Falke doesn't understand what a soft fork is.  What do we do about his post sitting in the mod queue?
  2 2016-01-03 02:26:57 <kanzure> reject
  3 2016-01-03 02:27:11 <kanzure> ask him to rewrite once he reads up on soft-forks (see bip99)
  4 2016-01-03 02:28:30 <rusty> kanzure: OK.
  5 2016-01-03 02:30:08 <PROTAX> has anyone had experience with Vaultoro?  Is it a scam?
  6 2016-01-03 12:57:02 <GamerSg> will disabling of serving bloom filters be in 0.12?
  7 2016-01-03 13:56:04 <Luke-Jr> GamerSg: #bitcoin
  8 2016-01-03 15:10:33 <jtoomim> I would like to reserve a protocol services bit for use with blocktorrent. Recommendations?
  9 2016-01-03 15:11:45 <sipa> my suggestion would be to first write it as a proxy, which can run over its own network, and to which you can connect using the regular p2p protocol
 10 2016-01-03 15:11:56 <sipa> like BlueMatt's relay protocol
 11 2016-01-03 15:12:46 <sipa> that means it's automatically compatible with all software
 12 2016-01-03 15:15:42 <jtoomim> i want access to the contents of mempool
 13 2016-01-03 15:16:05 <jtoomim> that means it may be a bit easier to write as a module inside bitcoind
 14 2016-01-03 15:16:19 <jtoomim> rather than have to keep a copy of the contents of mempool
 15 2016-01-03 15:16:24 <sipa> hmm, how can that work?
 16 2016-01-03 15:16:36 <sipa> the mempool cannot guarantee synchronization
 17 2016-01-03 15:16:40 <sipa> unless you control it yourself
 18 2016-01-03 15:17:02 <jtoomim> i intend to have a two-tier system for storing transactions
 19 2016-01-03 15:17:09 <jtoomim> so that we treat the mempool basically as a cache
 20 2016-01-03 15:17:25 <sipa> how do you know what is in the peer's mempool?
 21 2016-01-03 15:17:34 <jtoomim> if a transaction is in mempool, great, we copy it to the per-block blocktorrent storage just in case mempool evicts it, then proceed
 22 2016-01-03 15:17:59 <jtoomim> each peer's mempool? we don't necessarily. if they need it, and we don't guess correctly, then they ask
 23 2016-01-03 15:18:14 <sipa> does that not introduce extra roundtrips?
 24 2016-01-03 15:18:17 <jtoomim> isn't there a bloom filter in Core somewhere that tells us when a peer is likely to have a transaction?
 25 2016-01-03 15:18:25 <jtoomim> yes, if you guess incorrectly
 26 2016-01-03 15:18:35 <jtoomim> but with blocktorrent, most of the traffic is going to be over low-latency links
 27 2016-01-03 15:18:53 <jtoomim> and you can guess correctly a lot of the time
 28 2016-01-03 15:19:03 <jtoomim> e.g. using bloom filters
 29 2016-01-03 15:19:17 <jtoomim> or bitmaps
 30 2016-01-03 15:19:22 <sipa> you can guess, but the cost is large is you're wrong
 31 2016-01-03 15:19:25 <jtoomim> that are filled out based on inv messages
 32 2016-01-03 15:19:34 <jtoomim> well, the cost is one round trip if you're wrong
 33 2016-01-03 15:19:46 <jtoomim> which i think will be 100 ms in the typical case
 34 2016-01-03 15:19:55 <jtoomim> since we can find the low-latency peers and prefer them easily
 35 2016-01-03 15:20:04 <jtoomim> i think 100 ms is not "large", but more like "medium"
 36 2016-01-03 15:20:16 <sipa> in that time you can transfer a full block if the link is 100 Mbit/s
 37 2016-01-03 15:20:31 <jtoomim> not with the current MSG_BLOCK interface
 38 2016-01-03 15:20:34 <jtoomim> TCP is slow
 39 2016-01-03 15:20:41 <jtoomim> if we used UDP, then yeah
 40 2016-01-03 15:20:44 <sipa> fair enough, improvements are certainly possible
 41 2016-01-03 15:20:55 <sipa> what i'm getting at is: how is superior to the relay protocol?
 42 2016-01-03 15:21:04 <jtoomim> i'm trying to write blocktorrent in a way that it will have good performance for very large blocks while allowing optimizations for small blocks
 43 2016-01-03 15:21:17 <jtoomim> so i will probably add a lot of code for blasting out information on small blocks without it being requested
 44 2016-01-03 15:21:33 <jtoomim> and aiming for 20% of the data being sent redundantly and unnecessarily in order to reduce latency
 45 2016-01-03 15:21:36 <sipa> if you want the system to work in an adverserial condition, it's the worst case that counts
 46 2016-01-03 15:21:44 <jtoomim> and having that percentage controllable with a mapargs
 47 2016-01-03 15:21:57 <jtoomim> yep, and i think that's one of the places where blocktorrent shines
 48 2016-01-03 15:22:12 <jtoomim> since most of blocktorrent's advantages come from non-compression techniques
 49 2016-01-03 15:22:21 <sipa> do you know how the relay protocol works?
 50 2016-01-03 15:22:33 <sipa> BlueMatt's
 51 2016-01-03 15:22:39 <jtoomim> i think so
 52 2016-01-03 15:22:46 <jtoomim> you keep a record of the order in which each transaction is sent
 53 2016-01-03 15:22:53 <sipa> it never needs a second transmission
 54 2016-01-03 15:22:54 <jtoomim> and code each transaction as a lookup to that table
 55 2016-01-03 15:22:55 <jtoomim> 2 bytes
 56 2016-01-03 15:23:10 <sipa> because it has a guarantee about what the peer maintains
 57 2016-01-03 15:23:14 <jtoomim> which results in fairly heavy memory usage per per
 58 2016-01-03 15:23:23 <jtoomim> per peer
 59 2016-01-03 15:23:26 <jtoomim> sorry, bad "e" key
 60 2016-01-03 15:23:45 <sipa> yeah, it natually favors a hub-and-spoke model to reduce memory consumption (or at least: to share it)
 61 2016-01-03 15:24:09 <sipa> but if you're going for a probabilistic mechanism, where you try to guess, why not use IBLT immediately?
 62 2016-01-03 15:24:16 <jtoomim> i think that no-round-trips is a good design feature for small blocks with low-latency, but i want to build a protocol that is strong where the existing proposals are weak
 63 2016-01-03 15:24:27 <jtoomim> and isn't too weak where the existing protocols are strong
 64 2016-01-03 15:24:48 <jtoomim> so what i'm trying to build is a system in which the guesses aren't important
 65 2016-01-03 15:25:07 <jtoomim> where it's fast when there is zero data in mempool that can be used for the block
 66 2016-01-03 15:26:11 <jtoomim> where you can get a 10 MB block to the whole network in about two seconds, with no significant slowdown when crossing the GFW
 67 2016-01-03 15:26:16 <jtoomim> even in adversarial conditions
 68 2016-01-03 15:26:53 <jtoomim> with 100% unpublished transactions
 69 2016-01-03 15:27:01 <sipa> how can that work?
 70 2016-01-03 15:27:27 <jtoomim> let's say you are the originating peer
 71 2016-01-03 15:27:39 <jtoomim> you send a different chunk of the block to each of your 50-100 peers
 72 2016-01-03 15:27:46 <sipa> oh, ok
 73 2016-01-03 15:28:05 <jtoomim> you include enough merkle info for people to be able to prove that their chunk is part of the whole block
 74 2016-01-03 15:28:10 <jtoomim> they check PoW
 75 2016-01-03 15:28:12 <jtoomim> then share
 76 2016-01-03 15:28:22 <jtoomim> then you're basically done
 77 2016-01-03 15:28:35 <jtoomim> there are a ton of optimizations that can be done, though
 78 2016-01-03 15:28:40 <sipa> yeah, i tried that a few years ago, but the network stack in bitcoin core was too littered with waits back then, and it was many times slower than just sending the block
 79 2016-01-03 15:28:48 <jtoomim> for example, guessing when you can refer to a tx using only 4 bytes instead of 32 bytes or the full tx
 80 2016-01-03 15:28:49 <sipa> i'm sure it can work, and for larger blocks it may be necessary
 81 2016-01-03 15:29:05 <sipa> however, a few misbehaving peers can slow down the reconstruction a lot
 82 2016-01-03 15:29:06 <jtoomim> yeah, i'm planning on using the existing network stack mostly for peer discovery
 83 2016-01-03 15:29:16 <jtoomim> which is why i want the service bit
 84 2016-01-03 15:29:29 <jtoomim> i intend to do most of the implementation using UDP
 85 2016-01-03 15:30:16 <jtoomim> with maybe a parallel TCP channel used for control information
 86 2016-01-03 15:30:26 <jtoomim> and maybe an option for a TCP variant, i don't know
 87 2016-01-03 15:31:11 <jtoomim> do we have a list somewhere of what service bits are currently in use in production and testing code?
 88 2016-01-03 15:31:44 <sipa> see protocol.h
 89 2016-01-03 15:32:48 <jtoomim> i remember seeing a comment somewhere by someone who wanted to make use of another service bit about a month or two ago
 90 2016-01-03 15:32:53 <jtoomim> can't remember who, though
 91 2016-01-03 15:33:39 <jtoomim> so i guess i should just pick a number between 24-31, and hope it's not his (or verify)...
 92 2016-01-03 15:35:18 <sipa> how do you negotiate which peer will send what?
 93 2016-01-03 15:35:58 <sipa> the receiver chooses which fragment to request from each peer?
 94 2016-01-03 15:36:26 <sipa> how does the mempool fit in?
 95 2016-01-03 15:47:52 <jgarzik> jtoomim, FYI I did some UDP work years ago: https://github.com/jgarzik/bitcoin/tree/udp
 96 2016-01-03 15:53:58 <jtoomim> sipa with a mixture of volunteered/unrequested data (not to exceed some per-peer DoS limits) and round-trip requests
 97 2016-01-03 15:54:15 <jtoomim> but yes, the main code path is with the receiver requesting chunks
 98 2016-01-03 15:54:20 <jtoomim> or individual txs that are missing
 99 2016-01-03 15:54:40 <jtoomim> using bitmaps, indices, and/or hashes
100 2016-01-03 15:54:54 <jtoomim> jgarzik thanks, i'll take a look
101 2016-01-03 17:18:47 <jtoomim> sipa the mempool fits in because the receiver can choose not to request the full txs for hashes if the raw tx is already in mempool
102 2016-01-03 17:19:07 <jtoomim> and the sender can choose to relay the full tx without waiting for a request if it can guess that it is missing
103 2016-01-03 17:19:42 <jtoomim> http://toom.im/blocktorrent/
104 2016-01-03 17:20:18 <sipa> jtoomim: that means you first need to tell the receiver what transactions are in the block
105 2016-01-03 17:20:24 <jtoomim> that's about 3 months old, about 2 days after i first came up with the idea, so my ideas have evolved a bit since then
106 2016-01-03 17:20:34 <jtoomim> sipa correct, mostly
107 2016-01-03 17:20:47 <jtoomim> but that can be done using a chunkwise algorithm too
108 2016-01-03 17:20:50 <jtoomim> using the merkle tree
109 2016-01-03 17:21:14 <jtoomim> and you can request the transactions in the merkle tree one branch at a time, either up to the hash, or up to the full tx
110 2016-01-03 17:21:36 <sipa> if you introduce more roundtrips it's easy :)
111 2016-01-03 17:21:50 <jtoomim> and you can also use weak blocks to encode those branches, by encoding each branch as a merkle root, a start index, an end, and some diff
112 2016-01-03 17:22:28 <jtoomim> and for small blocks (e.g. 1 MB), you can skip a lot of those round trips by just broadcasting a lot of those first steps (since they'll be small, less than 1 packet)
113 2016-01-03 17:23:02 <jtoomim> the thing is, since you're doing these round trips to a lot of peers simultaneously, you're not stuck with round trips to distant peers
114 2016-01-03 17:23:15 <jtoomim> so you get much lower round trip times than the typical 200 ms you get with a random peer
115 2016-01-03 17:24:09 <jtoomim> for a 1 to 10 MB block in adversarial conditions, i'm expecting about 1/4 to 1/2 of our time budget to be taken up in round trip latencies, at least for the initial naive version
116 2016-01-03 17:24:39 <jtoomim> and that's okay, i'm not trying to get the best possible latency; relay network does that pretty well already, and IBLTs will do that better as well
117 2016-01-03 17:24:52 <sipa> do you mind walking me through the steps/
118 2016-01-03 17:24:56 <jtoomim> the point of blocktorrent is that the performance should never be *bad*
119 2016-01-03 17:25:06 <jtoomim> have you read http://toom.im/blocktorrent/ yet or not?
120 2016-01-03 17:25:12 <sipa> i have not; i'll do so later
121 2016-01-03 17:25:14 <jtoomim> (need to know where to start)
122 2016-01-03 17:25:16 <jtoomim> ok
123 2016-01-03 17:25:34 <jtoomim> 0. (Optional steps to build a Merkle cache; described later)
124 2016-01-03 17:25:34 <jtoomim> 1. A seed node mines a block.
125 2016-01-03 17:25:34 <jtoomim> 2. It notifies its peers that it has a new block with an extended version of inv.
126 2016-01-03 17:25:36 <jtoomim> 3. The leech peers request the block header.
127 2016-01-03 17:25:38 <jtoomim> 4. The seed sends the block header. The leech code path splits into two.
128 2016-01-03 17:25:40 <jtoomim> 5(a). The leeches verify the block header, including the PoW. If the header is valid,
129 2016-01-03 17:25:42 <jtoomim> 6(a). They notify their peers that they have a header for an unverified new block with an extended version of inv, looping back to 2. above. If it is invalid, they abort thread (b).
130 2016-01-03 17:25:44 <jtoomim> 5(b). The leeches request the Nth row (from the root) of the transaction Merkle tree, where N might typically be between 2 and 10. That corresponds to about 1/4th to 1/1024th of the transactions. The leeches also request a bitfield indicating which of the Merkle nodes the seed has leaves for. The seed supplies this (0xFFFF...).
131 2016-01-03 17:25:46 <jtoomim> 6(b). The leeches calculate all parent node hashes in the Merkle tree, and verify that the root hash is as described in the header.
132 2016-01-03 17:25:48 <jtoomim> 7. The leeches search their Merkle hash cache to see if they have the leaves (transaction hashes and/or transactions) for that node already.
133 2016-01-03 17:25:49 <sipa> no need to repeat everything here, i'll read it :)
134 2016-01-03 17:25:50 <jtoomim> 8. The leeches send a bitfield request to the node indicating which Merkle nodes they want the leaves for.
135 2016-01-03 17:25:52 <jtoomim> 9. The seed responds by sending leaves (either txn hashes or full transactions, depending on benchmark results) to the leeches in whatever order it decides is optimal for the network.
136 2016-01-03 17:25:54 <jtoomim> 10. The leeches verify that the leaves hash into the ancestor node hashes that they already have.
137 2016-01-03 17:25:56 <jtoomim> 11. The leeches begin sharing leaves with each other.
138 2016-01-03 17:25:58 <jtoomim> 12. If the leaves are txn hashes, they check their cache for the actual transactions. If they are missing it, they request the txns with a getdata, or all of the txns they're missing (as a list) with a few batch getdatas.
139 2016-01-03 17:26:00 <jtoomim> that's the steps as written in blocktorrent, trimmed to the essential parts
140 2016-01-03 17:26:32 <jtoomim> the short version: you download some intermediate level of the merkle tree nodes (e.g. the 3rd level), splitting the block into chunks of maybe 64 transactions each
141 2016-01-03 17:26:32 <sipa> steps 2-4 are already replaced by just announcing the header, so you can skip that :)
142 2016-01-03 17:27:10 <jtoomim> then you download chunks from a different peer in parallel
143 2016-01-03 17:27:19 <sipa> yeah, that was my assumptions; but you'll still need to reveal all the leaves (at least to some shorted hash) if nodes need to be able to recognize mempool txn
144 2016-01-03 17:27:26 <jtoomim> and when there are only a few chunks left, you download each chunk from multiple peers simultaneously to avoid bad/slow peers
145 2016-01-03 17:27:37 <jtoomim> sipa, correct
146 2016-01-03 17:28:16 <sipa> have you heard about gmaxwell's recoverable hash idea?
147 2016-01-03 17:28:25 <sipa> it may apply
148 2016-01-03 17:28:31 <jtoomim> not by that name, and possibly not at all
149 2016-01-03 17:29:01 <sipa> so instead of using sha256^2 for txids, you define another 'hash'
150 2016-01-03 17:29:03 <jtoomim> we had an idea where you can find the hash using only 4 or 5 bytes of it pretty reliably in non-adversarial conditions
151 2016-01-03 17:29:39 <jtoomim> and we could fall back to full hashes if collision attempts were detected, usually without a round-trip
152 2016-01-03 17:29:55 <sipa> you take the transaction, serialized in full, and encrypt it with aes-cbc with a fixed static key, then reverse the bytes, and encrypt the result again
153 2016-01-03 17:29:58 <jtoomim> that's 2x to 2.5x the relay network's size, but without the memory requirements
154 2016-01-03 17:30:14 <sipa> the result is a series of bytes that can be turned back into the txid
155 2016-01-03 17:30:34 <sipa> but every byte in the result also cryptographically depends on the whole transaction
156 2016-01-03 17:30:45 <jtoomim> ok
157 2016-01-03 17:31:00 <jtoomim> i was thinking of just using the last 4 or 5 bytes of the sha256^2
158 2016-01-03 17:31:02 <sipa> you can take the last 8 bytes from it (or 4 or whatever, as you suggest), and treat that as a hash
159 2016-01-03 17:31:14 <sipa> and now the actual data to relay is the rest, which is shorter
160 2016-01-03 17:31:17 <jtoomim> which is how the mempool is arranged anyway in the C++ map
161 2016-01-03 17:31:23 <jtoomim> red-black tree, i think
162 2016-01-03 17:31:30 <jtoomim> so it would make searching for collisions very fast
163 2016-01-03 17:31:43 <sipa> it's a boost multiindex now, and new indexes can always be added
164 2016-01-03 17:32:12 <sipa> anyway, not sure it's obvious what the abvantage is: the hash you already know actually contains transaction data
165 2016-01-03 17:32:25 <sipa> and you can relay that many bytes fewer when the full tx is needed
166 2016-01-03 17:32:36 <sipa> it's a cute optimization
167 2016-01-03 17:32:39 <sipa> nothing major
168 2016-01-03 17:34:27 <sipa> especially if you're only using 4 byte hashes
169 2016-01-03 17:34:28 <jtoomim> i can agree with "cute"
170 2016-01-03 17:34:40 <jtoomim> 4 or 5 bytes most of the time, yes
171 2016-01-03 17:34:49 <jtoomim> 5 bytes would have a lower accidental collision rate
172 2016-01-03 17:34:54 <jtoomim> but even 4 bytes is pretty low
173 2016-01-03 17:36:16 <jtoomim> for a 65535 transaction block with a 65535 transaction mempool, i think there would be a 37% probability of at least one collision with 4 byte hashes
174 2016-01-03 17:36:28 <jtoomim> oops, no, that's not right
175 2016-01-03 17:36:42 <jtoomim> 63% probability of collision
176 2016-01-03 17:36:56 <jtoomim> with 5 byte hashes, 0.4% probability of collision
177 2016-01-03 17:37:39 <jtoomim> 4 bytes is enough for 1-10 MB for about 1 collision per block or less, and 5 bytes is enough for nearly no collisions on up to around 100 MB, or something like that
178 2016-01-03 17:38:13 <jtoomim> (1- 65535./(2**40))**65535
179 2016-01-03 17:38:59 <jtoomim> anyway, the point behind blocktorrent is that we have a lot of bandwidth that we're not using, and if we make better use of that bandwidth, a lot of these compression techniques become unnecessary
180 2016-01-03 17:39:29 <jtoomim> the secondary point is that if we use the merkle tree structure of blocks, we can get a few extra compression techniques to be useful and safe
181 2016-01-03 17:40:09 <jtoomim> but i'm mostly going to be focusing on getting the actual bandwidth utilization to be better to serve as a solid and reliable base
182 2016-01-03 17:40:09 <sipa> if your aim is to reduce bandwidth, you're absolutely right
183 2016-01-03 17:41:56 <jtoomim> i like the idea of building a system that has very different optimization strategies to the system that's being built by others, so that we can get a sum of the parts that is more robust than any one component
184 2016-01-03 17:42:51 <jtoomim> but with weak blocks used to encode merkle nodes, I think this could compress down to a level that is almost as fast and small as IBLTs
185 2016-01-03 17:43:25 <jtoomim> but with worst-case performance that's about 50x better
186 2016-01-03 17:44:14 <jtoomim> yada yada yada. less chat, more code.
187 2016-01-03 17:44:34 <jtoomim> jgarzik thanks again for that udp code, it will probably save me some work.
188 2016-01-03 18:32:07 <murch> Hi, can someone refer me to what nick the github user MarcoFalke has here?
189 2016-01-03 18:34:42 <paveljanik> murch, look around at #bitcoin-core-dev
190 2016-01-03 18:35:04 <murch> paveljanik: thx
191 2016-01-03 20:20:02 <m> hi
192 2016-01-03 20:22:43 <m> I come to believe the overflow hypothesis. That is once blocks are full and fees high, commerce could start overflowing to the strongest altcoin
193 2016-01-03 20:25:46 <m> so the fee/KB ratio between bitcoin and altcoin 1 could be measured and this could be a valid indicator for a block size increase
194 2016-01-03 20:26:15 <m> I mean potential increase, I'm not a big blocks guy myself
195 2016-01-03 20:27:30 <m> ok bye
196 2016-01-03 20:47:25 <merlincorey> "errors": "This is a pre-release test build - use at your own risk - do not use for mining or merchant applications"
197 2016-01-03 20:47:33 <merlincorey> ^ I build on master very recently
198 2016-01-03 20:47:40 <merlincorey> which branch should I be building on?
199 2016-01-03 20:48:21 <merlincorey> or is this close enough to a release where the warning is not so dire?
200 2016-01-03 20:49:17 <pigeons> its not necessarily dire, but the release is tag v0.11.2 i think
201 2016-01-03 20:49:50 <merlincorey> okay thanks
202 2016-01-03 20:50:13 <merlincorey> if I'm only ever receiving transactions from the blockchain on this wallet, I'll probably be okay?
203 2016-01-03 20:50:20 <pigeons> yes
204 2016-01-03 20:50:30 <merlincorey> excellent thank you
205 2016-01-03 20:50:46 <merlincorey> where should I look to confirm the current release tag at a future date when I do update it?
206 2016-01-03 20:52:27 <pigeons> maybe https://github.com/bitcoin/bitcoin/releases
207 2016-01-03 23:34:25 <rusty> Luke-Jr: is there a simple way to identify Eligius blocks in retrospect?  Might be an interesting refinement to the "max stuck tx is 5 hours" from http://rusty.ozlabs.org/?p=573 )
208 2016-01-03 23:35:30 <wizkid057> rusty: Eligius blocks have "Eligius" in the coinbase
209 2016-01-03 23:35:42 <rusty> wizkid057: ah, subtle! :)  Thanks...
210 2016-01-03 23:35:46 <wizkid057> np