1 2015-07-20 04:25:25 <kanzure> gmaxwell: quick sanity check? because it touches a few of your commits https://github.com/bitcoin/bitcoin/issues/6454
  2 2015-07-20 06:38:45 <mjerr> Is there any major advantage of using P2SH over normal transactions in a lightning-network implementation?
  3 2015-07-20 06:40:42 <mjerr> While I like the idea of P2SH, I always felt it would require more bytes (min 20 add bytes for the redeemScriptHash), and storing the proper redeemScript properly is another risk of losing money
  4 2015-07-20 06:41:48 <mjerr> e.g. when the counterparty broadcasts an old channel, I still cannot claim the funds, if I did not stored the redeemScript for that exact channel, and coming up with it out of thin air might be very difficult
  5 2015-07-20 06:43:04 <mjerr> Meaning everybody will have to store all redeemScripts for all payments across all versions of the channel
  6 2015-07-20 06:47:08 <phantomcircuit> mjerr, it moves the cost of the bytes to the receiver which is better
  7 2015-07-20 06:47:19 <phantomcircuit> the protocol should have been native p2sh
  8 2015-07-20 06:48:46 <Luke-Jr> mjerr: eh, redeem scripts should be pretty deterministic when you know what you want
  9 2015-07-20 06:49:01 <Luke-Jr> so in practice you're basically complaining you need to store the private key :P
 10 2015-07-20 07:24:09 <mjerr> phantomcircuit, but ultimately, the blockchain has to store 33 bytes more per payment?
 11 2015-07-20 07:26:20 <mjerr> Luke-Jr, well, should is a stretchy term :P ... but is there any advantage? I can see that it works, but I also see any implementation will be more complex
 12 2015-07-20 07:26:43 <phantomcircuit> mjerr, it's a tradeoff worth making
 13 2015-07-20 07:27:36 <mjerr> also, for LN people might think that they can refill their channel by paying to the P2SH Address
 14 2015-07-20 07:31:29 <phantomcircuit> mjerr, just dont show them the address
 15 2015-07-20 07:32:01 <mjerr> They can always just go to the bc.info and check where their funds went and try to send more to the shown address
 16 2015-07-20 07:32:43 <mjerr> Which, knowing how people will not read any manual whatsoever, will definitely happen
 17 2015-07-20 07:44:20 <phantomcircuit> mjerr, how are they going to know where to look if you dont tell them?
 18 2015-07-20 07:44:35 <phantomcircuit> i guess LN should be safe against users doing silly things though
 19 2015-07-20 07:44:39 <phantomcircuit> iirc the current design is
 20 2015-07-20 07:46:15 <mjerr> They can trace the money from the wallet they initially used :P ... yea, might be right, but in the specific case of LN, I still don't see the advantage of P2SH, as it's based on much communication anyway
 21 2015-07-20 07:46:48 <mjerr> And any fees will be paid by the user aswell
 22 2015-07-20 07:47:41 <phantomcircuit> mjerr, lightning would be integrated into the wallet
 23 2015-07-20 07:47:55 <phantomcircuit> the wallet would simply not display the p2sh address
 24 2015-07-20 07:48:09 <mjerr> and how does the btc get into that wallet? ;)
 25 2015-07-20 07:48:25 <phantomcircuit> mjerr, ehhhh
 26 2015-07-20 07:49:53 <mjerr> I mean users will find a way to trace the btc
 27 2015-07-20 07:54:24 <mjerr> But currently we are just talking about mitigating some of the disadvantages, what are the advantages in the first place? (not in general, specific for LN case)
 28 2015-07-20 07:58:40 <phantomcircuit> mjerr, the current design has the coins sent to an address which is 2 of 2
 29 2015-07-20 07:58:53 <phantomcircuit> i think you would be basically just opening another channel if you sent more coins
 30 2015-07-20 08:01:05 <mjerr> phantomcircuit, unless the channel expired already (all timeouts are encoded in the address)
 31 2015-07-20 08:05:51 <phantomcircuit> mjerr, the timeout is relative
 32 2015-07-20 08:05:57 <phantomcircuit> so no
 33 2015-07-20 08:06:24 <extor> So is there a free btc woocommerce plugin that will work on a free wallet
 34 2015-07-20 08:06:56 <mjerr> phantomcircuit, but still, what are the adv? :P
 35 2015-07-20 08:07:34 <phantomcircuit> mjerr, proper accounting for where the expense lies with script execution
 36 2015-07-20 08:08:57 <jonasschnelli> Damit. They (datacenter) shut down my dns seeder!
 37 2015-07-20 08:11:45 <mjerr> phantomcircuit, but this is not really the case with LN, as all fees will be paid (one way or another) by the client
 38 2015-07-20 10:12:53 <wumpus> jonasschnelli: ugh :(
 39 2015-07-20 10:12:57 <wumpus> jonasschnelli: for what reason?
 40 2015-07-20 10:13:07 <jonasschnelli> wumpus: it's now running again...
 41 2015-07-20 10:13:18 <wumpus> great
 42 2015-07-20 10:13:22 <jonasschnelli> wumpus: the crawler does act like a portscan and portscans are forbidden by the TOC
 43 2015-07-20 10:13:40 <jonasschnelli> i now need to use tor or upload crawling data from another peer to my dns server
 44 2015-07-20 10:13:52 <jonasschnelli> at the moment the data served over my dns is static...
 45 2015-07-20 10:38:17 <wumpus> reminds me of aschildback's testnet seed, lots of complains of hosters about the dns seeders :/
 46 2015-07-20 10:45:19 <jonasschnelli> wumpus: yeah. aschildbach and i have the same datacenter provider
 47 2015-07-20 10:54:48 <jonasschnelli> i wonder how difficult it would be to add tor support for the bitcoin-seeder
 48 2015-07-20 11:03:22 <wumpus> it *has* tor support, right?
 49 2015-07-20 11:03:34 <wumpus> at least to 'scan' onion peers
 50 2015-07-20 11:10:02 <jonasschnelli> wumpus: i just start to explore the tor mode... i hope it's also for scanning through tor
 51 2015-07-20 11:11:11 <wumpus> sending all queries through tor may require a code change
 52 2015-07-20 11:30:46 <jonasschnelli> wumpus: i assume if i add the local tor proxy 127.0.0.1:9050 to the proxylist for ipv4 it goes over the tor proxy?
 53 2015-07-20 11:34:37 <wumpus> probably; but i'd check just to be sure
 54 2015-07-20 11:41:49 <jonasschnelli> wumpus: fixed... now it used Tor for crawling... hopefully the datacenter guys STFU now.
 55 2015-07-20 11:44:44 <jtimon> cfields: is there any problem in libconsensus depending on timedata.h ?
 56 2015-07-20 11:53:34 <netg> /
 57 2015-07-20 13:19:40 <Chris_Stewart_5> Is there an RPC command to check if your bitcoind is fully synced with the network?
 58 2015-07-20 13:21:07 <sipa> no, it cannot be known
 59 2015-07-20 13:22:06 <Chris_Stewart_5> Thanks Sipa. Is there a sidechains irc? I'm playing around with the alpha
 60 2015-07-20 13:24:22 <sipa> #sidechains-dev
 61 2015-07-20 14:04:14 <leakypat> Is there a lightning channel?
 62 2015-07-20 14:08:12 <Luke-Jr> leakypat: #lightning-dev
 63 2015-07-20 14:17:54 <jeremias> hmm, bitcoin directory stays locked, even when I shut the bitcoin daemon properly down
 64 2015-07-20 14:18:05 <jeremias> err, I mean debug.log says shutdown is done
 65 2015-07-20 14:18:50 <jeremias> there is bitcoin-shutoff process however
 66 2015-07-20 14:59:49 <wumpus> jeremias: weird, wonder what it was hanging on
 67 2015-07-20 15:04:46 <cfields> jtimon: see #6453
 68 2015-07-20 15:06:10 <jtimon> cfields: how is that relate to my question?
 69 2015-07-20 15:06:35 <cfields> jtimon: oh sorry, i s/time/memory/ in my head
 70 2015-07-20 15:07:30 <cfields> jtimon: yea, no timedata please. That's an app-state thing
 71 2015-07-20 15:10:28 <jtimon> thanks
 72 2015-07-20 15:30:14 <StephenM347> RPC command gettxout doesn't indicate whether the output is a member of the UTXO, is there any call that makes this information available?
 73 2015-07-20 15:33:20 <sipa> if the call returns something, it is part of the UTXO set
 74 2015-07-20 15:33:34 <sipa> if it's not part of the UTXO set, the call won't find it :)
 75 2015-07-20 15:34:24 <StephenM347> sipa: Hence the help says "Returns details about an unspent transaction output." Thanks!
 76 2015-07-20 15:34:53 <StephenM347> I guess I wish it had been named something that indicated that
 77 2015-07-20 15:34:58 <StephenM347> like getutxo
 78 2015-07-20 15:35:14 <StephenM347> or getunspenttxout
 79 2015-07-20 15:35:29 <sipa> i think the 'utxo' abbreviation wasn't very common at the time
 80 2015-07-20 15:35:32 <sipa> but yeah, i agree
 81 2015-07-20 15:49:51 <wumpus> sometimes it's better to see API call names as opaque identifiers :)
 82 2015-07-20 17:22:25 <mjerr> The new LN design from rusty does need OP_CLTV (which is merged already, so it will probably become active within some near future?), but I also see it needs OP_CHECKSEQUENCEVERIFY, and I find very few information on that. I found that it should be a delay, basically RCLTV, but are there any plans on implementing CSV?
 83 2015-07-20 17:24:43 <mjerr> While more effective than the original LN paper, it is still not possible to build an application on it without some major changes?
 84 2015-07-20 17:33:19 <morcos> sipa: petertodd: I think it would be helpful for mempool limiting if we first agreed on what the unconfirmed transaction dependency limits should be.
 85 2015-07-20 17:33:38 <petertodd> morcos: as in, how big a package?
 86 2015-07-20 17:33:53 <morcos> yes
 87 2015-07-20 17:34:04 <petertodd> morcos: I'd say it should be a big larger than the maximum allowed tx, which is 100KB, so 125KB seems fine to me
 88 2015-07-20 17:34:22 <morcos> how do you define that pacage
 89 2015-07-20 17:34:32 <morcos> if 1 tx has 200 outputs
 90 2015-07-20 17:34:48 <petertodd> morcos: as for # of txs, I said 100 basically as a "reasonable" amount, that clearly doesn't cause problems re: O(<bad>) blowup
 91 2015-07-20 17:35:07 <morcos> can you not have 200 independent 1k size transactions that each create a depth 2 chain off of that
 92 2015-07-20 17:35:22 <morcos> because if you kick the parent you have to kick 200k in size
 93 2015-07-20 17:35:31 <petertodd> morcos: I'm thinking in terms of what a miner might need to evaluate to decide if a bunch of unconfirmed should be mined
 94 2015-07-20 17:35:44 <morcos> yeah the issue is a problem in both directions
 95 2015-07-20 17:35:57 <petertodd> morcos: yeah
 96 2015-07-20 17:36:09 <morcos> sipa suggested thinking of them as one. where tx's are partioned into packages
 97 2015-07-20 17:36:24 <morcos> i dont' think its a bad idea
 98 2015-07-20 17:36:31 <petertodd> so, can we agree we're not going to allow huge "packages" of txs into the mempool at all?
 99 2015-07-20 17:36:35 <morcos> but i think a limit like 125 is going to prevent a LOT of current behavior
100 2015-07-20 17:37:02 <petertodd> well, given it's all just pointer following, a higher limit in terms of # of txs is ok, but in terms of total size...
101 2015-07-20 17:37:03 <morcos> well we need to agree on something, so the question is what can we get "consensus" on
102 2015-07-20 17:37:24 <petertodd> so, the goal here is don't waste bandwidth of txs whose chance of getting mined is dubious, right?
103 2015-07-20 17:37:48 <morcos> that's one goal
104 2015-07-20 17:37:59 <morcos> but in the example above, 1 parent and 200 children
105 2015-07-20 17:38:06 <morcos> they might all be likely to be mined relatively quickly
106 2015-07-20 17:38:17 <morcos> but they clog up the mempool entirely until they get mined
107 2015-07-20 17:38:17 <petertodd> right, and I guess another one, is don't put miners in positions where making an ideal block heavily needs to depend on knapsack crap, right?
108 2015-07-20 17:38:22 <morcos> bc they are unbootable
109 2015-07-20 17:38:26 <petertodd> right
110 2015-07-20 17:39:17 <petertodd> OTOH, given a *large* mempool, that situation where the "cloging" tx is going to be mined soon isn't going to happen
111 2015-07-20 17:39:34 <sipa> why not?
112 2015-07-20 17:39:41 <petertodd> (again, assumming the total size of the tx group is <125KB or so)
113 2015-07-20 17:45:02 <petertodd> sipa: so, with a 100MB mempool, the tx is only "clogging" up the pool in a significant way if the tx(s) are going to be mined soon, but that's obviously impossible because it'll take a few hours to clear that backlog
114 2015-07-20 17:46:32 <sipa> petertodd: depends what the variation of feerates is
115 2015-07-20 17:47:02 <sipa> assume for example that the highest feerate in the mempool is twice that of the lowest
116 2015-07-20 17:47:09 <morcos> what happens in practice with eviction, is the things you can evict are small tx.  so large tx's or packages with lowish fee rate filter to the bottom.  then when the mempool becomes full.  no transactions, no matter how high fee they are can get through until a block clears room.
117 2015-07-20 17:47:29 <Diablo-D3> morcos: you're doing the calculation wrong
118 2015-07-20 17:47:36 <sipa> a transaction (package) that is 3 times the normal tx size will never be kicked out
119 2015-07-20 17:47:50 <sipa> as the new tx can never pay for the fee difference
120 2015-07-20 17:47:51 <Diablo-D3> take your mempool, sort txen by bitcoins per byte
121 2015-07-20 17:47:51 <petertodd> sipa: true, that's a good point
122 2015-07-20 17:48:04 <Diablo-D3> then start evicting the least paying txen by that metric
123 2015-07-20 17:48:14 <sipa> i have been thinking about a pre-mempool
124 2015-07-20 17:48:32 <sipa> where unstructured unverified transactions are kept, sorted by feerate
125 2015-07-20 17:48:41 <morcos> Diablo-D3: see discussion on 6421 and 6455.   you have to evict dependent tx's and have to pay for their bandwidth
126 2015-07-20 17:48:44 <sipa> new invs arrive there, without being relayed
127 2015-07-20 17:48:55 <Diablo-D3> morcos: well, thats assuming bandwidth has a set cost for that
128 2015-07-20 17:48:59 <petertodd> sipa: that's not a bad idea
129 2015-07-20 17:49:03 <sipa> and the limitation code tries to substitute parts of the actual mempool by stuff in the premempool
130 2015-07-20 17:49:08 <morcos> ha i thought you said it was his idea!
131 2015-07-20 17:49:14 <sipa> at which point they get verified and relayed
132 2015-07-20 17:49:23 <Diablo-D3> sipa: so you mean only generate the mempool when its time to generate a block/
133 2015-07-20 17:49:36 <sipa> it's inspired by petertodd thinking about disconnecting relay and mempool
134 2015-07-20 17:49:41 <Diablo-D3> well yes
135 2015-07-20 17:49:42 <Diablo-D3> I agree
136 2015-07-20 17:49:48 <morcos> sipa: did you see what i pm'ed about using a soft cap to accomplish same goal?
137 2015-07-20 17:49:49 <Diablo-D3> in fact, I thought relay was already disconnected
138 2015-07-20 17:50:00 <sipa> yes, but i didn't quite grasp it
139 2015-07-20 17:50:03 <Diablo-D3> I thought it relayed everything already, and only stored locally into the mempool when it needed to
140 2015-07-20 17:50:20 <morcos> so suppose soft cap is 200MB, and you double every 20MB after that
141 2015-07-20 17:50:23 <Diablo-D3> if I wrote this code today, I'd have a table sorted by free/byte, and then generate the mempool on demand
142 2015-07-20 17:51:09 <morcos> at 201MB mempool sz you know you have 1MB of txs in your mempool that have not only paid minimum relay fee for themselves but an extra 1x minrelay fee, so if there is a large package that is paying less than min relay fee at the bottom, you can now freely evict that whole package
143 2015-07-20 17:51:27 <Diablo-D3> re: relaying
144 2015-07-20 17:51:36 <morcos> similarily if you are at 221MB you know have 1MB at 3x minRelayFee that you can use to pay for eviction
145 2015-07-20 17:51:54 <petertodd> sipa: so, if we expire txs out of the mempool by decreasing feerate over time, those big packages will naturally get replaced
146 2015-07-20 17:52:16 <sipa> hmm?
147 2015-07-20 17:52:28 <sipa> how do you mean, decrease feerate over time?
148 2015-07-20 17:52:30 <warren> is CPFP dead?  I haven't been following it lately.
149 2015-07-20 17:52:33 <petertodd> sipa: yup
150 2015-07-20 17:53:01 <petertodd> sipa: see, it gives roughly the same affect as a "relay stuff by coin days" system
151 2015-07-20 17:54:10 <petertodd> warren: CPFP isn't dead, it's just less inherently less efficient than FSS-RBF for most cases
152 2015-07-20 17:54:14 <sipa> you mean: treat feerates as something inflationary
153 2015-07-20 17:54:19 <sipa> ?
154 2015-07-20 17:54:25 <petertodd> sipa: lol, yeah kinda :)
155 2015-07-20 17:54:38 <petertodd> sipa: just as a way to implement expiration basically, rather than a hard time limit
156 2015-07-20 17:54:42 <sipa> that would not be hard to implement
157 2015-07-20 17:55:02 <petertodd> sipa: one trick is you'd want the mining code to use the real feerate, but otherwise it's simple enough
158 2015-07-20 17:55:02 <sipa> as we already have feerate and time
159 2015-07-20 17:55:25 <morcos> i think that sufferes from the same problem as the time only based expiration
160 2015-07-20 17:55:33 <petertodd> sipa: not time actually, you want to do this by block height, as that's the opportunity for blocks to get removed from the mempool
161 2015-07-20 17:55:58 <morcos> it might not happen fast enough to avoid clogging the mempool and also you have to be careful for the free relay opportunity you might be providing
162 2015-07-20 17:56:11 <sipa> sort of a: compare feerate/1.01^entryheight ?
163 2015-07-20 17:56:16 <petertodd> sipa: yeah
164 2015-07-20 17:57:04 <morcos> all of these things first require a limit on package sizes though
165 2015-07-20 17:57:07 <sipa> but we don't want a "avoid total clogging", also a "good transactions should replace worse transactions immediately"
166 2015-07-20 17:57:32 <petertodd> morcos: so, with a 125KB cap on total package size, it's at least we're looking at "just" a 625x decrease in relay fees if we replaced purely on feerate :)
167 2015-07-20 17:58:21 <morcos> i know!
168 2015-07-20 17:58:43 <morcos> and i think 125KB cap defined from the parent is not going to be easy to get agreement on
169 2015-07-20 17:59:17 <petertodd> morcos: what if that's done with sipa's pre-mempool suggestion, so the tx wouldn't get relayed immediately, but they would get relayed eventually
170 2015-07-20 17:59:44 <sipa> i think pre-mempool avoids the clogging problem entirely
171 2015-07-20 17:59:53 <morcos> we need someway of aggregating txs to quickly boot, yes either his pre-mempool idea, or the using the surplus over the soft-cap
172 2015-07-20 18:00:25 <morcos> my concern with the pre-mempool is if nodes are relaying txs very differently, it could lead to some sort of weird feedbacks in the p2p network
173 2015-07-20 18:00:29 <petertodd> sipa: how so?
174 2015-07-20 18:00:49 <petertodd> morcos: as in, a tx loop?
175 2015-07-20 18:00:56 <sipa> petertodd: once the premempool grows large enough, it would be able to kick out any large clog
176 2015-07-20 18:01:07 <sipa> because it both has high fee and high feerate
177 2015-07-20 18:01:15 <morcos> i think the surplus over the softcap is the same idea, but you relay it right away
178 2015-07-20 18:01:30 <petertodd> morcos: sorry, surplus over the softcap? not familiar
179 2015-07-20 18:01:40 <morcos> its what i was trying to describe above
180 2015-07-20 18:02:09 <morcos> put another way, you relay the premempool ahead of time, but you have a higher min relay fee for the premempool based on how big it is
181 2015-07-20 18:03:12 <petertodd> morcos: ah, right, so that's similar to my suggestion of just having a bandwidth-based min-relay-feerate?
182 2015-07-20 18:03:46 <morcos> yeah i'm basing it off your doubling of minrelay as the mempool grows
183 2015-07-20 18:04:26 <morcos> and so you know min fee rates that must have been paid by any tx's greater than the cap, so you can use that "surplus" fee to evict the bottom of the mempool
184 2015-07-20 18:05:46 <sipa> another possibility is remembering how much size_replaced every transaction in the mempool has (=the txsize of everything it kicked out to get it)
185 2015-07-20 18:05:56 <sipa> and not look at fee difference, but at combined size
186 2015-07-20 18:07:27 <petertodd> sipa: hmm, though that does mean if a tx has the bad luck to have to replace a big bunch of txs, it's at a disadvantage, if I understand that correctly
187 2015-07-20 18:09:12 <morcos> anyway, the point i wanted to make
188 2015-07-20 18:09:29 <morcos> is that testing any of these things is only reasonable if we know what the package limit will be
189 2015-07-20 18:09:45 <morcos> and i feel like thats going to be a difficult problem to solve
190 2015-07-20 18:09:56 <petertodd> morcos: well, concretely who is being affected by, say, a 125KB package limit?
191 2015-07-20 18:12:35 <morcos> i don't know, but i think the packages in the mempool now are quite large
192 2015-07-20 18:12:51 <dgenr8> there's no need to evict entire packages. we should be happy to evict just the deep dependents
193 2015-07-20 18:13:44 <morcos> petertodd: don't get me wrong.  i'm not against it.
194 2015-07-20 18:14:37 <petertodd> dgenr8: remember that the deep dependents may have high fees, which means they aren't going to get evicted
195 2015-07-20 18:15:08 <dgenr8> consider heavily penalizing them for being deep dependents
196 2015-07-20 18:15:09 <petertodd> dgenr8: like, with CPFP, those are perfectly good txs to mine, as a package
197 2015-07-20 18:15:39 <petertodd> morcos: I'm thinking pretty much only really big exchanges and the like are going to notice anything from a package limit
198 2015-07-20 18:20:55 <dgenr8> sipa: remember the size of totally unrelated txes that a tx kicked out?  that made sense for same-signer replacement but how does it in general?  it's not the txes fault
199 2015-07-20 18:21:44 <sipa> dgenr8: the purpose is to prevent relaying without paying fee for it
200 2015-07-20 18:22:21 <sipa> if you can keep replacing the bottom transaction without additional cost, you get infinite bandwidth for constant cost
201 2015-07-20 18:22:48 <dgenr8> i know but the the new tx author didn't get anything free ... it would be very hard for him to game my mempool
202 2015-07-20 18:23:52 <dgenr8> if CPFP were going to be big, i think more miners would be doing it already
203 2015-07-20 18:24:11 <sipa> dgenr8: the author of the replaced transaction got something for free
204 2015-07-20 18:24:56 <sipa> he got his transaction relayed, and it will (assuming the whole network uses the same policy of course) never be mined, so it never actually pays a fee
205 2015-07-20 18:25:14 <dgenr8> yes.  but that's not a reason to punish the replacing author
206 2015-07-20 18:25:38 <sipa> i agree; thatbwas the idea behind my floating fee
207 2015-07-20 18:25:48 <sipa> to avoid acxepting the original tx in the first place
208 2015-07-20 18:26:00 <sipa> by averaging out the effect of variation
209 2015-07-20 18:26:45 <dgenr8> i like that idea a lot, except i'm very interested in promoting consistent relay rules
210 2015-07-20 18:27:09 <sipa> i don't believe consistent relay rules are a meaninful or achievable goal
211 2015-07-20 18:27:10 <petertodd> dgenr8: well, we have to write code that works even if people aren't being consistent
212 2015-07-20 18:27:23 <sipa> ^
213 2015-07-20 18:28:10 <dgenr8> consistency will never be achieved 100%.  it is a very different kind of problem form writing an EC library
214 2015-07-20 18:28:53 <dgenr8> maybe your scheme wouldn't be that bad for consistency anyway
215 2015-07-20 18:29:51 <morcos> sipa: as far as floating the relay fee.   i'm not sure thats necessary...  if you're using fee estimation, then you have a sense of what fee you need to put on your tx to be mined
216 2015-07-20 18:30:38 <morcos> well maybe it helps to not let the mempool become clogged with low fee stuff if it happens to get cleared out a bit
217 2015-07-20 18:31:49 <dgenr8> +1 for leveraging the new fee estimation as much as possible and fixing any problems along the way
218 2015-07-20 18:31:55 <morcos> but in terms of predictiblity of what will get relayed, the minrelayfee is never going to help with that b/c you won't be able to predict whats needed to evict txs in full mempools
219 2015-07-20 18:35:24 <dgenr8> sipa: in your scheme, how about dependent-size += all-its-parent-sizes?  that'll get the trees flattened out.
220 2015-07-20 18:35:55 <petertodd> dgenr8: what do you mean by that?
221 2015-07-20 18:36:01 <sipa> dgenr8: that's the packaging approach morcos has been working on
222 2015-07-20 18:36:58 <sipa> but it gets complicated when one tx is a dependency of multiple parents
223 2015-07-20 18:38:29 <dgenr8> petertodd: p1 and p2 are inputs to d.  effectivesize(d) = truesize(d) + truesize(p1) + truesize(p2)
224 2015-07-20 18:39:20 <petertodd> dgenr8: yeah, that's what everyone is assuming - value of a tx has to take into account what it needs to get mined first
225 2015-07-20 18:40:07 <dgenr8> oh i definitely meant for it to be recursive: effectivesize(d) = truesize(d) + effectivesize(p1) + effectivesize(p2)
226 2015-07-20 18:40:22 <petertodd> dgenr8: yes, that's what we're proposing
227 2015-07-20 18:40:35 <dgenr8> great, i think that will work great
228 2015-07-20 18:40:57 <dgenr8> i don't think you even need to track packacges because dependents will sink right to the bottom
229 2015-07-20 18:41:41 <dgenr8> unless they pay huge fees
230 2015-07-20 18:45:15 <morcos> petertodd: just took a look at some of the blocking packages that got sorted to the bottom of the mempool over the last couple weeks.  there were many over 100k, and some over 1.5MB
231 2015-07-20 18:46:12 <petertodd> morcos: by blocking packages, what do you mean exactly?
232 2015-07-20 18:46:44 <morcos> i mean if you are looking a tx to evict... the total size of all the children you will now also have to evict
233 2015-07-20 18:46:56 <morcos> so what you want to limit to 125k i think
234 2015-07-20 18:47:01 <morcos> sorry, what we all want to limit!
235 2015-07-20 18:47:32 <petertodd> morcos: right, so you were just saying what was in the mempool, not trying to do any further analysis (e.g. CPFP fees)
236 2015-07-20 18:48:31 <morcos> i don't follow.  i wasn't looking at it the other way around, as would would need to be mined parent wise to get a child in a block, no.   but i was looking at the total fees and size you would have to pay the relay fees for in order to get your tx in the mempool
237 2015-07-20 18:48:53 <petertodd> morcos: yeah, that's what I figured you were looking at
238 2015-07-20 18:49:05 <morcos> we tried to put packages together the other way
239 2015-07-20 18:49:10 <morcos> it wasn't computationally feasible
240 2015-07-20 18:49:41 <petertodd> morcos: wait, sorry, maybe I'm missing it. So a package, is the most-child-tx edge right?
241 2015-07-20 18:51:33 <morcos> so we looked at two notions of packages.  each transaction represents a descendant package and an ancestor package.  (so they obviously overlap)  we gave up on the ancestors (whats needed for CPFP) and focused on the descendant package stats for each children.  Which is the cached value of the sizes and fees of all the children that are dependent on it in the mempool.
242 2015-07-20 18:52:20 <morcos> so if you sort by the descendant package feerate or fee, then you have a better chance of solving the knapsack problem than if you just sort by individual fee rate and discover the package as you go.
243 2015-07-20 18:52:50 <morcos> for small tx's we'd start by looking at the lowest fee packages, and for large tx's start by looking at the lowest feerate
244 2015-07-20 18:52:59 <petertodd> morcos: ah, so basically you're saying "if I include this tx, I have the option of including that tx(s)"
245 2015-07-20 18:53:17 <dgenr8> in formula above, parent's size has been paid for. if dependent qualifies to stay in mempool, all its parents and higher-level ancestors also qualify.  problem is solved by being appropriately brutal
246 2015-07-20 18:54:27 <morcos> petertodd: you could look at it that way, but not just direct children, all descendents, so you couldn't include them right away, and again this is for eviction, not mining.   so you're following the same logic as the recursive remove function in txmempool
247 2015-07-20 18:54:47 <petertodd> morcos: right
248 2015-07-20 18:55:13 <morcos> dgenr8: that doesn't work because then if you manage to get a bunch of low fee txs in the mempool, you can anchor them there forever with a large fee descendent, but it'll never be mined because as a package its unattractive...
249 2015-07-20 18:56:21 <dgenr8> morcos: no, as i said, all ancestors have been paid for
250 2015-07-20 18:56:56 <dgenr8> you do not need to track the packages at all, at eviction time
251 2015-07-20 18:56:57 <morcos> i don't follow
252 2015-07-20 18:57:26 <dgenr8> track an effective size in the dependent's mempool entry that includes all ancestors sizes
253 2015-07-20 18:57:35 <dgenr8> dependent's feerate uses that size
254 2015-07-20 18:58:08 <morcos> ah, ok, yes so thats the other way around that you need for CPFP mining
255 2015-07-20 18:58:13 <morcos> unfortunately thats very hard to do
256 2015-07-20 18:58:23 <dgenr8> why is it hard?  it's much easier
257 2015-07-20 18:58:49 <dgenr8> the complaint would be that the parent also pays a fee, so you're making child pay too much
258 2015-07-20 18:59:02 <dgenr8> i say to that ... them's the breaks
259 2015-07-20 18:59:03 <petertodd> morcos: are you sure it's the wrong way around? it's not going to always be 100% optimal, but pretty decent should be to just go down the list of packages from highest feerate to lowest and include them - if packages overlap that's just an unexpected bonus
260 2015-07-20 18:59:14 <morcos> i can't remember off the top of my head which is the hardest operation..  i think it might be returning things to the mempool from a disconnected block
261 2015-07-20 18:59:59 <morcos> petertodd: thats an interesting idea
262 2015-07-20 19:00:23 <morcos> but i think then you can sabotage your relative tx's
263 2015-07-20 19:00:24 <petertodd> morcos: basically, for every tx in the mempool, store nSumFees = nFees + sum(prevout nSumFees), and nSumSize = nSize + sum(prevout nSumSizes)
264 2015-07-20 19:00:52 <dgenr8> hey that works too ... so no brutality needed
265 2015-07-20 19:00:56 <petertodd> morcos: or if you don't want to overcount, just walk the whole parent set with some max_visits limit and compute it exactly
266 2015-07-20 19:01:21 <morcos> but thats the right way around, thats the ancestor package
267 2015-07-20 19:01:47 <petertodd> morcos: so long as you reject txs that depend on too many other txs before they can be mined, then assemlbing the block isn't hard and you'll get close-to-optimal results
268 2015-07-20 19:01:50 <morcos> i think some of these thigns can be made feasible once we have package size limits
269 2015-07-20 19:01:53 <morcos> yes
270 2015-07-20 19:02:04 <petertodd> morcos: ancestor package? as in, this tx and all its ancestors right?
271 2015-07-20 19:02:08 <morcos> yes
272 2015-07-20 19:02:12 <petertodd> ok good
273 2015-07-20 19:02:31 <morcos> ancestor package sort for mining, descendant package sort for eviction
274 2015-07-20 19:03:31 <morcos> unfortunately when you return things from a disconnected block to the mempool, it gets quite messy because you don't know that the limits you put in place are going to hold any more
275 2015-07-20 19:03:32 <dgenr8> when evicting, children must be evicted ofc.  but there's no reason to chase up to the big parent and evict that
276 2015-07-20 19:03:43 <morcos> dgenr8: yes agreed
277 2015-07-20 19:03:51 <dgenr8> ok great
278 2015-07-20 19:04:24 <petertodd> morcos: hang on, if you evict in ancestor order, and ignore txs with children, that works as well, modulo the "mempool is full of packages" case
279 2015-07-20 19:05:01 <morcos> what do you mean evict in ancestor order, evict the package thats least likely to be mined by CPFP?
280 2015-07-20 19:05:07 <petertodd> morcos: yes exactly
281 2015-07-20 19:05:41 <petertodd> morcos: obviously if the mempool is full of CPFP packages you could have a case where a tx probably should et in that can't, but that's a pretty rare circumstance
282 2015-07-20 19:05:42 <morcos> yeah i mean i guess thats what you want, but thats not the thing you're most likely to be able to evict is it?
283 2015-07-20 19:05:45 <dgenr8> petertodd: you can just evaluate all txes individually, dependent feerates having already been calculated to include ancestors.
284 2015-07-20 19:05:58 <morcos> dgenr8, thats what we're calling packages
285 2015-07-20 19:06:23 <dgenr8> ok, my point is you can also call a standalone tx a package
286 2015-07-20 19:06:30 <petertodd> morcos: why wouldn't I be able to evict it? remember we're only evaluating txs with no children
287 2015-07-20 19:06:41 <morcos> oh, sorry i missed that
288 2015-07-20 19:06:44 <dgenr8> petertodd: why
289 2015-07-20 19:07:07 <petertodd> dgenr8: so we never have to care about the children getting evicted, letting us just walk up the total feerate index
290 2015-07-20 19:07:57 <dgenr8> if children pay enough, they will have been passed over for eviction before you got to parent
291 2015-07-20 19:09:05 <petertodd> dgenr8: well, if the children pay enough, remember that's saying that you're not able to improve the feerate of the mempool by evicting
292 2015-07-20 19:09:40 <morcos> this all comes down to the fact that even exhaustive search over possiblities is insufficient if we don't have a way of aggregating to boot larger size packages
293 2015-07-20 19:10:03 <morcos> packages already clog the mempool
294 2015-07-20 19:10:15 <morcos> sure we can limit them a bit, but as you pointed out thats still a factor of 600
295 2015-07-20 19:10:24 <morcos> and as soon as we have this eviction mechanism
296 2015-07-20 19:10:35 <morcos> someone will intentionally design a tx flood to subvert it
297 2015-07-20 19:10:48 <petertodd> morcos: indeed they will
298 2015-07-20 19:11:45 <morcos> so i think there are a couple of next steps
299 2015-07-20 19:12:07 <morcos> 1) we need a proposal on what the dependant package size limits are
300 2015-07-20 19:12:30 <morcos> 2) we need an aggregation mechanism, either pre-mempool or my surplus over the soft cap idea
301 2015-07-20 19:12:35 <petertodd> oh hang on... so if we replaced by feerate only, then the pattern of large tx replaced by small replaced by large again is naturally limited, as each time the large tx is a bit smaller or it can't fit into the mempool
302 2015-07-20 19:13:07 <morcos> you mean if you didn't have to pay for the dependents fees
303 2015-07-20 19:13:30 <morcos> i had that same thought a couple of days ago, but i couldn't convince myself that it was safe
304 2015-07-20 19:14:04 <petertodd> morcos: well, I mean if you evaluated replacement of the bottom tx purely on whether or not the feerate was higher
305 2015-07-20 19:14:12 <morcos> of jsut that tx
306 2015-07-20 19:14:15 <petertodd> morcos: eys
307 2015-07-20 19:14:18 <petertodd> morcos: *yes
308 2015-07-20 19:14:42 <petertodd> morcos: remember that there's only ever going to be up to 100KB of txs being kicked out due to the max tx size
309 2015-07-20 19:14:55 <morcos> no
310 2015-07-20 19:15:02 <morcos> you kick out all dependents
311 2015-07-20 19:15:12 <morcos> like i said, right now its up to at least 1.5MB
312 2015-07-20 19:15:15 <petertodd> morcos: no, but remember I'm proposing we never kick out txs with dependents
313 2015-07-20 19:15:35 <petertodd> morcos: because we never let let large dependent chains into the mempool in the first place
314 2015-07-20 19:15:37 <morcos> oh i thought you were still kicking them just not evaluating onthat
315 2015-07-20 19:16:17 <petertodd> morcos: basically, if the nSumSize of a tx would become >125KB, it's just not accepted into the mempool
316 2015-07-20 19:16:34 <morcos> if you never kick out txs with dependents, then it's easy to clog the mempool but shoving in bunch of min fee txs whenever there is room and then chaining them larger at your leisure
317 2015-07-20 19:16:43 <morcos> yes, but so you make lots of them
318 2015-07-20 19:16:46 <morcos> 125 is still big!
319 2015-07-20 19:16:48 <petertodd> morcos: that way assuming miners have CPFP implemented similarly, we're only letting txs in that can be easily mined with a dumb CPFP implementation
320 2015-07-20 19:17:00 <morcos> and now they aren't even subject to potentially eviction
321 2015-07-20 19:17:35 <morcos> well can we agree 1) is a prereq for everything
322 2015-07-20 19:17:37 <petertodd> morcos: no, those min fee txs will get kicked out of the mempool early, because the total fee/kb needed to get them mined will be low
323 2015-07-20 19:17:46 <morcos> no b/c they will have children
324 2015-07-20 19:17:46 <petertodd> morcos: er, I mean total fee/kb of mining them
325 2015-07-20 19:18:05 <petertodd> morcos: right, but you see how the youngest child gets peeled off in my scheme?
326 2015-07-20 19:18:36 <petertodd> morcos: the only time when that doesn't happen is when the youngest child has a high fee, sufficient to pay for all parents, in which case in general there's no reason to evict it
327 2015-07-20 19:18:49 <morcos> thats the part i'm not getting
328 2015-07-20 19:19:03 <morcos> why does the youngest child have to have a fee sufficient to pay for all parents
329 2015-07-20 19:19:11 <morcos> how do you enforce that
330 2015-07-20 19:19:22 <petertodd> morcos: it's not that it *has* to, but rather, if it didn't it'd get evicted quickly
331 2015-07-20 19:19:39 <morcos> oh, b/c it is evicted based on its ancestor size
332 2015-07-20 19:19:42 <petertodd> morcos: yes
333 2015-07-20 19:19:48 <morcos> interesting..
334 2015-07-20 19:19:59 <morcos> so you might leave in signficantly worse parents
335 2015-07-20 19:20:10 <morcos> but what you've really done is opened them up for possible eviction themselves
336 2015-07-20 19:20:13 <petertodd> morcos: no, once the child is removed, evaluate the parents for eviction as well
337 2015-07-20 19:20:17 <petertodd> morcos: exactly
338 2015-07-20 19:20:28 <morcos> thats kind of neat
339 2015-07-20 19:20:51 <morcos> but then you've got to some how evaluate that guy getting in on the same criteria
340 2015-07-20 19:20:59 <morcos> otherwise he'll keep getting in and evicted over and over
341 2015-07-20 19:21:08 <morcos> (the high fee child)
342 2015-07-20 19:21:38 <petertodd> morcos: no, to get evicted another tx has to pay the sufficient fees to cover the cost of the tx getting evicted
343 2015-07-20 19:22:28 <morcos> now i'm confused again
344 2015-07-20 19:22:36 <petertodd> morcos: heck, it should be enough to just say that the feerate of the new tx has to be >= evicted_avg_feerate * k
345 2015-07-20 19:22:38 <morcos> i thought you only had to beat his ancestor package feerate
346 2015-07-20 19:22:52 <morcos> anyway, there are a lot of ideas
347 2015-07-20 19:23:01 <morcos> all depend on limited packages
348 2015-07-20 19:23:04 <helo> how exactly are transactions packaged up for cpfp?
349 2015-07-20 19:23:04 <morcos> so how do we move that forward
350 2015-07-20 19:23:10 <morcos> helo: they aren't now
351 2015-07-20 19:23:47 <helo> don't some miners that support it?
352 2015-07-20 19:23:50 <petertodd> morcos: ah, sorry, I missed something - what you want to do is sort this such that nSumSize is reset if the child tx would make the nTotalFeerate lower rather than higher
353 2015-07-20 19:25:03 <helo> does eligius just have a short-interval time based eviction window during which you can submit a not-sufficient-fee parent, and then its paying child before parent is evicted?
354 2015-07-20 19:25:08 <morcos> helo: there is an implementation, but it has some flaws.  i commented on the PR, 1647 i think.. (search for oldest)
355 2015-07-20 19:25:16 <petertodd> morcos: basically then you're *assuming* all the parents will get mined prior to the child being evaluated, and thus you only need to determine if replacing that child makes the situation better or worse
356 2015-07-20 19:43:22 <morcos> sipa: petertodd: how fast are we trying to solve this attack vector?  i keep going back to the idea that maybe we need something simpler until we can figure the best eviction algorithm
357 2015-07-20 19:43:50 <morcos> petertodd's exponentially increasing relay multiplier seems strictly better than what we have now
358 2015-07-20 19:44:02 <petertodd> yeah, it probably is
359 2015-07-20 19:44:15 <morcos> and we could combine that fairly safely with a long time horizon janitor
360 2015-07-20 19:44:52 <petertodd> morcos: yup
361 2015-07-20 19:44:57 <morcos> fee estimation can be used to figure out what you need to get relayed
362 2015-07-20 19:45:08 <petertodd> morcos: of course, screw it up and we'll quickly drive fees up a lot, but at least nodes won't crash
363 2015-07-20 19:45:30 <morcos> yeah they'll drive up to the point where people aren't willing to spam at that price
364 2015-07-20 19:45:45 <morcos> isn't that the only reasonable outcome anyway
365 2015-07-20 19:45:50 <petertodd> morcos: I mean, if we are in a situation where we're accepting txs that aren't getting mined
366 2015-07-20 19:46:18 <morcos> yeah so the janitor is a nice addition
367 2015-07-20 19:46:26 <petertodd> morcos: yes, at least the attack will go away
368 2015-07-20 19:47:34 <morcos> actually maybe the janitor is too complicated
369 2015-07-20 19:48:08 <morcos> it suffers from  being able to chain dependent txs from a tx that is about to time expire
370 2015-07-20 19:48:13 <petertodd> morcos: see, an issue with the exponentially increasing relay idea is you could "anchor" a set of low-fee txs, then put children spending those low fee txs in the higher bands, but to a miner who didn't get the low fee ones the children are orphans, so nothing relays at all
371 2015-07-20 19:48:45 <petertodd> morcos: well, *any* attempt at this stuff is going to make it possible to get into situations where txs are unlikely to be mined
372 2015-07-20 19:48:49 <morcos> ha, so maybe nothing is "simple"
373 2015-07-20 19:49:10 <petertodd> morcos: children make all this way more complex :)
374 2015-07-20 19:49:16 <petertodd> morcos: usually true in life...
375 2015-07-20 19:49:24 <morcos> :)
376 2015-07-20 19:49:49 <morcos> well i've seen you mention 0.11.1
377 2015-07-20 19:49:54 <morcos> but that just doesn't seem reasonable to me
378 2015-07-20 19:50:09 <petertodd> morcos: indeed, for v0.11.1 I think we should just revert the stupid 10x fee drop
379 2015-07-20 19:54:44 <helo> isn't it much more ideal for users to be able to rebroadcast the original transaction with an adequate fee?
380 2015-07-20 19:54:58 <helo> which is what relayfee does
381 2015-07-20 19:55:04 <petertodd> helo: that's orthogonal to what we're discussing here
382 2015-07-20 19:58:05 <helo> cpfp just seems like a mess that most nodes shouldn't care about. what user base is expected to commonly send a child to pay for a stalled parent?
383 2015-07-20 19:58:10 <CodeShark> We should probably be treating unconfirmed dependency chains as single objects somehow...
384 2015-07-20 19:58:21 <petertodd> CodeShark: we have to for relaying actually
385 2015-07-20 19:58:39 <petertodd> CodeShark: CPFP isn't reliable unless there's a P2P level notion of a tx package
386 2015-07-20 19:58:55 <helo> i would expect it to be services/businesses, which would do well to plug directly into miners with cpfp
387 2015-07-20 19:59:15 <CodeShark> right - is anything of the sort already implemented, petertodd?
388 2015-07-20 19:59:15 <petertodd> helo: if people are "plugging directly" into miners we're failing at decentralization
389 2015-07-20 19:59:22 <petertodd> CodeShark: no, not yet
390 2015-07-20 19:59:42 <petertodd> CodeShark: (maybe some special API's via hashing power contracts, but that's fucking dangerous for the ecosystem so...)
391 2015-07-20 20:00:18 <CodeShark> I was thinking perhaps txs that have unconfirmed dependencies could list the txids of the dependencies - then the receiving nodes could do a getdata on them
392 2015-07-20 20:00:36 <petertodd> CodeShark: yeah, you'd do something like that
393 2015-07-20 20:01:06 <CodeShark> so it could be implemented by extending the tx message
394 2015-07-20 20:01:17 <petertodd> CodeShark: would help if you told your peers what the total fee/size of the package was
395 2015-07-20 20:01:44 <petertodd> CodeShark: though actually, note how you could have nodes request unknown prevouts of txs they get too :)
396 2015-07-20 20:02:06 <petertodd> CodeShark: main thing, is you want to tell your peers *why* they should do that for some tx
397 2015-07-20 20:03:23 <CodeShark> petertodd: yes, very true - the data is already in the tx message...but the receiving node doesn't necessarily know whether it's because they're still missing dependencies or whether it's because the dependencies simply do not exist
398 2015-07-20 20:03:46 <petertodd> CodeShark: right
399 2015-07-20 20:03:56 <petertodd> CodeShark: which is something a package scheme can improve on
400 2015-07-20 20:06:06 <CodeShark> presumably nodes that advertise dependencies they cannot provide would be blacklisted
401 2015-07-20 20:06:12 <petertodd> yup
402 2015-07-20 20:06:54 <CodeShark> the fee calculation part is the harder issue
403 2015-07-20 20:07:20 <CodeShark> presumably rational miners are looking to optimize fee per kb
404 2015-07-20 20:09:07 <CodeShark> are we still looking to provide a "
405 2015-07-20 20:09:11 <CodeShark> "free" area?
406 2015-07-20 20:10:20 <CodeShark> or are we looking to ultimately just have a fee maximization strategy?
407 2015-07-20 20:11:20 <CodeShark> I guess for relay we only care about fee/kb for the whole tx package
408 2015-07-20 20:13:15 <helo> is decentralizing the most common use path not sufficient?
409 2015-07-20 20:15:03 <helo> there have (will) always been (be) entities that make deals with miners, and trying to implement whatever it is they would need to do that over p2p is probably not worthwhile
410 2015-07-20 20:37:57 <dgenr8> morcos: petertodd: Ah, you have to evaluate from high feerate to low, and INclude ancestor packages (positive logic).  When when you get to feerate < new_tx_feerate, you can evict everything else.
411 2015-07-20 20:38:04 <dgenr8> Any children left at this point do not pay at least new_tx_feerate for their ancestor package (or you would have seen them earlier)
412 2015-07-20 20:38:56 <morcos> and then you'll end up iterating the whole mempool over and over again
413 2015-07-20 20:39:04 <jgarzik> gavinand1esen, Are there any transactions >100K in the chain today?  (I'm presuming someone has already run this query)
414 2015-07-20 20:39:14 <gmaxwell> jgarzik: there are.
415 2015-07-20 20:41:10 <sipa> there is a >900k transaction even, no?
416 2015-07-20 20:42:09 <gmaxwell> Correct.
417 2015-07-20 20:45:33 <dgenr8> morcos: it can be optimized.  but you start with a provable algorithm.
418 2015-07-20 20:48:13 <gavinandresen> jgarzik: I ran a script to dump largest transactions in last 100,000 blocks: https://0bin.zertrin.org/paste/f58567db65a1e6dd3db10e65acfa687b9572fc2a#qPvD/aikpduoaZM821LDfW316jKy6zz1+f/Awytn5Qk=
419 2015-07-20 20:48:25 <gavinandresen> jgarzik: columns are size, block number, txid
420 2015-07-20 20:53:15 <gavinandresen> jgarzik: (and that pastebin is just the biggest 100 blocks, I'll re-run to dump the number of transactions > 100K....)
421 2015-07-20 21:09:43 <alexwaters> are there any options other than polling mining pools to get a matter of fact network hash rate?
422 2015-07-20 21:09:59 <sipa> ;;nethash
423 2015-07-20 21:10:00 <gribble> 375093928.423
424 2015-07-20 21:10:07 <sipa> :)
425 2015-07-20 21:10:24 <alexwaters> but is that just an approximation based on current target and average block times?
426 2015-07-20 21:10:50 <sipa> yes
427 2015-07-20 21:11:12 <sipa> not sure how you'd get more accurate by polling pools, knowing that not all pools are public
428 2015-07-20 21:11:14 <alexwaters> that is pretty cool though
429 2015-07-20 21:11:17 <alexwaters> ;;nethash
430 2015-07-20 21:11:17 <gribble> 375093928.423
431 2015-07-20 21:11:27 <sipa> it fetches from bitcoin.sipa.be/speed-3D.txt
432 2015-07-20 21:12:00 <sipa> the 3D is because it's based on block data with a 3-day exponentially decaying window
433 2015-07-20 21:12:30 <alexwaters> i saw what looked like a 100P drop in hashrate via https://tradeblock.com/mining which was later averaged out
434 2015-07-20 21:13:01 <alexwaters> i'm concerned that it's an important network health metric, and we don't have an accurate way to track it
435 2015-07-20 21:14:14 <alexwaters> just curious if there were any ideas on that front as to ways to hack an accurate stat
436 2015-07-20 21:14:44 <gmaxwell> alexwaters: I don't think this is interesting. We can observe the creation of blocks. This is an imporant element of system operations, some measurement of hashrate not so much.
437 2015-07-20 21:15:36 <alexwaters> the timeline of set reconciliation methodology effects, gaming retargeting, may not be captured in the averaging by approximation
438 2015-07-20 21:16:16 <gmaxwell> alexwaters: Have you considered oscillatring the hyperspatial cascades?
439 2015-07-20 21:16:32 <alexwaters> that sounds awesome, but i have no idea what that means
440 2015-07-20 21:16:54 <sipa> we should use a DHT
441 2015-07-20 21:17:25 <alexwaters> sipa: how would that address it?
442 2015-07-20 21:17:37 <CodeShark> by content, of course :)
443 2015-07-20 21:17:55 <CodeShark> content hashes
444 2015-07-20 21:18:00 <gmaxwell> Precisely.
445 2015-07-20 21:18:23 <alexwaters> so if I'm doing IBLTs, I'm broadcasting my intent to hash something?
446 2015-07-20 21:18:39 <gmaxwell> alexwaters: perhaps move this conversation to #bitcoin
447 2015-07-20 21:18:51 <alexwaters> gmaxwell: why?
448 2015-07-20 21:19:31 <CodeShark> no, not intent - content
449 2015-07-20 21:20:16 <CodeShark> you can easily model it in anti-de Sitter spaces
450 2015-07-20 21:21:27 <alexwaters> did #bitcoin-dev turn to magic?
451 2015-07-20 21:21:50 <sipa> alexwaters: i think you have no idea what you're talking about
452 2015-07-20 21:22:15 <sipa> what does set reconsiliation have to do with hashrate?
453 2015-07-20 21:24:51 <alexwaters> oh wow ok, cool
454 2015-07-20 21:30:10 <alexwaters> sipa: so keeping a mempool updated has no effect on hashrate?
455 2015-07-20 21:30:20 <alexwaters> s/hashrate/my hashrate
456 2015-07-20 21:31:49 <alexwaters> i'm just asking questions, I didn't know I was making statements that warranted hostility
457 2015-07-20 21:32:48 <sipa> no
458 2015-07-20 21:33:51 <alexwaters> sipa: so this is null and void? https://gist.github.com/gavinandresen/e20c3b5a1d4b97f79ac2
459 2015-07-20 21:34:17 <sipa> no, but it has nothing to do with hashrate
460 2015-07-20 21:36:28 <phantomcircuit> alexwaters, hostility no, a bit of mocking... maybe
461 2015-07-20 21:38:05 <alexwaters> phantomcircuit: maybe I'm missing something. I thought bitcoin did SHA256(SHA256(Block_Header)) where the blockheader contained a hash of the mempool. if i'm mistaken I want to know. that led me to believe that set reconciliation does in some way affect hashrate
462 2015-07-20 21:38:20 <alexwaters> if i'm wrong, i'll go away and continue being stupid
463 2015-07-20 21:38:49 <phantomcircuit> alexwaters, bitcoin does do sha256(sha256(block_header)) and block header contains a merkle tree root, which include (by reference) a set of transactions
464 2015-07-20 21:38:57 <phantomcircuit> the set of transactions is selected from the mempool
465 2015-07-20 21:39:03 <phantomcircuit> but it is an asynchronous operation
466 2015-07-20 21:39:08 <gmaxwell> Please, move the bitcoin 101 to #bitcoin.
467 2015-07-20 21:39:10 <phantomcircuit> so mempool size has no (direct) effect
468 2015-07-20 21:39:13 <jgarzik> alexwaters, the block header summarizes the transaction list for that block.  transaction list != mempool
469 2015-07-20 21:39:15 <phantomcircuit> oh we're in -dev
470 2015-07-20 21:39:16 <phantomcircuit> oops
471 2015-07-20 21:40:16 <alexwaters> ok thank you, sorry
472 2015-07-20 22:18:53 <jtimon> https://github.com/bitcoin/bitcoin/pull/6448#issuecomment-123064158
473 2015-07-20 23:13:24 <petertodd> sipa, morcos: quick-n-dirty sketch in python of a greedy cpfp supporting mempool: https://gist.github.com/petertodd/6717ae29cc3cc6e32916
474 2015-07-20 23:22:15 <morcos> petertodd: i wish sdaftuar and i had written down all we discussed about the various issues
475 2015-07-20 23:22:37 <morcos> what does your code do if a tx gets mined (that isn't a leaf tx)
476 2015-07-20 23:23:00 <morcos> also even more complicated is when a disconnected block is added back to the mempool and already has children
477 2015-07-20 23:23:19 <morcos> and one step further if that tx already has parents in the mempool (from the same block) so its completing a link
478 2015-07-20 23:24:11 <morcos> you end up potentially having to recalculate all the nSumFees and nSumSizes all over again from scratch
479 2015-07-20 23:24:37 <petertodd> morcos: well, again this is one of those things where limiting maximum depths/breadths makes sense so that those kinds of recalculations aren't too expensive
480 2015-07-20 23:25:12 <petertodd> similarly, notice how because we're talking about outstanding fees, the worst case is a tx looks more expensive than it actually is
481 2015-07-20 23:25:15 <morcos> yeah, but even if you limit, it gets difficult, unless you're limiting breadths even of confirmed txs
482 2015-07-20 23:25:41 <petertodd> I'll bet you lazily recalculating an entire mempool is more reasonable than it sounds...
483 2015-07-20 23:26:18 <petertodd> remember that it's just a single pass, and per-tx fees and size are cached, so it's just a bunch of pointer following and addition