1 2015-10-01 00:53:22 <FredEE> .
  2 2015-10-01 01:50:32 <OxADADA> do nodes connected with port-forwarded 8333 tend to recieve blocks faster from peers?
  3 2015-10-01 01:51:15 <OxADADA> i have a node connected without support of an open 8333 on the net, and it seems to be taking longer to sync with the latest block
  4 2015-10-01 01:52:44 <PRab> OxADADA: Bring the question over to #bitcoin and I'll be happy to answer.
  5 2015-10-01 02:17:12 <Guest1234> is 0.9 still mantained? are important bugfixes/features being backported to it? I'm interested in particular on BIP65, does that need to be backported?
  6 2015-10-01 02:21:12 <PRab> Guest1234: I'm not 100% qualified to answer, but https://github.com/bitcoin/bitcoin/pulls?utf8=%E2%9C%93&q=is%3Apr+backport implies to me that it isn't. I only see backports for 0.10 and 0.11 right now.
  7 2015-10-01 02:21:36 <PRab> This doesn't mean that nobody will do a backport, but I don't see one for now.
  8 2015-10-01 02:39:50 <Guest1234> thanks PRab!
  9 2015-10-01 06:17:19 <BlueMatt> Guest1234: do you have a specific need to still be using 0.9???
 10 2015-10-01 08:41:54 <Naphex> jgarzik: enjoy sinking with the shiteheads you're supporting
 11 2015-10-01 08:42:38 <jgarzik> Naphex, off topic for this channel.  Insulting me is ok on #bitcoin
 12 2015-10-01 08:42:53 <Naphex> nah that was enough for me
 13 2015-10-01 08:44:39 <Naphex> should've saved that line tough
 14 2015-10-01 10:21:05 <jgarzik> Trolling for reviews, add bip32 pub key serialization #6215 https://github.com/bitcoin/bitcoin/pull/6215
 15 2015-10-01 10:23:21 <jgarzik> Trolling for reviews, Bugfix: Fix testnet-in-a-box use case #5987 https://github.com/bitcoin/bitcoin/pull/5987
 16 2015-10-01 14:47:33 <zooko> Is hearn still banned from this channel?
 17 2015-10-01 14:55:49 <jcorgan> i don't think so
 18 2015-10-01 15:12:37 <Guest1234> BlueMatt no, I was just curious and if it was needed I could try and code it
 19 2015-10-01 15:13:00 <jgarzik> zooko, no - but disruptive behavior can be banned again
 20 2015-10-01 15:13:24 <jgarzik> IMNSHO it was an emotional reaction, better done by one of the other chanops
 21 2015-10-01 15:17:12 <zooko> tbh I didn't understand what happened. I guess the ban was motivated by other stuff from other forums, that I haven't read.
 22 2015-10-01 17:05:30 <maaku> what time does the meeting start?
 23 2015-10-01 17:08:07 <sipa> maaku: 19h UTC, so in 1h52m
 24 2015-10-01 17:10:00 <maaku> sipa: I'm gonna miss most of it then :(
 25 2015-10-01 17:35:14 <DocHex> Is there a better IRC channel for current network conditions and developing situations on the network?
 26 2015-10-01 17:36:01 <maaku> DocHex: more context necessary
 27 2015-10-01 17:36:48 <DocHex> like a place to discuss OMG I just saw xx on the mainnet; isnt' that a violation of BIPyyy
 28 2015-10-01 17:37:07 <maaku> #bitcoin
 29 2015-10-01 17:37:44 <DocHex> ok, thanks
 30 2015-10-01 17:40:04 <maaku> things like the "I think we should add a node to the relay network in kazakhstan because..." would be on topic here
 31 2015-10-01 17:40:18 <maaku> so long as there's data to back it up
 32 2015-10-01 17:42:38 <DocHex> hmm. as usual I don't feel safe chating in public about nothing
 33 2015-10-01 17:45:20 <DocHex> I think I'll just wait here until someone says "low S"
 34 2015-10-01 17:45:22 <ploopkazoo> is there a channel dedicated to the android bitcoin wallet app?
 35 2015-10-01 17:53:47 <maaku> DocHex: low S is not enforced
 36 2015-10-01 17:53:58 <maaku> it's not even non-standard :(
 37 2015-10-01 17:54:58 <DocHex> Yes, and that's a problem.
 38 2015-10-01 17:57:41 <jonasschnelli> ploopkazoo: you can try #bitcoinj
 39 2015-10-01 17:57:52 <ploopkazoo> with the j?
 40 2015-10-01 17:58:11 <jonasschnelli> bitcoinj is the library used in "Bitcoin Wallet" by A. Schildbach.
 41 2015-10-01 17:58:27 <jonasschnelli> j stands for java
 42 2015-10-01 17:58:44 <ploopkazoo> ah
 43 2015-10-01 17:58:46 <ploopkazoo> thanks
 44 2015-10-01 18:13:56 <maaku> Since I won't be here for the meeting, I'll say my piece now regarding CLTV/CSV/...
 45 2015-10-01 18:15:16 <maaku> #6312 (bip 68) needs ACKs. #6564 (check sequence verify) and #6566 (median time-past) needs reviews
 46 2015-10-01 18:15:34 <maaku> The latter two are not as controversial in the implementation so review should be easy.
 47 2015-10-01 18:16:19 <maaku> There's an ugly part about #6312 non-consensus pathways, but doing things "properly" would be too big of a change to back-port as it requires modifying view structures and a good deal of wallet logic
 48 2015-10-01 18:16:22 <kanzure> can you elaborate again on the confusion re: process about which way to merge or which thing to merge first
 49 2015-10-01 18:16:31 <kanzure> saw your comment but have already lost it
 50 2015-10-01 18:17:09 <maaku> #6312 and #6564 can merge now. They have no code dependencies on anything.
 51 2015-10-01 18:17:25 <maaku> #6566 builds on #6312.
 52 2015-10-01 18:17:43 <btcdrak> maaku did you sort out your github notifications?
 53 2015-10-01 18:18:22 <maaku> no, but I github-stalked sipa to find all his nits
 54 2015-10-01 18:18:37 <maaku> i get notifications... just not from sipa on that patch set
 55 2015-10-01 18:18:50 <sipa> heh?
 56 2015-10-01 18:19:22 <maaku> sipa: i had to go to your profile page to see 'recent activity' to read what you had commented, since I wasn't getting email notifications of your code comments
 57 2015-10-01 18:19:38 <sipa> spam box.
 58 2015-10-01 18:19:39 <sipa> ?
 59 2015-10-01 18:19:49 <maaku> empty.. bizarre, I know
 60 2015-10-01 18:19:54 <btcdrak> accidental ignore
 61 2015-10-01 18:19:55 <btcdrak> ?
 62 2015-10-01 18:20:05 <maaku> maybe, i'll investigate
 63 2015-10-01 18:20:20 <sipa> I would feel sad :(
 64 2015-10-01 18:21:26 <maaku> I thought it was push related. I force-pushed a history modifying change and that's when the notifications stopped
 65 2015-10-01 18:21:44 <sipa> still weird
 66 2015-10-01 18:25:02 <btcdrak> maaku: check on https://github.com/sipa, there's a (!) dropdown, does it have "unblock" there?
 67 2015-10-01 18:27:24 <maaku> nope, sipa is not blocked
 68 2015-10-01 19:00:30 <dstadulis> I have a google doc for the meeting if there isn't one going already:
 69 2015-10-01 19:00:31 <dstadulis> https://docs.google.com/document/d/10yfs6_DeF1CnrkpyubNCpIVVPSqBE-tV_KebsI_O41A/edit?usp=sharing
 70 2015-10-01 19:00:43 <wumpus> nice
 71 2015-10-01 19:01:05 <dstadulis> Please include items you would like to discuss
 72 2015-10-01 19:03:34 <jtimon> meeting time?
 73 2015-10-01 19:03:36 <wumpus> we have some topics from last time: mempool managent, BIP68. and I think we should discuss this time softfork deployment (esp CLTV which is essentially ready to go
 74 2015-10-01 19:03:59 <morcos> is BlueMatt here?
 75 2015-10-01 19:04:05 <BlueMatt> yes
 76 2015-10-01 19:04:25 <BlueMatt> cant say I can think straight yet, but I'm here
 77 2015-10-01 19:04:33 <wumpus> ok, let's start with mempool management then
 78 2015-10-01 19:04:51 <morcos> ok well i guess i'd be interested if anybody else has reviewed 6557 and 6722 in depth and has opinions on direction
 79 2015-10-01 19:05:03 <CodeShark> in depth, no - concept ACK
 80 2015-10-01 19:05:12 <sipa> CodeShark: but which one of the two!
 81 2015-10-01 19:05:12 <wumpus> next time wel'll have a fancy IRC bot for keeping meeting topics etc
 82 2015-10-01 19:05:24 <morcos> personally i don't think there is a large difference in how effective either fo them are, but 6722 seems conceptually significantly simpler
 83 2015-10-01 19:05:34 <morcos> so barring any problems with it, thats the one i'd lean towards
 84 2015-10-01 19:06:08 <sdaftuar> i agree with morcos
 85 2015-10-01 19:06:29 <morcos> for those who hate PR number (sipa) 6557 is the one that allows a reserve space with exponentially increasing relay fee and new tx have to pay for the entire package size of evicted txs and rhte reserve space can be used in aggregate to trim the mempool
 86 2015-10-01 19:06:39 <warren> Comment last week was about confusion in the meeting due to discussing multiple topics simultaneously.
 87 2015-10-01 19:06:51 <warren> Could we put the current topic in /topic and stick to that until we agree to move on?
 88 2015-10-01 19:06:53 <sipa> so, only one topic at a time now
 89 2015-10-01 19:06:54 <BlueMatt> there are potentially still issues with it, but i think its the right direction to push for more review
 90 2015-10-01 19:07:02 <sipa> ### TOPIC: MEMPOOL MANAGEMENT
 91 2015-10-01 19:07:12 <gmaxwell> sipa is topicbot
 92 2015-10-01 19:07:16 <CodeShark> lol
 93 2015-10-01 19:07:17 <morcos> 6722 is the one where txs can boot a package of lower fee rate without paying for its total fee, but the effective min relay rate is now boosted to the cost of the evicted tx + minRelayRate with a decay
 94 2015-10-01 19:07:25 <dstadulis> warren +1 to topic announcement
 95 2015-10-01 19:07:35 <crescendo> ACK on topic management technique
 96 2015-10-01 19:07:42 <jgarzik> [meta]  For discussion PRs, please paste info disambiguation info, e.g. "Limit mempool by throwing away the cheapest txn and setting min relay fee to it #6722 https://github.com/bitcoin/bitcoin/pull/6722"
 97 2015-10-01 19:07:57 <sipa> ACK jgarzik
 98 2015-10-01 19:08:12 <morcos> does anybody have concerns with 6722 approach?
 99 2015-10-01 19:08:31 <gmaxwell> When I looked at 6722 before I thought it would take too long to return, but I see that this has already come up in the PR discussion.
100 2015-10-01 19:08:49 <sdaftuar> gmaxwell: i think there is a potential issue with how much work it could do
101 2015-10-01 19:08:54 <sdaftuar> but i think that can be mitigated
102 2015-10-01 19:09:16 <btcdrak> sdaftuar: how?
103 2015-10-01 19:09:28 <BlueMatt> gmaxwell: did you look at it pre-package-tracking or post?
104 2015-10-01 19:09:28 <jgarzik> the amount of work is also mitigated by raising minfee.....
105 2015-10-01 19:09:33 <Luke-Jr> 6722 doesn't seem perfect, but ACK on the basis of clear improvement
106 2015-10-01 19:09:34 <jgarzik> which 6722 does
107 2015-10-01 19:09:47 <sdaftuar> the last thing i suggested to BlueMatt is that we just be willing to evict even if we can't make enough space for the new transaction, and in that event we bump the relay fee up to the level of the last evicted package
108 2015-10-01 19:09:53 <Luke-Jr> concept ACK*
109 2015-10-01 19:10:49 <BlueMatt> sdaftuar: yea, I think thats probably OK
110 2015-10-01 19:10:53 <sipa> it seems like there are quite a bit of edge cases to consider in 6722
111 2015-10-01 19:11:03 <sipa> but the same is true for alternative approaches, i guess
112 2015-10-01 19:11:11 <BlueMatt> sipa: there are in all approaches yet proposed :(
113 2015-10-01 19:11:12 <morcos> sipa:  yes for sure!  but i f anything less edge cases than 6557
114 2015-10-01 19:11:16 <sdaftuar> the idea is that if we can evict every time we find something with a lower fee rate, then we end up re-sorting the rest of the mempool, so we avoid doing too much work (by following long chains that fail)
115 2015-10-01 19:11:25 <BlueMatt> hence why I really wanted a simpler approach, which 6722 is :/
116 2015-10-01 19:11:30 <jgarzik> sipa, agree - though in flood conditions imperfect hueristics are inevitable
117 2015-10-01 19:11:48 <jgarzik> and remember eviction is logically impossible to get perfect anyway, due to miner policy variation etc.
118 2015-10-01 19:12:26 <sipa> i think the aim should be that if miners would run with this code, they'd get good feerates in their mempools :)
119 2015-10-01 19:12:29 <jgarzik> client always has responsibility to resend, thus dropping is ok as a fallback
120 2015-10-01 19:12:29 <morcos> yes we agreed to add time based eviction as well
121 2015-10-01 19:12:39 <sipa> that way the network has an incentive to also run that code, to approximate it
122 2015-10-01 19:12:42 <morcos> sipa: i was discussing that with BlueMatt
123 2015-10-01 19:12:47 <gmaxwell> jgarzik: yes, but this isn't a question about perfection; it's a question about unexplorered corner cases that have potentially bad behavior that we're missing. :)
124 2015-10-01 19:12:47 <jgarzik> you can get eviction wrong... and client picks up the slack
125 2015-10-01 19:12:50 <morcos> its important to consider two aims
126 2015-10-01 19:13:03 <morcos> optimizing whats in a miners mempool and protecting against attacks
127 2015-10-01 19:13:11 <btcdrak> sipa: good idea
128 2015-10-01 19:13:13 <morcos> miners can run with much larger mempools
129 2015-10-01 19:13:18 <jtimon> I would like the dynamic min fee rate to be part of the fee estimator
130 2015-10-01 19:13:18 <sipa> yeah, it's obviously a compromise between those two
131 2015-10-01 19:13:25 <morcos> so the important thing is that more things get relayed to them
132 2015-10-01 19:13:38 <morcos> not that the eviction code is optimal on what gets kicked out from their viewpoint
133 2015-10-01 19:14:00 <morcos> if you can clog up full nodes mempools or boost the effective relay rate, thats a worse problem
134 2015-10-01 19:14:02 <jtimon> but I guess that's just a policy encapsulation nit
135 2015-10-01 19:14:05 <BlueMatt> 6722 has some corner cases where the selection of what to evict is optimal for maximum-value-of-fees-in-mempool, but not optimal for maximum-value-of-fees-in-next-block-I'd-mine
136 2015-10-01 19:14:16 <morcos> than the 10th percentile tx accidentally gets booted before the 0th
137 2015-10-01 19:14:23 <gmaxwell> (as far as the impossiblity of perfection, I don't actually agree at least in the sense that we could have a scheme which is 'perfect' with respect to at least _some_ miner policy, and thats perhaps better than being correct relative to none) :)   As has been previously noted non-miners don't technically need mempools at all, so long as our behavior doesn't allow nasty relay denying or relay amplific
138 2015-10-01 19:14:29 <gmaxwell> ation attacks, we're doing okay from the perspective of non-miners, I think.
139 2015-10-01 19:14:35 <sipa> jtimon: i don't think any of the currently explored alternatives use the fee estimator at all
140 2015-10-01 19:14:48 <jgarzik> gmaxwell, sure you have to be in -rough- sync otherwise nothing happens
141 2015-10-01 19:15:12 <BlueMatt> gmaxwell: 6722 is optimal for maximum-value-of-fees-in-my-mempool
142 2015-10-01 19:15:15 <morcos> yes, unspoken in all of these mempool limiting approaches is that the default min relay fee is hardcoded
143 2015-10-01 19:15:20 <BlueMatt> it is perfect for that metric
144 2015-10-01 19:15:27 <morcos> and that is not a good long term solution
145 2015-10-01 19:15:37 <BlueMatt> oh, except for a few games you can play to kick things out in the bottom package-size txn
146 2015-10-01 19:15:56 <gmaxwell> BlueMatt: I was about to say, only assuming no dependant transactions, and no funny business with conflicts.
147 2015-10-01 19:15:58 <morcos> but i think it is a separate fix to find out how to over the long term evolve the min relay fee
148 2015-10-01 19:15:59 <jtimon> sipa I was planning on implementing a dynamic minRelayFee using the fee estimator because I was assuming this would just use the static minRelayFee and some kind of delta
149 2015-10-01 19:16:02 <gmaxwell> which is probably fine.
150 2015-10-01 19:16:03 <BlueMatt> so, i guess its optimal for maximum-value-of-fees-in-my-mempool-for-a-mempool-sized-(mempool size - maximum package size)
151 2015-10-01 19:16:06 <morcos> its possible we can use fee estimation for that
152 2015-10-01 19:16:14 <morcos> but for now 1000 as a hard coded value is fine
153 2015-10-01 19:16:21 <jtimon> sipa: but I should read more about this new approach
154 2015-10-01 19:16:27 <BlueMatt> jtimon: I think I did that at one point
155 2015-10-01 19:16:37 <BlueMatt> its not so good, since its a really laggy indicator
156 2015-10-01 19:16:48 <BlueMatt> you'd limit mempool a few blocks after your mempool caused your node to crash from OOM
157 2015-10-01 19:16:49 <CodeShark> a long term solution will require direct compensation of relayer...given that's not the current situation, what gmaxwell said :)
158 2015-10-01 19:16:54 <gmaxwell> jtimon: I believe the fee estimator needs to be largely orthorgonal with this. This needs to depend on the size of your own mempool to achieve its goals.
159 2015-10-01 19:17:00 <sipa> i think we can use a very-slowly-adjusting estimator for the minimum relay cost
160 2015-10-01 19:17:10 <BlueMatt> gmaxwell: no, it should be optimal even assuming crazy dependent transactions and crazy funny business with conflicts
161 2015-10-01 19:17:13 <morcos> laggy is ok, its not good because it can't return information about data points it doesn't have, so if most txs are happening at a higher fee rate, its hard to predict what should ACTUALLY be the min relay rate
162 2015-10-01 19:17:24 <sipa> because all of these techniques require a good estimate of "cost of the the network of an increment"
163 2015-10-01 19:17:29 <BlueMatt> gmaxwell: I have yet to see/come up with an issue with that
164 2015-10-01 19:17:29 <jgarzik> KISS
165 2015-10-01 19:17:32 <morcos> gmaxwell: yes
166 2015-10-01 19:17:46 <sipa> jgarzik: well, i think many simple techniques have been explorer already and found faulty
167 2015-10-01 19:17:51 <gmaxwell> BlueMatt: I mean I can feed you a set of transations that makes you evict a parent of a later higher value transaction I give you.
168 2015-10-01 19:17:56 <jtimon> BlueMatt I remember jgarzik's implementation which used another thread instead of the fee estimator
169 2015-10-01 19:18:21 <morcos> ok, if i can move it along, no objections so far.  so lets let BlueMatt get his code a bit more complete, and then we can all keep trying to attack it
170 2015-10-01 19:18:27 <morcos> next subtopic is chain limits
171 2015-10-01 19:18:27 <sipa> morcos: ack
172 2015-10-01 19:18:29 <gmaxwell> OK.
173 2015-10-01 19:18:31 <jgarzik> ack
174 2015-10-01 19:18:32 <BlueMatt> jtimon: I responded on that pull with something that used the fee estimator, but never submitted it as a pr
175 2015-10-01 19:18:47 <CodeShark> ack
176 2015-10-01 19:18:50 <morcos> all of these approachs are much more attackable the bigger package sizes can be
177 2015-10-01 19:18:52 <BlueMatt> gmaxwell: yes, that is something else we need to discuss
178 2015-10-01 19:18:59 <morcos> the proposed limits that we implemented already are too generous
179 2015-10-01 19:19:01 <sipa> ### SUBTOPIC: CHAIN LIMITS
180 2015-10-01 19:19:03 <BlueMatt> two more things to discuss: a) lower package size limits
181 2015-10-01 19:19:06 <jtimon> gmaxwell: I agree that this should be mostly orthogonal to the fee estimator, but at the same time I believe a dynamic minRelayFee should be calculated withing the estimator, that's my concern
182 2015-10-01 19:19:11 <BlueMatt> b) accepting multiple txn at once
183 2015-10-01 19:19:25 <morcos> oh please lets hold off on b)
184 2015-10-01 19:19:35 <morcos> thats a whole differnt ball of wax
185 2015-10-01 19:19:45 <morcos> and not needed right now
186 2015-10-01 19:19:49 <jgarzik> +1
187 2015-10-01 19:19:51 <BlueMatt> oh, and, maybe, decay constant for #6722, but maybe thats too implementation-detail for this talk
188 2015-10-01 19:19:58 <BlueMatt> morcos: I think its quite simple
189 2015-10-01 19:20:05 <BlueMatt> use mapOrphans to accept multiple txn at once
190 2015-10-01 19:20:15 <BlueMatt> i just wanted to mention it to make sure no one would complain if i did that
191 2015-10-01 19:20:20 <gmaxwell> BlueMatt: stop. offtopic now. What the heck is the new topic?
192 2015-10-01 19:20:26 <morcos> can that PLEASE be a separate pull
193 2015-10-01 19:20:31 <jgarzik> <sipa> ### SUBTOPIC: CHAIN LIMITS
194 2015-10-01 19:20:32 <sipa> ack morcos
195 2015-10-01 19:20:36 <BlueMatt> morcos: ok
196 2015-10-01 19:20:39 <jgarzik> +1 morcos
197 2015-10-01 19:20:40 <BlueMatt> hence why i mentioned it :p
198 2015-10-01 19:20:44 <BlueMatt> anyway, so chain limits.....
199 2015-10-01 19:20:47 <sipa> morcos: elaborate on topic
200 2015-10-01 19:20:54 <morcos> hmm
201 2015-10-01 19:20:56 <BlueMatt> all of the mempool limiting stuff is wayyyyy easier to attack if you have bigger chain limits
202 2015-10-01 19:21:06 <BlueMatt> ratio of max-chain-size to block size
203 2015-10-01 19:21:06 <sipa> BlueMatt: you mean smaller chain limits :)
204 2015-10-01 19:21:16 <sipa> ah, attackable
205 2015-10-01 19:21:16 <sipa> yes
206 2015-10-01 19:21:17 <morcos> ok one problem is that you can construct funny looking packages that have a very high descendant package feerate and a very low ancestor package feerate
207 2015-10-01 19:21:22 <BlueMatt> ratio of max-chain-size to reasonable tx size, etc
208 2015-10-01 19:21:24 <BlueMatt> all matter a lot
209 2015-10-01 19:21:27 <morcos> these packages will not be mined any time soon , nor evicted
210 2015-10-01 19:21:42 <morcos> they serve to clog up your mempool (until time based eviction)
211 2015-10-01 19:21:50 <gmaxwell> I thought just limiting depth was reasonably prudent.
212 2015-10-01 19:21:54 <morcos> this is of course extremely bad right now without CPFP mining code
213 2015-10-01 19:21:56 <CodeShark> shouldn't the total fee calculation weigh ancestors more than descendants?
214 2015-10-01 19:21:59 <jgarzik> +1 gmaxwell
215 2015-10-01 19:22:09 <sdaftuar> gmaxwell: what exactly do you mean by depth?
216 2015-10-01 19:22:44 <jgarzik> I'm highly biased, but I think time limiting should go sooner rather than later as it serves as a natural limit for many of these other attributes
217 2015-10-01 19:22:46 <gmaxwell> I am not aware of any applications that require very long chains of unconfirmed transactions right now. Though I realize that both size and depth are an issue, because of tx soft size limits, if the depth can not be longer than 10 you can still fit in a block.
218 2015-10-01 19:22:46 <morcos> right now the effective limits in place are 100 ancestors 900kb ancestor package size, 1000 descendants, 2500kb descendant package size
219 2015-10-01 19:23:11 <gmaxwell> but I was just just thinking in terms of a linear chain, and thats .. a problem.
220 2015-10-01 19:23:13 <jgarzik> vast majority of user applications do not construct long chains at all -- they avoid it
221 2015-10-01 19:23:14 <gmaxwell> :(
222 2015-10-01 19:23:15 <CodeShark> the metric should be something like total fees / expected time to confirmation (or something like that)
223 2015-10-01 19:23:15 <morcos> the reaosn to have larger descendant packages is you can't control that yourself,  somebody pays you and bob, and bob chains off a million descendants and he ends up screwing you
224 2015-10-01 19:23:22 <jgarzik> let's not over-optimize for an unusual user case
225 2015-10-01 19:23:53 <morcos> so i'd propose that we reduce at least to : ancestor = 25/250kb   descendant = 50/500kb
226 2015-10-01 19:23:59 <BlueMatt> total size of packages effects whether your decision making in 6722 is optimal for more cases or just vaguely optimal
227 2015-10-01 19:24:08 <BlueMatt> total package tx count is more related to runtime
228 2015-10-01 19:24:08 <gmaxwell> jgarzik: some of this is due to malleability which is being fixed over time; so we should not be too agressive there.
229 2015-10-01 19:24:21 <jgarzik> there are user risks inherent in long chains that incentivize against building them at all
230 2015-10-01 19:24:22 <morcos> using data from april and may of this year, each of those limits on its own would affect no more than 5% of txs
231 2015-10-01 19:24:25 <sipa> jgarzik: i think time limiting should definitely be part of the release, but as some of the alternative limiting code patches already have that integrated, it may not be useful to pre-merge that part already
232 2015-10-01 19:24:29 <jgarzik> gmaxwell, agree; it's a balance
233 2015-10-01 19:24:56 <gmaxwell> There are two issues as I understand it, runtime blowup from deep chains;  and packages that can't be mined in one go (over 1mb) (or at least are so big that quantization in mining decisions really lowers their prospects of being mined)
234 2015-10-01 19:24:59 <jgarzik> sipa, nod - time-limiting PR is closed until dust settles
235 2015-10-01 19:25:02 <jgarzik> can be re-created easily
236 2015-10-01 19:25:11 <jgarzik> just waiting for the moment :)
237 2015-10-01 19:25:26 <jgarzik> notably packages
238 2015-10-01 19:25:28 <morcos> whether those limits are sufficiently aggresive or not remains to be seen, but if we think those are politcially palpable it will be helpful to think about what attacks we can construe on 6722 if constrained to those limits
239 2015-10-01 19:25:42 <morcos> gmaxwell: its worse than that
240 2015-10-01 19:25:47 <sipa> we can start with relatively lax limits
241 2015-10-01 19:26:08 <sipa> if specific attacks are found that cause large packages/chains to cause probably, we can recosinder the defaults
242 2015-10-01 19:26:21 <morcos> if you have a say 900kb ancestor package limit, then even if the ancestor fee rate is reasonably high, default mining code is likely going to find 100kb of very high fee txs to include first, and then there won't be room for your ancestor package
243 2015-10-01 19:26:25 <BlueMatt> I think almost every attack we've come up with against 6722 is directly limited by package size/volume
244 2015-10-01 19:26:41 <jgarzik> time limiting will wind up evicting an early ancestor of a long chain, of course
245 2015-10-01 19:26:46 <morcos> sipa: what is relatively lax?
246 2015-10-01 19:26:50 <gmaxwell> I think limits that allow 3 or 4 deep linear chains of ordinary dependant transactions are sufficient to cover all current usecases that I'm aware of... (e.g. not unduely limiting)
247 2015-10-01 19:26:57 <sipa> morcos: what we have now? :)
248 2015-10-01 19:26:58 <jgarzik> that mitigates some attacks, which is good.
249 2015-10-01 19:27:10 <jgarzik> +1 gmaxwell
250 2015-10-01 19:27:11 <morcos> i'd call those VERY lax
251 2015-10-01 19:27:12 <CodeShark> so no more satoshi dice? ;)
252 2015-10-01 19:27:15 <gmaxwell> morcos: sorry, thats what I was going for with "quantization effects",  I just mean the transaction is in or out, and a big package is more costly than its size implies.
253 2015-10-01 19:27:25 <jgarzik> CodeShark: those guys wait for confirmations these days
254 2015-10-01 19:27:33 <jgarzik> too many thefts
255 2015-10-01 19:27:55 <morcos> sipa, you think my new proposal is too strict:  ancestor = 25/250kb   descendant = 50/500kb
256 2015-10-01 19:28:19 <BlueMatt> what you really want is smaller package size, not tx count...that part worries me more
257 2015-10-01 19:28:19 <sipa> all good to me... i think they're all way beyond what is needed
258 2015-10-01 19:28:25 <BlueMatt> and you really cant go very small
259 2015-10-01 19:28:37 <BlueMatt> since one of the important metrics is ratio of package size to block size :(
260 2015-10-01 19:28:46 <jgarzik> I think morcos limits are lax :)
261 2015-10-01 19:28:47 <BlueMatt> 500kb is much larger than you wait, ideally, I think
262 2015-10-01 19:28:53 <BlueMatt> want
263 2015-10-01 19:28:57 <morcos> the way i view it is after months of working on this stuff , we keep finding slightly more obscure attacks and fixing them , if the limits are just significantly smaller to begin with it just gives us a bigger safety buffer
264 2015-10-01 19:29:00 <BlueMatt> but probably a reasonable limit
265 2015-10-01 19:29:12 <sipa> morcos: ack then
266 2015-10-01 19:29:14 <sdaftuar> +1 morcos
267 2015-10-01 19:29:24 <jgarzik> ack morcos limits - and would be ok with smaller limits
268 2015-10-01 19:29:33 <sipa> i have no good intuition for what is needed here... i would think much smaller is acceptable
269 2015-10-01 19:29:41 <sipa> but no need to unnecessarily restrict things
270 2015-10-01 19:30:09 <morcos> so i don't wnat to email the dev list and propose different limits over and over
271 2015-10-01 19:30:09 <sdaftuar> seems like next step is to email the dev list and see if anyone has use cases we
272 2015-10-01 19:30:12 <sdaftuar> 're missing?
273 2015-10-01 19:30:14 <morcos> should i aim for even smaller?
274 2015-10-01 19:30:54 <sipa> i think 25/50k 50/100k would also be fine :)
275 2015-10-01 19:31:09 <sdaftuar> so 100k -> 50k for standard transactions?
276 2015-10-01 19:31:10 <CodeShark> not sure I understand that notation - what's the numerator?
277 2015-10-01 19:31:15 <sipa> CodeShark: number of transactions
278 2015-10-01 19:31:37 <BlueMatt> 100k would be ideal for mempool limiting stuff, imo
279 2015-10-01 19:31:40 <CodeShark> so allocate 50k for 25 transactions, allocate 100k for 50?
280 2015-10-01 19:31:44 <BlueMatt> but I think thats more constricting than we want to be?
281 2015-10-01 19:31:52 <BlueMatt> does someone have numbers for how many single txn are over 50kb?
282 2015-10-01 19:31:54 <sipa> ### MEETING AT 50%
283 2015-10-01 19:32:10 <morcos> ok, i will email the dev list with several proposals
284 2015-10-01 19:32:11 <gmaxwell> sdaftuar: when emailing the dev list please pass your message by a second set of eyes to make sure you don't accidentally make people think things will be forbidden completely when instead they's just have to wait for the first layer to confirm.
285 2015-10-01 19:32:31 <morcos> gmaxwell: yes good to emphasize
286 2015-10-01 19:32:34 <sdaftuar> gmaxwell: yep.  i'll let morcos do it anyway :)
287 2015-10-01 19:32:43 <sipa> anything else to discuss about chain limits?
288 2015-10-01 19:32:44 <morcos> i think not reducing the std tx limit of 100k makes sense right?
289 2015-10-01 19:32:59 <morcos> and if you want to chain one thing of that, might it not need to be of almost similar size?
290 2015-10-01 19:33:00 <gmaxwell> sdaftuar: (we've suffered this with the dust limit stuff; where people _constantly_ think the dust limit prevents you from _spending_ low value utxo, partially because initial communications were unclear and triggered some confused press)
291 2015-10-01 19:33:09 <morcos> i guess you only need to spend one of its outputs
292 2015-10-01 19:33:28 <jgarzik> gmaxwell, a high hashrate miner was even confused about that....
293 2015-10-01 19:33:47 <morcos> ok will post proposed email here tomorrow, with some stats on historical txs
294 2015-10-01 19:33:48 <gmaxwell> morcos: std limit is largely for legacy reasons (old validation code loaded all input txn into memory), but it  turns out to be useful in many surprising places (e.g. capping sighash cost), so I think we should not touch it. We've had very few complaints about it.
295 2015-10-01 19:34:01 <sdaftuar> ok if we're done with chain limits, can we talk briefly about default mempool size?
296 2015-10-01 19:34:17 <sipa> ### SUBTOPIC: default mempool size
297 2015-10-01 19:34:22 <morcos> think BlueMatt agreed to go at least 300MB, thats seems enough for me for now
298 2015-10-01 19:34:39 <jgarzik> Previously discussions have measured size in days first, size second.
299 2015-10-01 19:34:39 <morcos> but i do think 100MB is too low for default
300 2015-10-01 19:34:49 <morcos> 300MB is less thna 1 day
301 2015-10-01 19:34:51 <sdaftuar> well we're under a day still
302 2015-10-01 19:34:57 <sdaftuar> but 300MB seems much better to me
303 2015-10-01 19:34:58 <morcos> but not REALLY because its a day backlog
304 2015-10-01 19:35:01 <jgarzik> 2 days, was the past discussion
305 2015-10-01 19:35:24 <morcos> if you are the bottom of the 300MB mempool you're probably 2 days away from getting confirmed
306 2015-10-01 19:35:25 <gmaxwell> This has some interaction with the minimum supported amount of memory to run bitcoin core, but 300mb sounds fine to me.  We should perhaps autoscale it on low memory systems; though I don't think we have any portable way to accomidate that.  I think for now lets just do something stupid/simple.
307 2015-10-01 19:35:25 <jonasschnelli> the current default is "unlimited", i think it should be at least 300MB.
308 2015-10-01 19:35:37 <sipa> jonasschnelli: 2 days, at max block size, would be 900 MB usage
309 2015-10-01 19:35:49 <sipa> eh, jgarzik ^
310 2015-10-01 19:36:05 <wumpus> 900MB is unacceptable imo
311 2015-10-01 19:36:09 <morcos> lets stick with 300... its plenty for the current usage we're seeing.  its a command line option
312 2015-10-01 19:36:11 <sdaftuar> ok if 300MB is our current proposal that's fine with me (move along)...
313 2015-10-01 19:36:14 <wumpus> yes
314 2015-10-01 19:36:22 <jonasschnelli> ack
315 2015-10-01 19:36:25 <BlueMatt> yea, I dont like 900, 300 ack
316 2015-10-01 19:36:27 <morcos> future changes where people want to really take advantage of weekly cycles can consider ways to reasonably make the mempool biger
317 2015-10-01 19:36:30 <BlueMatt> hence why I picked that :p
318 2015-10-01 19:36:40 <sipa> topic mempool limiting done?
319 2015-10-01 19:36:43 <morcos> ack
320 2015-10-01 19:36:44 <BlueMatt> ack
321 2015-10-01 19:36:44 <sdaftuar> ack
322 2015-10-01 19:36:45 <gmaxwell> ack
323 2015-10-01 19:36:52 <jgarzik> +1
324 2015-10-01 19:36:52 <wumpus> next topic, upcoming CLTV softfork?
325 2015-10-01 19:36:58 <Luke-Jr> is there a reliable way to measure system RAM?
326 2015-10-01 19:37:02 <Luke-Jr> oh well, nm
327 2015-10-01 19:37:05 <sipa> ### TOPIC: CLTV softfork
328 2015-10-01 19:37:08 <wumpus> Luke-Jr: let's take that offline
329 2015-10-01 19:37:48 <maaku> sipa: clarity on topic? we discussing softfork deployment strategy, or content of softfork
330 2015-10-01 19:37:48 <sipa> anyone has anything to say on CLTV?
331 2015-10-01 19:37:49 <btcdrak> So the question is, do we want to push out CLTV using IsSuperMajority now, or wait for BIP68+CSV to be finished?
332 2015-10-01 19:37:50 <Luke-Jr> so is versionbits ready? or would we be triggering on >=?
333 2015-10-01 19:38:01 <sipa> maaku: deployment
334 2015-10-01 19:38:01 <wumpus> so, CLTV is ready for merge, including backport PRs. But there are some people talking about including other stuff in the softfork as well such as CSV.
335 2015-10-01 19:38:34 <wumpus> also the content I suppose, as it affects deployment
336 2015-10-01 19:38:36 <btcdrak> The other question is are we going to mask out BIP101's blockversion or stick with a test of blockver >= 4 with no mask
337 2015-10-01 19:38:38 <Luke-Jr> wumpus: if we use versionbits, CMV might as well be independent
338 2015-10-01 19:38:40 <wumpus> CLTV could  be run out now
339 2015-10-01 19:38:58 <sipa> if versionbits is deployed for a future softfork, we'll need to wait until all ISM-based softforks are over
340 2015-10-01 19:39:02 <maaku> regarding content, I'd much prefer explaining the soft-fork with all the various lock-time stuff together
341 2015-10-01 19:39:10 <gmaxwell> CLTV BIP is a year old today, the design is older still. It's gone through a lot of maturation, and is clearly quite mature and ready (there is even some altcoin deployment).  So, obviously I am ACK on the view that its ready for short term deployment.
342 2015-10-01 19:39:12 <wumpus> versionbits has no code yet, so I think it's off the table for this one
343 2015-10-01 19:39:17 <Luke-Jr> btcdrak: it's important that we don't make blockver>=4 a consensus rule long-term or we lose a bit for versionbits
344 2015-10-01 19:39:18 <maaku> it'd be a lot of wasted effort to pull of two soft-forks
345 2015-10-01 19:39:34 <CodeShark> versionbits has code - but not complete ;)
346 2015-10-01 19:39:37 <wumpus> it will need to be written, tested, reviewed, and we'll be a few months further
347 2015-10-01 19:39:38 <sipa> IMHO, once versionbits exists, softforks can be iterated much more quickly
348 2015-10-01 19:39:39 <gmaxwell> maaku: we need to become more adept at keeping a pipeline of soft forks going, though.. A thing to keep in mind.
349 2015-10-01 19:39:52 <wumpus> that's true sipa
350 2015-10-01 19:39:56 <sipa> and once versionbits exists, there is less overhead from "scheduling another softfork"
351 2015-10-01 19:39:57 <bsm1175321> +1 gmaxwell
352 2015-10-01 19:39:58 <btcdrak> maaku how far away would BIP68/112/113 be from merge?
353 2015-10-01 19:40:09 <jl2012> sipa: no need for waiting if the future softfork includes CLTV
354 2015-10-01 19:40:15 <maaku> btcdrak: as far as I am aware they are all merge-ready
355 2015-10-01 19:40:21 <jgarzik> If it's ready go to, go
356 2015-10-01 19:40:35 <sipa> Luke-Jr: indeed, i noted that too
357 2015-10-01 19:40:44 <sipa> though i don't have a strong opinion
358 2015-10-01 19:40:46 <jtimon> btcdrak I think your plan of waaiting for 0.12 and move with CLTV (and RCLTV and nVersion bits too if they are ready on time) was good
359 2015-10-01 19:40:46 <morcos> Luke-Jr: no verstionbits can just = versionbits +CLTV
360 2015-10-01 19:40:48 <CodeShark> about how long do we expect deployment to take?
361 2015-10-01 19:40:51 <gmaxwell> Maturity of CLTV is such that I think we could achieve very fast deployment. There is strong demand for it.
362 2015-10-01 19:41:06 <btcdrak> if the CSV still will be ready in the next month/6 weeks I would hold off deployment of CLTV personally.
363 2015-10-01 19:41:10 <gmaxwell> CodeShark: I believe certantly no more time than BIP66.
364 2015-10-01 19:41:10 <wumpus> I don't like binding 0.12 to softfork deployment
365 2015-10-01 19:41:11 <Luke-Jr> morcos: ?
366 2015-10-01 19:41:19 <CodeShark> if deployment is a matter of a month or two, I'll wait - if it's going to take six months, I'd rather do version bits first
367 2015-10-01 19:41:24 <warren> What is the maturity of RCLTV?
368 2015-10-01 19:41:25 <sipa> indeed, softfork deployment can be done at any time, including with minor releases
369 2015-10-01 19:41:28 <btcdrak> but I've no strong opinion to block it. We need CLTV out sooner than later.
370 2015-10-01 19:41:30 <gmaxwell> We shouldn't bind major releases and soft-fork deployment.
371 2015-10-01 19:41:33 <sipa> warren: you mean checksequenceverify?
372 2015-10-01 19:41:34 <wumpus> it needs to be backported to 0.10+ anyhow, so it can be done any time
373 2015-10-01 19:41:58 <wumpus> 0.12 could obviously have some mempool-only softforks already in, but that's a different thing
374 2015-10-01 19:42:05 <sipa> argument in favor of IsSuperMajority-based trigger: versionbits may be harder to backport
375 2015-10-01 19:42:07 <btcdrak> but if it's going to hold up future deployment of versionbits deployments, it means we have to push CSV out with blockver >= 5 also
376 2015-10-01 19:42:16 <btcdrak> sipa: +1
377 2015-10-01 19:42:32 <jtimon> wumpus: softfork deployment i not coupled with major version, if CSV is not ready by then we can make a minor version for it (maybe using versionbits)
378 2015-10-01 19:42:40 <btcdrak> in fact, considering the backporting issue, I think we cannot reasonably expect to deploy CLTV or CSV with versionbits now
379 2015-10-01 19:42:45 <gmaxwell> On a general matter of principle I do not prefer binding multiple proposals. If multiple proposals happen to coincide I don't see a problem with it. But CLTV has 6 months to a year maturity advantage on the related locking proposals.
380 2015-10-01 19:42:47 <wumpus> jtimon: right
381 2015-10-01 19:42:54 <sipa> IsSuperMajority is ready, CLTV is ready... let's go for it
382 2015-10-01 19:43:02 <maaku> i'm in favor of doing IsSuperMajoirty for this, just not rushing CLTV without CSV (which is ready already)
383 2015-10-01 19:43:02 <wumpus> agreed gmaxwell, CLTV is ready, let's do it
384 2015-10-01 19:43:11 <gmaxwell> As an aside ignoring version bits for a moment, we can also begin a v=5 rollout while v=4 is in progress so long as the v5 is strictly cumulative.
385 2015-10-01 19:43:12 <jgarzik> +1 gmaxwell
386 2015-10-01 19:43:20 <morcos> I was under the impression that we can deploy CLTV with SuperMajority now, then whenever the next soft fork is ready, assuming we all still think we want both of them, we just condition the second soft fork on also including CLTV
387 2015-10-01 19:43:20 <sipa> gmaxwell: true!
388 2015-10-01 19:43:27 <morcos> yes gmaxwell, thats what i was trying to say poorly
389 2015-10-01 19:43:30 <jtimon> sipa: why make a minor version now if we're about to do a major one?
390 2015-10-01 19:43:30 <maaku> gmaxwell: my argument is no that it is technically difficult, but that it is hard to explain
391 2015-10-01 19:43:33 <warren> sipa: oops, yes
392 2015-10-01 19:44:06 <wumpus> ok: so one BIPxx softfork at a time (don't bind them together), use IsMajority for now, CLTV first
393 2015-10-01 19:44:07 <gmaxwell> maaku: I think it is no more difficult than 4,5 non-overlapping.
394 2015-10-01 19:44:07 <jgarzik> binding unrelated proposals together creates a logjam
395 2015-10-01 19:44:07 <maaku> i don't want to go back to a miner a week or two later with _another_ soft fork point release
396 2015-10-01 19:44:07 <sipa> but CLTV is useful, even without CSV?
397 2015-10-01 19:44:11 <btcdrak> I think we've got consensus to roll out CLTV
398 2015-10-01 19:44:21 <jgarzik> wumpus, ack
399 2015-10-01 19:44:30 <BlueMatt> mehhhh
400 2015-10-01 19:44:36 <jgarzik> CLTV first, versionbits next
401 2015-10-01 19:44:36 <maaku> nack
402 2015-10-01 19:44:39 <BlueMatt> csv is muchhh more useful
403 2015-10-01 19:44:49 <BlueMatt> and i dont really want to push it out another six months
404 2015-10-01 19:44:49 <sipa> BlueMatt: but not nearly as ready
405 2015-10-01 19:44:52 <gmaxwell> maaku: as I said ealier, we need to develop a cadance in general.  Expirence with BIP66 was that miner adoption was good even though the benefits of BIP66 were vague to most people.
406 2015-10-01 19:44:53 <BlueMatt> i know
407 2015-10-01 19:44:56 <wumpus> this means #6706 and #6707 (backport CLTV to 0.10 and 0.11 respectivel) need review ASAP
408 2015-10-01 19:45:00 <sipa> BlueMatt: it isn't being pushed out another 6 motnhs
409 2015-10-01 19:45:09 <BlueMatt> gmaxwell: cadence is great....when we have versionbits
410 2015-10-01 19:45:09 <maaku> what is holding up merging CSV?
411 2015-10-01 19:45:17 <btcdrak> jgarzik: nack, csv would need issupermajority rollout due to versionbits backporting difficulty
412 2015-10-01 19:45:27 <CodeShark> about how long do we expect CLTV using IsSuperMajority to take to deploy (if we go that route)?
413 2015-10-01 19:45:31 <jtimon> so, again, we're doing a minor version only for CLTV instead of waiting for 0.12 which is just around the corner?
414 2015-10-01 19:45:34 <jgarzik> rollout will happen quickly
415 2015-10-01 19:45:38 <jgarzik> weeks not months
416 2015-10-01 19:45:42 <btcdrak> maaku: good question: why havent there been more reviews on CVS?
417 2015-10-01 19:45:42 <sipa> well... the first versiobits rollout will trigger all ISM-based rollouts too
418 2015-10-01 19:45:47 <BlueMatt> sipa: in theory its not, in reality it probably is
419 2015-10-01 19:45:47 <morcos> why don't we set a deadline for CSV to be fully reviewed
420 2015-10-01 19:45:50 <sipa> jgarzik: we don't know that
421 2015-10-01 19:45:51 <wumpus> yes, no waiting for 0.12
422 2015-10-01 19:46:05 <wumpus> don't want to couple softfork with a major release
423 2015-10-01 19:46:11 <gmaxwell> BlueMatt: We can deploy CLTV basically immediately ... like, within days if we really wanted to, the same is not true for versionbits, and I think it is far less obviously true for CSV, as it's just a less mature proposal.
424 2015-10-01 19:46:23 <jgarzik> +1
425 2015-10-01 19:46:24 <BlueMatt> jgarzik: rollout will not take weeks
426 2015-10-01 19:46:31 <sipa> BIP66 took 5 months
427 2015-10-01 19:46:35 <BlueMatt> gmaxwell: I highly doubt cltv can softfork in weeks
428 2015-10-01 19:46:37 <BlueMatt> that is insane
429 2015-10-01 19:46:37 <btcdrak> gmaxwell: I would like to see it as a test of how united the mining community can be.
430 2015-10-01 19:46:37 <jtimon> morcos: btcdrak's plan was to make 0.12 that deadline (and for versionbits to be used in CLTV or not too)
431 2015-10-01 19:46:55 <gmaxwell> I believe that very rapid rollout is possible and not unlikely. There is clear pent up demand.
432 2015-10-01 19:47:06 <sipa> gmaxwell: i'm unsure about that
433 2015-10-01 19:47:06 <wumpus> yes, the mailing list was clear on that
434 2015-10-01 19:47:14 <BlueMatt> gmaxwell: bip66 took a long time with devs hammering on miners to do it quick
435 2015-10-01 19:47:16 <maaku> what is the imputus to rush this? why are people not reviewing CSV?
436 2015-10-01 19:47:18 <BlueMatt> and yet it didnt happen
437 2015-10-01 19:47:19 <bsm1175321> Proposal: create windows for and deadlines for soft forks.  One every 3 months?  6?  Window for testnet inclusion, and intended deployment.
438 2015-10-01 19:47:19 <gmaxwell> weeks is unrealistic just on the activation thresholds, but 2 months I think is possible.
439 2015-10-01 19:47:24 <BlueMatt> that is not gonna happen with cltv
440 2015-10-01 19:47:27 <sipa> here is something to consider: the first versionbits rollout will trigger all ISM-based rollouts
441 2015-10-01 19:47:33 <jgarzik> I think 8-12 weeks
442 2015-10-01 19:47:48 <sipa> we can implement versionbits, and use it for CSV, even before the ISM-based CLTV softfork occurred
443 2015-10-01 19:47:57 <maaku> I'm not in favor of versionbits for CSV
444 2015-10-01 19:47:59 <CodeShark> what are ISM-based rollouts?
445 2015-10-01 19:48:00 <sipa> it will just mean that they may trigger simultaneously
446 2015-10-01 19:48:02 <BlueMatt> sipa: hmm, indeed
447 2015-10-01 19:48:03 <CodeShark> oh
448 2015-10-01 19:48:05 <Luke-Jr> maaku: wtf?
449 2015-10-01 19:48:06 <michagogo> Does ISM happen faster on testnet?
450 2015-10-01 19:48:06 <sipa> CodeShark: IsSuperMajority
451 2015-10-01 19:48:07 <wumpus> in any case I think it's better to make progress, one step at a time
452 2015-10-01 19:48:08 <CodeShark> IsSuperMajority - duh ;P
453 2015-10-01 19:48:09 <btcdrak> CodeShark: issuprtmajority
454 2015-10-01 19:48:09 <gmaxwell> sipa: so long as it includes CLTV, which I think there is zero concern that it wouldn't.
455 2015-10-01 19:48:15 <sipa> gmaxwell: indeed
456 2015-10-01 19:48:18 <maaku> I'm in favor of IsSuperMajoirty with all lock-time soft-forks together
457 2015-10-01 19:48:34 <Luke-Jr> maaku: why is ISM preferred over versionbits?
458 2015-10-01 19:48:36 <morcos> proposal, end of Oct, release ISM soft fork with CLTV and CSV if its ready
459 2015-10-01 19:48:40 <Luke-Jr> they seem to me to be on the same timeframe
460 2015-10-01 19:48:45 <morcos> 3 months later release 0.12
461 2015-10-01 19:48:53 <maaku> Luke-Jr: versionbits isn't written. CSV is
462 2015-10-01 19:48:54 <sipa> morcos: sounds good to me
463 2015-10-01 19:48:56 <morcos> 3 month later version bits soft fork
464 2015-10-01 19:48:57 <gmaxwell> maaku: even if that means delaying CLTV 6 months, ... a timespan when almost certantly could have been deployed by itself?
465 2015-10-01 19:49:00 <jgarzik> sequence number stuff + versionbits stuff is a lot to swallow
466 2015-10-01 19:49:10 <jtimon> wumpus: we can always create a minor version 2 weeks before the major release, would that solve your "don't couple softforks with major releases" concerns? if your concern is delaying 0.12, that's not going to happen with btcdrak's plan
467 2015-10-01 19:49:24 <gmaxwell> jgarzik: versionbits will always come along with at least some softfork! :)
468 2015-10-01 19:49:28 <maaku> gmaxwell: where is your 6months coming from?
469 2015-10-01 19:49:32 <jgarzik> true
470 2015-10-01 19:49:39 <morcos> i think the concern of giving people too many upgrades in quick succession is valid
471 2015-10-01 19:49:43 <maaku> I am very, very confused here. Do you think it will take 6mo to merge CSV?
472 2015-10-01 19:49:52 <btcdrak> wumpus: Are the CLTV backports ready to merge?
473 2015-10-01 19:49:53 <maaku> If so let me know so I can stop wasting my time.
474 2015-10-01 19:49:58 <wumpus> my concern is not delaying 0.12, it is just that I prefer to keep them administratively separate, so that people explicitly upgrade for the softfork not for other things
475 2015-10-01 19:49:59 <sipa> maaku: i don't think so
476 2015-10-01 19:50:00 <gmaxwell> maaku: it was a benchamrk number.  CSV is more than six months younger than CLTV.
477 2015-10-01 19:50:08 <wumpus> btcdrak: no, they still need ACKs
478 2015-10-01 19:50:14 <maaku> gmaxwell: that is meaningless
479 2015-10-01 19:50:20 <gmaxwell> maaku: I don't think it will, but lets imagine for a moment that it takes six months to reach the same maturity as CLTV.
480 2015-10-01 19:50:21 <maaku> we had a six month wait for BIP 66
481 2015-10-01 19:50:24 <sipa> i think focus and priorities are much different now, so no, i think CSV can be merged pretty soon
482 2015-10-01 19:50:27 <sipa> (weeks?)
483 2015-10-01 19:50:32 <btcdrak> CSV should not take more than a few weeks more surely?! omg
484 2015-10-01 19:50:48 <BlueMatt> sipa: ack
485 2015-10-01 19:50:48 <gmaxwell> btcdrak: control your behavior.
486 2015-10-01 19:50:51 <wumpus> also the backports?
487 2015-10-01 19:51:14 <btcdrak> gmaxwell :)
488 2015-10-01 19:51:23 <gmaxwell> maaku: people weren't entirely sitting on their hands during that time.
489 2015-10-01 19:51:24 <wumpus> I'm fine with including CSV if it can be done quickly, but it's certainly a risk if hurried too much at once
490 2015-10-01 19:51:31 <jgarzik> +1
491 2015-10-01 19:51:36 <jtimon> no, if CSV is not ready for CLTV_deadline (be it 0.12 or some time sooner), CSV doesn't have to wait for 0.13, that's the decoupling, no need to wait 6 months
492 2015-10-01 19:51:38 <sipa> proposal: we don't gate anything on versionbits, but encourage CSV and related PRs review, for mempool-only; in 3 weeks we see what's ready to go in a softfork?
493 2015-10-01 19:51:41 <BlueMatt> wumpus: ack
494 2015-10-01 19:51:54 <maaku> jtimon: no one is gating anything on 0.12 release
495 2015-10-01 19:51:56 <wumpus> jtimon: no, CSV doesn't have to wait for 0.13, no one is saying that
496 2015-10-01 19:51:59 <jgarzik> IMO redefining behavior in a soft fork carries a higher risk in general (ref: seq numbers)
497 2015-10-01 19:52:05 <BlueMatt> ACK morcos' original proposal
498 2015-10-01 19:52:10 <BlueMatt> end of oct, softfork
499 2015-10-01 19:52:13 <btcdrak> right, softforks are not dependent on major releases
500 2015-10-01 19:52:17 <BlueMatt> if it includes csv, great, otherwise oh well
501 2015-10-01 19:52:28 <sipa> sounds good to me
502 2015-10-01 19:52:33 <btcdrak> BlueMatt, that's a decent compromise
503 2015-10-01 19:52:33 <gmaxwell> maaku: please, I am not saying it will take six months. But I would consider that an upper bound on how long it might take based on the best available reference case we have.  We could have concurrently deployed CLTV and BIP66 and we did not.
504 2015-10-01 19:52:40 <gmaxwell> that sounds okay to me too.
505 2015-10-01 19:52:51 <jtimon> maaku: btcdrak's plan (which several people ACKed) was just like sipa's replacing the 3 week deadline with 0.12 at most as a deadline
506 2015-10-01 19:53:00 <gmaxwell> I just do not want us creating more than the most trivial delays due to tying, as these things tend to creep and snowball.
507 2015-10-01 19:53:01 <wumpus> end of oct sounds fine to me
508 2015-10-01 19:53:03 <maaku> what does CSV need to be merged?
509 2015-10-01 19:53:10 <wumpus> agreed gmaxwell, bad experiences with this
510 2015-10-01 19:53:19 <jgarzik> +1
511 2015-10-01 19:53:26 <jgarzik> +1 end of oct, also
512 2015-10-01 19:53:29 <jtimon> maaku: of course even following that plan, if everything is ready in 3 weeks we can create the minor version
513 2015-10-01 19:53:37 <wumpus> but the backports need to be reviewed as well, so it's not like we can do the release right now immediately, so end of oct is still fine IMO
514 2015-10-01 19:53:42 <jonasschnelli> as far as i know CSV has no test /unit or rpc
515 2015-10-01 19:53:50 <btcdrak> wumpus +1
516 2015-10-01 19:53:53 <maaku> jonasschnelli: it has unit tests
517 2015-10-01 19:53:53 <wumpus> jonasschnelli: it certainly needs that
518 2015-10-01 19:53:59 <btcdrak> Let's move to the next topic of the CSV bips.
519 2015-10-01 19:54:14 <BlueMatt> maaku: so in the next two weeks you hold the ACK-whip
520 2015-10-01 19:54:14 <jgarzik> ### 6 minutes left
521 2015-10-01 19:54:31 <maaku> BlueMatt: I've been doing that the past 2 months to no avail
522 2015-10-01 19:54:34 <BlueMatt> maaku: push for ACKs on this stuff and hopefully it'll be in in two or three weeks and then it gets pushed at the end of oct
523 2015-10-01 19:54:37 <btcdrak> sipabot: next topic please
524 2015-10-01 19:54:49 <sipa> ### ALL GOALS FOR 0.12 RELEASE
525 2015-10-01 19:54:57 <sipa> who added that?
526 2015-10-01 19:55:04 <dstadulis> was hold over from last week
527 2015-10-01 19:55:04 <maaku> I'm am seriously to the point of forking bitcoin out of frustration :\
528 2015-10-01 19:55:11 <dstadulis> should go to libconsensus
529 2015-10-01 19:55:12 <dstadulis> I believ
530 2015-10-01 19:55:22 <btcdrak> I thought next topic was CSV
531 2015-10-01 19:55:31 <sipa> we just talked about CSV...
532 2015-10-01 19:55:35 <wumpus> what is to discuss about CSV?
533 2015-10-01 19:55:47 <wumpus> BIP is finalized?
534 2015-10-01 19:55:52 <wumpus> BIP112 I mean
535 2015-10-01 19:56:36 <dstadulis> ### Proposed Topic: libconsensus merge time window
536 2015-10-01 19:56:39 <btcdrak> BIP112 is final, the wording needs to be tweaked a little
537 2015-10-01 19:56:50 <sipa> ### TOPIC: libconsensus merge time window
538 2015-10-01 19:56:59 <btcdrak> wumpus: just what is remaining for review points to get ACKs
539 2015-10-01 19:57:10 <jgarzik> would like to concentrate refactors into a pre-announced time window if possible
540 2015-10-01 19:57:17 <jgarzik> merge libconsensus moves quickly
541 2015-10-01 19:57:19 <sipa> jgarzik: that sounds good
542 2015-10-01 19:57:28 <sipa> before or after the normal merge window?
543 2015-10-01 19:57:32 <jgarzik> proposed: week before, or after, translation freeze
544 2015-10-01 19:57:41 <jtimon> I strongly oppose to impose such a restriction on anything related to libconsensus
545 2015-10-01 19:57:41 <sipa> as in: right after branching off, or after the feature freeze?
546 2015-10-01 19:57:53 <sipa> jtimon: s/libconsensus/refactors/
547 2015-10-01 19:57:57 <wumpus> move-only refactors would be OK before branch split-off
548 2015-10-01 19:58:05 <jgarzik> sipa, yes
549 2015-10-01 19:58:06 <wumpus> if it's more involved/risky, it needs to be after
550 2015-10-01 19:58:10 <jgarzik> sipa, but I think earlier is better
551 2015-10-01 19:58:21 <jgarzik> more time to rebase and push more important stuff
552 2015-10-01 19:58:27 <Luke-Jr> IMO the restriction shouldn't be on libconsensus, but on code movement/churn
553 2015-10-01 19:58:30 <sipa> Luke-Jr: ack
554 2015-10-01 19:58:32 <jgarzik> Luke-Jr, ack
555 2015-10-01 19:58:36 <jtimon> sipa: yes, there's small refactors (ie removing dependencies like CChainParams from certain functions) that don't need to wait
556 2015-10-01 19:58:47 <jgarzik> Resolved:  s/libconsensus/refactor/
557 2015-10-01 19:58:50 <jgarzik> continue
558 2015-10-01 19:58:52 <wumpus> not going to merge anything with significant risk just before a release
559 2015-10-01 19:59:11 <btcdrak> wumpus: the time to merge something like that is at the beginning of a release cycle
560 2015-10-01 19:59:14 <jgarzik> wumpus, thus my proposal for around the first freeze (string freeze)
561 2015-10-01 19:59:18 <btcdrak> not at the end
562 2015-10-01 19:59:20 <wumpus> (which any serious refactoring to consensus code would be)
563 2015-10-01 19:59:23 <morcos> btcdrak: +1
564 2015-10-01 19:59:41 <jgarzik> btcdrak, That's what Linux kernel does -- 0.1 seconds after a release version is posted to the Internet, merge window opens.
565 2015-10-01 19:59:44 <jgarzik> including refactors
566 2015-10-01 19:59:47 <sipa> so there is also a change (more than just code movement) that i'd like to do: create a CBlockDB, which manages all mapBlockIndex/CBlockIndex/file location stuff, without a private lock, so that doesn't need cs_main anymore
567 2015-10-01 19:59:48 <wumpus> right, the beginning for 0.13, after 0.12 is split off is the time for more impactful changes
568 2015-10-01 20:00:00 <jtimon> I'm fine with having a period for medium/big moveonlies, but remember last time we did that was for "right after major versions", we passed through "right after 0.10" and "right after 0.11" and nothing happened
569 2015-10-01 20:00:04 <btcdrak> jgarzik +1
570 2015-10-01 20:00:26 <jgarzik> jtimon, it's up to you to follow that process as a peer developer :)
571 2015-10-01 20:00:30 <sipa> jtimon: things do need to be ready and reviewed at that point
572 2015-10-01 20:00:32 <jgarzik> self-enforcement :)
573 2015-10-01 20:00:32 <wumpus> jtimon: that's because your changes didn't have enough review/agreement
574 2015-10-01 20:00:52 <morcos> does it make sense to do it right after 0.12 splits off or right after 0.12 is released.  it seems like after the split there are still a lot of backports, and maybe we shouldn't make those harder
575 2015-10-01 20:01:02 <jtimon> wumpus: I disagree, for people rebasing on top of major versions it's much easier if big refactors happen right before major releases rather than right after them
576 2015-10-01 20:01:10 <jgarzik> right after a branch can be a more painful than right after a full release.
577 2015-10-01 20:01:26 <wumpus> jtimon: AS I said, that's OK for just move-only stuff, but not for anything that incurs risk@
578 2015-10-01 20:01:29 <jgarzik> last minute bug fixes become more onerous to backport.
579 2015-10-01 20:01:44 <jtimon> jgarzik: I was nagging almost everyone with that clearly uneffective ACK-whip, next time I will make sure I nagg you as well
580 2015-10-01 20:01:46 <CodeShark> if we were to just break up main.cpp (consensus-focused or not) it would go to great lengths to reducing rebase headaches ;)
581 2015-10-01 20:02:01 <jgarzik> jtimon, I think it is moving in the right direction now - your issue PR
582 2015-10-01 20:02:09 <sipa> CodeShark: no disagreement there, but not relevant
583 2015-10-01 20:02:12 <jgarzik> the code movement can be created from scratch in minutes.
584 2015-10-01 20:02:20 <CodeShark> sipa: how is that not relevant?
585 2015-10-01 20:02:39 <wumpus> specific strategy on splitting up main is too detailed for the current discussion
586 2015-10-01 20:02:48 <jtimon> jgarzik: I thought it was moving in the right direction when people finally decided to review some libconsensus-related code
587 2015-10-01 20:02:49 <sipa> CodeShark: if there was no cost to that refactor is would happen instantly
588 2015-10-01 20:03:05 <jgarzik> ### meeting T+3 minutes overtime
589 2015-10-01 20:03:26 <jtimon> jgarzik: I'll provide more documentation, but I'm increasingly pesimistic about libconsensus being ever completed
590 2015-10-01 20:03:27 <sipa> meeting over?
591 2015-10-01 20:03:31 <jgarzik> Proposed:  last week of October for refactor window.
592 2015-10-01 20:03:43 <sipa> what is the timeframe for 0.12?
593 2015-10-01 20:03:51 <jgarzik> feb 2016
594 2015-10-01 20:04:03 <wumpus> the branch split-off is only december 1 that I remember
595 2015-10-01 20:04:05 <jtimon> as said I certainly prefer before-major than after-major for big refactors
596 2015-10-01 20:04:14 <jtimon> but wumpus disagrees
597 2015-10-01 20:04:31 <jgarzik> Nov 1 close i18n
598 2015-10-01 20:04:34 <jgarzik> Dec 1 feature freeze
599 2015-10-01 20:04:36 <gmaxwell> We are over time.
600 2015-10-01 20:04:42 <sipa> ### MEETING OVER
601 2015-10-01 20:04:48 <wumpus> yes, I disagree. I don't want to merge anything big just before a release. Very bad experiences with that.
602 2015-10-01 20:04:53 <sipa> (but keep talking...)
603 2015-10-01 20:05:08 <bsm1175321> Great meeting folks. I hope I can offer more next time.
604 2015-10-01 20:05:10 <jtimon> well the "big" part are moveonly commits
605 2015-10-01 20:05:16 <jgarzik> Thus proposed refactor window ~Oct 23 is well before Feb 1 release.
606 2015-10-01 20:05:27 <jgarzik> then next refactor window, proposed: Feb 2
607 2015-10-01 20:05:34 <wumpus> Im ean before *splitoff* not release so much
608 2015-10-01 20:05:58 <jtimon> jgarzik: I prefer that over wumpus alternative
609 2015-10-01 20:05:59 <wumpus> after the split off, development for 0.12 will go on a branch anyway, so changes on master can be more impactful, lots of time to test etc
610 2015-10-01 20:06:28 <jgarzik> having refactor changes in 0.12 will mitigate later pain
611 2015-10-01 20:06:45 <dstadulis> Please amend google doc so I can send minutes out to list https://docs.google.com/document/d/10yfs6_DeF1CnrkpyubNCpIVVPSqBE-tV_KebsI_O41A/edit?usp=sharing
612 2015-10-01 20:06:55 <jtimon> wumpus: exactly the opposite! you don't want the 2 branches to differ a lot!!
613 2015-10-01 20:07:04 <jgarzik> +1
614 2015-10-01 20:07:22 <wumpus> splitoff of 0.12 branch is scheduled for 2016-01-06
615 2015-10-01 20:07:36 <wumpus> http://lists.linuxfoundation.org/pipermail/bitcoin-dev/2015-September/011182.html
616 2015-10-01 20:07:38 <jtimon> after the major release is frozen, then it's fine to have big differences again
617 2015-10-01 20:07:44 <jgarzik> no need to wait until Jan 2016 for libconsensus moves
618 2015-10-01 20:07:53 <jgarzik> late Oct is fine
619 2015-10-01 20:08:07 <jgarzik> keep the pace up
620 2015-10-01 20:08:14 <jgarzik> keep the diffs small
621 2015-10-01 20:08:19 <wumpus> as long as it is move-only.. I'm really scared of introducing consensus bugs
622 2015-10-01 20:08:27 <jtimon> specially in the context of having waited for jan 2015 before and then not having moved anything...
623 2015-10-01 20:08:37 <wumpus> they shouldn't just make it into a major release
624 2015-10-01 20:08:42 <jonasschnelli> maaku: CSV has tests. My mystake. Sorry. Although a RPC test like CLTV provides would be nice (and would allow better testing [by adapting the rpc test] and quicker ACKing)
625 2015-10-01 20:08:58 <CodeShark> let's move everything before 0.12 and make all changes afterwards
626 2015-10-01 20:08:59 <jgarzik> late-Oct merge gives you months of testing for libconsensus code moves
627 2015-10-01 20:09:02 <jgarzik> which are easy to verify
628 2015-10-01 20:09:03 <jtimon> wumpus: that's why we have release candidates, no?
629 2015-10-01 20:09:03 <sipa> jtimon: i think having this meeting helps... it makes it easier to reviewers to know whether other reviewers are on-board with a goal
630 2015-10-01 20:09:09 <wumpus> ok, never mind
631 2015-10-01 20:09:10 <jgarzik> jtimon, +1
632 2015-10-01 20:09:13 <wumpus> do it your way,I'll shut up
633 2015-10-01 20:09:41 <wumpus> jtimon: there's no guarantee at all that subtle problems in refactoring will be found in a RC!
634 2015-10-01 20:09:52 <sipa> let's not be overly pessimistic
635 2015-10-01 20:10:01 <CodeShark> the sooner we do this the less we'll have to worry about it ;)
636 2015-10-01 20:10:01 <jgarzik> The Internet is a great test lab
637 2015-10-01 20:10:02 <sipa> the majority of jtimon's changes _are_ moveonly
638 2015-10-01 20:10:05 <jgarzik> nod
639 2015-10-01 20:10:07 <wumpus> I'm very much against doing this before the splitoff
640 2015-10-01 20:10:18 <sipa> i think we just need to agree on where we want the refactors to go
641 2015-10-01 20:10:20 <CodeShark> move-only is easy to review
642 2015-10-01 20:10:23 <jtimon> wumpus: there's no guarantee they will be found no matter which month of the year you chose to merge them
643 2015-10-01 20:10:25 <jgarzik> I'm with jtimon on this one
644 2015-10-01 20:10:28 <sipa> a few people to look at the move-only changes
645 2015-10-01 20:10:29 <wumpus> move-only is, but they're not move-only
646 2015-10-01 20:10:38 <wumpus> not just move-only I mean
647 2015-10-01 20:10:46 <sipa> i think we can manage
648 2015-10-01 20:11:05 <sipa> no opinion on when, but i do like the idea of a dedicated refactoring space in the release schedule
649 2015-10-01 20:11:15 <jtimon> wumpus: I have tried everything I believe, remove most dependencies first then moveonly, moveonly first then remove most dependencies...nothing seemed to be acceptable so far...
650 2015-10-01 20:11:26 <wumpus> yes I like the principle too, but I don't like doing it just before a release splitoff...
651 2015-10-01 20:11:29 <sipa> jtimon: you were just spamming people with code...
652 2015-10-01 20:11:39 <jgarzik> _months_ before a release splitoff
653 2015-10-01 20:11:43 <jgarzik> which is basically dev
654 2015-10-01 20:11:50 <jgarzik> we need a window before 0.12
655 2015-10-01 20:12:05 <CodeShark> go easy, sipa ;)
656 2015-10-01 20:12:05 <jgarzik> there will be other refactors we want to do
657 2015-10-01 20:12:05 <morcos> jtimon: i noticed you said you still owe the high level doc.  i'm waiting for that.
658 2015-10-01 20:12:15 <jtimon> if you want pure moveonly, I can rewirte that in half an hour at  any point in time when it's going to be reviewed and potentially merged, no problem
659 2015-10-01 20:12:31 <bsm1175321> Is there a dedicated window where the 0.12rc becomes the testnet code?  That rc moves to fixes-only?
660 2015-10-01 20:12:39 <jgarzik> bsm117532: yes
661 2015-10-01 20:12:46 <bsm1175321> What is it, where is it posted?
662 2015-10-01 20:12:47 <jgarzik> bsm117532: branch is fixes only
663 2015-10-01 20:12:51 <jgarzik> bsm117532: mailing list
664 2015-10-01 20:13:04 <morcos> jgarzik: HA! half of the pulls are "fixes"
665 2015-10-01 20:13:16 <sipa> jtimon: sorry, that last thing i said was unnecessary
666 2015-10-01 20:13:25 <jtimon> morcos: well, I believe I explained most of what the doc is going to contain in our conversation after the meeting last week, but you asked for pictures...
667 2015-10-01 20:13:37 <jgarzik> morcos, :)
668 2015-10-01 20:14:19 <jtimon> sipa: it's fine, I was in fact spamming people with code hoping that they would give ANY kind of feedback
669 2015-10-01 20:14:22 <jgarzik> post-branch changes permitted tend to be: docs, obvious bug fixes, oh sh*t network problems.  More involved bug fixes are a years-long project :)
670 2015-10-01 20:14:22 <morcos> jtimon: i know i'm newer to the code than other people, but it still not clear to me what the end goal (in terms of code movement / class structure) is that i shoudl be evaluating your pulls on
671 2015-10-01 20:14:45 <morcos> so i can evaluate them for correctness, but i feel like i might be wasting my time if other people might not approve them for direction reasons
672 2015-10-01 20:15:28 <CodeShark> I think we have several somewhat conflated (but at the same time complementary) goals here
673 2015-10-01 20:15:31 <morcos> i'd like to see a high level doc (in an issue or something) with a bunch of ACK's
674 2015-10-01 20:15:39 <jgarzik> morcos, jtimon opened an issue
675 2015-10-01 20:15:46 <morcos> i know, thats what im referring to
676 2015-10-01 20:16:00 <wumpus> isn't that https://github.com/bitcoin/bitcoin/issues/6714 ?
677 2015-10-01 20:16:01 <jtimon> well, the issue now includes an approximate API, everything can be explained just looking at that list: you need to build stuff independently from main.cpp, chainparams.cpp, util.cpp, etc
678 2015-10-01 20:16:02 <morcos> its progress, but its still not enough
679 2015-10-01 20:16:15 <Luke-Jr> can we teach a non-developer how to safely evaluate move-only commits, so we don't need to do it?
680 2015-10-01 20:16:24 <sipa> jtimon: i think you're misunderstanding what morcos is asking for (morcos, correct me if i'm wrong)
681 2015-10-01 20:16:58 <jtimon> morcos: what I was told was missing last time, is a list of files, a description of the final desired directory structure and things like that
682 2015-10-01 20:17:00 <sipa> jtimon: i think he wants to see a description of what the responsibility of what directory, what file, is, how they are connected and depend on each other, ...
683 2015-10-01 20:17:10 <morcos> yes and yes
684 2015-10-01 20:17:18 <sipa> not API
685 2015-10-01 20:17:21 <jtimon> if you think there's more missing things for it to "be enough", please tell them to me
686 2015-10-01 20:17:32 <morcos> not complainign about the API, i liek that that is there as part of the issue as well
687 2015-10-01 20:17:53 <wumpus> Luke-Jr: probably easier to write a script :p
688 2015-10-01 20:18:26 <CodeShark> movebot
689 2015-10-01 20:18:31 <sipa> jtimon: i still don't see anything about that
690 2015-10-01 20:18:33 <wumpus> but I don't think evaluating move-only is the bottleneck, move-only is super easy to review manually as well
691 2015-10-01 20:18:35 <jtimon> I think I undesrtood last time, but you guys will have a chance to ask for more things in the doc
692 2015-10-01 20:18:48 <morcos> jtimon: his last comment 3 hours ago said he still owed the pictures doc
693 2015-10-01 20:18:49 <sipa> jtimon: apart from "Separate consensus critical code into different files"
694 2015-10-01 20:18:53 <morcos> sipa i mena
695 2015-10-01 20:18:56 <jtimon> sipa: it will be a different document with pictures
696 2015-10-01 20:18:58 <morcos> i can't type
697 2015-10-01 20:19:13 <sipa> ok!
698 2015-10-01 20:19:21 <wumpus> once we agree where what things should be moved
699 2015-10-01 20:19:26 <jgarzik> +1 on seeing directories/files
700 2015-10-01 20:19:29 <jtimon> wumpus: Luke-Jr: that script seems harder to write than the identical-build script...
701 2015-10-01 20:19:43 <wumpus> I have an identical-build script
702 2015-10-01 20:19:49 <morcos> yes so jtimon, once you have that doc (and people think its enough info for them) then we need ACK's on it!
703 2015-10-01 20:19:50 <wumpus> but if you move code around, it's not identiacl
704 2015-10-01 20:19:53 <sipa> most code movement doesn't result in identical builds
705 2015-10-01 20:20:17 <wumpus> it ends up in different plaes in the excutable, pointers will be different, etc
706 2015-10-01 20:20:42 <CodeShark> even if we were to break up main.cpp into files shorter than n lines and include them all in a single .cpp it would make rebasing less painful (although it would be a bit ugly) :p
707 2015-10-01 20:20:59 <sipa> CodeShark: i don't believe that
708 2015-10-01 20:21:00 <jtimon> pesonally I read a moveonly git diff better than a document describing those movements, but if people want that, I will give them that
709 2015-10-01 20:21:08 <sipa> CodeShark: the hard part is disentangling things
710 2015-10-01 20:21:19 <Luke-Jr> hmm, once upon a time I wrote a script that moved code from one big .c file to many smaller ones, that was intended to avoid breaking third-party patches :P
711 2015-10-01 20:21:27 <wumpus> jtimon: what people would like is higher-level information, why what moves where
712 2015-10-01 20:21:28 <sipa> CodeShark: whether they move to different files or not is not relevant; git's refactoring is good enough to combine changes
713 2015-10-01 20:22:02 <dstadulis> Nice meeting all, will send out minutes later
714 2015-10-01 20:22:02 <jgarzik> sipa, actually it is -- when the diff is smaller and lower percentage chance of conflict, there is increased productivity
715 2015-10-01 20:22:11 <wumpus> thanks dstadulis
716 2015-10-01 20:22:33 <sipa> jgarzik: i mean that one large file that is well organized, or many small files... makes no difference for how much conflict there is
717 2015-10-01 20:23:01 <jtimon> sipa: identical builds wouldn't be for moveonly commits but mainly to break montrous giant switchs that any checkstyle tool would complain about (see #4646 and #5153 )
718 2015-10-01 20:23:02 <sipa> so the question is about turning an entangled mess into something well organized... whether that's in one file or not
719 2015-10-01 20:23:17 <jgarzik> sipa, it does, for subtle reasons - developers will iterate files at different rates
720 2015-10-01 20:23:24 <sipa> jgarzik: perhaps yes
721 2015-10-01 20:23:29 <wumpus> if you divide up the file randomly (as CodeShark proposes jokingly), any change will probably affect not one file but many of them
722 2015-10-01 20:23:34 <wumpus> it doesn't help localize changes
723 2015-10-01 20:23:44 <sipa> wumpus words what i wanted to say better :)
724 2015-10-01 20:23:59 <CodeShark> yes, I said it somewhat facetiously
725 2015-10-01 20:24:00 <jgarzik> the initial chunky division into separate files will not be random but purposeful and shapes future per-file iteration speed.
726 2015-10-01 20:24:28 <CodeShark> of course the division shouldn't be random
727 2015-10-01 20:24:35 <wumpus> right
728 2015-10-01 20:24:37 <CodeShark> but it doesn't have to be very fine-grained either
729 2015-10-01 20:24:52 <jgarzik> the initial division should aim for minimizing future cross-file movement.
730 2015-10-01 20:24:57 <wumpus> the problem is, as I remember, is avoiding circular dependencies
731 2015-10-01 20:25:07 <jgarzik> yep
732 2015-10-01 20:25:34 <jtimon> mhmm, I have an idea for a logical division, why don't we start moving consensus and policy code out of main.cpp ?
733 2015-10-01 20:25:39 <jtimon> oh, never mind...
734 2015-10-01 20:25:39 <wumpus> main.cpp is a web of functions calling each other, if you just move a few, you'll have two compilation units that dpend on each other
735 2015-10-01 20:25:55 <Luke-Jr> jtimon: did you make that map suggested at the last meeting?
736 2015-10-01 20:26:01 <Luke-Jr> (if so, I missed it, please link)
737 2015-10-01 20:26:08 <wumpus> but anyhow if the goal is move-only changes that may not be that bad
738 2015-10-01 20:26:21 <wumpus> if it can be disentangled later
739 2015-10-01 20:26:24 <jtimon> What map, a graph of calls?
740 2015-10-01 20:26:47 <jtimon> Luke-Jr: ^
741 2015-10-01 20:27:18 <Luke-Jr> jtimon: no.. a proposed directory layout, how they relate to each other, what code each file contains, etc
742 2015-10-01 20:27:32 <sipa> Luke-Jr: we were just talking about that
743 2015-10-01 20:27:33 <jtimon> another goal IMO is breaking giant functions into smaller ones (like main::ProcessMessage() )
744 2015-10-01 20:27:50 <Luke-Jr> jtimon: do that separately.
745 2015-10-01 20:27:52 <jtimon> Luke-Jr: not yet as we we're commenting right now
746 2015-10-01 20:28:01 <CodeShark> ProcessMessage should be replaced by a signaling system...but that's a separate issue
747 2015-10-01 20:28:12 <wumpus> yes, that's a separate issue, that's far from move-only-land
748 2015-10-01 20:28:29 <morcos> Hmm
749 2015-10-01 20:28:34 <morcos> so here is where there is some confusion
750 2015-10-01 20:28:44 <morcos> I think all of these thigns should be part of the master plan
751 2015-10-01 20:28:51 <morcos> in one location
752 2015-10-01 20:29:01 <wumpus> yes
753 2015-10-01 20:29:06 <morcos> these are the goals we'd like to move the code archticture in
754 2015-10-01 20:29:13 <morcos> consensus API looks like this
755 2015-10-01 20:29:17 <wumpus> that's true
756 2015-10-01 20:29:20 <jtimon> Luke-Jr: about the call graphs, I some times do a small part of it manually ofr things that interest me, but there's proper tools to do so (I believe it was gwillen who showed me the resulting graph for main...)
757 2015-10-01 20:29:21 <morcos> file class layout liek this
758 2015-10-01 20:29:31 <wumpus> jtimon: he's not asking for a call graph
759 2015-10-01 20:29:37 <morcos> ok ...
760 2015-10-01 20:29:56 <CodeShark> for a call graph use doxygen...but that's not what we're after
761 2015-10-01 20:29:57 <wumpus> call graphs can be generated, I think they are even in my generated doxygen documentation, but they're not really informative
762 2015-10-01 20:30:03 <jtimon> wumpus: he said "do that separately"
763 2015-10-01 20:30:21 <wumpus> jtimon: he was talkling about splitting up PRocessMessage not callgraphs
764 2015-10-01 20:30:39 <jgarzik> the blocker on ProcessMessage is someone writing hooks
765 2015-10-01 20:31:06 <wumpus> which should be in the master plan, but is not the first step (as it's more involved and not move-only)
766 2015-10-01 20:31:10 <phantomcircuit> ProcessMessages is certainly in need of a refactor
767 2015-10-01 20:31:12 <jtimon> Dividing ProcessMessage in many separate functions (maybe inline at first for the identical-build check) would help with any later work related to that
768 2015-10-01 20:31:21 <jgarzik> wumpus, move only PR was rejected...
769 2015-10-01 20:31:26 <sipa> jgarzik: i had a plan for doing that, but i don't want to interfere with cfields's libevent refactor there (which i think will include that)
770 2015-10-01 20:31:38 <phantomcircuit> the current version has a while loop that executes exactly once
771 2015-10-01 20:31:42 <phantomcircuit> (my fault)
772 2015-10-01 20:31:48 <wumpus> yes, we shouldn't trample on cfields's work
773 2015-10-01 20:32:00 <wumpus> which involves the network code
774 2015-10-01 20:32:15 <wumpus> though I"m not sure in how much it involves the high-level part
775 2015-10-01 20:32:45 <wumpus> that'd be more useful to discuss with him here
776 2015-10-01 20:33:12 <sipa> if cfields' project would take too long, i think the solution would be to move individual pieces of logic handling out, one by one (for example the askfor handing, pingpong handing, tx propagation, block propagation)
777 2015-10-01 20:33:35 <CodeShark> I've long been proposing something along these lines: https://github.com/ciphrex/mSIGNA/blob/master/deps/CoinQ/src/CoinQ_peer_io.h
778 2015-10-01 20:33:39 <jgarzik> Move message processing to new 'procmsg' module. #4646 https://github.com/bitcoin/bitcoin/pull/4646
779 2015-10-01 20:33:57 <sipa> jgarzik: i'm still neutral on that
780 2015-10-01 20:34:46 <jtimon> jgarzik: I think you could have done it with an identical-build commit if you had separate it first in inline functions before moving anything out of main.cpp
781 2015-10-01 20:34:55 <sipa> i think the temporary state it creates is worse than what we have now (something circularly dependent on main and everything else), and the solution to that is more moving, until procmsg is empty again
782 2015-10-01 20:35:03 <wumpus> inline functions won't usually give you identical builds
783 2015-10-01 20:35:23 <wumpus> you underestimate the fragility of compiler results :)
784 2015-10-01 20:35:41 <jgarzik> Where should the ProcessMessage() code live - in which files? msg-inv.cc, msg-block.cc, ?
785 2015-10-01 20:35:41 <jtimon> jgarzik: and doing that (breaking up that function but not moving anything) doesn't suffer from sipa's circular dependencies concerns
786 2015-10-01 20:36:08 <sipa> jgarzik: imho, in handler_tx_propagation, handling_block_propagation, handling_ibd, handling_pingpong, ...
787 2015-10-01 20:36:15 <jgarzik> filenames, please
788 2015-10-01 20:36:20 <sipa> add .cpp
789 2015-10-01 20:36:21 <sipa> :)
790 2015-10-01 20:36:38 <sipa> jgarzik: not per message - sometimes one messages has effects on multiple modules
791 2015-10-01 20:36:42 <jgarzik> *boggle*
792 2015-10-01 20:36:55 <jtimon> as said we don't need to know where to move it for breaking up the function...
793 2015-10-01 20:36:55 <wumpus> not per message, but per concern
794 2015-10-01 20:37:31 <sipa> jgarzik: what i envision is some hook mechanism, where you can install zero or more signal handlers for each message
795 2015-10-01 20:37:46 <jgarzik> sure.  I envision filenames shorter than 32 chars
796 2015-10-01 20:37:58 <sipa> remove the handle_ prefix :)
797 2015-10-01 20:38:14 <wumpus> change handler to a directory name
798 2015-10-01 20:38:19 <jtimon> then we can start separating by concern (in fact the parameters of the newly created functions will probably give hints)
799 2015-10-01 20:38:20 <CodeShark> that's exactly what I was proposing, sipa
800 2015-10-01 20:38:39 <sipa> jgarzik: and for example block propagation needs a global view... it needs to prevent downloading from multiple peers at once, so it needs some state lock that is shared across peers
801 2015-10-01 20:38:47 <jtimon> we can have a p2p or messages directory...
802 2015-10-01 20:39:12 <sipa> jgarzik: but that lock should not be cs_main, and when everything related to that concern is in one place, it's much easier to see what needs access to the same shared data
803 2015-10-01 20:39:26 <wumpus> +1 sipa
804 2015-10-01 20:39:27 <sipa> but pingpong handling has no shared state at all
805 2015-10-01 20:39:39 <sipa> so it could run concurrently for multiple peers
806 2015-10-01 20:39:44 <jgarzik> sure.  cs_main is the Big Kernel Lock of old-BSD and old-Linux and needs to go in general.
807 2015-10-01 20:39:47 <jtimon> of course removing the use of globals would make those hints more valuable...
808 2015-10-01 20:39:54 <CodeShark> the architecture I had in mind: peer_io (handles connection to a single peer, provides message subscription) -> peer_manager (creates multiple peer connections, queues messages) -> sync_manager (does synchronization tasks)
809 2015-10-01 20:40:12 <sipa> jgarzik: absolutely!
810 2015-10-01 20:40:42 <sipa> i'd like to begin by encapsulating mapBlockIndex into a separate class with its own lock, that manages block references and information about them
811 2015-10-01 20:40:45 <jgarzik> peel back the locks just like was done for RPC.
812 2015-10-01 20:40:53 <CodeShark> sipa: yes!
813 2015-10-01 20:40:57 <CodeShark> let's please do that
814 2015-10-01 20:41:03 <sipa> i'm afraid however that it will break every other patch...
815 2015-10-01 20:41:07 <jgarzik> sipa, +1
816 2015-10-01 20:41:17 <jgarzik> sipa, do it in late October :)
817 2015-10-01 20:41:27 <jtimon> well, you can "begin" with that while other people "begin" with other parts...
818 2015-10-01 20:41:40 <sipa> jtimon: of course!
819 2015-10-01 20:41:47 <sipa> jtimon: i don't want to conflict with your changes
820 2015-10-01 20:43:12 <jtimon> never mind, I don't mean my changes, it's just something I don't like "let's block anything mempool until X or Y is merged, even if it's code that both X and Y share"...like in the mempool case...
821 2015-10-01 20:43:20 <jgarzik> IMO that's the point of a refactor window - it is _ok_ to conflict with your changes, that's the week to resolve and order and merge things like that.
822 2015-10-01 20:43:33 <jgarzik> Obviously you prioritize user impactful features above refactors, and don't block those.
823 2015-10-01 20:43:51 <sipa> jtimon: well imho mempool limiting is way higher priority than any refactoring
824 2015-10-01 20:43:52 <jgarzik> it's a zen balance of devs ;p
825 2015-10-01 20:44:03 <jgarzik> +1
826 2015-10-01 20:44:05 <sipa> jtimon: it's a security concern of the network
827 2015-10-01 20:44:25 <wumpus> yes, mempool limiting is absolutely highest priority now
828 2015-10-01 20:44:32 <jtimon> sipa: agree, but some refactors were helpful for mempool limiting development and review!
829 2015-10-01 20:45:21 <wumpus> (and CLTV/CSV, if we want them ready for end october)
830 2015-10-01 20:45:49 <CodeShark> the block index refactor will greatly help version bits
831 2015-10-01 20:45:58 <sipa> CodeShark: don't assume that
832 2015-10-01 20:45:59 <jtimon> anyway, never mind, just "i'd like to begin by encapsulating mapBlockInde..." reminded me of you doing that kind of "blocking"
833 2015-10-01 20:46:12 <CodeShark> sipa: why not?
834 2015-10-01 20:46:13 <sipa> jtimon: by "begin" i mean, "begin of reducing locks":
835 2015-10-01 20:46:29 <sipa> CodeShark: because versionbits will need to be backported to code that doesn't have that refactor, if it happens at all
836 2015-10-01 20:46:39 <sipa> jtimon: as i said, i don't want to block any existing work here
837 2015-10-01 20:46:41 <jtimon> exactly, you can begin with that while other people begin reducing locks in other parts was my point
838 2015-10-01 20:46:57 <CodeShark> sipa: I wasn't talking in a dependency sense - I was talking in a code cleanliness sense (ultimately)
839 2015-10-01 20:47:10 <sipa> CodeShark: well you'll need to write versionbits without it!
840 2015-10-01 20:47:15 <CodeShark> yes, absolutely
841 2015-10-01 20:47:20 <sipa> so it won't "help" at all
842 2015-10-01 20:47:33 <CodeShark> it won't help writing it - it will help it be better implemented (ultimately)
843 2015-10-01 20:47:53 <CodeShark> yes, it will require backporting
844 2015-10-01 20:48:04 <CodeShark> but it's the more pleasant type of backporting, as backporting goes
845 2015-10-01 20:48:17 <sipa> jtimon: likewise!
846 2015-10-01 20:48:23 <sipa> CodeShark: i think you're confused
847 2015-10-01 20:48:28 <sipa> CodeShark: i'm talking about a refactor
848 2015-10-01 20:48:34 <sipa> it won't make anything "better", just cleaner
849 2015-10-01 20:48:46 <CodeShark> "better" depends on the criteria you use
850 2015-10-01 20:48:53 <jtimon> isn't cleaner better?
851 2015-10-01 20:48:57 <CodeShark> if one of your criteria is "clean" then yes, better -> cleaner
852 2015-10-01 20:48:58 <sipa> and you need to make it work now, not with a mythical refactor that doesn't exist
853 2015-10-01 20:49:16 <CodeShark> sipa, read what I wrote
854 2015-10-01 20:49:16 <sipa> CodeShark: i'm challenging that it "helps" with anything related to versionbits
855 2015-10-01 20:50:57 <sipa> CodeShark: i don't understand, but i don't think it matters :)
856 2015-10-01 20:51:43 <CodeShark> sipa: I might need to provide more background for it to make sense...but we'll do that later ;)
857 2015-10-01 20:53:13 <CodeShark> why so many lightningbots?!?
858 2015-10-01 20:54:00 <sipa> i wonder the same thing
859 2015-10-01 20:54:08 <Luke-Jr> /mode #bitcoin-dev +b ⁇⁇⁇⁇?bot
860 2015-10-01 20:54:12 <Luke-Jr> err, without the Unicode..
861 2015-10-01 20:54:22 <Luke-Jr> /mode #bitcoin-dev +b ?????????bot
862 2015-10-01 20:54:47 <Luke-Jr> maybe a * at the end
863 2015-10-01 20:57:28 <Luke-Jr> please?
864 2015-10-01 20:58:06 <Luke-Jr> also, thoughts on Travis building with -Werror ? :p
865 2015-10-01 20:58:22 <Luke-Jr> more importantly: why does master not build? :/
866 2015-10-01 21:00:47 <phantomcircuit> wumpus, ^ yup doesn't build
867 2015-10-01 21:04:53 <phantomcircuit> huh
868 2015-10-01 21:04:57 <phantomcircuit> Luke-Jr, you sure?
869 2015-10-01 21:05:13 <Luke-Jr> phantomcircuit: looks like the autotools logic to re-run configure is just broken
870 2015-10-01 21:05:19 <Luke-Jr> manually re-configuring fixed it
871 2015-10-01 21:12:21 <wumpus> master builds fine, however you need to re-autoconf and configure after #6637
872 2015-10-01 21:13:48 <phantomcircuit> ah im on the wrong branch :)
873 2015-10-01 21:17:21 <Luke-Jr> wumpus: autotools is *supposed* to automatically take care of that when you run make FWIW ;)
874 2015-10-01 21:17:31 <Luke-Jr> but I'll nag cfields on that
875 2015-10-01 21:36:07 <wumpus> for most changes it does, but for big impactful changes to the build system it doesn't
876 2015-10-01 21:36:27 <wumpus> and moving univalue to a subtree is quite impactful as these things go
877 2015-10-01 22:52:52 <XexorZ> Anyone here really understand the inner workings of the blockchain?  I've a likely hairbrained idea I'd like to discuss if you're bored and need a laugh.
878 2015-10-01 22:55:40 <Luke-Jr> wumpus: ping
879 2015-10-01 22:56:00 <Luke-Jr> wumpus: I am trying to address https://github.com/bitcoin/bitcoin/pull/6349#discussion-diff-35016896 , but I don't see a way to do it.
880 2015-10-01 22:56:25 <Luke-Jr> GCC doesn't like simply moving it to util.cpp (even removing static and adding an extern in the header)
881 2015-10-01 22:56:34 <XexorZ> So the blockchain is getting massive.  I was thinking, why not just consolodate all current up to the last 100 or so blocks into some sort of prefix-index at an agreed upon time...  All sum-zero transactions would go away.  Tons of other stuff would go away.  I think things would get mighty small then... no?  Likely totally worthless idea but I thought I'd ask folks who had a clue.
882 2015-10-01 22:57:53 <deego> XexorZ: iiuc, that's the first thing anyone proposes. :) But, that's about all i know
883 2015-10-01 22:57:54 <Luke-Jr> XexorZ: if you want to trust some centralised group, that works
884 2015-10-01 22:58:32 <XexorZ> Luke-Jr: no need to trust a centralized group, the consolidation index can be hashed too, no?
885 2015-10-01 22:58:44 <Luke-Jr> XexorZ: if you want to make it a regular thing as part of blocks using the consensus system to keep it secure (ie, make the trusted group "all full nodes"), then it's complicated to implement
886 2015-10-01 22:58:56 <Luke-Jr> XexorZ: hashing it doesn't prove it is correct
887 2015-10-01 22:59:54 <XexorZ> Luke-Jr  would be a consensus of all full nodes.  Would hapen at pre-determined period.  Last "N" number of blocks would remain so current transactions could continue (I think).  Of course I'm speaking out of my arse - I know very little about all of this and appreciate your thoughts on the same.
888 2015-10-01 23:00:09 <XexorZ> Luke-Jr Right, but consensus does prove it correct, no?
889 2015-10-01 23:00:28 <Luke-Jr> XexorZ: no, but it proves a large number of nodes checked it
890 2015-10-01 23:00:41 <Luke-Jr> I don't think delaying it in groups of N blocks makes it any less complicated
891 2015-10-01 23:00:47 <Luke-Jr> might actually make it more complicated
892 2015-10-01 23:01:05 <XexorZ> I figured it would allow for current transactions to complete -  perhaps that's foolish.
893 2015-10-01 23:01:22 <Luke-Jr> XexorZ: well, a one block delay would do that
894 2015-10-01 23:01:45 <Luke-Jr> but it's undetermined if that's a good idea or not
895 2015-10-01 23:02:05 <XexorZ> Oh? I had no idea.  I thought some low-urgency transactions stalled for hours or days
896 2015-10-01 23:04:38 <XexorZ> Anywho thanks for your thoughts on the idea.  I wanted to get it out before I lost it in case it had some sort of use.
897 2015-10-01 23:06:26 <maaku> XexorZ: it's been proposed. it is strictly speaking a break from bitcoin's security model
898 2015-10-01 23:06:49 <maaku> but a variant of that idea is what we'll probably end up with once the block chain gets unreasonably large (fetch utxo set a year back and validate from there)
899 2015-10-01 23:07:19 <Luke-Jr> (opt-in of course)
900 2015-10-01 23:07:25 <XexorZ> (y) Thank you Maaku.  I assumed someone else thought it up already as it seems so simple to me, but again, I don't grasp 1/100th the details of how things work in the blockchain.
901 2015-10-01 23:08:01 <berndj> XexorZ, most recently that i'm aware of: http://lists.linuxfoundation.org/pipermail/bitcoin-dev/2015-September/011065.html
902 2015-10-01 23:10:51 <XexorZ> Thank you Berndj, that sounds about what I'm talking about if I understand what is being said there correctly.
903 2015-10-01 23:14:32 <maaku> XexorZ: there's a number of reasons it's good to commit to the validaiton state, e.g. the utxo set. and once you do that something like this becomes possible, whether it is a good idea or not..