1 2015-10-22 01:16:01 <rusty> kanzure: you have GPG?
  2 2015-10-22 01:16:22 <kanzure> http://heybryan.org/pubkey.txt
  3 2015-10-22 01:25:18 <rusty> kanzure: sent
  4 2015-10-22 01:25:32 <rusty> jgarzik: OK, should we pull the trigger and send the post?
  5 2015-10-22 01:25:39 <kanzure> rusty: received
  6 2015-10-22 01:25:41 <jgarzik> rusty, ack
  7 2015-10-22 01:26:25 <jgarzik> release early, release often, rapidly iterate based on feedback received :)
  8 2015-10-22 01:26:39 <jgarzik> blame me if things go wrong
  9 2015-10-22 01:26:45 <kanzure> and all iterations funded by jgarzik
 10 2015-10-22 01:28:11 <jcorgan> http://pgp.mit.edu/pks/lookup?op=vindex&search=0x473077BC671DA2F7
 11 2015-10-22 01:28:52 <rusty> OK, everyone is moderated.  Now I'll send post (which I shiould have to manually let through)
 12 2015-10-22 01:30:35 <kanzure> where's the moderator interface? or only by mail reply?
 13 2015-10-22 01:32:20 <rusty> kanzure: https://lists.linuxfoundation.org/mailman/admindb/bitcoin-dev
 14 2015-10-22 01:32:36 <rusty> I'm despamming the queue now...
 15 2015-10-22 01:32:38 <kanzure> are pending moderation requests sent by email?
 16 2015-10-22 01:33:18 <rusty> kanzure: hmm, they should.  There's a setting where you can say where to sned IIRC
 17 2015-10-22 01:34:15 <kanzure> are the rejects automatically sent to the correct mailing list?
 18 2015-10-22 01:36:47 <rusty> kanzure: no, you need to click the "forward" button and put in the address.  Let me send a test for you...
 19 2015-10-22 01:37:51 <kanzure> ah that is an important detail. i would have neglected that.
 20 2015-10-22 01:41:11 <rusty> kanzure: OK, look now.  Forward to  
 21 2015-10-22 01:43:12 <kanzure> requests: (1) modification request notifications, (2) automatic forwarding, or at least a default in the admin interface
 22 2015-10-22 01:43:18 <kanzure> s/modification/moderation
 23 2015-10-22 01:45:25 <rusty> kanzure: you can set these yourself?
 24 2015-10-22 01:45:33 <morcos> thanks all of you for doing this!
 25 2015-10-22 01:45:47 <kanzure> rusty: probably requires access to the mailman implementation? i don't have this.
 26 2015-10-22 01:47:11 <jcorgan> rusty: in mailman you'll need to add our email addresses to the moderator config box for it to send us email notifications of pending moderation requests
 27 2015-10-22 02:08:46 <rusty> jcorgan: you could do that, too... one sec, seding password
 28 2015-10-22 02:09:00 <rusty> Sorry, was AFK
 29 2015-10-22 02:09:15 <jcorgan> i have it
 30 2015-10-22 02:09:20 <jcorgan> i'll do mine
 31 2015-10-22 02:09:48 <jcorgan> done
 32 2015-10-22 02:14:00 <rusty> jcorgan: me too.  I also put mdoeration rules into welcome message.
 33 2015-10-22 02:15:23 <jcorgan> istr there was a plan to have a URL with each list and its policies on it somewhere, maybe at LF
 34 2015-10-22 02:15:34 <jcorgan> if so we could link to that
 35 2015-10-22 02:23:38 <rusty> jcorgan: that's a good idea, for the moment just point to the opening post at http://lists.linuxfoundation.org/pipermail/bitcoin-dev/2015-October/011591.html
 36 2015-10-22 02:24:13 <jcorgan> wfm
 37 2015-10-22 07:13:08 <btcdrak> sipa: gmaxwell: sdaftuar: any chance you could take a look at https://github.com/bitcoin/bitcoin/pull/6566
 38 2015-10-22 16:38:53 <FullAnGel> 1Cx2yGrC5c1QUXFMPNKarpgURQhUwCVBz2
 39 2015-10-22 16:40:36 <jcorgan> jgarzik: do we have a policy for -dev ML regarding cross-posting to other lists?  I don't think it is an issue, but wanted to check.
 40 2015-10-22 16:40:57 <jgarzik> that's fine
 41 2015-10-22 16:48:53 <btcdrak> jgarzik: I think it's fine too but I'm not sure it continues to be fine when the discussion should be moved to github as is the case with the current memory leak thread
 42 2015-10-22 16:49:45 <jgarzik> sure. and that's not a CC issue that's a "stop sending to this list (and/or any other lists) and send to github"
 43 2015-10-22 16:49:46 <btcdrak> jgarzik: i mean it's turned into a semi support thread. I think we should say something tbh
 44 2015-10-22 16:49:50 <jgarzik> imo
 45 2015-10-22 16:50:27 <btcdrak> jgarzik: yes, I agree.
 46 2015-10-22 16:53:41 <btcdrak> I might post something like "I think this thread has gotten to the stage where it should be moved to an issue tracker and not continue to CC bitcoin-dev list (or any other list tbh)."
 47 2015-10-22 19:00:05 <sipa> meeting time?
 48 2015-10-22 19:00:09 <gmaxwell> Thu Oct 22 19:00:00 UTC 2015
 49 2015-10-22 19:00:25 <btcdrak> yup
 50 2015-10-22 19:00:36 <sipa> brb
 51 2015-10-22 19:00:37 <btcdrak> ping wumpus
 52 2015-10-22 19:00:41 <petertodd> hi
 53 2015-10-22 19:00:57 <btcdrak> greetings petertodd
 54 2015-10-22 19:01:19 <btcdrak> petertodd: how's the RBF rebase going?
 55 2015-10-22 19:01:39 <petertodd> btcdrak: was just working on that...
 56 2015-10-22 19:02:09 <petertodd> btcdrak: was more of a rewrite, because the new tx package tracking is useful for it
 57 2015-10-22 19:02:58 <sipa> suggested topic: mempool memory usage due to chainstate cache
 58 2015-10-22 19:03:01 <sdaftuar> petertodd: not sure if it's useful, but i'm working on ancestor package tracking as well
 59 2015-10-22 19:03:08 <BlueMatt> sipa: yea, was just about to say that
 60 2015-10-22 19:03:13 <sipa> suggested topic: leveldb seems unmaintained, do we look for replacements?
 61 2015-10-22 19:03:19 <gmaxwell> sipa: seconded.
 62 2015-10-22 19:03:22 <petertodd> sdaftuar: descendent tracking is the only thing rbf needs as far as i can tell
 63 2015-10-22 19:03:23 <BlueMatt> answer: yes
 64 2015-10-22 19:03:25 <BlueMatt> wait, not yet
 65 2015-10-22 19:03:49 <cfields> jonasschnelli might like to weigh in here
 66 2015-10-22 19:03:51 <btcdrak> ping maaku
 67 2015-10-22 19:04:01 <jgarzik> suggested topic: FYI bitcoin-dev now moderated
 68 2015-10-22 19:04:10 <btcdrak> who is going to operate the meetingbot?
 69 2015-10-22 19:04:11 <gmaxwell> To be clear,  seconded, on Mempool/chainstate, leveldb.
 70 2015-10-22 19:04:30 <gmaxwell> jgarzik: Any discussion on bitcoin-dev or should we just note that?
 71 2015-10-22 19:05:11 <btcdrak> #startmeeting
 72 2015-10-22 19:05:11 <lightningbot`> Meeting started Thu Oct 22 19:05:11 2015 UTC.  The chair is btcdrak. Information about MeetBot at http://wiki.debian.org/MeetBot.
 73 2015-10-22 19:05:11 <lightningbot`> Useful Commands: #action #agreed #help #info #idea #link #topic.
 74 2015-10-22 19:05:50 <jgarzik> gmaxwell, just note
 75 2015-10-22 19:06:10 <jonasschnelli> replacement for leveldb is a good topic IMO
 76 2015-10-22 19:06:12 <jgarzik> Just letting devs know to resubscribe :)     (unless someone wants to discuss)
 77 2015-10-22 19:06:43 <jonasschnelli> so. Topics?
 78 2015-10-22 19:07:03 <btcdrak> let's start with mempool memory usage
 79 2015-10-22 19:07:12 <btcdrak> #topic mempool memory usage
 80 2015-10-22 19:07:15 <sipa> so
 81 2015-10-22 19:07:25 <sipa> a few things that BlueMatt and i discovered
 82 2015-10-22 19:07:26 <jgarzik> paste wumpus's gist
 83 2015-10-22 19:07:48 <sipa> there is a discrepancy between the mempool limit that is configured and actual memory usage
 84 2015-10-22 19:08:08 <sipa> investigating turned up that just transaction processing can pull in large amounts of utxo cache data
 85 2015-10-22 19:08:15 <sipa> which is only flushed after block processing
 86 2015-10-22 19:08:24 <sipa> so it can temporarily exceed the limit
 87 2015-10-22 19:08:52 <sipa> also, -checkmempool pulls in all mempool dependencies into the utxo cache, which also can blow it out of bounds
 88 2015-10-22 19:09:01 <morcos> sipa: to be clear there are two limits, utxo dbcache limit and mempool limit and the utxo dbcache limit is exceeded right?
 89 2015-10-22 19:09:07 <sipa> morcos: correct
 90 2015-10-22 19:09:22 <sipa> there is an obvious solution to this, namely just enforcing the utxo limit the whole time
 91 2015-10-22 19:09:45 <sipa> however, that means that if you misconfigure your mempool limit, an attack can basically blow away your utxo cache
 92 2015-10-22 19:09:51 <BlueMatt> well, there are two obvious solutions: always flush utxo cache, or limit mempool in parallel with utxo cache
 93 2015-10-22 19:09:56 <BlueMatt> eg https://github.com/TheBlueMatt/bitcoin/commit/2315cbca8b74197aac022126f0dc0527024c64c9
 94 2015-10-22 19:09:58 <sipa> which has a significant impact on block validation/propagation
 95 2015-10-22 19:10:17 <BlueMatt> (but this opens you up to txn in mempool eating WAY more "space" than others, allowing then to more easily kick out other txn)
 96 2015-10-22 19:10:44 <morcos> aren't there in between answers
 97 2015-10-22 19:10:45 <gmaxwell> sipa: since the utxo cache impact of a mempool transaction can be larger than the transaction itself; do we have an issue that strictly enforcing that limit would slow block verification by resulting in not all mempool txn being cached?
 98 2015-10-22 19:11:02 <morcos> such as not putting into pcoinstip cache utxos for txs which weren't actually accepted into the mempool
 99 2015-10-22 19:11:04 <BlueMatt> morcos: yes
100 2015-10-22 19:11:30 <morcos> also if the mempool limit and utxo limit are closer to the same size... then it should be hard for the actual mempool to blow up the utxo limit, modulo eviction
101 2015-10-22 19:11:33 <sipa> gmaxwell: indeed.. basically... we want the chainstate cache to closely match what is in the mempool as an expectation of the next block
102 2015-10-22 19:11:35 <sdaftuar> we could also be smarter about what we flush, based on what is in the mempool/likelhiood of being mined?
103 2015-10-22 19:11:43 <sipa> so it makes sense to only have one limit that governs both
104 2015-10-22 19:12:07 <gmaxwell> morcos: though if we reject them and they are still mined, that again will result in lower block verification speed.  (though I agree they're second class citizens for sure)
105 2015-10-22 19:12:11 <morcos> sdaftuar: yes, exactly.  first approx, whats actually in the mempool (and not things that got checked but didn't make it) .  next remove things that are evicted
106 2015-10-22 19:12:18 <BlueMatt> morcos: well, yes, you can do thinks like only kick out cache entries as the txn leave mempool (eg https://github.com/TheBlueMatt/bitcoin/commit/5b90ba0da0e6bb086bf5e793634bd105b2bc45ca )
107 2015-10-22 19:12:31 <sipa> two ideas so far for getting closer to that: when a tx is evicted from the mempool, also kick them out of the chainstate cache (if they are not dirty)
108 2015-10-22 19:12:50 <BlueMatt> the above commit does that
109 2015-10-22 19:13:04 <gmaxwell> BlueMatt: what about ones that never go in?
110 2015-10-22 19:13:05 <sipa> and not letting failed-to-accept-to-mempool's dependencies go into the cache
111 2015-10-22 19:13:12 <BlueMatt> gmaxwell: dont think so
112 2015-10-22 19:13:14 <BlueMatt> fixing now
113 2015-10-22 19:13:46 <sipa> i think a better solution is to turn the chainstate cache into a LRU, and be able to on the fly evict things from it
114 2015-10-22 19:13:55 <sipa> when there are better things to put in
115 2015-10-22 19:13:58 <BlueMatt> ehhh, i dont think that fixes it?
116 2015-10-22 19:14:03 <sipa> it's part of it
117 2015-10-22 19:14:08 <BlueMatt> you dont care about lru here, you care about used-in-high-feerate-txn
118 2015-10-22 19:14:15 <sdaftuar> BlueMatt: agreed
119 2015-10-22 19:14:21 <sipa> right, that's independent
120 2015-10-22 19:14:22 <morcos> BlueMatt: by defniniton that will be LRU
121 2015-10-22 19:14:26 <morcos> otherwise it'll be mined
122 2015-10-22 19:14:30 <BlueMatt> morcos: huh? no
123 2015-10-22 19:14:40 <sipa> you want to give priority to the cache for things in the mempool
124 2015-10-22 19:14:45 <BlueMatt> morcos: over long timespans, sure, but what about something that barely doesnt get mined for a while
125 2015-10-22 19:14:56 <BlueMatt> we want to keep it, because its more likely than something for which there is a lot of churn at the bottom of the mempool
126 2015-10-22 19:14:59 <sipa> but outside of that, you want to be able to evict things from the cache when more space in it is needed for mempool txn
127 2015-10-22 19:15:01 <BlueMatt> but it may be a while before its mined
128 2015-10-22 19:15:25 <morcos> how much of a practical problem is this right now
129 2015-10-22 19:15:52 <morcos> lets say you set dbcache to 2x mempool limit
130 2015-10-22 19:16:06 <BlueMatt> right now utxo cache is unlimited
131 2015-10-22 19:16:11 <morcos> do you really get much of a problem, i think there might be more low hanging fruit
132 2015-10-22 19:16:19 <morcos> only unlimited between blocks though
133 2015-10-22 19:16:22 <BlueMatt> sure
134 2015-10-22 19:16:23 <sipa> the problem is that the utxo cache is per tx
135 2015-10-22 19:16:25 <sipa> not per txout
136 2015-10-22 19:16:43 <BlueMatt> but if you create a tx that spends 100 txos from different very-large-txo-set txn, you can pull in a lot of memory
137 2015-10-22 19:16:43 <sipa> so if you are spending outputs from large transactions, the impact on the cache is proportionally way larger
138 2015-10-22 19:17:06 <morcos> i think if we could do the simple thing of not letting non-accepted txs populated the utxo cache, we'll have closed most of the problem
139 2015-10-22 19:17:10 <sipa> i believe i've seen single transactions pull in over 10 MB of chainstate cache
140 2015-10-22 19:17:29 <sipa> though i need to investigate further
141 2015-10-22 19:17:36 <morcos> still, i mean what kind of attack is that
142 2015-10-22 19:18:06 <morcos> its pretty tough to do that with a big chunk of txs from the mempool
143 2015-10-22 19:18:52 <morcos> anyway, i should rebase #5967, as i think it might be good for doing some of this
144 2015-10-22 19:19:03 <morcos> that allows the intermediate cache to be flushed
145 2015-10-22 19:19:24 <morcos> so pcoinstip doesn't need to have coins just b/c viewmempool does
146 2015-10-22 19:19:24 <sipa> ok
147 2015-10-22 19:20:08 <jgarzik> ### 1/3 of meeting complete
148 2015-10-22 19:20:29 <btcdrak> so are there any action points for this topic?
149 2015-10-22 19:20:41 <gmaxwell> okay I think people are going to continue existing work to research and optimize here.
150 2015-10-22 19:20:49 <BlueMatt> yea, lets move forward
151 2015-10-22 19:20:52 <btcdrak> #action continue research
152 2015-10-22 19:20:57 <btcdrak> next topic
153 2015-10-22 19:21:03 <btcdrak> #topic leveldb replacement
154 2015-10-22 19:21:12 <CodeShark> is that really a topic? :)
155 2015-10-22 19:21:21 <sipa> being discussed in #bitcoin-core-dev now
156 2015-10-22 19:21:25 <sipa> it's more a mention
157 2015-10-22 19:21:28 <CodeShark> I know - lol
158 2015-10-22 19:21:37 <sipa> seems several people are interested in investigating this
159 2015-10-22 19:21:51 <gmaxwell> This is pretty bitcoin core specific. Leveldb durrability is not good, and we also suffer from its performance... it is also not (apparently) actively maintained anymore (and the FS layer stuff seems to have not really been maintained for sometime).
160 2015-10-22 19:21:55 <sipa> not sure there is much to say about it any more
161 2015-10-22 19:21:57 <gmaxwell> People are investigating alternatives.
162 2015-10-22 19:21:58 <btcdrak> jeff is researching sqlite as an option
163 2015-10-22 19:22:03 <jgarzik> Argument - leveldb not so well maintained.     I should have a patch for sqlite by the end of the day.
164 2015-10-22 19:22:12 <CodeShark> SQLite is a fine replacement for bdb for the wallet stuff - but for a blockchain store, not sure
165 2015-10-22 19:22:17 <sipa> i look forward to benchmarking
166 2015-10-22 19:22:24 <kanzure_> so would that be sqlite + leveldb?
167 2015-10-22 19:22:30 <sipa> without benchmark results, any discussing is probably worthless
168 2015-10-22 19:22:37 <btcdrak> the problem is leveldb is not being maintained.
169 2015-10-22 19:22:40 <gmaxwell> (also I vind the leveldb code to be very difficult to review :) )
170 2015-10-22 19:22:41 <kanzure_> and also, would there be an automatic migration from leveldb to sqlite
171 2015-10-22 19:22:45 <BlueMatt> so i think the action is benchmark lots of things, come back and report (in -core-dev) with results
172 2015-10-22 19:22:46 <jgarzik> sqlite appears to perform better than bdb for large numbers of records, according to the sqlite key/value store wiki benchmarks page.
173 2015-10-22 19:23:02 <jcorgan> it's pretty much a guarantee that sqlite perf will be much lower than leveldb, but i'd be happy to be surprised
174 2015-10-22 19:23:08 <luke-jr_> UTXO storage db is part of the consensus protocol, so it really isn't Core-specific
175 2015-10-22 19:23:10 <gmaxwell> btcdrak: I don't really agree there, in any case. I think all we need to do is note this is being discussed and explorered.
176 2015-10-22 19:23:16 <jgarzik> jcorgan, the benchmarks pages disagrees
177 2015-10-22 19:23:23 <kanzure_> sqlite also has in-memory database which could be strangely useful for unit testing (sqlite://:memory:)
178 2015-10-22 19:23:32 <sipa> sqlite is better for synchronous storage, slower for batch updates, faster for random-access reads
179 2015-10-22 19:23:41 <sipa> all of which are things we rely on very heavily
180 2015-10-22 19:23:45 <jonasschnelli> sqlite seems not to be the type we'd like to have for a leveldb replace. What about other non-sql solutions?
181 2015-10-22 19:23:46 <sipa> so i don't know the result
182 2015-10-22 19:23:49 <jgarzik> kanzure_, yes.  that is easily wrapped into the current wrapper, newly s/CLevelDBWrapper/CDBWrapper/
183 2015-10-22 19:23:54 <jcorgan> as i mentioned before, my main reason for advocating sqlite is to be able to use the sqlcipher fork for full db encryption
184 2015-10-22 19:23:54 <maaku> sipa: benchmarks https://www.sqlite.org/cvstrac/wiki?p=KeyValueDatabase
185 2015-10-22 19:24:04 <maaku> (sqlite is crazy slow compared with bdb)
186 2015-10-22 19:24:17 <kanzure_> jcorgan: and where would the keys go?
187 2015-10-22 19:24:20 <kanzure_> oh, wallet. nevermind.
188 2015-10-22 19:24:25 <jgarzik> maaku, except for large numbers of records, as shown on that page
189 2015-10-22 19:24:37 <sipa> jcorgan: encryption for public data seems pointless
190 2015-10-22 19:25:03 <kanzure_> well maybe for watches or something..
191 2015-10-22 19:25:11 <jonasschnelli> I agree with sipa: first we need some research, there are other solutions like RocksDB, etc. We need an overview with benchmarks to evaluate more
192 2015-10-22 19:25:19 <sipa> ok
193 2015-10-22 19:25:22 <btcdrak> #action continue research and benchmarking
194 2015-10-22 19:25:22 <sipa> next topic?
195 2015-10-22 19:25:33 <btcdrak> #topic versionbits
196 2015-10-22 19:25:42 <btcdrak> CodeShark wanted to discuss his implementation
197 2015-10-22 19:25:58 <CodeShark> #6816 needs more reviews :)
198 2015-10-22 19:26:05 <kanzure_> gmaxwell: how much better is sqlite source code for legibility?
199 2015-10-22 19:26:37 <btcdrak> CodeShark, are you planning to add some RPC tests?
200 2015-10-22 19:26:41 <gmaxwell> kanzure_: we're on a new topic. will respond later.
201 2015-10-22 19:27:00 <CodeShark> btcdrak: I added versionbits info to the getblockchaininfo RPC
202 2015-10-22 19:27:01 <btcdrak> Also, for the record, what's the status of the implementation?
203 2015-10-22 19:27:21 <CodeShark> the implementation should be ready for integration barring any issues I'm unaware of
204 2015-10-22 19:28:06 <btcdrak> ok is there anything else to discuss on this topic?
205 2015-10-22 19:28:17 <sipa> we should review implementation
206 2015-10-22 19:28:25 <btcdrak> #action review versionbits implementation https://github.com/bitcoin/bitcoin/pull/6816
207 2015-10-22 19:28:25 <sipa> that includes me
208 2015-10-22 19:28:42 <btcdrak> next topic?
209 2015-10-22 19:28:51 <btcdrak> I'd like to mention the mailing list for the record
210 2015-10-22 19:28:58 <btcdrak> #topic mailing list
211 2015-10-22 19:29:14 <kanzure_> have users been moved to bitcoin-discuss, or is that voluntary sign-up only?
212 2015-10-22 19:29:20 <btcdrak> We finalised the moderation rules for bitcoin-dev
213 2015-10-22 19:29:27 <btcdrak> and have started a 3 month moderation period
214 2015-10-22 19:29:38 <btcdrak> We#d encourage devs to join again
215 2015-10-22 19:29:38 <CodeShark> have those rules been posted anywhere?
216 2015-10-22 19:29:45 <btcdrak> #info http://lists.linuxfoundation.org/pipermail/bitcoin-dev/2015-October/011591.html
217 2015-10-22 19:29:46 <petertodd> btcdrak: by moderation, does this mean posts have to be manually approved?
218 2015-10-22 19:29:58 <CodeShark> is there a white list?
219 2015-10-22 19:30:01 <btcdrak> see the link above
220 2015-10-22 19:30:09 <gmaxwell> On the subject of "take note" -- bitcoin.org had incorrect release notes for 0.11.1.  It's corrected now. They had posted the release notes for the initial RC and not updated them. Process wise it would be good to watch out for that in the future.
221 2015-10-22 19:30:11 <kanzure_> petertodd: yes, i have had to manually approve email today
222 2015-10-22 19:30:21 <jcorgan> only the first
223 2015-10-22 19:30:29 <petertodd> btcdrak: Ah, I missed the "mod bit" part of that message, thanks.
224 2015-10-22 19:30:32 <jcorgan> then we take the mod bit off that user
225 2015-10-22 19:30:57 <btcdrak> ok next topic, anyone?
226 2015-10-22 19:31:03 <Luke-Jr> gmaxwell: my bad for not bringing it up. I had assumed the only difference was minor, but I should have compared when I noticed it was not identical. :/
227 2015-10-22 19:31:25 <kanzure_> if we want to have actively used bitcoin-discuss there should be someone who currates content for that mailing list, otherwise i doubt people will use it
228 2015-10-22 19:31:31 <kanzure_> actually what is the status, has traffic actually been redirected there
229 2015-10-22 19:31:45 <kanzure_> "No messages have been posted to this list yet, so the archives are currently empty."
230 2015-10-22 19:31:46 <btcdrak> kanzure_ please see the posted link
231 2015-10-22 19:31:49 <sipa> note: there is still a discuss mailing list on SF
232 2015-10-22 19:31:59 <gmaxwell> Luke-Jr: in particular it did not have the relay fee bump.
233 2015-10-22 19:32:00 <sipa> there have been several messages sent there this year
234 2015-10-22 19:32:01 <kanzure_> which discuss mailing list should have priority?
235 2015-10-22 19:32:06 <btcdrak> sipa: the new official list is now bitcoin-discuss at LF
236 2015-10-22 19:32:12 <kanzure_> linuxfoundation has been working well afaik
237 2015-10-22 19:32:15 <kanzure_> for mailing list hosting
238 2015-10-22 19:32:34 <jgarzik> #action tell SF list about new lists
239 2015-10-22 19:32:38 <btcdrak> #link lists.linuxfoundation.org/mailman/listinfo/bitcoin-discuss
240 2015-10-22 19:32:47 <kanzure_> we should find someone who wants to actively run -discuss or something
241 2015-10-22 19:32:52 <sipa> i did hide the old dev list from SF
242 2015-10-22 19:32:54 <kanzure_> or replay old -dev messages to -discuss that would have otherwise been moderated
243 2015-10-22 19:33:06 <sipa> i haven't deleted it yet should we want its archives for whatever reason
244 2015-10-22 19:33:33 <btcdrak> ok I think next topic. I've got one ;)
245 2015-10-22 19:33:33 <kanzure_> (for the record i am not interested in running or moderating -discuss, but "go here where nobody is sending email/reading" is not likely to win anyone over)
246 2015-10-22 19:33:50 <btcdrak> #topic Median Past locktime
247 2015-10-22 19:34:00 <btcdrak> #link PR https://github.com/bitcoin/bitcoin/pull/6566
248 2015-10-22 19:34:12 <btcdrak> Reviews have been good since last week
249 2015-10-22 19:34:42 <btcdrak> maaku: there are a couple of nits from wumpus and gmaxwell. Has anyone else got concerns for this PR?
250 2015-10-22 19:34:54 <sipa> concept ack
251 2015-10-22 19:34:57 <gmaxwell> I am a strong concept ack.
252 2015-10-22 19:35:03 <gmaxwell> (which I'll also go make clear there)
253 2015-10-22 19:35:09 <kanzure_> yes same with sipa please
254 2015-10-22 19:35:26 <Luke-Jr> I'll try to finish reviewing it ASAP
255 2015-10-22 19:36:10 <btcdrak> #action review https://github.com/bitcoin/bitcoin/pull/6566
256 2015-10-22 19:36:12 <gmaxwell> There is lots going on in lots of PRs, but I can't think of any that need to be elevated to here.
257 2015-10-22 19:36:32 <btcdrak> I only bring up 6566 because we talked about adding it to the CLTV softfork
258 2015-10-22 19:36:43 <btcdrak> we're getting close to that now
259 2015-10-22 19:36:53 <gmaxwell> Topic suggestion: Where are we on CLTV soft-fork deployment?
260 2015-10-22 19:37:03 <btcdrak> #topic CLTV softfork deployment
261 2015-10-22 19:37:15 <petertodd> btcdrak: re: CLTV softfork, I'm likely not going to have time to be doing more rebasing/rework on that going forward, so either merge it or someone else should be prepared to take over
262 2015-10-22 19:37:20 <gmaxwell> btcdrak: that PR is just mempool behavior.
263 2015-10-22 19:37:29 <petertodd> (I no longer have funding for bitcoin core work)
264 2015-10-22 19:38:04 <CodeShark> fwiw, #6816 will make merging the soft fork logic itself pretty straightforward ;)
265 2015-10-22 19:38:33 <btcdrak> We agreed to roll out the CLTV softfork at the end of October.
266 2015-10-22 19:39:16 <gmaxwell> btcdrak: no one forgets what the objective was there, but what is the status?
267 2015-10-22 19:39:33 <btcdrak> I believe for master the PR is ready to merge
268 2015-10-22 19:39:35 <sipa> what needs to be done?
269 2015-10-22 19:39:44 <btcdrak> there are 0.10 and 0.11 PRs for the packports also
270 2015-10-22 19:39:50 <btcdrak> so they just need to be merged.
271 2015-10-22 19:39:57 <sipa> merge PR, do v4 block ISM softwarez done?
272 2015-10-22 19:40:05 <btcdrak> sipa: yes
273 2015-10-22 19:40:07 <sipa> i will review then
274 2015-10-22 19:40:15 <sipa> but i expect i will be fine
275 2015-10-22 19:40:24 <sipa> this is the sort of thing to discuss on the ML though
276 2015-10-22 19:40:28 <btcdrak> https://github.com/bitcoin/bitcoin/pull/6707
277 2015-10-22 19:40:35 <btcdrak> https://github.com/bitcoin/bitcoin/pull/6706
278 2015-10-22 19:40:53 <btcdrak> https://github.com/bitcoin/bitcoin/pull/6351
279 2015-10-22 19:41:10 <btcdrak> Those are the PRs. If petertodd isnt around to rebase, one of us can just do it.
280 2015-10-22 19:41:19 <sipa> yeah
281 2015-10-22 19:41:36 <btcdrak> if we want to add media-past-locktime then there is still time
282 2015-10-22 19:42:01 <gmaxwell> #action review #6707 #6706 #6351 for CLTV deployment.
283 2015-10-22 19:42:24 <gmaxwell> Has code even been writeen for MPL enforcement as part of that soft fork? if not, someone needs to take the task of doing it.
284 2015-10-22 19:42:28 <btcdrak> do we want to try for median-past-locktime with this or just wait?
285 2015-10-22 19:42:52 <btcdrak> I would have assume maaku would do it, he was just waiting for the mempool PR to be merged
286 2015-10-22 19:42:52 <petertodd> btcdrak: median past is the type of boring change that'd be an excellent way to deploy versionbits actually
287 2015-10-22 19:43:11 <gmaxwell> btcdrak: there are three possibilities in my mind.
288 2015-10-22 19:43:16 <petertodd> btcdrak: there's no rush to do it and it's uncontroversial
289 2015-10-22 19:43:24 <gmaxwell> no MPL. MPL mempool only. MPL in softfork.
290 2015-10-22 19:43:32 <gmaxwell> I strongly believe we should not do the first.
291 2015-10-22 19:43:45 <gmaxwell> well somewhat strongly.
292 2015-10-22 19:43:59 <kanzure> so short-term median-past-locktime mempool only, then median-past-locktime soft-fork using versionbits?
293 2015-10-22 19:44:00 <btcdrak> gmaxwell: at the least, I would love MPL as mempool together with BIP68 and OP_CSV as mempool.
294 2015-10-22 19:44:11 <maaku> gmaxwell: the MPL pull is specifically written to not require further code for soft-fork, just setting a flag bit undre a ISM condition
295 2015-10-22 19:44:19 <gmaxwell> actually, for same reason, MPL-mempool only should be done instead of the softfork. Rational: right now miners will currently mine MPL violations.
296 2015-10-22 19:44:25 <btcdrak> deployment is easy
297 2015-10-22 19:44:40 <sipa> yes, MPL should be deployed mempool-only first
298 2015-10-22 19:44:56 <petertodd> gmaxwell: oh right, that's a good point... further delays MPL deployment, which again is an argument to do it with versionbit
299 2015-10-22 19:45:04 <gmaxwell> maaku: I know I read the code. But there is no patch to actually set the flag and enforce it. But thats okay at the moment. We should mempool only it first.
300 2015-10-22 19:45:15 <sipa> same with BIP68, no?
301 2015-10-22 19:45:27 <gmaxwell> sipa: CSV enforcement is flagged.
302 2015-10-22 19:45:31 <sipa> current miners will accept bip68 violations
303 2015-10-22 19:45:46 <sdaftuar> BIP68 is only active for higher tx version right?
304 2015-10-22 19:45:48 <petertodd> sipa: requires nVersion=2
305 2015-10-22 19:45:49 <gmaxwell> sipa: no, requires a tx version flag.
306 2015-10-22 19:45:55 <sipa> right, separate tx version
307 2015-10-22 19:46:00 <sipa> which is already nonstandard
308 2015-10-22 19:46:01 <sipa> nvm!
309 2015-10-22 19:46:05 <maaku> gmaxwell: there's no patch because it'd presumably be part of #6351 if rolled out together, so I didn't create a separate PR
310 2015-10-22 19:46:36 <gmaxwell> maaku: thats okay, I think based on the observation that miners would currently mine violations we'll only do mempool only immediately.
311 2015-10-22 19:47:08 <gmaxwell> (which also still has the positive effect of stopping new CLTV locktime users from depending on the to-be-replaced behavior.
312 2015-10-22 19:47:13 <gmaxwell> )
313 2015-10-22 19:47:33 <btcdrak> how about we merge all three mempool PRs and release them with the CLTV deployment to get them out in the wild. We can then consider another ISM or versionbits deployment after?
314 2015-10-22 19:47:57 <gmaxwell> btcdrak: the mempool related PRs are more or less 0.12 only.
315 2015-10-22 19:48:23 <btcdrak> gmaxwell: but we can backport?
316 2015-10-22 19:48:26 <gmaxwell> They depend on git master exclusive major changes to the mempool code.
317 2015-10-22 19:48:31 <gmaxwell> btcdrak: I don't think that would be prudent.
318 2015-10-22 19:48:39 <btcdrak> we have to backport for softfork deployments anyway
319 2015-10-22 19:48:40 <Luke-Jr> we can release backported policy patches
320 2015-10-22 19:48:49 <Luke-Jr> btcdrak: consensus and policy are not the same
321 2015-10-22 19:48:56 <gmaxwell> I would recommend 'backporting' TCP_NODELAY.  (which I will gladly do, of course) :P  if you're looking for upgrade sweetener.
322 2015-10-22 19:48:58 <btcdrak> Luke-Jr: ah, I understand
323 2015-10-22 19:49:05 <CodeShark> should I start working on #6816 backports?
324 2015-10-22 19:49:25 <btcdrak> CodeShark I would get #6816 merged first.
325 2015-10-22 19:49:39 <gmaxwell> CodeShark: perhaps wait until the first round of review unless you're itching and won't mind redoing it if you get asked to change the design?
326 2015-10-22 19:50:14 <CodeShark> the backports should be pretty easy, I think - so it can wait
327 2015-10-22 19:50:15 <kanzure> does median-past-locktime mempool-only also require lots of master-only 0.12 mempool changes that are not in the older versions? re: backporting
328 2015-10-22 19:50:27 <CodeShark> the backports can wait, that is - the reviews can't ;)
329 2015-10-22 19:50:46 <gmaxwell> Kireji: no.
330 2015-10-22 19:50:48 <gmaxwell> er
331 2015-10-22 19:50:49 <gmaxwell> kanzure: no.
332 2015-10-22 19:50:58 <gmaxwell> are there any in flight backportable bug fixes which we should hammer down before we do backport updates for the CLTV soft-fork?
333 2015-10-22 19:51:17 <sipa> gmaxwell: which "mempool related PRs" are not backportable?
334 2015-10-22 19:51:55 <gmaxwell> sipa: I assume btcdrak is mostly talking about matt's limiter, which depends on the major data structure changes in the mempool.  Do you think we should actually backport all that?
335 2015-10-22 19:52:08 <sipa> gmaxwell: hell no
336 2015-10-22 19:52:14 <petertodd> sipa: +1
337 2015-10-22 19:52:14 <sipa> but we're talking about softforks here
338 2015-10-22 19:52:27 <btcdrak> I think the point is, if we get the mempool PRs merged in master, then we can start working on backporting them. The actual softfork deployment can be done at any time thereafter at our convenience.
339 2015-10-22 19:52:42 <gmaxwell> oh dear! okay. I was wires crossed. I thought btcdrap was talking about mempool capacity management.
340 2015-10-22 19:52:45 <btcdrak> this way we're doing a pure CLTV release first.
341 2015-10-22 19:52:54 <sipa> i think that btcdrak is referring to the BIP68/112/113 mempool-only changes with "mempool PRs"
342 2015-10-22 19:52:56 <btcdrak> gmaxwell: no :)
343 2015-10-22 19:52:58 <maaku> kanzure: I don't believe any of #6312, #6564 or #6566 depend on master changes
344 2015-10-22 19:53:01 <petertodd> also, if a backport is going to take a lot of work, you might want to ask miners first if they really want it...
345 2015-10-22 19:53:11 <maaku> I'm not sure what gmaxwell is referring to
346 2015-10-22 19:53:12 <btcdrak> yes, sorry that's my mistake, I was referring to BIUP68/112/113
347 2015-10-22 19:53:48 <gmaxwell> as I said, I thought he was referring to the mempool limits and friends.
348 2015-10-22 19:53:53 <sipa> OK
349 2015-10-22 19:53:56 <btcdrak> thing is MPL is quite a trivial patch, it would easily backport and be part of the CLTV softfork...
350 2015-10-22 19:54:14 <gmaxwell> I do not think there is a reason to mempool only any of them except median right now, as they're already non-standard.
351 2015-10-22 19:54:33 <gmaxwell> The concern I have is that if we deploy a mempool only CSV then now we have a problem if the interperation of the field changes.
352 2015-10-22 19:54:52 <gmaxwell> (because we may need to first make it non-standard again)
353 2015-10-22 19:54:55 <maaku> do we have reason to believe the interpretation of the field will change?
354 2015-10-22 19:55:18 <CodeShark> maaku: the last few weeks :p
355 2015-10-22 19:55:30 <gmaxwell> Deploying it as mempool only in the soft fork backport when its already non-standard carries basically only risk.
356 2015-10-22 19:55:35 <btcdrak> BIP68 has been refined.
357 2015-10-22 19:56:02 <btcdrak> I think everyone nits were addressed for BIP68, so itr should be considered final
358 2015-10-22 19:56:10 <petertodd> gmaxwell: w/ CLTV, the plan was to s/OP_NOP2/OP_NOP3/ - the equivalent for CSV's redefinition of what nSequence means might be to use nVersion>=3 rather than nVersion>=2
359 2015-10-22 19:57:05 <gmaxwell> petertodd: indeed, thats true.
360 2015-10-22 19:57:14 <gmaxwell> But again, I do not see the gain.
361 2015-10-22 19:57:25 <gmaxwell> At least in the context of the CLTV softfork release.
362 2015-10-22 19:57:39 <petertodd> gmaxwell: it's not a gain, just a backup plan if a mempool-only release turns out to need to be changed because the behavior was wrong
363 2015-10-22 19:57:42 <btcdrak> gmaxwell: so you're suggesting we properly softfork MPL, and bip68/CSV right off the bat
364 2015-10-22 19:58:08 <gmaxwell> petertodd: I meant I do not see the gain in putting out CSV mempool only at the same time as CLTV softfork.
365 2015-10-22 19:58:16 <petertodd> gmaxwell: ah, yeah, neither do I
366 2015-10-22 19:58:18 <gmaxwell> But I agree with your backup plan, good observation.
367 2015-10-22 19:58:26 <btcdrak> gmaxwell: the original discussion is to have CLTV + MPL
368 2015-10-22 19:58:33 <gmaxwell> btcdrak: No. backup.
369 2015-10-22 19:58:50 <gmaxwell> btcdrak: I am saying, there will be a CLTV soft fork release which also has mempool only MPL.
370 2015-10-22 19:59:14 <btcdrak> gmaxwell: ok, understood.
371 2015-10-22 19:59:15 <gmaxwell> The latter because MPL violates current 'standard' behavior, so we would prefer to have that violation dead in the network before MPL soft-fork moves forward.
372 2015-10-22 19:59:30 <gmaxwell> For CSV mempool it's already non-standard, so the same argument doesn't apply.
373 2015-10-22 19:59:45 <btcdrak> in that case, maaku needs to work on backporting MPL to 0.11 and 0.10 asap.
374 2015-10-22 20:00:05 <gmaxwell> And we're awfully close right now to the last semantic changes, so I'd rather hold for that. (could still go in mempool only first, but just not in the CLTV soft fork release)
375 2015-10-22 20:00:18 <kanzure> was the mempool limiting stuff something that needs to be discussed? the thing that was misconfused above.
376 2015-10-22 20:00:19 <gmaxwell> btcdrak: fortunate those backports will be trivial code copying, I believe.
377 2015-10-22 20:00:34 <btcdrak> ok so I'll log that as an action point then
378 2015-10-22 20:00:46 <gmaxwell> kanzure: my wires were crossed, I thought btcdrak brought them up as deployment sweetener for CLTV.
379 2015-10-22 20:00:51 <kanzure> right
380 2015-10-22 20:00:59 <kanzure> it was wrong, but does that need to be discussed anyway
381 2015-10-22 20:01:02 <btcdrak> #action backport MPL to 0.10 and 0.11
382 2015-10-22 20:01:02 <jgarzik> ###
383 2015-10-22 20:01:12 <sipa> time's up
384 2015-10-22 20:01:17 <gmaxwell> we are past time. Thanks everyone!
385 2015-10-22 20:01:21 <jgarzik> meeting fini
386 2015-10-22 20:01:24 <btcdrak> #endmeeting
387 2015-10-22 20:01:24 <lightningbot`> Log:            http://www.erisian.com.au/meetbot/bitcoin-dev/2015/bitcoin-dev.2015-10-22-19.05.log.html
388 2015-10-22 20:01:24 <lightningbot`> Meeting ended Thu Oct 22 20:01:24 2015 UTC.  Information about MeetBot at http://wiki.debian.org/MeetBot . (v 0.1.4)
389 2015-10-22 20:01:24 <lightningbot`> Minutes:        http://www.erisian.com.au/meetbot/bitcoin-dev/2015/bitcoin-dev.2015-10-22-19.05.html
390 2015-10-22 20:01:24 <lightningbot`> Minutes (text): http://www.erisian.com.au/meetbot/bitcoin-dev/2015/bitcoin-dev.2015-10-22-19.05.txt
391 2015-10-22 20:01:52 <btcdrak> haha gmaxwell that was hectic for a bit, I was confused as hell :)
392 2015-10-22 20:02:33 <sdaftuar> gmaxwell: now that the meeting is over, just wanted to ping you to see if anything is holding up assigning a BIP number for the sendheaders BIP?
393 2015-10-22 20:02:39 <gmaxwell> kanzure: as far as sqllite, yes I think the code is quite readable, and extensively commented. Their testing is also very impressive.  But I really do not expect the performance to be acceptable...
394 2015-10-22 20:02:55 <jonasschnelli> I think RocksDB could be a way to go
395 2015-10-22 20:02:55 <kanzure> haven't looked at either leveldb or sqlite source code
396 2015-10-22 20:03:23 <jonasschnelli> Check: https://github.com/facebook/rocksdb/wiki/Performance-Benchmarks
397 2015-10-22 20:03:28 <jonasschnelli> https://influxdb.com/blog/2014/06/20/leveldb_vs_rocksdb_vs_hyperleveldb_vs_lmdb_performance.html
398 2015-10-22 20:03:31 <bsm1175321> I re-ran those performance benchmarks sipa mentioned: https://gist.github.com/mcelrath/6952eab246a7c705a0fb
399 2015-10-22 20:03:48 <jonasschnelli> RocksDB seems to be maintained.
400 2015-10-22 20:04:35 <sipa> jonasschnelli: is it extensively tested on windows?
401 2015-10-22 20:04:43 <kanzure> i wonder if there's any mozilla people working on sqlite. hm.
402 2015-10-22 20:05:00 <jonasschnelli> windows: who cares. :) ... no: i think not.
403 2015-10-22 20:05:14 <sipa> that's the massive problem.with leveldb
404 2015-10-22 20:05:18 <maaku> jonasschnelli: RocksDB is Leveldb descendent
405 2015-10-22 20:05:24 <sipa> it seems to consistently corrupt on windows
406 2015-10-22 20:05:25 <btcdrak> gmaxwell: that was my impression about sqlite but according to the chat in #bitcoin-core-dev, it might perform much better than leveldb for what we need
407 2015-10-22 20:05:51 <maaku> btcdrak: ?
408 2015-10-22 20:05:52 <sipa> it may also be much worse
409 2015-10-22 20:05:58 <maaku> sqlite cannot match leveldb performance
410 2015-10-22 20:06:01 <sipa> let's move discussion there
411 2015-10-22 20:06:05 <jonasschnelli> sipa: +1
412 2015-10-22 20:06:09 <btcdrak> +!
413 2015-10-22 20:06:10 <btcdrak> +1
414 2015-10-22 20:06:23 <jonasschnelli> for a wallet, yes. But not for the blockstore
415 2015-10-22 20:06:54 <jonasschnelli> (even for a wallet we would consider using a hashed append only db like sipas logdb patch)
416 2015-10-22 20:07:11 <sipa> we are not talking about the wallet at all
417 2015-10-22 20:07:19 <jonasschnelli> right
418 2015-10-22 20:08:41 <Luke-Jr> disagree with moving it to #bitcoin-core-dev, as this is a protocol issue
419 2015-10-22 20:08:42 <btcdrak> For reference this was the discussion today about sqlite: https://botbot.me/freenode/bitcoin-core-dev/2015-10-22/?msg=52492157&page=2
420 2015-10-22 20:10:57 <BlueMatt> Luke-Jr: HUH?
421 2015-10-22 20:12:25 <maaku> BlueMatt: consensus code
422 2015-10-22 22:42:19 <kanzure> phantomcircuit: trying to decide whether to reject your two-sentence email. tough decision!
423 2015-10-22 22:42:51 <phantomcircuit> kanzure, benchmarks :) need more benchmarks
424 2015-10-22 22:43:14 <kanzure> yeah but... two sentence "+1" email, against how many subscribers? dunno.
425 2015-10-22 22:43:17 <kanzure> approving anyway..
426 2015-10-22 22:43:46 <btcdrak> I thought +1's were discouraged
427 2015-10-22 22:43:54 <kanzure> it's not really a +1
428 2015-10-22 22:44:00 <kanzure> but it's close :-)
429 2015-10-22 22:47:11 <Luke-Jr> this moderation thing isn't working out. even now noise is getting approved. :P jk
430 2015-10-22 22:47:28 <phantomcircuit> ha
431 2015-10-22 23:04:39 <btcdrak> kanzure: Luke_Jr: yeah I think we need to be quite strict to set the bar. Certainly that is how I understood Rusty's take on it.