1 2015-11-19 12:41:08 <therethere> Hello can anyone tell me some tips to reduce memory usage of the bitcoin client? I'm gonna re-compile it now
  2 2015-11-19 12:41:51 <sipa> therethere: master branch has significant memory reductions, but is probably not ready for production
  3 2015-11-19 12:42:54 <wumpus> therethere: https://gist.github.com/laanwj/efe29c7661ce9b6620a7
  4 2015-11-19 12:43:43 <sipa> wumpus: you may want to remove the "maxorphanblocks" there :)
  5 2015-11-19 12:43:52 <wumpus> good idea
  6 2015-11-19 12:44:32 <therethere> I tried the dbcache trick, it improved the memory usage but not substantially
  7 2015-11-19 12:44:51 <therethere> It still gets to ±2GB after 10hrs aprox
  8 2015-11-19 12:44:53 <therethere> which is huge
  9 2015-11-19 12:45:06 <sipa> therethere: try increasing the minimum relay fee
 10 2015-11-19 12:45:20 <wumpus> I see I haven't included the minrelayfee suggestion from the 0.11 release notes
 11 2015-11-19 12:45:36 <sipa> therethere: 0.12 will automatically do that, though
 12 2015-11-19 12:47:12 <therethere> I'll up the minrelayfee some more I guess as for the "export MALLOC_ARENA_MAX=1" any chances of this messing up other programs?
 13 2015-11-19 12:48:22 <wumpus> not if you only set it when executing bitcoind
 14 2015-11-19 12:48:46 <therethere> And if there's something I can modify directly in code can anyone direct me to it before I compile this new version? I ran several full nodes and while I'd like to support the network further, their main purpose is for wallet functionality and if the RAM usage becomes problematic I might have to switch to something else and shut down the nodes :(
 15 2015-11-19 12:48:48 <wumpus> no guarantees what will happen if you add it to some top-level script (eg .profile or rc.local) so don't do that
 16 2015-11-19 12:49:07 <therethere> Is that doable in a cron?
 17 2015-11-19 12:49:09 <therethere> oh ok
 18 2015-11-19 12:49:41 <sipa> therethere: if you're running a production wallet, better not use the master branch
 19 2015-11-19 12:49:41 <wumpus> just make a script that sets it and then calls bitcoind
 20 2015-11-19 12:49:57 <sipa> therethere: in which case you can just use 0.11.3, no need to recompile
 21 2015-11-19 12:53:01 <therethere> Yes this is for production wallets. So should I just get latest binary (0.11.2 ?). I do like squeezing every last bit of performance which compiling should achieve in theory
 22 2015-11-19 12:54:15 <sipa> therethere: it won't, not unless taking significant risks by compiling with experimental optimization flags
 23 2015-11-19 12:56:03 <therethere> Alright. Thanks a lot for the help!
 24 2015-11-19 12:56:36 <wumpus> also there's nothing in manual compiling that will reduce memory usage significantly, e.g. making the executable smaller would only count for very little
 25 2015-11-19 12:57:42 <sipa> therethere: though by all means feel free to try the master branch, it should be both significantly faster and use less memory
 26 2015-11-19 12:57:47 <sipa> s/try/test/
 27 2015-11-19 12:59:38 <therethere> Thanks sipa I am compiling it now and will definitely test it out and in the meantime I'll go with stable releases + higher minrelayfee for production.
 28 2015-11-19 13:01:48 <wumpus> if it is a wallet with a lot of transactions that may cause lots of memory usage too
 29 2015-11-19 13:07:32 <therethere> So should I "reset" the wallet every x transactions?
 30 2015-11-19 13:11:27 <sipa> therethere: how many transactions are we talking about?
 31 2015-11-19 13:11:56 <wumpus> there are companies doing that, yes, but that's talking about 100,000+ transactions
 32 2015-11-19 13:12:58 <wumpus> IMO, 2GB of memory usage shouldn't be something you should worry about in a serious production setting, although monitoring it is good
 33 2015-11-19 13:21:03 <therethere> I have under 10k right now
 34 2015-11-19 13:21:55 <therethere> Well 2GB is alright now considering before it used to climb up to 3+ and my systems run on 4GB
 35 2015-11-19 13:23:12 <sipa> 10k shouldn't hurt
 36 2015-11-19 13:24:00 <therethere> Finished compiling it , I get this  tor: Disconnected from Tor control port 127.0.0.1:9051, trying to reconnect
 37 2015-11-19 13:24:10 <therethere> Any ideea what this is?
 38 2015-11-19 13:24:53 <wumpus> just ignore it
 39 2015-11-19 13:25:47 <wumpus> that's https://github.com/bitcoin/bitcoin/pull/7035
 40 2015-11-19 13:25:56 <noamh> does anyone know how I can regenerate the src/test/data/*.json files?
 41 2015-11-19 13:26:16 <wumpus> noamh: which ones specifically?
 42 2015-11-19 13:26:30 <sipa> noamh: some are generated with code, but most are hand written
 43 2015-11-19 13:26:32 <wumpus> there are some of the json files (base58, part of the tx tests) that can be generated, the rest is human-written
 44 2015-11-19 13:26:36 <wumpus> right
 45 2015-11-19 13:26:50 <noamh> transaction_tests for example
 46 2015-11-19 13:27:39 <noamh> II know how to do it for script_tests using the define UPDATE_JSON_TESTS
 47 2015-11-19 13:27:53 <sipa> those are hand written, afaik
 48 2015-11-19 13:28:17 <wumpus> right, if it doesn't have anything like UPDATE_JSON_TESTS then they are hand written
 49 2015-11-19 13:28:44 <wumpus> if it doesn't have anything like UPDATE_JSON_TESTS then they are hand written, I was confused with the script tests then, but check the respective source file to be sure
 50 2015-11-19 13:28:58 <wumpus> better question: why are you trying to do this?
 51 2015-11-19 13:30:40 <noamh> experimenting with some serialization stuff
 52 2015-11-19 13:32:33 <wumpus> ok
 53 2015-11-19 14:44:53 <akrmn> Can some one please explain the ordering of the blocks in the blk*.dat files? For example, I run over blk00000.dat and I keep getting blocks in order hashing the previous blocks, but then after block 2288, instead of block 2289, I get block 2305 according to blockchain.info.
 54 2015-11-19 14:45:37 <sipa> akrmn: they are stored in the order they are received
 55 2015-11-19 14:45:38 <akrmn> Do I need to use the leveldb databse to tell me where in the files to find the blocks?
 56 2015-11-19 14:45:57 <akrmn> o ok...
 57 2015-11-19 14:45:58 <sipa> akrmn: you can scan for block markers, but you'll need to build the tree yourself
 58 2015-11-19 14:47:05 <akrmn> ya there was also problems I had with not finding the next magic number after skipping the number of bytes indicated by the blocksize, but that's another issue
 59 2015-11-19 14:47:26 <sipa> there can be garbage in between blocks too
 60 2015-11-19 14:47:52 <akrmn> how?
 61 2015-11-19 14:48:16 <akrmn> doesn't the program check that it's valid before writing to the file?
 62 2015-11-19 14:51:45 <wumpus> it does, but it is possible for there to be padding in some cases, to properly scan the block files you need to look for the block markers; see e.g. the scripts in contrib/linearize for an example
 63 2015-11-19 14:58:58 <morcos> wumpus: you can add to your low memory gist that unless you have reason to , don't ever call getblocktemplate.
 64 2015-11-19 14:59:29 <morcos> of course the effect is much reduced with 1 memory arena, but its still pretty bad if you have a decent sized mempool
 65 2015-11-19 14:59:42 <morcos> that should be fixed in 0.12 as well
 66 2015-11-19 15:12:11 <wumpus> I don't expect people will be calling getblocktemplate without a good reason
 67 2015-11-19 19:00:11 <wumpus> #meetingstart
 68 2015-11-19 19:00:37 <wumpus> #startmeeting
 69 2015-11-19 19:00:38 <lightningbot> Meeting started Thu Nov 19 19:00:38 2015 UTC.  The chair is wumpus. Information about MeetBot at http://wiki.debian.org/MeetBot.
 70 2015-11-19 19:00:38 <lightningbot> Useful Commands: #action #agreed #help #info #idea #link #topic.
 71 2015-11-19 19:01:16 <wumpus> ok, any topics?
 72 2015-11-19 19:01:18 <sipa> suggested topic: priority clarification
 73 2015-11-19 19:01:42 <wumpus> #topic priority clarification
 74 2015-11-19 19:01:44 <sipa> suggested topic: dealing with mempool eviction
 75 2015-11-19 19:02:01 <sipa> so it wasn't entirely clear to me after last weekend what the plan was wrt priority
 76 2015-11-19 19:02:45 <sipa> but imho, either we stop supporting it for tx creation entirely (meaning estimatepriority etc go away)
 77 2015-11-19 19:02:48 <sipa> or it keeps working
 78 2015-11-19 19:03:20 <sipa> which also means a mempool area for priority, or there's no way to keep wallet-created priority transactions in
 79 2015-11-19 19:03:23 <morcos> sipa: i am ok with either.  i vote for stop supporting it for tx creation entirely.
 80 2015-11-19 19:04:14 <morcos> i feel like we will keep going round and round on this issue release after release unless we finally start pulling off the bandaid
 81 2015-11-19 19:04:28 <morcos> if we develop a better framework for supporting a similar type metric we can add it back
 82 2015-11-19 19:04:47 <morcos> but we waste too much time debating it.
 83 2015-11-19 19:05:04 <wumpus> action point for that for last week re: priority was to look at https://github.com/bitcoin/bitcoin/pull/6134
 84 2015-11-19 19:05:47 <jgarzik> it sounded like luke-jr was the only one opposed to ceasing support for it
 85 2015-11-19 19:06:12 <btcdrak> morcos: we should pull the bandaid now imo
 86 2015-11-19 19:06:12 <wumpus> he's also the person that introduced the priority deltas IIRC
 87 2015-11-19 19:06:27 <morcos> wumpus: yes #6134 is required either way.  if we decide to start including priority support, we can make a simple modification to that pull to return the estimate maxed with mempools min priority instead of infinity
 88 2015-11-19 19:06:47 <morcos> i think other people (including myself) think the notion has some benefit
 89 2015-11-19 19:07:17 <morcos> but its a matter of introducing additional complicatin at this point to continue supportin git
 90 2015-11-19 19:08:21 <gmaxwell> I've decided to stop disliking the complete removal of priority.
 91 2015-11-19 19:08:40 <jtimon> I won't insist on this, but everything (including 6134) would just be simpler without having to worry about the priority code
 92 2015-11-19 19:08:44 <CodeShark> I never really liked it in the first place
 93 2015-11-19 19:08:44 <jgarzik> yay!
 94 2015-11-19 19:09:08 <BlueMatt> yay!
 95 2015-11-19 19:09:13 <btcdrak> do it
 96 2015-11-19 19:09:36 <jtimon> iirc Luke-Jr wasn't so happy about removing it, but not now nor eventually
 97 2015-11-19 19:09:39 <BlueMatt> wumpus: we have fee deltas that work in the same way, though
 98 2015-11-19 19:10:22 <wumpus> yes, we can still support a similar delta
 99 2015-11-19 19:10:33 <sipa> i'd still like to try adding a "count very small fraction of bitcoin-days-destroyed as extra fee"
100 2015-11-19 19:10:46 <wumpus> txfee rate delta
101 2015-11-19 19:10:57 <morcos> If this is the path we follow (which I agree with), then I would suggest for the mining code we use the lowerBoundPriority concept from 7008 for 0.12.
102 2015-11-19 19:10:57 <sipa> wumpus: fee delta
103 2015-11-19 19:11:09 <CodeShark> clarification: I never really liked the priority code - I only disliked removing it because it was too tangled up
104 2015-11-19 19:11:29 <sipa> morcos: sounds good to me
105 2015-11-19 19:11:30 <BlueMatt> sipa: I think we can only do that if there is a very low cap
106 2015-11-19 19:11:33 <BlueMatt> at which point its useless
107 2015-11-19 19:11:35 <BlueMatt> so....why?
108 2015-11-19 19:11:42 <sipa> BlueMatt: i know
109 2015-11-19 19:11:46 <sipa> BlueMatt: i said try
110 2015-11-19 19:11:52 <BlueMatt> fair enough
111 2015-11-19 19:12:25 <wumpus> ok - next topic?
112 2015-11-19 19:12:33 <Luke-Jr> I will discourage all miners from upgrading if priority is removed.
113 2015-11-19 19:12:43 <sdaftuar> so do we actually need to modify 6134?  i think it's fine in it's current form.  we -could- take out futher priority support, but i think it works pretty well as is.
114 2015-11-19 19:12:55 <btcdrak> suggested topic: merge #6312 sequence numbers
115 2015-11-19 19:12:56 <sipa> Luke-Jr: it's not removed from mining, only from the wallet
116 2015-11-19 19:12:59 <Luke-Jr> sipa: ok
117 2015-11-19 19:13:13 <sipa> Luke-Jr: though you'll need a sufficiently large mempool to keep priority txn in
118 2015-11-19 19:13:14 <Luke-Jr> that seems acceptable, since nobody is really maintaining the wallet
119 2015-11-19 19:13:21 <morcos> sdaftuar: I think if we remove priority from other tx creation places it'll be equivalently easy to move in from 6134
120 2015-11-19 19:13:42 <morcos> i'd merge 6134 first (after it gets reviewed) and then let people remove priority
121 2015-11-19 19:13:47 <morcos> but 6134 has important fee estimation changes
122 2015-11-19 19:13:51 <sdaftuar> morcos: ok that makes sense to me
123 2015-11-19 19:14:03 <jtimon> Luke-Jr: my offer to help adapting any codebase to a non-priority future still stands (even/spcially if you think that case it's "impossible")
124 2015-11-19 19:14:06 <sipa> wumpus: suggested topic: dealing with evicted wallet transactions
125 2015-11-19 19:14:07 <gmaxwell> sipa: I don't dislike that "count as extra fee", though I think morcos convinced me that it won't make the big improvement I thought it would. (and if we were going to bias a bit it could be utxo destroyed)
126 2015-11-19 19:14:10 <BlueMatt> morcos: i thought this was always the plan?
127 2015-11-19 19:14:52 <sipa> gmaxwell: he convinced me too... but... numbers after trying say more
128 2015-11-19 19:15:53 <wumpus> #topic dealing with mempool eviction
129 2015-11-19 19:16:22 <sipa> so: problem: currently when a wallet transaction is evicted, the wallet considers the resulting transaction as "conflicting" and will happily respend the inputs
130 2015-11-19 19:16:43 <BlueMatt> add a timeout, done
131 2015-11-19 19:16:48 <gmaxwell> We knew when we added the conflicting stuff using the mempool that it would be problematic later. (I checked logs)
132 2015-11-19 19:16:51 <sipa> suggested solution: only when accepttomemorypool fails due to non-existing inputs does the wallet treat the result as conflicting
133 2015-11-19 19:16:53 <BlueMatt> makes the wallet shitty, but we need rbf
134 2015-11-19 19:16:58 <wumpus> I think that's the wrong behavior, it shouldn't automatically regard transactions that you created yourself as respendable
135 2015-11-19 19:17:17 <BlueMatt> sipa: it should consider it respendable at some point later on, though
136 2015-11-19 19:17:23 <jtimon> what's the definition of an evicted tx?
137 2015-11-19 19:17:35 <Luke-Jr> the wallet code needs to understand when transactions are *actually* conflicted
138 2015-11-19 19:17:39 <Luke-Jr> fix that and the rest just works
139 2015-11-19 19:17:40 <wumpus> just add a way to manually remove transactions if the user doesn't care aout it anymore
140 2015-11-19 19:17:41 <sipa> + add a removewallettransaction function (GUI/RPC) which removes unconfirmed transactions (the GUI version can warn if it's still in the mempool at that point)
141 2015-11-19 19:18:08 <sipa> Luke-Jr: that corresponds exactly (afaik) to failing with missing inputs
142 2015-11-19 19:18:18 <gmaxwell> the 'remove' should perhaps not remove but tag as removed. I am uncomfortable with anything that destroys records.
143 2015-11-19 19:18:26 <jgarzik> That will be a big usability improvement for Bitcoin Core users...
144 2015-11-19 19:18:26 <wumpus> Luke-Jr: that would be good, too, but I don't like anything involving a timeout
145 2015-11-19 19:18:35 <jgarzik> rename if not remove
146 2015-11-19 19:18:42 <jonasschnelli> removewallettransaction: tag as removed and remove it withing the next x hours when no confirmation come in?
147 2015-11-19 19:18:47 <jonasschnelli> *comes
148 2015-11-19 19:18:52 <Luke-Jr> wumpus: I agree, a timeout would not fix the logic here. And even if we added a timeout, we would still need to fix the logic.
149 2015-11-19 19:18:52 <wumpus> gmaxwell: well as long as it's no longer visible
150 2015-11-19 19:18:58 <gmaxwell> jonasschnelli: any reason to remove vs move to another list?
151 2015-11-19 19:19:04 <jgarzik> s/remove/store it as archived but invisible/
152 2015-11-19 19:19:09 <gmaxwell> archive would be fine.
153 2015-11-19 19:19:10 <petertodd> sipa: rather than remove, maybe have a atomic thing to force acceptance of a double-spend?
154 2015-11-19 19:19:16 <wumpus> as long as it's *effectively* removed
155 2015-11-19 19:19:22 <jgarzik> archivewallettransactions
156 2015-11-19 19:19:32 <petertodd> sipa: remove could give the impression the tx can't be mined in the future...
157 2015-11-19 19:19:45 <wumpus> I mean people use -zapwallettxes now, this deletes all unconfirmed transactions
158 2015-11-19 19:19:46 <Luke-Jr> if we don't have a "is this transaction conflicted" function, it probably isn't hard to write one..
159 2015-11-19 19:19:46 <sipa> maybe we need something separate that just marks a tx as respendable
160 2015-11-19 19:20:03 <gmaxwell> yes, archive then, and effectively removes it.   I'm concerned that either will be seen as canceled but we could allow archival only once confirmed or conflicted?
161 2015-11-19 19:20:03 <Luke-Jr> might be more expensive though, if we're not careful
162 2015-11-19 19:20:20 <gmaxwell> lets perhaps not do design in the meeting (/me guilty look)
163 2015-11-19 19:20:52 <jonasschnelli> should we agree on adding a function (RPC/GUI) that can remove/archive an transaction? specs. done later?
164 2015-11-19 19:20:54 <jtimon> sipa: yeah, like an opt-in RBF or something but for non-respendable transactions...
165 2015-11-19 19:21:22 <sipa> well the important part is being able to mark a transaction as respendable
166 2015-11-19 19:21:32 <sipa> whether that also archives/deletes/hides... is extra functionality
167 2015-11-19 19:21:48 <petertodd> giveupontx
168 2015-11-19 19:21:55 <jtimon> yeah, sorry the important part is to free the inputs
169 2015-11-19 19:22:26 <jtimon> freeinputstransaction ?
170 2015-11-19 19:22:30 <gmaxwell> indeed, it shouldn't be silent because you may need to resend the payment.
171 2015-11-19 19:22:30 <wumpus> bleh
172 2015-11-19 19:22:41 <gmaxwell> archive sounded fine to me.
173 2015-11-19 19:22:47 <petertodd> jtimon: free is way too overloaded for this space :)
174 2015-11-19 19:22:53 <sipa> bitcoin wallet for android has a "respend with higher fee" function
175 2015-11-19 19:22:58 <sipa> i think
176 2015-11-19 19:22:59 <gmaxwell> also archiving transactions could (eventually) improve performance of listtransactions and such.
177 2015-11-19 19:22:59 <jtimon> petertodd: fair enough
178 2015-11-19 19:23:16 <Luke-Jr> sipa: that would be a nice feature regardless of removal
179 2015-11-19 19:23:22 <wumpus> I would like an option to just forget about a transaction completely - but yes, that's unrelated to the mempool eviction thing
180 2015-11-19 19:23:24 <sipa> but we need a minimal viable idea for 0.12 (as morcos said)
181 2015-11-19 19:23:40 <jtimon> yeah respendtransaction seems cooler than archivetransaction
182 2015-11-19 19:24:36 <Luke-Jr> minimum viable idea IMO is to fix the "isconflicted" method
183 2015-11-19 19:24:50 <Luke-Jr> whatever we do otherwise, we still need that fixed
184 2015-11-19 19:24:51 <gmaxwell> do we have other topics (we can discuss this more later.   I think minimum viable is what sipa suggests.
185 2015-11-19 19:24:54 <sipa> Luke-Jr: that means you still get coins that may be locked up for infinity
186 2015-11-19 19:25:01 <gmaxwell> that we detect conflict instead of non-mempooling.
187 2015-11-19 19:25:08 <jtimon> but archivetransaction and then spend seems equivalent (assuming you can chose one of the inputs from the archived transaction)
188 2015-11-19 19:25:13 <Luke-Jr> sipa: yes, that's not a regression
189 2015-11-19 19:25:45 <sipa> Luke-Jr: it's still a regression wrt 0.11, where wallet transactions wouldn't be kicked out of the mempool until they were actually conflicted
190 2015-11-19 19:25:54 <gmaxwell> sipa: luke means the and-immediately-respendable, part as we have now.
191 2015-11-19 19:26:00 <sipa> ok
192 2015-11-19 19:26:18 <gmaxwell> sipa: luke is saying we detect actual conflict instead of non-mempoolability and leave the coins immediately respendable.
193 2015-11-19 19:26:21 <sipa> no other suggested topicks afaik
194 2015-11-19 19:26:28 <wumpus> I don't think we have any other topics
195 2015-11-19 19:26:31 <gmaxwell> Which I agree is the essential first step.
196 2015-11-19 19:26:35 <btcdrak> suggested topic: sequence numbers..
197 2015-11-19 19:26:48 <jonasschnelli> suggested topic: bdb replacement
198 2015-11-19 19:26:54 <morcos> gmaxwell: important though that if we add that step, we note we've now locked up inputs that perviously would have been freed
199 2015-11-19 19:27:05 <sipa> yes, that was my point too ^
200 2015-11-19 19:27:06 <Luke-Jr> jonasschnelli: that's a Core-specific matter
201 2015-11-19 19:27:06 <morcos> also given the tight deadlines, hopefully somebody volunteers to work on this
202 2015-11-19 19:27:07 <jgarzik> I like both those topics :)
203 2015-11-19 19:27:22 <jgarzik> Luke-Jr, we handle plenty of Core-specific stuff in this meeting
204 2015-11-19 19:27:22 <Luke-Jr> (also, FWIW I need to go in 4 minutes.)
205 2015-11-19 19:27:25 <jonasschnelli> Luke-Jr : agreed.
206 2015-11-19 19:27:42 <wumpus> #topic sequence numbers
207 2015-11-19 19:28:08 <btcdrak> what's left to get this merged?
208 2015-11-19 19:29:01 <sipa> btcdrak: why do we need it merged? we need to wait until BIP113 is deployed as standardness so 68/112/113 can go in a softfork
209 2015-11-19 19:29:05 <gmaxwell> because of soft fork feature binding I think I want to move in a direction where we don't soft fork in major versions.
210 2015-11-19 19:29:12 <jtimon> I just posted a little nit to sipa's latest commit
211 2015-11-19 19:29:27 <jgarzik> As mentioned in one of the PRs, my biggest concern was nascent projects already using sequence numbers, and this would introduce a new behavior into the field
212 2015-11-19 19:29:28 <gmaxwell> but we should still be moving this code to maturity.
213 2015-11-19 19:29:30 <CodeShark> sipa: I think he's just sick of rebasing it :)
214 2015-11-19 19:29:43 <btcdrak> BIP113 standardness already is being mined by 36% of the network and rising fast
215 2015-11-19 19:30:38 <sipa> gmaxwell: absolutely
216 2015-11-19 19:30:41 <jtimon> well, merging bip68, would also make bip112 esier to review
217 2015-11-19 19:30:43 <petertodd> btcdrak: what do you mean there?
218 2015-11-19 19:30:43 <sipa> CodeShark: understably!
219 2015-11-19 19:30:59 <btcdrak> *bad language, I mean, 36% of miners so far appear to be applying policy
220 2015-11-19 19:31:05 <petertodd> btcdrak: ah, thanks
221 2015-11-19 19:31:29 <sipa> CodeShark: that's a portmonteau of "understandably" and "stable"
222 2015-11-19 19:31:42 <CodeShark> :)
223 2015-11-19 19:32:14 <btcdrak> I would like to see bip68 and 112 merged, then we dont have to worry about the PRs rotting or keeping up with all refactoring.
224 2015-11-19 19:32:18 <gmaxwell> going back to my earlier comment, 68/112 could go in as standardness rules though (113 already is)
225 2015-11-19 19:32:33 <gmaxwell> if we feel they are sufficiently reviewed and mature.
226 2015-11-19 19:32:46 <btcdrak> cltv sat merged for 11 months before we deployed the softfork.
227 2015-11-19 19:32:46 <jgarzik> downstream payment channels code -does- make use of lower sequence numbers, but that is usually pre-broadcast
228 2015-11-19 19:32:55 <sipa> btcdrak: we'll need to rebase it for backports anyway... no need to keep it up the whole time
229 2015-11-19 19:33:06 <sipa> (though if people ack it, it should go in, of course)
230 2015-11-19 19:33:07 <jgarzik> +1 for standardness rules
231 2015-11-19 19:33:08 <gmaxwell> btcdrak: do we have a summary of final gripes about the design of them?
232 2015-11-19 19:33:37 <btcdrak> gmaxwell: well as of today, as I understood it, sipa patched it according to his preference.
233 2015-11-19 19:33:42 <gmaxwell> It's also the case that we can add them as unused code even without the standardness rule.
234 2015-11-19 19:33:51 <btcdrak> so to my knowledge it's good to go.
235 2015-11-19 19:34:13 <jgarzik> +1 on standardness roll out
236 2015-11-19 19:34:25 <gmaxwell> btcdrak: sipa's gribes were implementation related, not the protocol design. If there are no gripes about the protocol design we haven't looked hard enough, there always should be some. :)
237 2015-11-19 19:34:59 <CodeShark> or we've just given up on doing anything about them :p
238 2015-11-19 19:35:06 <sipa> i have little opinion about bip68 itself; if people feel it's good, it's fine
239 2015-11-19 19:35:07 <gmaxwell> We should perhaps call for gripes on the design. I believe petertodd had some but they were really mild.
240 2015-11-19 19:35:24 <gmaxwell> CodeShark: well thats the point, its fine to go forward with residual gripes but we should know that we are and do so intentionally.
241 2015-11-19 19:36:01 <jtimon> well, even as unused code I think would simplify things, but I still don't see the "wait for bip113 to be widely used to merge bip68 as a policy thing in master"
242 2015-11-19 19:36:20 <jtimon> we need the backports for consensus, not for policy
243 2015-11-19 19:36:35 <morcos> but we all agree we don't need the policy before the soft fork right?
244 2015-11-19 19:36:36 <CodeShark> I personally don't really care how relative locktime is deployed anymore as long as the functionality is there and I can wrap it in my code so that I don't have to look at the ugliness :p
245 2015-11-19 19:36:36 <sipa> jtimon: bip68 is already nonstandard
246 2015-11-19 19:36:42 <morcos> it might not hurt, but its not required
247 2015-11-19 19:37:09 <gmaxwell> Seperate concerns, we should not isstandard enforce unless we're reasonably confident the protocol design will not change.
248 2015-11-19 19:37:22 <gmaxwell> morcos: right we don't need policy before the soft fork, but it can be better than merging dead code.
249 2015-11-19 19:37:41 <gmaxwell> and allows more (insecure) testing of the protocol.
250 2015-11-19 19:38:12 <jtimon> sipa: ??
251 2015-11-19 19:38:19 <sipa> jtimon: it needs v2 transactions
252 2015-11-19 19:38:26 <sipa> jtimon: which are nonstandard
253 2015-11-19 19:38:55 <gmaxwell> Okay so I think we should try to see if we can find any gripes, and move towards merging, and turn on as standard after next meeting if the gripes we can find are not convincing at all.
254 2015-11-19 19:38:56 <jtimon> but what is exactly the harm if it gets merged before bip113 is widely used?
255 2015-11-19 19:39:00 <jgarzik> Related - any opinion on reserving bits for TX versions, like version bits does for blocks?
256 2015-11-19 19:39:03 <morcos> wait wait
257 2015-11-19 19:39:04 <sipa> jtimon: nothing
258 2015-11-19 19:39:11 <morcos> sorry i haven't looked at BIP68 in a while
259 2015-11-19 19:39:25 <morcos> but the text does not reflect what i thought was the agreed upon semantics, i haven't checked the code
260 2015-11-19 19:39:27 <gmaxwell> jgarzik: the widely used comment was we cannot softfork 113 until 113 is desployed as isstandardness (well we can but it's preferrable to not)
261 2015-11-19 19:39:44 <sipa> jtimon: i'm saying we don't need a long wait between deploying bip68 as standardness and doing a softfork for it (like with bip113)
262 2015-11-19 19:39:50 <gmaxwell> morcos: see. okay thats fine, so lets address that.
263 2015-11-19 19:39:51 <jtimon> sipa: I got confused with this "we need to wait until BIP113 is deployed as standardness so 68/112/113 can go in a softfork"
264 2015-11-19 19:40:03 <morcos> i thought there was going to be 512 second resolution on the time part
265 2015-11-19 19:40:15 <morcos> thats the last thing maaku discussed before leaving the project
266 2015-11-19 19:40:16 <sipa> jtimon: bip113 needs to be deployed as standardness rule before we can softfork it, to make sure nothing breaks
267 2015-11-19 19:40:32 <morcos> i'm not sure whats in the code, but the BIP needs to be what we all agree on before we even talk about merging the code
268 2015-11-19 19:40:46 <jtimon> sipa: yeah thanks now I get what you meant
269 2015-11-19 19:40:52 <gmaxwell> morcos: yup yup. okay, so we need to take a spin on that.
270 2015-11-19 19:41:30 <sipa> seconds_offset is 5
271 2015-11-19 19:41:39 <sipa> so time is specified in a multiple of 30 seconds?
272 2015-11-19 19:41:45 <sipa> eh, 32
273 2015-11-19 19:41:46 <morcos> the code appears to have the 512 seconds
274 2015-11-19 19:42:12 <morcos> but i refuse to even review the code until the BIP is right, how can you check if the code properly implements the BIP if they describe different things
275 2015-11-19 19:42:22 <sipa> ok, so look at the bip
276 2015-11-19 19:42:59 <morcos> this was the same complaint i had a couple of months ago and i just blew off continuing to look at it because everytime i checked the BIP and the code were describing different things and iw anted to wait for the design to settle down
277 2015-11-19 19:43:07 <morcos> i'm not sure if the design has settled or not
278 2015-11-19 19:43:16 <morcos> and if it has settled, can someone tell me what it settled on
279 2015-11-19 19:43:27 <sipa> it certainly hasn't changed in a long time, and afaik, there are also no other outstanding concerns
280 2015-11-19 19:43:31 <morcos> sorry for getting a bit worked up about this... but why do we keep talking about merging something
281 2015-11-19 19:43:40 <morcos> what hasn't changed in a long time?
282 2015-11-19 19:43:47 <sipa> bip68
283 2015-11-19 19:43:57 <morcos> does it match the code, maybe i just read it wrong?
284 2015-11-19 19:44:06 <jtimon> morcos: only implementation details have changed recently
285 2015-11-19 19:44:23 <jtimon> morcos: if it doesn't please say so
286 2015-11-19 19:44:48 <sipa> it doesn't match
287 2015-11-19 19:44:56 <sipa> i was not aware
288 2015-11-19 19:45:30 <btcdrak> I thought maaku had updated that. The PR has the right details. In any case, this is minor to update.
289 2015-11-19 19:45:30 <sipa> bip uses shift 5, implementing shift 9
290 2015-11-19 19:45:40 <sipa> well, that's not good!
291 2015-11-19 19:45:43 <jtimon> I also got lost in the time resolution discussion so I assumed the code would contain whatever was decided
292 2015-11-19 19:45:47 <wumpus> good to discover that at least before merging...
293 2015-11-19 19:45:51 <morcos> sipa, no, bip uses shift 5, implementing is shift 5+9 with granularity 9
294 2015-11-19 19:46:08 <morcos> i think.
295 2015-11-19 19:46:33 <sipa> so does anyone even know what "the plan" is?
296 2015-11-19 19:46:36 <morcos> oh maybe i'm wrong
297 2015-11-19 19:46:43 <morcos> i assume the code is "the plan"
298 2015-11-19 19:46:53 <btcdrak> what do you mean "the plan". The code is the correct specification.
299 2015-11-19 19:47:00 <jtimon> morcos: let's assume they have to say the same thing
300 2015-11-19 19:47:03 <morcos> but "the plan" should be described in the BIP so we can all decide if we agree on it
301 2015-11-19 19:47:17 <gmaxwell> yes, BIP must be correct.
302 2015-11-19 19:47:25 <btcdrak> It got chanegd so much according to people nits regarding the spec. The last change was the granularity so that BIP68 could be supported on chains with faster blocktimes
303 2015-11-19 19:47:26 <sipa> btcdrak: then what's the point of having a BIP?
304 2015-11-19 19:47:34 <gmaxwell> (or rather there at least must be no known errors in the thing, even if the code is normative!)
305 2015-11-19 19:47:37 <morcos> jtimon: i just meant the current notion of the plan is the same as whats in the code right now, i think
306 2015-11-19 19:47:48 <gmaxwell> also the code is only normative once its network consensus. :P
307 2015-11-19 19:47:50 <btcdrak> I will edit the bip, it's just an oversight, I thought it has been updated by maaku
308 2015-11-19 19:48:07 <morcos> yeah but this is important to get right
309 2015-11-19 19:48:11 <jtimon> morcos: then the specification must be corrected, no?
310 2015-11-19 19:48:53 <morcos> sipa, i think the code and the BIP are actually completely different
311 2015-11-19 19:49:07 <sipa> why is the SECONDS_FLAG = (1 << 22)?
312 2015-11-19 19:49:08 <morcos> the code looks like the low order 16 bits describe the number of 512 second units
313 2015-11-19 19:49:11 <sipa> where does that number come from
314 2015-11-19 19:49:43 <morcos> my numbers, i don't know.... i'm quickly skimming the code now, probably not the best use of everyones time
315 2015-11-19 19:49:55 <sipa> no, i'm looking at the code
316 2015-11-19 19:50:13 <morcos> yeah thats why we need a BIP
317 2015-11-19 19:50:18 <wumpus> anyhow, so the code and BIP looked to be good at, it's certainly not ready yet
318 2015-11-19 19:50:20 <btcdrak> sipa: there was a long conversation months about about conserving as many bits in nsequence as possible
319 2015-11-19 19:50:35 <morcos> right but the conserved bits are kind of all over the place now
320 2015-11-19 19:50:36 <jgarzik> +1 wumpus
321 2015-11-19 19:50:46 <petertodd> wumpus: ack
322 2015-11-19 19:50:49 <btcdrak> wumpus: no I disagree. the code is ready, the BIP needs to be syncronised. It wont take an hour
323 2015-11-19 19:50:53 <morcos> which might be ok, but it needs to be clearly described
324 2015-11-19 19:51:06 <morcos> btcdrak: this has to be NACK for 0.12.  this is crazy
325 2015-11-19 19:51:22 <petertodd> NACK for v0.12.0 doesn't mean a NACK for v0.12.1 after all
326 2015-11-19 19:51:27 <sipa> petertodd: +1
327 2015-11-19 19:51:29 <wumpus> petertodd: right
328 2015-11-19 19:51:38 <btcdrak> morcos: sorry, I disagree. You are making a big fuss over an administration error
329 2015-11-19 19:51:39 <morcos> we have to agreed on the design specification first.  some people maybe think they were agreeing on the BIP and soe think they were agreein gon the code
330 2015-11-19 19:51:51 <gmaxwell> in any case, we really can't merge without the bip being believed consistent-- I agree.
331 2015-11-19 19:51:52 <btcdrak> morcos: we already agreed the design specification.
332 2015-11-19 19:51:52 <morcos> i agree i'm making a big fuss :)
333 2015-11-19 19:51:54 <sipa> btcdrak: this is not an administration error. people don't know what the proposal or its intent is
334 2015-11-19 19:52:14 <gmaxwell> btcdrak: I'm sure many people only read the
335 2015-11-19 19:52:15 <gmaxwell> BIP
336 2015-11-19 19:52:17 <btcdrak> sipa: I'll dig out the mailing list posts, it was agreed. discussed on the IRC meets too
337 2015-11-19 19:52:19 <gmaxwell> and other people only read the code.
338 2015-11-19 19:52:36 <morcos> i'm not blaming anyone btw... we all want this functionality merged.  and you're the only one thats even stepping up to the plate at all, so don't get me wrong
339 2015-11-19 19:52:38 <wumpus> #action look closely at BIP68 and make sure it matches the implementation #6312
340 2015-11-19 19:52:47 <gmaxwell> and other people don't know what they agreed to. Which is part of why I was suggesting another round of gripe gathering, but that can't happen unless we think the code and bip agree.
341 2015-11-19 19:52:50 <morcos> but i think we have to be a little bit more careful with this stuff
342 2015-11-19 19:52:55 <gmaxwell> wumpus: ya. thanks.
343 2015-11-19 19:53:37 <petertodd> fwiw, I'm going to be doing some python demo's of CSV functionality, which will help get clarity that it all works as expected
344 2015-11-19 19:53:44 <gmaxwell> Currently I don't know the actual behavior, I stopped looking when it kept changing. (Though I doubt I have an opinion about those details if they've already satisfied multiple people)
345 2015-11-19 19:53:51 <jonasschnelli> petertodd: +1!
346 2015-11-19 19:53:52 <gmaxwell> petertodd: that will help.
347 2015-11-19 19:53:56 <jgarzik> yep
348 2015-11-19 19:53:59 <morcos> i'll try to go back and see what the last thing maaku and i discussed on IRC is, and hopefully that matches the code
349 2015-11-19 19:54:10 <wumpus> good idea petertodd
350 2015-11-19 19:54:22 <petertodd> wumpus: btcdrak's idea :)
351 2015-11-19 19:54:39 <jgarzik> 330 seconds left
352 2015-11-19 19:54:47 <gmaxwell> Do we have any versionbits things to discuss?  I prodded for some start time (due to BIP65 expirence), and progress was made
353 2015-11-19 19:54:52 <jtimon> suggested topic: optin RBF
354 2015-11-19 19:55:15 <gmaxwell> Is there anything left to say on optin RBF? I think it's mostly just code pipeline right now.
355 2015-11-19 19:55:33 <petertodd> jtimon: not much to say there other than I posted a think to the dev list announcing it and got no responses
356 2015-11-19 19:55:34 <jtimon> gmaxwell: fair enough
357 2015-11-19 19:55:46 <gavinand1esen> are any wallet devs onboard with optin RBF?  Any actively participating in spec’ing ?
358 2015-11-19 19:55:50 <gmaxwell> We were GO last week on it, Rusty was confused about the condition and suggested some description change.
359 2015-11-19 19:56:00 <gmaxwell> gavinand1esen: yes, see the pull req.
360 2015-11-19 19:56:07 <wumpus> #topic optin RBF
361 2015-11-19 19:56:20 <jtimon> yea, I was more like asking what was missing for it to get merged?
362 2015-11-19 19:56:28 <jtimon> more review maybe?
363 2015-11-19 19:56:35 <gmaxwell> Greenaddress is all over it, and I believe we intend to support replacement in core once its deployed.
364 2015-11-19 19:56:57 <gmaxwell> Greenaddress I think also did FSS RBF support before or was working on it, but found it not very useful.
365 2015-11-19 19:57:04 <petertodd> jtimon: mostly it has concept or utACKs; another full ACK would be nice
366 2015-11-19 19:57:17 <sipa> i haven't looked at the code yet; i will
367 2015-11-19 19:57:21 <petertodd> sipa: thanks!
368 2015-11-19 19:57:36 <wumpus> #action #6871 (nSequence-based Full-RBF opt-in) needs more review and testing
369 2015-11-19 19:57:46 <sipa> (though upcoming mempool changes may cause it to need rebase, and simplify?)
370 2015-11-19 19:57:49 <petertodd> https://github.com/petertodd/replace-by-fee-tools#incremental-send-many <- testing tools
371 2015-11-19 19:58:01 <jtimon> sipa or viceversa?
372 2015-11-19 19:58:19 <sipa> jtimon: removing priority should only simplify things i think :)
373 2015-11-19 19:58:20 <wumpus> although I think it looks pretty well already
374 2015-11-19 19:58:27 <petertodd> sipa: why would it simplify? it's independent of priority
375 2015-11-19 19:58:32 <jtimon> sipa: sure
376 2015-11-19 19:58:43 <sipa> petertodd: i added a question mark
377 2015-11-19 19:58:50 <sipa> jtimon: oh, but nvm, only wallet
378 2015-11-19 19:58:52 <jtimon> sipa: just no need to plan what comes first
379 2015-11-19 19:59:02 <sdaftuar> sipa: i'm not aware of anything coming up that would affect it fwiw
380 2015-11-19 19:59:09 <sipa> sdaftuar: ok!
381 2015-11-19 19:59:13 <sipa> that's good to know
382 2015-11-19 20:00:01 <sipa> *BANG*
383 2015-11-19 20:00:04 <sdaftuar> er.  i might have spoken too soon.
384 2015-11-19 20:00:10 <sipa> too late
385 2015-11-19 20:00:12 <gmaxwell> #doommeeting
386 2015-11-19 20:00:17 <jonasschnelli> logo
387 2015-11-19 20:00:20 <jonasschnelli> lol
388 2015-11-19 20:00:23 <gmaxwell> #rollcredits
389 2015-11-19 20:00:23 <lightningbot> Log:            http://www.erisian.com.au/meetbot/bitcoin-dev/2015/bitcoin-dev.2015-11-19-19.00.log.html
390 2015-11-19 20:00:23 <lightningbot> Meeting ended Thu Nov 19 20:00:23 2015 UTC.  Information about MeetBot at http://wiki.debian.org/MeetBot . (v 0.1.4)
391 2015-11-19 20:00:23 <lightningbot> Minutes:        http://www.erisian.com.au/meetbot/bitcoin-dev/2015/bitcoin-dev.2015-11-19-19.00.html
392 2015-11-19 20:00:23 <lightningbot> Minutes (text): http://www.erisian.com.au/meetbot/bitcoin-dev/2015/bitcoin-dev.2015-11-19-19.00.txt
393 2015-11-19 20:00:23 <wumpus> #endmeeting
394 2015-11-19 20:00:25 <sdaftuar> petertodd: can you take a look at 7062?
395 2015-11-19 20:00:47 <petertodd> sdaftuar: sure
396 2015-11-19 20:00:53 <jtimon> sipa: so what were you worried #6068 would affect?
397 2015-11-19 20:01:46 <sipa> jtimon: RBF, for example; that doesn't fit in that design
398 2015-11-19 20:02:11 <sdaftuar> petertodd: i need to re-review RBF but maybe it should use the GetModifiedFee() stuff I'm adding in 7062
399 2015-11-19 20:02:14 <jtimon> sipa: I'm pretty sure the rebase will be trivial
400 2015-11-19 20:02:39 <jtimon> what design?
401 2015-11-19 20:02:43 <petertodd> sdaftuar: what does GetModifiedFee() do?
402 2015-11-19 20:02:53 <sdaftuar> it takes into account any fee delta created using prioritisetransaction
403 2015-11-19 20:03:10 <btcdrak> re sequence numbers, the last change to the specification was this, requested by morcos and agreed by maaku https://lists.linuxfoundation.org/pipermail/bitcoin-dev/2015-October/011552.html
404 2015-11-19 20:03:45 <jtimon> sipa: #6871 and #6068 hardly affect each other
405 2015-11-19 20:03:50 <sdaftuar> the idea was to make the eviction code match miner policy
406 2015-11-19 20:04:14 <petertodd> sdaftuar: ah, yeah, that'd be good to use
407 2015-11-19 20:07:33 <sdaftuar> well, i'd be happy to fix up 7062 to update the function calls, i don't really think it should hold up merging 6871 (which has obviously gotten more review than the pull i just opened today)
408 2015-11-19 20:13:16 <morcos> sipa: btcdrak: see http://bitcoinstats.com/irc/bitcoin-dev/logs/2015/10/15 starting at 16:54 for discussion on the latest change to BIP 68
409 2015-11-19 20:14:02 <morcos> That was where maaku proposed using bit 23 for the seconds flag.  I didn't reread the whole discussion yet, but I believe it was my opinion at the time that any further discussion over bit locations was bikeshedding
410 2015-11-19 20:15:08 <morcos> so i have no reason to believe that the code isn't the spec that i thought made sense, but i haven't reviewed it yet, and the BIP is very out of date.   i think it makes sense to make the BIP what we want it to be and have everyone agree on that before we can merge policy
411 2015-11-19 20:16:17 <morcos> I think the BIP should include some reasoning about how future soft forks both within and without the BIP 68 paradigm would use the unused bits
412 2015-11-19 20:17:03 <morcos> maaku and I both got confused earlier in the process about whether things like increasing or decreasing granularity could be soft forks...
413 2015-11-19 20:20:53 <morcos> btcdrak: that irc conversation should give you some good material for the logic behind the choices for the BIP
414 2015-11-19 20:22:21 <btcdrak> morcos: I'll re-read it, my mind is a blank atm :)
415 2015-11-19 22:07:14 <Abstrct> Has anybody come across a practical limit to transaction size that they don't bother trying to broadcast? would a 130kB tx with a plentiful miner fee have any issues getting confirmed?
416 2015-11-19 22:14:37 <Prattler> Abstrct
417 2015-11-19 22:14:38 <Prattler> static const unsigned int MAX_STANDARD_TX_SIZE = 100000;
418 2015-11-19 22:14:54 <Prattler> /** The maximum size for transactions we're willing to relay/mine */
419 2015-11-19 22:15:02 <Prattler> that's the standard bitcoin-core client
420 2015-11-19 22:15:12 <Prattler> so most full nodes wouldn't relay tx over 100k
421 2015-11-19 22:15:19 <Prattler> 100 kB
422 2015-11-19 22:15:22 <Abstrct> perfect
423 2015-11-19 22:16:49 <Abstrct> Thanks for the prompt/accurate answer!
424 2015-11-19 22:17:11 <Prattler> glad to help :)