1 2015-06-18 04:34:20 <dermoth> hey there... I'm trying better understand a change in rpc protocol... in 10.2 going a getblockhash on non-existing block returns -8, while I think in 10.0 it returned -1. Anyone would be able to run it on 10.0 to confirm (or even better know the exact commit that changed that)??
  2 2015-06-18 04:35:57 <wumpus> dermoth: git bisect may help
  3 2015-06-18 04:36:42 <dermoth> i'm cloning the repo now... if i'm lucky git log -S<pattern> will save the day :)
  4 2015-06-18 04:37:03 <dermoth> that's called "pickaxe" fwiw
  5 2015-06-18 04:37:44 <wumpus> yes, that could help as well
  6 2015-06-18 04:38:23 <wumpus> RPC_INVALID_PARAMETER           = -8,  //! Invalid, missing or duplicate parameter
  7 2015-06-18 04:38:32 <wumpus> that's a strange error code to return for a non-existing block
  8 2015-06-18 04:38:39 <dermoth> https://github.com/bitcoin/bitcoin.git
  9 2015-06-18 04:38:41 <wumpus> are you providing a valid hash?
 10 2015-06-18 04:38:45 <dermoth> oops
 11 2015-06-18 04:38:47 <dermoth> 2ffdf21ce39fc3133fc
 12 2015-06-18 04:38:50 <dermoth> that's the commit
 13 2015-06-18 04:39:16 <dermoth> parameter is a block number, app used to rely on error -1 to know when it reached the most recent block
 14 2015-06-18 04:39:39 <wumpus> -1 is the general error, -8 is a more specialized one
 15 2015-06-18 04:39:44 <dermoth> I see
 16 2015-06-18 04:40:26 <gmaxwell> error: {"code":-8,"message":"Block height out of range"}
 17 2015-06-18 04:40:40 <wumpus> I think a NOT_FOUND code would be more appropriate than invalid parameter (which hints at a parse error),but don't be afraid I won't change it again...
 18 2015-06-18 04:40:59 <dermoth> agreed
 19 2015-06-18 04:41:11 <wumpus> at least the message is clear
 20 2015-06-18 04:41:24 <dermoth> lets' break all clients again :p
 21 2015-06-18 04:41:45 <wumpus> these RPC return codes evolved from a swamp of semi-randomly assigned codes in Satoshi's code base
 22 2015-06-18 04:41:48 <dermoth> all but mine, since i'll add both :)
 23 2015-06-18 04:42:16 <gmaxwell> a flag to randomize errors would not be totally pointless. :P
 24 2015-06-18 04:42:19 <wumpus> at some point I gave them names and put them into an enum RPCErrorCode, but there is very little consistency
 25 2015-06-18 04:42:20 <gmaxwell> -rpcfuzz=1
 26 2015-06-18 04:42:42 <dermoth> rofl
 27 2015-06-18 04:42:47 <gmaxwell> though probably 99% of the time you'd think to test with that, you probably also handled most of the cases in your code already.
 28 2015-06-18 04:42:47 <wumpus> hehe
 29 2015-06-18 04:43:10 <gmaxwell> (or you only think to test with it after it breaks... too late! :) )
 30 2015-06-18 04:43:32 <wumpus> heh, nah, could be solved with a simple python proxy script too.. just 10% of the time, don't proxy and return an error
 31 2015-06-18 04:46:35 <fanquake> cfields New mechanism to fetch the gitian signature works well. Good stuff.
 32 2015-06-18 04:47:35 <dermoth> wumpus, did you mean http not found (404) or adding a new RPC error?
 33 2015-06-18 04:47:50 <wumpus> dermoth: where?
 34 2015-06-18 04:47:59 <dermoth> https://github.com/bitcoin/bitcoin.git
 35 2015-06-18 04:48:04 <dermoth> damn copy paste :(
 36 2015-06-18 04:48:07 <dermoth> src/rpcprotocol.h
 37 2015-06-18 04:48:17 <wumpus> I'm not implying changing anything
 38 2015-06-18 04:48:40 <dermoth> well there is no NOT_FOUND code already
 39 2015-06-18 04:48:49 <dermoth> I mean yet
 40 2015-06-18 04:48:55 <wumpus> but yes, add a RPC_NOT_FOUND error, and use that. There's been more places where we could use one.
 41 2015-06-18 04:49:01 <wumpus> using HTTP 404 would be a layer violation
 42 2015-06-18 04:49:10 <dermoth> well excluding the RPC_METHOD_NOT_FOUND which is inappropriate
 43 2015-06-18 04:49:36 <wumpus> that one would be even worse, yes :)
 44 2015-06-18 04:49:56 <dermoth> closest error IMHO, in existing code, is RPC_INVALID_ADDRESS_OR_KEY
 45 2015-06-18 04:50:16 <wumpus> indeed. We've been using that as a NOT_FOUND.
 46 2015-06-18 04:50:45 <wumpus> maybe renaming it would be an idea
 47 2015-06-18 04:51:08 <wumpus> as said, those error names are back-projected by me, the original codebase used bare numbers without any explanation
 48 2015-06-18 04:52:34 <dermoth> would you keep the current one as an alias? I can submit a pr
 49 2015-06-18 04:57:44 <wumpus> yes, good idea
 50 2015-06-18 04:59:35 <dermoth> if (nHeight < 0... should be NOT FOUND too?
 51 2015-06-18 04:59:52 <dermoth> I see where the invalid param comes from, maybe split it up then
 52 2015-06-18 05:00:27 <wumpus> bleh.
 53 2015-06-18 05:00:47 <wumpus> see why this was never addressed? you lift it in one place, and notice the whole fabric is coming apart :-)
 54 2015-06-18 05:01:48 <wumpus> I mean I'm sure RPC_INVALID_ADDRESS_OR_KEY is also used in many cases where it really means RPC_INVALID_PARAMETER
 55 2015-06-18 05:01:56 <wumpus> before you know it, you're breaking the entire API
 56 2015-06-18 05:04:36 <wumpus> (and maybe that's for the best, but then it will be a fully fledged RPC return values redesign, and you would have accomplished exactly what you were coming here to complain about in the first place - hehe)
 57 2015-06-18 05:05:24 <wumpus> but maybe there's a middle way, I don't know, thanks for looking into it.
 58 2015-06-18 05:09:48 <dermoth> wumpus, do you want to command before I submit a PR? https://github.com/dermoth/bitcoin/commits/master
 59 2015-06-18 05:11:10 <wumpus> sure, I'll have a look
 60 2015-06-18 05:11:32 <dermoth> i'll try building, i'd be surprised If I have all I need the first time though
 61 2015-06-18 05:12:45 <wumpus> maybe try compiling with the alias commented out
 62 2015-06-18 05:13:12 <wumpus> that will turn up any uses of the old name
 63 2015-06-18 05:13:34 <dermoth> my command should have caught them all though
 64 2015-06-18 05:13:50 <dermoth> grep -R RPC_INVALID_ADDRESS_OR_KEY *|cut -d: -f1|uniq|xargs perl -pi -e 's/RPC_INVALID_ADDRESS_OR_KEY/RPC_NOT_FOUND/cg;'
 65 2015-06-18 05:13:52 <wumpus> yes. 'should'. would be just a sanity check
 66 2015-06-18 05:14:00 <wumpus> 'git grep' is even better for  that
 67 2015-06-18 05:15:15 <dermoth> nice, I should've known that
 68 2015-06-18 05:21:34 <dermoth> exactly what I guessed... first lib I install I need a specific version not packaged by debian... seems like building is going to be painful
 69 2015-06-18 05:22:50 <dermoth> is there a os flavor/version that would give me the right deps straight up? or a dev setup script/doc for the deps?
 70 2015-06-18 05:23:30 <wumpus> what dep gives difficulties? in principle it should just work with debian if you follow instructions in build-unix
 71 2015-06-18 05:26:50 <dermoth> configure: error: Found Berkeley DB other than 4.8, required for portable wallets (--with-incompatible-bdb to ignore or --disable-wallet to disable wallet functionality)
 72 2015-06-18 05:26:55 <wumpus> if bdb gives you trouble just pass --with-incompatible-bdb
 73 2015-06-18 05:27:01 <dermoth> k
 74 2015-06-18 05:27:31 <wumpus> (just be careful to move your wallet out of the way when testing, or use an alternative data directory, as wallets touched by newer bdb won' t be compatible with old bdb anymore)
 75 2015-06-18 05:28:40 <dermoth> It's been a long time I haven't stored any fund in a bitcoin-core wallet
 76 2015-06-18 05:28:55 <wumpus> no risk in that case
 77 2015-06-18 05:38:36 <dermoth> got libs sorted out, build time :)
 78 2015-06-18 05:44:43 <dermoth> well, take 2 with QT gui now (I'm lazy)
 79 2015-06-18 06:05:08 <dermoth> wumpus, tested ok
 80 2015-06-18 06:05:22 <dermoth> $ ~/bitcoin/bitcoin/bin/bitcoin-cli getblockhash -1
 81 2015-06-18 06:05:22 <dermoth> error: {"code":-8,"message":"Negative block height"}
 82 2015-06-18 06:05:28 <wumpus> great, time to submit a PR then
 83 2015-06-18 06:05:34 <dermoth> $ ~/bitcoin/bitcoin/bin/bitcoin-cli getblockhash 361428
 84 2015-06-18 06:05:34 <dermoth> error: {"code":-5,"message":"Block height out of range"}
 85 2015-06-18 06:05:42 <dermoth> 000000000000000004fd86600f6a12e5186ec18f11d7502afba3b7f7e00cae1a
 86 2015-06-18 06:05:42 <dermoth> $ ~/bitcoin/bitcoin/bin/bitcoin-cli getblockhash 361424
 87 2015-06-18 06:06:14 <dermoth> used the 10.2 client though, but was runnign the 0.11-blarhs-dirty one (dirty bc of the commented out alias)
 88 2015-06-18 06:06:37 <dermoth> v0.11.99.0-fa19502-dirty (64-bit)
 89 2015-06-18 06:08:22 <wumpus> using an older -cli is generally not a problem, it can cause misparsing for new commands, but has no effect on the output (which is just passed through)
 90 2015-06-18 06:21:44 <dermoth> https://github.com/bitcoin/bitcoin/pull/6300
 91 2015-06-18 06:57:38 <jonasschnelli> mempool limit: wouldn't it be sufficient to just add RBF and sweep everything which is older then 24h out of the mempool? DisconnectTip tx need to reset the 24h counter though. Or is there a reason to allow dozing tx >24h in the mempool?
 92 2015-06-18 07:00:40 <wumpus> I think the main reason for the mempool limiting would be to put up resource limits. Time-based expiry doesn't guarantee that...
 93 2015-06-18 07:02:06 <wumpus> aren't you effectievly describing jgarzik's poolman?
 94 2015-06-18 07:02:28 <jonasschnelli> wumpus: Right. Yes. But you could do the same thing with sweep oldest tx if mempool.size > mempool.sizelimit. Well, it could end up that you sweep relatively new transactions .
 95 2015-06-18 07:03:32 <jonasschnelli> wumpus: right. jgarzik poolman works like this.
 96 2015-06-18 07:03:35 <wumpus> sure, some criterion has to be chosen to expire transactions, e.g. time has been suggested a lot, but also priority
 97 2015-06-18 07:05:15 <wumpus> well if some safe default max size is chosen, e.g. 100MB, i wouldn't expect too many transactions to get dropped that would be hopeful to be mined in the first place - but yes, that's a risk that has caused this to be considered very  carefully
 98 2015-06-18 07:07:52 <jonasschnelli> wumpus: is it really crucial if you tx get mined and swept out of the mempool? That would probably mean it couldn't successfully compete with other tx. Therefor RBF would be a good thing to make sure you can rebroadcast your tx and add fee to gain a better chance of getting mined.
 99 2015-06-18 07:08:01 <jonasschnelli> tx *not* get mined
100 2015-06-18 07:08:16 <wumpus> at least with some kind of replace-by-fee it would be possible to retransmit transactions with a higher fee if they don't get mined
101 2015-06-18 07:08:21 <wumpus> so I think that is important
102 2015-06-18 07:08:52 <wumpus> (retransmitting transactions as-is doesn't help much, as some nodes can still have the transaction in the mempool so won't re-relay it)
103 2015-06-18 07:09:55 <jonasschnelli> wumpus: that's why i think something like replace-by-fee is important, this could give the transaction owner/sender the possibility to making sure that transaction gets re-relayed.
104 2015-06-18 07:10:25 <wumpus> yes, exactly. After that it's a matter of wallet UI
105 2015-06-18 07:32:12 <phantomcircuit> jonasschnelli, there's maybe some approach which takes into account the age of the transaction
106 2015-06-18 07:32:25 <phantomcircuit> im currently leaning towards nothing but fees
107 2015-06-18 07:32:36 <phantomcircuit> (well feerate rather)
108 2015-06-18 07:39:39 <s3gfault> I was about to issue a pull request to github, but I'm worried that Gavin will take Mike Hearn's advice and delete commit access for all the core devs.  Should I continue contributing to code on github, or will it be pointless when Mike Hearn and Gavin unilaterally delete all the other core devs and sabotage this open source project?  Please advise.
109 2015-06-18 07:41:08 <wumpus> s3gfault: if that happens, we'll happily continue somewhere else - this is a drama-free zone. Just keep on contributing :)
110 2015-06-18 07:42:05 <s3gfault> Thanks for the info. Will do.
111 2015-06-18 07:56:47 <jonasschnelli> phantomcircuit = pstratem ?
112 2015-06-18 08:09:19 <phantomcircuit> jonasschnelli, yes
113 2015-06-18 08:09:31 <jonasschnelli> okay.
114 2015-06-18 08:11:55 <phantomcircuit> jonasschnelli, the basic issue is that you need to solve two problems one of which is maybe impossible
115 2015-06-18 08:12:13 <phantomcircuit> avoid DoS issues related to transactions flapping
116 2015-06-18 08:12:37 <phantomcircuit> roughly model the priority calculation of miners
117 2015-06-18 08:12:58 <phantomcircuit> maybe the only way to do this is to completely come up with a new priority score :/
118 2015-06-18 08:13:11 <phantomcircuit> since it *should* take into account the age of the transaction
119 2015-06-18 08:13:19 <phantomcircuit> but that clearly shouldn't be the only factor
120 2015-06-18 08:14:16 <sipa> why should it take age into account?
121 2015-06-18 08:14:32 <jonasschnelli> phantomcircuit: Agreed. But from the wallet perspective, the mempool, chances of getting mined, is uncontrollable. It's a fee market.
122 2015-06-18 08:14:38 <phantomcircuit> sipa, if you fail to model the miners correctly age is a nice proxy for that
123 2015-06-18 08:15:28 <phantomcircuit> it's otherwise uninteresting
124 2015-06-18 08:16:41 <sipa> phantomcircuit: ignoring the priority area, fee is all that matters
125 2015-06-18 08:17:02 <jonasschnelli> phantomcircuit: but how can you model the priority calculations of miners. They can have complete different priorities (lets say some miners priorise tx from china).
126 2015-06-18 08:17:35 <jonasschnelli> sipa: agree with "fee is all that matters"
127 2015-06-18 08:17:48 <phantomcircuit> sipa, i intend to ignore the priority area as i dont expect it's a useful metric a 48 hour window
128 2015-06-18 08:18:06 <phantomcircuit> jonasschnelli, you cant thus using age as a proxy
129 2015-06-18 08:18:31 <jonasschnelli> phantomcircuit: because of the DoS mempool size?
130 2015-06-18 08:18:36 <phantomcircuit> jonasschnelli, priority(feerate, age)
131 2015-06-18 08:20:40 <jonasschnelli> from the wallet perspective, two things should happen: a) users need to see the fee before sending a tx, b) wallet needs to estimate fees for different levels, aprox. feerate for getting mined within the next block, approx. feerate for getting mined within next 3 blocks, etc.
132 2015-06-18 08:21:58 <jonasschnelli> and b is just a estimation, users should be aware of that fees are a market and therefore it can always happen that your tx get stuck. The wallet UI should repesent this in some way... and walle users should be able to increase fees if they see that it's tx get not confirmed.
133 2015-06-18 08:22:34 <phantomcircuit> jonasschnelli, default limit is 48 hours worth of full blocks
134 2015-06-18 08:22:45 <phantomcircuit> im not seeing anybody trying to estimate fees beyond that
135 2015-06-18 08:24:27 <jonasschnelli> for a estimation 48h of full blocks makes sense.
136 2015-06-18 08:24:47 <phantomcircuit> jonasschnelli, that's the default for the mempool limit
137 2015-06-18 08:25:05 <phantomcircuit> 288mB
138 2015-06-18 08:25:19 <jonasschnelli> As wallet users i would like to see the fee market visually, i could see my unconfirmed, tx, i see the mempools avg. fee rate, if the tx get not mined within the next block i can increase fees, etc.
139 2015-06-18 08:25:47 <sipa> mempool limit should be computed on actual memory usage, not on serialized size
140 2015-06-18 08:27:02 <phantomcircuit> sipa, is there already an estimate of that?
141 2015-06-18 08:27:55 <wumpus> it'd have to work similar as the estimation in CCoinsViewCache
142 2015-06-18 08:28:56 <sipa> OR
143 2015-06-18 08:29:07 <sipa> we could make the menpool store the serialized form
144 2015-06-18 08:29:10 <sipa> :)
145 2015-06-18 08:29:28 <sipa> which is around 8 times smaller on 64 bit platforms
146 2015-06-18 08:29:49 <sipa> the backreference index remains large, though
147 2015-06-18 08:29:50 <wumpus> that'd certainly save space - I wonder how much it would slow down queries against it, though
148 2015-06-18 08:30:03 <wumpus> right, an index
149 2015-06-18 08:30:27 <sipa> unsure how much we actually query it
150 2015-06-18 08:31:00 <wumpus> well at least for self-consistency, to find transactions to mine, and from the wallet
151 2015-06-18 08:32:40 <volante> hey devs.  fyi i'm working on a patch to change the mempool's mapTx into a boost::multi_index_container
152 2015-06-18 08:33:14 <wumpus> anyhow if the win for serialized is about 8x,  we could afford some extra indexing structures to speed up those, if necessary.
153 2015-06-18 08:33:50 <sipa> well mapPrev is still there
154 2015-06-18 08:33:58 <sipa> volante: cool
155 2015-06-18 08:34:26 <phantomcircuit> wumpus, iirc most queries are "is this transction in the mempool"
156 2015-06-18 08:36:56 <wumpus> phantomcircuit: and usually by txid - and sometimes by inputs, to check whether inputs already spent
157 2015-06-18 09:07:21 <midnightmagic> cfields: hello, is the 0.11.0rc2 signature tarball up somewhere?
158 2015-06-18 09:09:02 <wumpus> midnightmagic: yes, it's in a special repository now, you don't need to retrieve it manually anymore, just follow release-process.md :)
159 2015-06-18 09:18:55 <midnightmagic> wumpus: ah, right, thanks.
160 2015-06-18 09:23:15 <bengt_> ah, was just about to ask aswell, thanks
161 2015-06-18 10:44:24 <s3gfault_> has there been any serious consideration to keeping blocks at 1MB, but reducing block time to 2 minutes?
162 2015-06-18 10:44:34 <sipa> far worse
163 2015-06-18 10:44:58 <s3gfault_> because of orphans?
164 2015-06-18 10:45:14 <hearn> it breaks every SPV wallet
165 2015-06-18 10:45:18 <hearn> amongst other problems
166 2015-06-18 10:45:39 <sipa> because what matters for convergance and miner centralization pressure is the ratio between block interval time and propagation delay
167 2015-06-18 10:46:05 <s3gfault_> i saw some analysis about a year ago saying 2 minutes was more then enough time for global propagation.  would only increase orphans by a small amount.  perhaps it was flawed
168 2015-06-18 10:46:05 <sipa> propagation delay is network latency + block size dependent transfer and processing time
169 2015-06-18 10:46:34 <sturles> It will speed up inflation a lot as well.  Unless the block reward is reduced at the same time.
170 2015-06-18 10:47:03 <sipa> by making processing, network speed up, and increasing block.size accordnlingly, you make the latency relatively less important
171 2015-06-18 10:47:11 <s3gfault_> yes, im assuming reward decrease
172 2015-06-18 10:47:27 <sipa> by reducing the block time, you multiply the impact of latencu
173 2015-06-18 10:48:46 <sipa> as now you get its effect 5 times per 10 minutes rather than once
174 2015-06-18 10:48:47 <s3gfault_> sipa: yes, but there must be a cost/benefit tradeoff where block time is most efficient.  10 minutes was chosen arbitrarly as a guess. i.e. i could have been 9.59 or 10.01.  not much research done on initial parameters.