1 2015-07-28 00:28:56 <Sapex> Luke-Jr: ok, can I ask here about bitcoind compilation errors?
  2 2015-07-28 00:29:02 <Luke-Jr> no
  3 2015-07-28 00:29:06 <Luke-Jr> that's a user topic
  4 2015-07-28 00:29:27 <Sapex> Luke-Jr: ok, got it
  5 2015-07-28 00:36:27 <harding> leakypat: I am now.
  6 2015-07-28 00:36:58 <leakypat> Hey, (I'm from Ninki wallet btw)
  7 2015-07-28 00:38:38 <leakypat> I was thinking about Bitcoin.org and wondered what your views were on splitting the wallets into different levels of experience, eg . A wallet for total newbies, a wallet for average Bitcoin Joe and more advanced wallets for experienced Bitcoin users
  8 2015-07-28 00:39:20 <leakypat> I will PR to change the description on my wallet to "for intermediate users"
  9 2015-07-28 00:40:09 <leakypat> But I was thinking in general, that users who are completely clueless shouldn't be pointed to multisig wallets or heavy user dependent tech/sec wallets
 10 2015-07-28 00:40:52 <leakypat> (Just today I have had one guy email me all his private keys from all his walets, for example, and another guy who setup 2fa using his friends phone)
 11 2015-07-28 00:43:03 <leakypat> As a result they have had a bad Bitcoin experience
 12 2015-07-28 00:45:00 <harding> leakypat: hey, sorry, my Internet connection just dropped.  We should probably discuss this elsewhere.  Would a PM be ok?
 13 2015-07-28 00:45:07 <leakypat> Sure
 14 2015-07-28 01:25:59 <Luke-Jr> leakypat: why not just fix those UI bugs? ;)
 15 2015-07-28 01:27:30 <leakypat> Well, the only thing I can do is make it easier to reset 2fa, which I didn't want to do
 16 2015-07-28 01:27:45 <leakypat> And def need a FAQ and better recovery tools
 17 2015-07-28 01:28:15 <leakypat> But a lot of it is people who don't know what an address is jumping in headfirst with a 2 of 3 multisig wallet
 18 2015-07-28 01:28:40 <leakypat> Luke-Jr: unless you have found some UI bugs?
 19 2015-07-28 01:29:43 <Luke-Jr> leakypat: giving the end user easy access to private keys is a UI bug.
 20 2015-07-28 01:30:00 <Luke-Jr> 2FA can potentially explain more when pairing another phone
 21 2015-07-28 01:30:02 <leakypat> They sent their phrases
 22 2015-07-28 01:30:09 <leakypat> From their backups
 23 2015-07-28 01:30:13 <Luke-Jr> ok, so tell users not to do that?
 24 2015-07-28 01:30:19 <leakypat> Not only form my wallet but from their breadwallet too
 25 2015-07-28 01:30:25 <Luke-Jr> >_<
 26 2015-07-28 01:30:39 <leakypat> Of course, but I'm just observing the level of knowledge vs the systems thy are using
 27 2015-07-28 01:30:59 <Luke-Jr> I guess some users are just hopeless
 28 2015-07-28 01:31:04 <leakypat> And trying to figure out how to get them to use a non Bitcoin wallet first while they learn
 29 2015-07-28 01:31:11 <leakypat> Eg coinbase
 30 2015-07-28 01:31:53 <leakypat> They can at least learn about addresses and sending transactions , qr code scanning etc.
 31 2015-07-28 01:33:51 <leakypat> I mean, a big adoption increase- forget about blocksize throughput, worry about opsec Armageddon
 32 2015-07-28 05:01:39 <wumpus> https://lists.linuxfoundation.org/pipermail/bitcoin-dev/2015-July/009689.html  I think we need to organise moderation in some way, this mailing list has been getting out of hand since the block size debate
 33 2015-07-28 05:02:24 <wumpus> signal/noise ratio has reached a worrying low
 34 2015-07-28 05:04:26 <moa> i agree
 35 2015-07-28 05:04:55 <jcorgan> you could always organize a website that allows people to up/downvote comments, i'm sure that would work out really well
 36 2015-07-28 05:04:58 <moa> there is extensive electrum/opem-alias discussion that seems out of place also
 37 2015-07-28 05:05:13 <wumpus> jcorgan: lol.
 38 2015-07-28 05:08:35 <wumpus> moa: at least that still relates to development of bitcoin related software and protocols
 39 2015-07-28 05:09:21 <wumpus> moa: (though weakly sometimes, granted)
 40 2015-07-28 05:28:55 <Luke-Jr> wumpus: since before the block size stuff IMO :/
 41 2015-07-28 05:37:26 <kanzure> CodeShark: wtf ?
 42 2015-07-28 05:37:35 <kanzure> CodeShark: where's the snarky sarcasm?
 43 2015-07-28 05:37:56 <kanzure> re: https://github.com/CryptoConsortium/CCSS/issues/15#issuecomment-125448659
 44 2015-07-28 05:38:10 <CodeShark> "so that in the event of a fork they do not have to enforce one certain fork on all customers"
 45 2015-07-28 05:38:27 <CodeShark> that's not snarky sarcasm? :)
 46 2015-07-28 05:38:31 <kanzure> not at all
 47 2015-07-28 05:38:50 <kanzure> it's a real suggestion
 48 2015-07-28 05:38:59 <CodeShark> if there's a fork and different forks are being imposed on different customers, I think it's pretty safe to say the system completely broke
 49 2015-07-28 05:39:00 <kanzure> businesses *do* care about which side of a fork they end up on
 50 2015-07-28 05:39:07 <kanzure> sometimes their preferences differ from other customers
 51 2015-07-28 05:39:27 <kanzure> you have to be fault-tolerant
 52 2015-07-28 05:39:40 <kanzure> reorgs are a part of fault-tolerance
 53 2015-07-28 05:39:46 <kanzure> or er, i mean, reorg tolerance is a part of fault-tolerance
 54 2015-07-28 05:39:49 <CodeShark> oh....you're talking about reorgs...
 55 2015-07-28 05:39:53 <kanzure> and even reorgs-while-hanging-out-on-a-weird-fork
 56 2015-07-28 05:39:55 <CodeShark> I thought you were referring to consensus rule differences
 57 2015-07-28 05:40:04 <kanzure> yes, i am also talking about consensus-related forks
 58 2015-07-28 05:40:36 <kanzure> plus, it would be stupid to have your third-party blockchain api provider not upgrade to interpret some soft-fork rule, just because one of their legacy customers can't support it yet or something
 59 2015-07-28 05:41:04 <CodeShark> the system is designed to handle reorgs...but consensus-related forks are pretty much "all bets are off" at this point
 60 2015-07-28 05:41:06 <kanzure> the api provider could mask this in a bunch of ways, but the only way to be sure is if they are running the consensus rules
 61 2015-07-28 05:41:24 <kanzure> ok, so second issue
 62 2015-07-28 05:41:35 <kanzure> i wasn't being snarky or sarcastic, but even if i was, what does that have to do with elitism...?
 63 2015-07-28 05:42:08 <CodeShark> I apologize if I misread your comment
 64 2015-07-28 05:42:16 <kanzure> still confused yo
 65 2015-07-28 05:42:23 <kanzure> oh well
 66 2015-07-28 05:42:53 <CodeShark> it sounded like you were making fun of people trusting third party blockchain data providers
 67 2015-07-28 05:43:29 <kanzure> ok. and that would be elitist?
 68 2015-07-28 05:43:50 <kanzure> i'm just trying to follow the train of thought here :-)
 69 2015-07-28 05:44:12 <CodeShark> there is a certain tech elitist attitude inherent in thinking that other people should be sufficiently skilled to do things you find obvious...but perhaps I completely was off the mark
 70 2015-07-28 05:44:58 <kanzure> ok.
 71 2015-07-28 05:45:10 <CodeShark> FWIW, if you had intended the statement as snarky sarcasm it would have made for a good troll :p
 72 2015-07-28 05:45:21 <kanzure> i'll leave that to petertodd for now
 73 2015-07-28 05:45:47 <kanzure> i'd be interested in seeing some actual analysis into claims about bitcoin-core being too difficult for companies to use
 74 2015-07-28 05:45:59 <kanzure> so far i haven't seen good reasons, just marketing from bitgo.com/chain.com and a few others
 75 2015-07-28 05:46:54 <CodeShark> bitcoin core does not provide an industrial strength wallet
 76 2015-07-28 05:47:09 <CodeShark> (not that others that are mentioned or not mentioned do either)
 77 2015-07-28 05:47:21 <kanzure> that's an okay reason to not use it as a wallet, but that's not a good excuse for not using it atall
 78 2015-07-28 05:47:23 <wumpus> at this point there is no open source industrial strength wallet, period
 79 2015-07-28 05:47:25 <kanzure> *at all
 80 2015-07-28 05:48:08 <kanzure> one reason that i would accept is "rpc crashes all the time when we send it a bajillion rpc requests", but i haven't actually heard this complaint from anyone that insists bitcoin-core is too hard to use! so i dunno
 81 2015-07-28 05:48:12 <wumpus> these things don't materialize out thin air from lots of complaining
 82 2015-07-28 05:48:40 <CodeShark> thing is the RPC was really designed around the wallet...not around validation in the more general sense. To use bitcoin core for validation in the more general sense you need to use the p2p protocol
 83 2015-07-28 05:49:02 <kanzure> which validation rpc calls are broken.... ? i don't understand.
 84 2015-07-28 05:49:15 <wumpus> but please move this to #bitcoin, this is meta discussion
 85 2015-07-28 05:54:51 <CodeShark> wumpus: I hope you didn't read my comments as complaints - I see them more as opportunities ;)
 86 2015-07-28 05:54:52 <wumpus> it would help concretely in bringing the wallet forward if people would review and test pull requests (e.g. jonasschnelli's) that improve bitcoin core's wallet. It's not that no work happens.
 87 2015-07-28 05:55:37 <CodeShark> I also am happy jonas is doing the work he's doing
 88 2015-07-28 05:55:44 <wumpus> CodeShark: I was not referring to you specificially. But the bitcoin community in general has a very hands-off-and-complain attitude instead of an involved open source development one. Most people in this channel are obviously excluded from that.
 89 2015-07-28 05:55:45 <jonasschnelli> thanks
 90 2015-07-28 05:56:39 <berndj> wumpus, i agree re moderation, as much as i dislike having to do that. clearly self-regulation is failing on the list, people are using it as a soapbox for bashing $enemy
 91 2015-07-28 05:57:29 <berndj> isn't there some moderation-queue-for-first-time-posters mode? i'm guesstimating that most of the NSR comes from first-time posters
 92 2015-07-28 06:00:17 <wumpus> berndj: right - as much as I hate it, it may be that we need a concrete list of rules as to what the list is for and what is acceptable. Self regulation has clearly failed.
 93 2015-07-28 06:00:41 <CodeShark> I can only imagine all the crap you have to deal with, wumpus - for the record I think you're doing a great job
 94 2015-07-28 06:01:58 <wumpus> CodeShark: you were in the same position; you tried to improve the wallet (e.g. add multiwallet support) but hardly anyone paid attention, certainly not the 'oh no bitcoin wallet is limited and not industrial strength' people
 95 2015-07-28 06:02:11 <wumpus> berndj: good suggestion, will have a look
 96 2015-07-28 06:03:41 <phantomcircuit> <wumpus> these things don't materialize out thin air from lots of complaining
 97 2015-07-28 06:03:46 <phantomcircuit> if they did we'd be set though
 98 2015-07-28 06:04:18 <CodeShark> hmm...sounds like a pretty dystopic society...where everyone gets their way by complaining :p
 99 2015-07-28 06:04:25 <CodeShark> *dystopian
100 2015-07-28 06:04:26 <wumpus> phantomcircuit: heh.
101 2015-07-28 06:05:10 <phantomcircuit> CodeShark, "THIS IS STUPID EVERYBODY SHOULD GET THEIR WAY!"
102 2015-07-28 06:05:10 <wumpus> CodeShark: brave new world 2 - AI nanny solves all issues, all humanity still has to do is complain!
103 2015-07-28 06:05:11 <phantomcircuit> solved
104 2015-07-28 06:12:04 <berndj> let me know if you people would prefer i delete this, if it would be counterproductive. I deliberately didn't link to the list or any posts https://www.reddit.com/r/Bitcoin/comments/3ev9kt/can_all_the_derps_with_nothing_to_contribute/
105 2015-07-28 06:13:15 <wumpus> "I have no mouth and I must complain"
106 2015-07-28 06:13:36 <jcorgan> heh
107 2015-07-28 06:14:55 <wumpus> berndj: besides that I'm not sure reddit is a good meta-discussion place for the ML, good and to-the-point post.
108 2015-07-28 06:15:32 <berndj> agreed, but i have a *suspicion* that a lot of the crap comes from /r/bitcoin mouthbreathers
109 2015-07-28 06:15:42 <wumpus> sure.
110 2015-07-28 06:20:01 <CodeShark> perhaps we can have a mailing list for core dev caricatures :p
111 2015-07-28 06:20:17 <drazisil> This may be stiring the pot, and I'll shut up if so, but did anyone notice that twitter linked on the list wasn't the real one?
112 2015-07-28 06:21:29 <drazisil> I wasn't sure if people missed that, or if it was a troll.
113 2015-07-28 06:22:34 <CodeShark> I think these people got confused - they were looking for r/Buttcoin
114 2015-07-28 06:22:46 <CodeShark> they typed in the wrong URL
115 2015-07-28 06:25:47 <wumpus> drazisil: it was a troll, hence my reply
116 2015-07-28 06:39:17 <wumpus> <CodeShark> perhaps we can have a mailing list for core dev caricatures :p <- sure :-)
117 2015-07-28 09:05:50 <jonasschnelli> needs wallet TAG https://github.com/bitcoin/bitcoin/pull/6482
118 2015-07-28 09:08:14 <drazisil> Looking to parse out the first tx from getblock using python, it appears too big for json.loads. Thoughts or advise?
119 2015-07-28 09:09:36 <jonasschnelli> drazisil: REST interface
120 2015-07-28 09:10:17 <jonasschnelli> drazisil: if you don't fear the additional work, your probably better of with the binary format which can be served over REST
121 2015-07-28 09:10:33 <jonasschnelli> https://github.com/bitcoin/bitcoin/blob/master/doc/REST-interface.md#blocks
122 2015-07-28 09:11:11 <jonasschnelli> If you really only need the first tx, just load some bytes over HTTP (maybe max tx size + header) and rip out the tx.
123 2015-07-28 09:11:44 <jonasschnelli> most JSON parsers are not memory optimized... there are servel C json parsers which just pass around pointer to the strings...
124 2015-07-28 09:11:54 <jonasschnelli> s/servel/serval
125 2015-07-28 09:14:32 <drazisil> jonasschnelli: I'm looking to get it it into a webpage, so not sure how well c will work. How does rest work, just enable it and do calls on :18332 with no auth? I've only mucked with rpc so far.
126 2015-07-28 09:15:23 <jonasschnelli> drazisil: yeah. Just enable it with `-rest` ... then call something like this: curl localhost:18332/rest/getutxos/checkmempool/b2cdfd7b89def827ff8af7cd9bff7627ff72e5e8b0f71210f92ea7a4000c5d75-0.json 2>/dev/null | json_pp
127 2015-07-28 09:16:05 <jonasschnelli> you can also get JSON blocks...
128 2015-07-28 09:16:14 <jonasschnelli> mind the /notxdetails/  if you only like to get the tx id...
129 2015-07-28 09:16:31 <jonasschnelli> if you have enabled -txindex you then could only load this particular tx over GET /rest/tx/<TX-HASH>.<bin|hex|json>
130 2015-07-28 09:18:26 <drazisil> I do have txindex on. I'm mainly trying to grab the coinbase at this time, is there a better way then getblockcount, getblockhash, getblock, getrawtransaction?
131 2015-07-28 09:20:22 <jonasschnelli> i think most easy if you don't wand to fiddle with the bytestream would be: ... /rest/chaininfo.json (get the best block)
132 2015-07-28 09:20:27 <jonasschnelli> then
133 2015-07-28 09:20:42 <jonasschnelli> GET /rest/block/notxdetails/<BLOCK-HASH>.json
134 2015-07-28 09:21:07 <jonasschnelli> get the first transaction hash, do GET /rest/tx/<txhash>.json
135 2015-07-28 09:21:45 <wumpus> if you're working from python, why not request the raw block and use python-bitcoinlib to do everything clientside?
136 2015-07-28 09:21:54 <jonasschnelli> because it's public data, better use REST than RPC. You don't need to expose your RPC password in your app
137 2015-07-28 09:22:20 <wumpus> I've always found that easier to work with, and saves a lot of roundtrips to the RPC server for pure utility functions
138 2015-07-28 09:22:43 <jonasschnelli> wumpuses approach would also be good. If you like to keep the p2p deserialization away from you, you can use JSON (but with a parsing overhead in mem/cpu)
139 2015-07-28 09:25:48 <wumpus> jonasschnelli: I had indeed added the wrong label for #6482, fixed now
140 2015-07-28 09:25:56 <jonasschnelli> thanks
141 2015-07-28 09:26:08 <drazisil> I would actually love to use rest. I'm concerned I would run into cors issues though if I tried to use javascript, do you know if that's an issue?
142 2015-07-28 09:26:23 <jonasschnelli> it's also partial "validation".. but mostly only for tx coming out of the internal wallet,..
143 2015-07-28 09:27:22 <drazisil> the rpcpass isn't a huge issue as I'm having it pull from the bitcoin.conf file so it's not actually stored/displayed anywhere.
144 2015-07-28 09:27:40 <jonasschnelli> drazisil: so javascript to connect to REST (AJAX)... yeah. Why not... but mind that you should expose REST to the public.. don't push a website where you do a AJAX connection to your IP's rest. Hide it somehow (rev. proxy, caching, etc.)
145 2015-07-28 09:29:59 <drazisil> I know I run into cors issues when I try to connect to the rpc via javascript (darn security). do you know if I would have that issue with the rest interface?
146 2015-07-28 09:31:08 <jonasschnelli> You can connect with javascript to REST. But just don't do it on a public website so that the remote browser connect to your REST interface.
147 2015-07-28 09:31:13 <jonasschnelli> Better cache it somehow.
148 2015-07-28 09:31:13 <wumpus> leakypat: you're from Ninki wallet? I'm getting automated responses from "Ninki Support" for mails I've posted to the mailing list, that can't be right.
149 2015-07-28 09:31:34 <jonasschnelli> or run a apache with a reverse proxy and memcached.
150 2015-07-28 09:32:44 <leakypat> wumpus: sorry, just switched off forwarding
151 2015-07-28 09:32:53 <leakypat> I didn't think that would happen
152 2015-07-28 09:34:06 <drazisil> I can probably make a simple python proxy that can control the commands it proxies though. Thank you very much for the help jonasschnelli and wumpus
153 2015-07-28 09:35:33 <CodeShark> at one time I had written an HTTP proxy for access controls to the RPC...but hell if I can locate it anymore :p
154 2015-07-28 09:36:36 <jonasschnelli> common guys,... use apache (or another proven httpd) if you place it public available. :)
155 2015-07-28 09:36:53 <jonasschnelli> plenty scalability options.
156 2015-07-28 09:37:20 <CodeShark> I use a completely different architecture now...so the issue is moot ;)
157 2015-07-28 09:38:49 <drazisil> jonasschnelli: sure, when I do it correctly/production. For now it just needs to work on my local box :)
158 2015-07-28 09:39:17 <jonasschnelli> Sure.
159 2015-07-28 09:41:16 <CodeShark> just make calls using the CLI if it's not for production
160 2015-07-28 09:44:26 <drazisil> CodeShark: that doesn't get it into my webpage
161 2015-07-28 09:44:33 <CodeShark> oh...
162 2015-07-28 14:50:28 <morcos> Luke-Jr: Can you take a look at #6470 and let me know how prioritization deltas should fit into this?  Is the idea that if something is prioritized it should make it into the mempool end of story?  Should it bypass all the checks?
163 2015-07-28 14:55:40 <jgarzik> morcos, a delta is a local application of policy, a manual override
164 2015-07-28 14:56:11 <jgarzik> morcos, you can use fee deltas to implement any sort of local prioritization policy, in theory
165 2015-07-28 14:56:29 <jgarzik> (or priority deltas etc.)
166 2015-07-28 14:58:05 <morcos> jgarzik: so in master, the rate limiter still applies regardless of fee deltas if the actual fee is less than min relay rate right?  is that just oversight?
167 2015-07-28 14:58:39 <morcos> i'm trying to figure out whether those txs with fee deltas should just go in to the mempool regarless of size limits, or they should just evict something, or what
168 2015-07-28 14:58:53 <jgarzik> morcos, in master there are actually two deltas, priority and fee :)
169 2015-07-28 14:59:08 <jgarzik> morcos, priorisetransaction RPC tweaks these
170 2015-07-28 14:59:26 <morcos> yes but in the mempool code they only serve to tell you the minRelayFee should be effectively 0 for those txs, they deltas aren't actually applied until mining
171 2015-07-28 14:59:36 <jgarzik> morcos, the general intention should be to avoid post-delta-processing, IMO
172 2015-07-28 14:59:44 <jgarzik> er
173 2015-07-28 14:59:51 <jgarzik> morcos, the general intention should be to _evict_ post-delta-processing, IMO
174 2015-07-28 15:00:20 <morcos> that seems complicated and a departure from how it works now
175 2015-07-28 15:00:21 <jgarzik> eviction view is more complete following application of local tweaks via delta(s)
176 2015-07-28 15:00:31 <jgarzik> that's how it works now
177 2015-07-28 15:00:33 <morcos> no
178 2015-07-28 15:00:45 <morcos> if you apply 1BTC fee delta to a free transaction
179 2015-07-28 15:00:53 <morcos> it still might be rejected by the rate limiter right now
180 2015-07-28 15:01:27 <gavinandresen> morcos: that's probably oversight, but luke-jr might have insight (he wrote priortisetransaction if I recall correctly)
181 2015-07-28 15:01:34 <jgarzik> by definition applying to tx A means you already have tx A
182 2015-07-28 15:01:50 <morcos> jgarzik: thats why i'm asking all these questions
183 2015-07-28 15:01:54 <morcos> it doesn't actually meant that
184 2015-07-28 15:01:59 <morcos> but i was thinking it did before
185 2015-07-28 15:02:03 <jgarzik> deltas are definitely more on the miners/block template side right now
186 2015-07-28 15:07:01 <morcos> i think this opens a whole can of worms, so i'm not going to touch it for now.  there are a lot of open questions on how it should work with the eviction code and i think it depends on the real world usage of these prioritisations as to what makes the most sense.
187 2015-07-28 15:08:23 <morcos> can prioritized things be evicted?  are their deltas taken into account for that?  etc..   and since its already not internally very consistent, its not obvious to me what to do.  so for now at least, i'll leave it alone.
188 2015-07-28 15:51:51 <schwach> Apps that use coinbase apis?
189 2015-07-28 15:57:47 <morcos> sipa: i'm pushing some fixes to #6470 today and i want to tweak the defaults.  i think the maxmempool default should be as high as is reasonable, given the blowup of txsize -> mempool usage, can we get higher than 300 MB?  500MB?  also i think the expiry threshold should be much longer, at least a week = 168
190 2015-07-28 16:18:37 <jtimon> sipa what was 17b11428 's PR ?
191 2015-07-28 16:18:43 <jtimon> oh, no sipa around...
192 2015-07-28 16:19:31 <jtimon> never mind, it was 6077
193 2015-07-28 17:01:11 <morcos> sipa: wumpus: i have a question about 6077, i'm sure i'm missing something silly, but if tx A comes in to the mempool and is cached as valid.  then conflicting tx A' comes in in a block.   What happens if the next should-be-invalid block contains tx A?
194 2015-07-28 17:01:39 <morcos> block 2 in this case building on block 1
195 2015-07-28 17:06:42 <helo> morcos: block 2 contains a transaction spending an already-spent (by A') output, and be classically invalid
196 2015-07-28 17:07:10 <helo> oh, mempool eviction
197 2015-07-28 17:07:15 <morcos> helo: yeah i think i'm just misundertstanding how 6077 works
198 2015-07-28 17:19:29 <helo> morcos: hah, so am i... when uncached-valid block transactions conflict with cached valid transactions, the cached valid transaction should be similarly removed from the cache... i don't see where this happens, but i'm not very well versed
199 2015-07-28 17:20:15 <morcos> helo: the thing i was missing is the HaveInputs check is done every time.  So you'll fail on that check before you do CheckInputs whose output is cached
200 2015-07-28 17:20:38 <helo> ohhh, thanks :)
201 2015-07-28 17:22:03 <morcos> but i'm a bit concerned if there are any issues around coinbase maturity, which may have passed before and not now, or same thing with CheckLockTimeVerify (with height or time) as I think any of these things can go backwards
202 2015-07-28 17:36:19 <sdaftuar> yeah, i have a test that demonstrates that after 6077, the coinbase maturity consensus check can be broken -- say if a transaction is accepted at a given height but would be invalid at earlier heights
203 2015-07-28 17:36:30 <sdaftuar> accepted into the mempool i mean, and therefore transaction validation is cached
204 2015-07-28 17:36:56 <sdaftuar> if there's a reorg, then at a lower block height we'd accept a block as valid even if it includes that transaction at too early a height
205 2015-07-28 17:38:36 <sdaftuar> test here: https://gist.github.com/sdaftuar/23a3db523e68541a3195
206 2015-07-28 17:41:43 <gavinandresen> Safest thing is probably for DisconnectTip to trigger a re-evaluation of every transaction in the mempool, and any invalid removed and flushed from the validation cache
207 2015-07-28 17:43:36 <gavinandresen> ... and maybe DisconnectTip should flush the validation cache...
208 2015-07-28 17:43:55 <sdaftuar> i think maybe just flushing would be best
209 2015-07-28 17:46:11 <morcos> Do we have a clear model of what things can affect the validity of a script/tx?
210 2015-07-28 17:46:59 <morcos> So if we HaveInputs() and the time >= old time and height >= old height.  We know those are the only things that can affect the validity?
211 2015-07-28 17:48:45 <gavinandresen> morcos: uhh... locktime gets a time/height dependency. The state of the UTXO set obviously affects validity (your inputs must be unspent when you go into a block).
212 2015-07-28 17:49:04 <morcos> For instance, in the event of a soft fork, something accepted into the mempool before soft fork activates could become invalid after the soft fork activates, but have been cached as valid
213 2015-07-28 17:49:18 <morcos> This whole approach just seems a bit risky for consensus  code to me
214 2015-07-28 17:50:15 <morcos> Did we ever benchmark how much of an improvement this was?
215 2015-07-28 17:50:49 <gavinandresen> Should be a big win, but I never got around to running benchmarks.
216 2015-07-28 17:52:37 <morcos> oh i guess he caught the soft fork issue, by checking that you're using a superset of the flags
217 2015-07-28 17:57:32 <sipa> sdaftuar: ouch
218 2015-07-28 17:58:42 <Chris_St6> is Full-RBF going to be implemented in Bitcoin Core?
219 2015-07-28 18:08:59 <jtimon> sdaftuar: thoughts on #6445 as a solution to the problem you found in #6077 ?
220 2015-07-28 18:09:31 <sdaftuar> jtimon: i'll take a look
221 2015-07-28 18:12:09 <jtimon> to be honest, I don't like #6077 much...only saves you repeating the checks in CheckInputs, but not the checks in CheckTransaction or IsStandardTx, for example
222 2015-07-28 18:13:33 <jtimon> cfields: around?
223 2015-07-28 18:17:48 <sipa> sdaftuar: reverted
224 2015-07-28 18:17:50 <sipa> Chris_St6: no
225 2015-07-28 18:20:24 <sdaftuar> sipa: ack.  jtimon: not sure i understand what you mean about #6445, doesn't seem like there's any caching in that code?  sorry if i'm missing something obvious...
226 2015-07-28 18:31:50 <jtimon> sdaftuar: the problem is that you're caching the maturity checks when you shouldn't right?
227 2015-07-28 18:33:06 <jtimon> if you combine #6077 with #6445 then you are not caching that anymore: you would be only caching the script checks (the most expensive ones)
228 2015-07-28 18:33:14 <sipa> sdaftuar: i think i acted too fast
229 2015-07-28 18:33:15 <jtimon> until we move to a smarter cache
230 2015-07-28 18:33:54 <sipa> nope, nevermind
231 2015-07-28 18:34:13 <jtimon> IMO, the cache should store an enum or something where FULLY_VALID_ONCE is one of the values
232 2015-07-28 18:34:17 <sdaftuar> sipa: do you think there's an easy fix?
233 2015-07-28 18:34:36 <sipa> sdaftuar: storing the height of the validation
234 2015-07-28 18:34:49 <sdaftuar> i think height and time?
235 2015-07-28 18:34:56 <sipa> sdaftuar: and requiring that upon cache lookup, it isn't lower
236 2015-07-28 18:35:22 <sipa> how does time matter?
237 2015-07-28 18:35:31 <sipa> (without cltv)
238 2015-07-28 18:35:36 <jtimon> sipa: isn't #6445 an even easier fix? (that saves you repeating the expensive script checks when the height changes?
239 2015-07-28 18:36:01 <sdaftuar> can't you break locktime that way, by reorging to a block with an earlier timestamp?
240 2015-07-28 18:36:11 <sipa> locktime isn't checked in checktxinput
241 2015-07-28 18:36:28 <sipa> as it doesn't need chainstate
242 2015-07-28 18:37:21 <sipa> jtimon: how does it cache?
243 2015-07-28 18:37:28 <sdaftuar> sipa: yes i see, right
244 2015-07-28 18:38:49 <jtimon> sipa: just like 6077 ? the code #6445 has right now (just rebased on top of #6077 ) shouldn't have the problem sdaftuar found
245 2015-07-28 18:39:35 <sipa> jtimon: i see no cache in their (note that github does not force pushes to pullreq branches when the pullreq is closed)
246 2015-07-28 18:39:38 <sipa> in there
247 2015-07-28 18:40:26 <jtimon> sipa: I force rebased, force pushed, then realized that now my change was taking consensus::checktxinputs out of the caching and closed
248 2015-07-28 18:40:47 <jtimon> the cache code is just yours (merged on master)
249 2015-07-28 18:41:00 <sipa> jtimon: link to branch?
250 2015-07-28 18:41:29 <sipa> got it
251 2015-07-28 18:47:32 <sdaftuar> sipa: if we do something like 6077+height caching, how do you think CLTV will work?
252 2015-07-28 18:48:13 <jtimon> sdaftuar: more fun later when we add op_maturity/rcltv/seq_verify
253 2015-07-28 18:48:46 <sipa> sdaftuar: yeah
254 2015-07-28 18:49:06 <sdaftuar> heh, i think fun == risky!  seems like we need a better model for script verification dependencies
255 2015-07-28 18:49:11 <sipa> i dislike the need for comparing state
256 2015-07-28 18:50:08 <jtimon> sdaftuar: yeah I was being sarcastic, sorry
257 2015-07-28 18:50:34 <jtimon> sipa: what do you mean comparing state?
258 2015-07-28 18:50:36 <sdaftuar> seems like at the least we'd want to flush the cache on disconnect tip then...
259 2015-07-28 18:51:31 <sipa> sdaftuar: another is caching the prevblock at the time the validation is done
260 2015-07-28 18:51:52 <sipa> sdaftuar: and trusting the cache when the current prevbloc is a descendant of it
261 2015-07-28 18:51:56 <jtimon> I really think the cache should be in acceptToMemmoryPool and ConnectBlock separately instead of CheckInputs
262 2015-07-28 18:52:15 <morcos> jtimon: doesn't it lose all its value then?
263 2015-07-28 18:53:13 <morcos> but i agree that the whole idea seems risky to me, i liked the simplicity of it doesn't really matter what you do with your mempool, its all just kind of an optimization, that you're checking things from scratch on blocks.  the signature caching i think was a simple enough abstraction that we can be convinced its correct
264 2015-07-28 18:53:51 <sipa> i think prevblock caching should work... as successors are not allowed to invalidate transactions
265 2015-07-28 18:53:55 <jtimon> that way, also, the cache can be used to omit previous checks (like CheckTransaction), not only the ones in CheckInputs
266 2015-07-28 18:53:57 <morcos> but tx validation seems like it opens a can of worms, and evne if we catch them all now, how do we know something else isn't going to pop up down the road
267 2015-07-28 18:54:02 <sipa> except by double spending
268 2015-07-28 18:54:11 <sipa> which you can test for separately
269 2015-07-28 18:54:23 <morcos> sipa: is that clearly the validation model?
270 2015-07-28 18:54:34 <morcos> successors are not allowed to invalidate tx's?
271 2015-07-28 18:54:36 <jtimon> morcos: how so? on the contrary, you get to use it in more places
272 2015-07-28 18:54:52 <morcos> jtimon: oh i thought you meant separate caches...
273 2015-07-28 18:55:25 <jtimon> the one thing we need to have clear is that the caching means "fully valid once", which is not the same as "fully valid now"
274 2015-07-28 18:55:42 <jtimon> I would just repeat the checks that can change instead of storing the height
275 2015-07-28 18:56:44 <jtimon> and use the knowledge of "fully valid once" to skip as many checks as is completely safe to skip, but not more
276 2015-07-28 18:57:06 <morcos> sipa: you might be right, but i think we should just spell that all out very clearly, so in the future if anyone wants to add more opcodes or functionality, we are aware of whether its breaking this model
277 2015-07-28 18:57:29 <morcos> for instance i see why expiring transactions are a bad idea, but i wouldn't have been aware they break a fundamental model of how we think about validation
278 2015-07-28 18:58:00 <jtimon> (I would also reuse the same cache for the reject reasons [and use the reject reason to skip as many tests as is completely safe to skip])
279 2015-07-28 18:58:03 <sipa> morcos: they do - there is a basic principle that nothing can accidentally can become invalid
280 2015-07-28 18:58:46 <sipa> it is the reason for the maturity period, by the way
281 2015-07-28 18:59:32 <jtimon> (but then again, nobody seemed to like anything from https://github.com/jtimon/bitcoin/commit/935ee1ec875308f27339418363c787ec061d335f )
282 2015-07-28 19:12:56 <sdaftuar> hmm so i guess CLTV should work just fine even with 6077 -- looking at it again, it just relies on locktime working properly (neat!)
283 2015-07-28 19:24:13 <jtimon> sdaftuar: yes, it would just work the same way, that's why I mentioned scv
284 2015-07-28 19:24:49 <jtimon> or csv, I can never remember the new name, I was so happy with the initial op_maturity name...
285 2015-07-28 19:25:30 <sipa> CSV i guess
286 2015-07-28 19:26:16 <jtimon> whatever, for rcltv, you would ideally put the new checks in consensus::checktxinputs but...you don't want that to be cached
287 2015-07-28 19:31:50 <jtimon> so, anyway, for the new version of #6077 (given that reusing the same txCheck cache for both full-success-once and rejections is apparently such a bad idea)
288 2015-07-28 19:32:36 <jtimon> can we please use the cache for skiping more tests than just  Consensus::CheckTxInputs and the script tests ? for example, CheckTransaction and IsStandardTx ?
289 2015-07-28 19:32:48 <sipa> jtimon: that's a good question
290 2015-07-28 19:33:10 <sipa> in theory, i would say yes - it should skip everything except the do-all-inputs-exist test
291 2015-07-28 19:33:44 <jtimon> but would be more work today and more total work later?
292 2015-07-28 19:34:03 <cfields> jtimon: i am now
293 2015-07-28 19:34:59 <jtimon> sipa: I mean, if you don't do it now and leave it for later, you're slowing down libconsensus' verifyTx, you know that, right?
294 2015-07-28 19:36:11 <sipa> i haven't checked the code to see how hard it would be to do
295 2015-07-28 19:38:57 <jtimon> sipa well, the most basic version is declaring a bool fFullyValidOnce at the beginning of ConnectBlock and AcceptToMemoryPool and then interactive query-replace s/if (/if (!fFullyValidOnce && /
296 2015-07-28 19:40:11 <jtimon> I wanted to store the reject reason too, so that was more complicated (I needed for all the failing checks to use CValidationState first)
297 2015-07-28 19:41:21 <sipa> jtimon: well fullyvalidonce isn't enough
298 2015-07-28 19:41:32 <sipa> you need to know in which context it was valid
299 2015-07-28 19:41:55 <jtimon> in all of them or you don't store it
300 2015-07-28 19:42:11 <jtimon> I'm fine keeping the insert to the cache in CheckInputs
301 2015-07-28 19:42:27 <jtimon> just where it is
302 2015-07-28 19:43:20 <jtimon> if you don't make it there, you don't get cached, that's why we have the non-redundant-for-some-reason-I-miss rejection cache too, right?
303 2015-07-28 19:44:08 <sipa> how do you mean "in all of them" ?
304 2015-07-28 19:45:03 <sipa> i think that context currently is only height
305 2015-07-28 19:45:14 <sipa> it may also need block version
306 2015-07-28 19:47:09 <sipa> uh, softforks complicate this
307 2015-07-28 19:47:25 <sipa> as they break the "once valid, forever valid" principle
308 2015-07-28 19:59:00 <BlueMatt> whats the status of removing the priority field of txn entirely? (jgarzik)
309 2015-07-28 19:59:08 <jtimon> oh, I think I see what you're saying
310 2015-07-28 19:59:43 <BlueMatt> seems there are a ton of outstanding mempool changes that could be simplified some with that merged in first
311 2015-07-28 20:00:06 <jtimon> "you need to know in which context it was valid" I thought you meant the state in the validation process, my answer was "we only care if the tx makes it to the end"
312 2015-07-28 20:00:23 <sipa> BlueMatt: i think we first want limited mempool
313 2015-07-28 20:00:25 <jtimon> sipa you were just justifying why using the height
314 2015-07-28 20:00:42 <sipa> jtimon: yeah, changed my mind... i don't think height is enough
315 2015-07-28 20:00:45 <BlueMatt> sipa: why? limited mempool is simpler without priority :)
316 2015-07-28 20:00:58 <jtimon> yeah, that makes sense, but the simplest solution is to just repeat all the checks that don't follow the "once valid, forever valid"
317 2015-07-28 20:01:01 <jtimon> rule
318 2015-07-28 20:01:10 <sipa> BlueMatt: all limited mempool code being written ignores priority entirely
319 2015-07-28 20:01:18 <BlueMatt> oh, well that works, then :)
320 2015-07-28 20:01:35 <jtimon> sipa: like any fee check if we move to dynamic relay fee
321 2015-07-28 20:02:59 <jtimon> you just use fFullyValidOnce where you can and don't use the cache at all on the other cases
322 2015-07-28 20:03:29 <sipa> unfortunately, that's not very interesting
323 2015-07-28 20:03:43 <sipa> as script checks already fall outside that category
324 2015-07-28 20:04:03 <jtimon> for cltv?
325 2015-07-28 20:04:15 <sipa> because of softforks
326 2015-07-28 20:04:28 <sipa> the flags for validation can change from one block to another
327 2015-07-28 20:04:38 <jtimon> oh, I see
328 2015-07-28 20:04:55 <sipa> 6077, which worked at a lower level, cached the signature validation checks used
329 2015-07-28 20:05:59 <sipa> doesn't change a thing
330 2015-07-28 20:06:45 <jtimon> well, if you're going to end up using the height everywhere anyway, why use block.ntime in many places too?
331 2015-07-28 20:06:57 <jtimon> but anyway, side conversation...
332 2015-07-28 20:07:49 <jtimon> sipa: "6077, which worked at a lower level, cached the signature validation checks used" where is that in the code?
333 2015-07-28 20:08:33 <sipa> https://github.com/bitcoin/bitcoin/pull/6077/files#diff-7ec3c68a81efff79b6ca22ac1f1eabbaR1400
334 2015-07-28 20:08:47 <sipa> the script validation flags are the mapped-to type in the cache
335 2015-07-28 20:09:44 <jtimon> BlueMatt: I agree refactoring/removing freeTx stuff first would simplify things...but even though there's so many special cases for accepting/rejecting free tx (at least 4), nobody seems to agree that is prioritary
336 2015-07-28 20:11:40 <jtimon> oh, I see, you mean you are storing the flags used in the cache entry, you can still do that (and even storing the reject reasons too if you didn't had an independent txid index for that already)
337 2015-07-28 20:12:00 <sipa> right, but it's not very clean
338 2015-07-28 20:12:11 <jtimon> what is not very clean?
339 2015-07-28 20:12:16 <sipa> and it's heavily specialized for script validation
340 2015-07-28 20:12:30 <sipa> if we want something that can skip all checks
341 2015-07-28 20:13:48 <jtimon> well for all cache-safe checks will be "did you made it to the end once?" for scripts will be "did you made it to the end once with flags X?"
342 2015-07-28 20:14:36 <jtimon> it is special-casing scripts, sure, but they are special: f#%& expensive
343 2015-07-28 20:15:29 <jtimon> other checks that aren't always cache-safe are not special-cased and always repeated instead
344 2015-07-28 20:23:30 <jtimon> btw sipa what is reusing the same cache for reject reasons a bad idea?
345 2015-07-28 20:23:37 <jtimon> s/what/why
346 2015-07-28 22:04:59 <waxwing> is there anything special i have to do to get sendrawtransaction to work in regtest?
347 2015-07-28 22:05:11 <waxwing> i get rpc error -25, but can't make sense of the error code
348 2015-07-28 22:06:05 <belcher> what does debug.log say
349 2015-07-28 22:06:13 <belcher> theres nothing special you have to do iirc
350 2015-07-28 22:08:04 <waxwing> belcher: yeah i realised after i wrote it that the Q doesn't make sense
351 2015-07-28 22:08:11 <waxwing> already working in regtest script
352 2015-07-28 22:10:16 <waxwing> belcher: good call, insane fees I think :)
353 2015-07-28 22:52:02 <CodeShark> how long does it take for a new thread to the mailing list to post?
354 2015-07-28 22:52:20 <Luke-Jr> jgarzik: the deltas map is independent from the mempool specifically because you don't need to have the tx yet when you prioritise