1 2015-08-07 00:05:27 <gmaxwell> jonasschnelli: Why is PR6256 using public derrivation to produce consecutive outputs of a hdgetaddress by default? This is vulnerable because the private keys can be exported and they are linearly related. BIP32 expressly specifies that hardened derrivation should be used for top level wallet keys, specifically to close this vulnerability. (generally the public derrivation should only be used for sp
2 2015-08-07 00:05:33 <gmaxwell> ecial cases like repeated payments, multisig, merchant servers, etc.)
3 2015-08-07 06:46:34 <jonasschnelli> gmaxwell: by default it follows bip44 IMO...
4 2015-08-07 06:47:06 <jonasschnelli> so it derives over private key derrivation c(0 and 1) in m/44'/0'/0'/c
5 2015-08-07 06:47:49 <jonasschnelli> then it uses public key derrivation to generate consecutive addresses in m/44'/0'/0'/0/n (where n is the consecutive index)
6 2015-08-07 06:50:13 <gmaxwell> This is insecure. Refer to BIP32: it specifies the use of hardened derrivation for a reason. All public derrivation siblings are effectively (though not obviously to the user) the same private key. It's especially important to avoid public derrivation at any point where a private key could possibly be exported.
7 2015-08-07 06:50:51 <Luke-Jr> private keys shouldn't be exported, so this seems like a feature to me ;)
8 2015-08-07 06:51:03 <jonasschnelli> gmaxwell: but would that mean to access the private key (unlock the wallet) if one needs to generate addresses?
9 2015-08-07 06:52:08 <gmaxwell> jonasschnelli: It should keypool, that needs to be done in any case for scanning reasons (e.g. if another copy of the wallet has been used, and they're somewhat desynced)
10 2015-08-07 06:53:50 <warren> jonasschnelli: there are legit uses of public key derivation, but it should have scary names or something because it's insecure.
11 2015-08-07 06:54:06 <wumpus> agree with warren
12 2015-08-07 06:54:23 <Luke-Jr> exporting private keys is what is insecure IMO, more than the derivation used..
13 2015-08-07 06:54:24 <jonasschnelli> okay so instead of m/44'/0'/0'/c/n it should do m/44'/0'/0'/c'/n'?
14 2015-08-07 06:54:57 <jonasschnelli> (which means to go back to keypools)
15 2015-08-07 06:55:01 <wumpus> but it's not even proven that the derivation is secure
16 2015-08-07 06:55:03 <gmaxwell> The public derrivation is intended for special applications where there is no great alternative; they come with a pretty significant security tradeoff (and one which is very surprising to users).
17 2015-08-07 06:55:11 <wumpus> but yes, exporting is catastrophic
18 2015-08-07 06:56:12 <gmaxwell> A better way to think about public derrivation is that it's a single key; which can be anonymized for privacy or invoice tracking purposes.
19 2015-08-07 06:57:22 <gmaxwell> I kind of regret introducing the idea now; because the applications where it is really useful have hardly been developed at all.. and its getting applied ubiqutiously in other cases. :(
20 2015-08-07 06:57:43 <Luke-Jr> if the derivation is weak in itself, then that's a reason to avoid it - but the private key stuff is just something that's always been a risk in unexpected ways anyway
21 2015-08-07 06:57:46 <gmaxwell> E.g. AFAIK there is no simple CGI you can drop on your webserver and it'll churn out more addresses for you. Even after all this time.
22 2015-08-07 06:58:20 <Luke-Jr> gmaxwell: they is as soon as bitcoind can do a watch on one
23 2015-08-07 06:58:24 <Luke-Jr> there is*
24 2015-08-07 06:59:00 <jonasschnelli> so it would make sense to keep the public derivation in the Bip32 PR but use private derivation as default?
25 2015-08-07 06:59:11 <gmaxwell> Luke-Jr: yea, sure users burn themselves with private keys; but public derrivation makes it much worse. Changes mishandling some paper wallet and burning some coins to total loss of the wallet.
26 2015-08-07 06:59:14 <jonasschnelli> would also require two keypools (internal / external)
27 2015-08-07 06:59:14 <warren> gmaxwell: not sure you want to encourage people to blindly churn out addresses on a web server CGI =)
28 2015-08-07 06:59:45 <gmaxwell> jonasschnelli: yes, it makes sense to support it-- thats why it's there.
29 2015-08-07 06:59:46 <Luke-Jr> warren: donations..
30 2015-08-07 07:00:07 <warren> Luke-Jr: hack the web server, replace the generator to send coins elsewhere ...
31 2015-08-07 07:00:13 <gmaxwell> warren: you can happily put it behind a captcha and/or IP limiter, yadda yadda.
32 2015-08-07 07:00:21 <Luke-Jr> warren: so? it's strictly better than any other option today
33 2015-08-07 07:00:26 <gmaxwell> warren: better than having private keys on the webserver. There is nothing around what you're describing. :)
34 2015-08-07 07:00:32 <warren> sure
35 2015-08-07 07:00:52 <gmaxwell> jonasschnelli: yea, the expectation was that there would be a keypool for each internal and external as otherwise there are gap issues, thats why they're seperate chains.
36 2015-08-07 07:01:31 <Luke-Jr> warren: theymos should have a way to put an xpub for donations in your profile (which generates addresses and keeps the xpub itself private)
37 2015-08-07 07:01:37 <leakypat> I think the way we implemented it in Ninki is ok, no xpriv export and users can exchange xpubs and derive addressees for each other, it's a fantastic scheme, I wouldn't regret introducing it
38 2015-08-07 07:01:38 <jonasschnelli> okay... i'll try to switch to private derrivation and if one stats a new wallet with a xpubkey it uses public key derivation (watch only)
39 2015-08-07 07:01:44 <warren> jonasschnelli: what does the rpc look like for this now?
40 2015-08-07 07:02:04 <gmaxwell> Being able to add a 'chain-address' where the address it returns is an xpub and it watches children, would be good evenutally-- but it's a more advanced feature for sure.
41 2015-08-07 07:02:31 <jonasschnelli> warren: the bip32 PR (PR6256) is a backupplan if corewallet (the new wallet) https://github.com/jonasschnelli/bitcoin won't succeed.
42 2015-08-07 07:02:33 <Luke-Jr> jonasschnelli: well, obviously to be useful we need a way to do derivation with a private key too
43 2015-08-07 07:03:31 <jonasschnelli> gmaxwell: i think there are usecases where people might want to start a wallet with master xpubkey and derive addresses, receive coins, but can't sign.
44 2015-08-07 07:04:23 <jonasschnelli> warren: the RPC interface of PR6256 is 'hdaddchain', 'hdgetaddress' and for now 'hdsendtoaddress'
45 2015-08-07 07:04:26 <gmaxwell> jonasschnelli: Yes, I pointed that out specifically earlier, :) . There are special use cases for which the functionality was intended.
46 2015-08-07 07:05:17 <jonasschnelli> Okay. Let me work on it... i see the point... it just liked the way of not touching the master private key (any private key) unless you wan't to sign something.
47 2015-08-07 07:05:35 <warren> jonasschnelli: how about hdaddinsecurechain and hdgetinsecureaddress? Same thing, just scary sounding for public deriv
48 2015-08-07 07:05:38 <gmaxwell> jonasschnelli: but I think the right thing to do for those applications is that those are 'single addresses' (xpubs). E.g. you create a new xpub, and it creates a new subchain, and you can go import the xpub on another host.
49 2015-08-07 07:06:16 <jonasschnelli> What are the risks when you expose a non hardened child pubkey?
50 2015-08-07 07:06:19 <gmaxwell> e.g. you want to createchain 'tag' and then getaddress 'tag' to get the next address under that particular chain.
51 2015-08-07 07:07:16 <gmaxwell> The full pubkey? that you lose privacy-- someone can assc all the transactions. ... and if they also have one of the sibling privates, that they can steal all the coins in that subtree.
52 2015-08-07 07:08:18 <jonasschnelli> So the risk is losing a non hardened child pub key together with any child priv key.
53 2015-08-07 07:08:30 <jonasschnelli> How could you leak a child priv key?
54 2015-08-07 07:09:00 <Luke-Jr> users are stupid and will find a way :|
55 2015-08-07 07:09:06 <jonasschnelli> i assume most wallets do not stored them (in mem derivation)
56 2015-08-07 07:09:12 <gmaxwell> jonasschnelli: by exporting and handing someone a private key, or using it another problem (somewhat surprisingly users do this all the time).
57 2015-08-07 07:10:31 <gmaxwell> (to sign with alternative programs, to import coins into web services, to list some of the more sane reasons people do this ... but also seemingly because the moon has called to them.)
58 2015-08-07 07:10:55 <jonasschnelli> Okay. Will switch to priv derivation by default while still allow a way of pub der.
59 2015-08-07 07:11:24 <gmaxwell> importchain <tag> <xpub> / createchain <tag> (returns xpub) / getchainaddress <tag> <opt index> might be a reasonable API style.. also extends reasonably well to multisig.
60 2015-08-07 07:11:27 <jonasschnelli> Also need to check in detail how the mem is cleansed during private derrivation
61 2015-08-07 07:12:13 <gmaxwell> jonasschnelli: don't count on that to work.
62 2015-08-07 07:12:28 <jonasschnelli> gmaxwell: for now i was avoding tags and uses a DoubleSha256 from the xpubkey as chain identifier... but the tag would be more user friendly
63 2015-08-07 07:13:07 <gmaxwell> Well it's very useful when transactions show up in your wallet that some metadata shows up with them so you know what the heck they are, esp as you may have never seen the 'address' before.
64 2015-08-07 07:13:47 <jonasschnelli> gmaxwell: Hmm.. you mean the ability to see over what bip32 chain you have received that specific wtx?
65 2015-08-07 07:14:00 <gmaxwell> (Re clense: it's basically impossible to guarentee memory clearing from C and C++, you can only make a best effort... part of the reason I think all signing and key management should end up in a seperate process. :) )
66 2015-08-07 07:14:38 <jonasschnelli> gmaxwell: clense: Agree. That's why i'm also woking on a detached signing process (Bip) to support hardware wallets.
67 2015-08-07 07:15:27 <gmaxwell> jonasschnelli: more specifically to be able to get an automatic human understanble tag connected. E.g. if I create an xpub chain and stick it on the bitcoin talk profile.. and later donations start coming in, when the transactions show up in my wallet I want them already taged with "bct donation".
68 2015-08-07 07:15:56 <gmaxwell> otherwise tax time turns into suicide watch. :)
69 2015-08-07 07:16:10 <jonasschnelli> gmaxwell: yes. Would be nice. The problem than start if the wtx has two outputs to two specific bip32 chain tags. :)
70 2015-08-07 07:16:46 <gmaxwell> Whats the problem? when a tx has multiple outputs that pay you, they show up as seperate recieved entries in the transaction list.
71 2015-08-07 07:17:12 <jonasschnelli> isn't it one data row = one wtx?
72 2015-08-07 07:17:29 <jonasschnelli> (sure, can be changed)
73 2015-08-07 07:18:50 <gmaxwell> No, as I just said-- if you have multiple outputs paying you they already show up as multiple entries! :) Go try it.
74 2015-08-07 07:18:51 <jonasschnelli> gmaxwell: what would interest me is how you would imaging the backup process? Your proposal "createchain <tag> (returns xpub)" could also extended with createchain <tag> (returns xpub, return xprivkey) but obviously it's a hight risk to pass that back over JSON to (probably) a console?
75 2015-08-07 07:19:54 <jonasschnelli> over the GUI it could be a printer backup (like armory) with a 8 char password which you need to write down to the printed paper.
76 2015-08-07 07:20:00 <gmaxwell> jonasschnelli: I would expect createchain to create a hardened child off your wallet's current master branch. e.g. a sibling of what getnewaddress would give.
77 2015-08-07 07:20:27 <gmaxwell> a createchain is just like a getnewaddress there, except its explicitly notifying the wallet that the address it returns will be used multiple times.
78 2015-08-07 07:22:02 <jonasschnelli> gmaxwell: okay. So no exporting of private keys at all... no bip39 phrase. Means: backup needs to go the old way by backing up the wallet.dat.
79 2015-08-07 07:22:34 <gmaxwell> what no?
80 2015-08-07 07:23:32 <gmaxwell> What is unacceptable is to export private keys of nodes inside a public derrived chain. Exporting a master secret for backup sure. Exporting private keys for interior hardened nodes--- sure (though it should be an advanced feature with some amount of warnings around it)
81 2015-08-07 07:24:00 <jonasschnelli> Okay... i see.
82 2015-08-07 07:24:25 <Luke-Jr> backups always need to go the "old way".. no amount of seed data will save the metadata with it
83 2015-08-07 07:24:30 <gmaxwell> (I say advanced feature just because manual private key handling has resulted in a surprising number of loss events)
84 2015-08-07 07:24:48 <gmaxwell> But indeed, there is no escaping the need to have ways to backup metadata. Though it could be handled much better than now.
85 2015-08-07 07:24:51 <Luke-Jr> HD just makes a stale backup less damaging
86 2015-08-07 07:25:47 <gmaxwell> As I've pointed out, loss of metadata could result in criminal prosecution in many places (including the US). Losing it is not good at all, losing it _and_ access to the coins is worse, of course.
87 2015-08-07 07:27:33 <gmaxwell> If, in the future, we adopt the scheme for generating a metadata backup key as a branch off the master secret, then it would be pretty easy to integrate online backups for metadata.
88 2015-08-07 07:27:49 <jonasschnelli> The PR currently supports generating of different chains with different paths (like m/45' along with a m/44'/0'/0'/1' chain),.. so i think "createchain" could always seed a new maser and generate a new internal/external chain key pair.
89 2015-08-07 07:27:50 <da2ce7> You can always encrypt the meta-data to the private key's it is assioated with and upload it somwhere as-you-go.
90 2015-08-07 07:28:41 <da2ce7> maybe sombody can have a public storage for small-payment type server.
91 2015-08-07 07:29:27 <gmaxwell> jonasschnelli: Well then you have a backup problem then, every time a new chain is setup for reoccuring payments. That seems unfortunate to me.
92 2015-08-07 07:30:03 <jonasschnelli> gmaxwell: so one chain per wallet would make more sense?... hmm..
93 2015-08-07 07:30:24 <jonasschnelli> (one master seed per wallet, sry)
94 2015-08-07 07:30:50 <gmaxwell> jonasschnelli: I think one master seed per wallet probably makes the most sense, and eventually (perhaps in future wallet software) supporting multiple wallets.
95 2015-08-07 07:32:10 <gmaxwell> (and good multiple wallet support means both supporting isolating them, and being able to make transactions that span them, if you choose). At least from a backup perspective having wallet = atomic unit of backup makes sense.
96 2015-08-07 07:32:14 <jonasschnelli> metadata: using a special keypath like m/123'/0' to encrypt the metadata would allow one to recover/decrypt the metadata if he still can access m/123'/0' (over a printed out backup of the masterseed as example)... yes. makes sense.
97 2015-08-07 07:32:31 <gmaxwell> Right.
98 2015-08-07 07:32:58 <jonasschnelli> how would you encrypt with m/123'/0' ,.. derive a AES_256_CBC key?
99 2015-08-07 07:34:25 <jonasschnelli> ([...]being able to make transactions that span them[...]): does that mean having a option to form a transaction with inputs from multiple wallets?
100 2015-08-07 07:34:33 <gmaxwell> It should be authenicated as well, but sure, something like that.
101 2015-08-07 07:35:39 <gmaxwell> jonasschnelli: Yes, otherwise you can get in a situation where you have enough funds but can't make a payment, and have to do silly things like make multiple transactions. E.g. you have an online wallet and 4 cold storage wallets, and you bring 2 cold storage wallets online and want to make a transaction that spends all the coins in the only and the two cold ones.
102 2015-08-07 07:36:29 <gmaxwell> not so different than banks supporting covering checking account transactions using funds in a savings account. :)
103 2015-08-07 07:36:29 <jonasschnelli> Yes. I see. Moving the coins first to one wallet would make you loose funds in fees.. agreed.
104 2015-08-07 07:37:00 <gmaxwell> yea, lose funds in fees, waste time watiting for confirmations, more chances to fat-finger an address.. etc.
105 2015-08-07 07:37:26 <gmaxwell> (and really, more pressure on the user to not have seperate wallets as a result)
106 2015-08-07 07:37:59 <jonasschnelli> metadata: Will think about the medata-enc. Would really like something like that,.. this could allow auto-backup to a insecure space... maybe an option to increase the child index from m/123/0 to m/123/1 after a certain trigger...
107 2015-08-07 07:39:01 <gmaxwell> well probably every time its written out it should be reencrypted with a new IV or someone watching the insecure space will learn some thing things about what changed in the wallet.
108 2015-08-07 07:39:27 <jonasschnelli> now i come back to the multiple bip32 chain (master xkeys) per wallet. I have implemented this initially because we where once talking about key-rotation...
109 2015-08-07 07:39:44 <jonasschnelli> wouldn't this allow key rotation at the deepest point?
110 2015-08-07 07:40:16 <jonasschnelli> if you don't trust a master seed anymore, add a new one, swap funds.
111 2015-08-07 07:40:26 <jonasschnelli> not swap ... move over (sry)
112 2015-08-07 07:41:53 <gmaxwell> Yes, supporting multiple for rotation also is sensible. Hm. The issue you want to avoid is where the "wallet" as a whole has a mixed integrity story.. e.g. some things more backed up than others in ways that will surprise users. Rotation is explicit and has obvious known effects which are understandable.
113 2015-08-07 07:42:45 <gmaxwell> Users elsewhere have been burned by 'deterministic wallets' that also allowed import.. they imported keys, and then later wondered where their funds went after they erased their wallet and tried to recover from their 'master seed'.
114 2015-08-07 07:43:03 <gmaxwell> for that PR this is kind of a stopgap thing, so having that problem is okay.
115 2015-08-07 07:43:52 <gmaxwell> Though, to be honest, you could have a much simpler approach for a stopgrap thing: change nothig visible, add a master key if there isn't one there, and use it to fill the keypools. (unless disabled).
116 2015-08-07 07:44:23 <jonasschnelli> Hmm.. good point. I originally thought to allow single key together with hdchains for corewallet. But right. the recovery use case over a hd master key would not cover the single keys and could harm..
117 2015-08-07 07:45:36 <gmaxwell> s/stopgrap/stopgap/
118 2015-08-07 07:45:36 <jonasschnelli> gmaxwell: would also work,.. the keypool approach. But my goal was to reuse the stuff written for corewallet and seperate both worlds to not interfere ones single key wallet.
119 2015-08-07 07:46:00 <jonasschnelli> both worlds = deterministic and non-deterministic within one wallet
120 2015-08-07 07:46:34 <gmaxwell> Thats ~generally~ undesirable. Or at least I think the user shouldn't think their wallet is "determinstic" unless it really is.
121 2015-08-07 07:46:38 <gmaxwell> (unless all of it is)
122 2015-08-07 07:47:18 <gmaxwell> esp since its not clear which keys have funds. You might think your backup has all your coins, but really those payements you got last week went to a one off key.
123 2015-08-07 07:47:19 <wumpus> maybe show a flag
124 2015-08-07 07:47:24 <jonasschnelli> gmaxwell: Yes. Truly. But i switching over the bip32 completely would require first to swipe over funds to a bip32 wallet?
125 2015-08-07 07:47:46 <wumpus> if the wallet is *completely* deterministic, I mean
126 2015-08-07 07:48:02 <jonasschnelli> i was looking for a way of a save transition for existing wallets.
127 2015-08-07 07:48:18 <jonasschnelli> (which indeed could do harm in case of security)
128 2015-08-07 07:48:23 <wumpus> I think that should be optional, at least.
129 2015-08-07 07:48:58 <wumpus> it shouldn't just automatically sweep the entire wallet during an upgrade process or such
130 2015-08-07 07:49:08 <gmaxwell> jonasschnelli: had you not started on that PR what I would have suggested is, add a master, use it for new keys. Don't tell the user, functionality doesn't change. Newly created wallets would just do that at birth, ... and there could be a flag that displays if there are any loose keys loaded in it.
131 2015-08-07 07:49:13 <gmaxwell> No I don't think it should sweep.
132 2015-08-07 07:49:17 <jonasschnelli> I have a slight feeling that my bip32 PR does not make sense for the current wallet. The new wallet approach makes more sende and looks more sane.
133 2015-08-07 07:49:38 <wumpus> I'd love a 'sweep' functionality, but it should not happen automatically, at least not without the user asking for it.
134 2015-08-07 07:50:07 <gmaxwell> Perfect shouldn't be the enemy of good.
135 2015-08-07 07:50:12 <wumpus> right
136 2015-08-07 07:50:50 <wumpus> makes sense to do what gmaxwell says, use the seed for new keys
137 2015-08-07 07:50:52 <gmaxwell> What I'm describing is maybe only 50 lines of functionality (over whats already in the codebase); and would make the wallet far less of a trap for most users.
138 2015-08-07 07:51:03 <wumpus> then when you make a backup *from then on* it will be deterministic from there
139 2015-08-07 07:51:05 <gmaxwell> It's not where things need to be for sure.
140 2015-08-07 07:51:35 <wumpus> (assuming the user doesn't use importing)
141 2015-08-07 07:51:47 <jonasschnelli> gmaxwell: you need also the on-the-fly-derrivation.. needs another 100+ lines of code... or would you store the derived private keys as you store the non-deterministic priv keys?
142 2015-08-07 07:52:14 <gmaxwell> jonasschnelli: would just use the existing key pool code, just change where its getting the private key from.
143 2015-08-07 07:52:16 <jonasschnelli> and what about supporting flexible chainpath (m/1 or m/44/0/0/0?
144 2015-08-07 07:52:37 <jonasschnelli> gmaxwell: yes. That would work... but would expose every private key in the wallet database (maybe unencrypted)
145 2015-08-07 07:52:46 <gmaxwell> jonasschnelli: all of that has major design considerations, user impact, api impact, etc. probably shouldn't be done too casually. :)
146 2015-08-07 07:53:07 <jonasschnelli> right...
147 2015-08-07 07:53:08 <gmaxwell> jonasschnelli: I'm not following your latest comment?
148 2015-08-07 07:54:10 <jonasschnelli> If we would just change the keypool approach (hd deriving instead of single key) it would loose one benefit IMO, that is, deriving private keys on the fly over the xprivkey of the external or internal chain.
149 2015-08-07 07:54:36 <gmaxwell> jonasschnelli: What is the specific benefit you believe it would lose?
150 2015-08-07 07:55:16 <jonasschnelli> xpubkey watch only wallets, not exposing child private keys
151 2015-08-07 07:56:03 <jonasschnelli> but right,... as long as the master xpriv is exposed it doesn't matter that much if a child priv key is stored in the same database.
152 2015-08-07 07:56:11 <gmaxwell> Yes, it wouldn't be supporting that. But as discussed above, that should be supported in an intelligent way that is savvy to the application and such. By no means am I suggesting that the simple thing is a replacement for more complete support.
153 2015-08-07 07:56:38 <gmaxwell> But if something is going to wait for a new wallet design, the xpub stuff should-- as that has lots of considerations to get right, and is visible to the user.
154 2015-08-07 07:56:58 <jonasschnelli> Agreed.
155 2015-08-07 07:57:47 <gmaxwell> I imagine that any new wallet software would consider one of these wallets to really just be the same as an old bag of keys wallet, it just turns out that that the keypool fills deterministically in it.
156 2015-08-07 07:59:33 <jonasschnelli> gmaxwell: do i understand that right that the only benefit of the keypool is that one can "genrate" (lets say use) keys in a encrypted wallet without unlock?
157 2015-08-07 08:00:47 <gmaxwell> jonasschnelli: currently? no! the keypool is to make backups useful at all. (if you backup once every $keypool keys used, you won't be exposed to funds loss)
158 2015-08-07 08:01:20 <jonasschnelli> gmaxwell: right.. i was thinking in the bip32 world... there backup benefit is no longer given?
159 2015-08-07 08:02:01 <gmaxwell> In the BIP32 feeds keypool model, the purpose of the keypool is for scanning lookahead so you won't miss transactions if two copies of the same wallet get used, and for generation while unlocked too, indeed. Also just because it would minimize changes to the software.
160 2015-08-07 08:03:05 <gmaxwell> there would also be no harm in making it a fair amount bigger in that case, by default, because there is no longer a useful key rotation tradeoff.
161 2015-08-07 08:03:29 <gmaxwell> only downside in making it bigger is somewhat slower wallet creation/loading and a somewhat larger file.
162 2015-08-07 08:05:16 <jonasschnelli> Would it be required to store the derived private key? Why not only store the public keys and derive the private keys if we need to sign?
163 2015-08-07 08:06:59 <gmaxwell> It wouldn't --- it would certantly be more efficient to not store those private keys, but it would make the change more complex. (e.g. take it out of the realm of just a couple lines of code).
164 2015-08-07 08:07:51 <gmaxwell> I'm not sure what the overhead ends up being on storing the private keys. But it really should only make it store half the data.
165 2015-08-07 08:08:44 <gmaxwell> jonasschnelli: oh another comment I had about this stuff, the existing code we have for key generation is hardened against error; e.g. after generating a pubkey it goes and makes sure it can sign and verify with the private key. Some hardening should be done of the derrivation.
166 2015-08-07 08:09:14 <gmaxwell> If nothing else it should probably just do it twice and compare. Otherwise a single bit error can cause supermassive losses, when the private key cannot be generated again.
167 2015-08-07 08:09:54 <jonasschnelli> Agreed. That needs to be implemented... the CKey.VerifyPubKey after a CExtKey derivation...
168 2015-08-07 08:10:23 <olalonde> Anyone can help with block locator message?
169 2015-08-07 08:10:40 <jonasschnelli> gmaxwell: Okay... need to think about this. Thanks you very much for the discussion.
170 2015-08-07 08:10:47 <olalonde> I'm trying to grasp the expected behaviour
171 2015-08-07 08:11:18 <gmaxwell> while worrying about this a couple years ago I went and wrote an implementation of SHA512 that was bit reversed, e.g. you gave it the compliment of the input and it gave you the compliment of the hash output. with the though that it could be a parallel case to compare to. But thats perhaps a little extreme.
172 2015-08-07 08:11:46 <gmaxwell> (though it was neet that sha2 was alegbraically structured enough that I could flip the sense of all the bits and still have it work with not too much effort)
173 2015-08-07 08:11:50 <gmaxwell> jonasschnelli: thanks!
174 2015-08-07 08:12:34 <olalonde> I'm using bitcore-p2p and sending those hashes: `[ genesis block hash ], stop: hash at height 6`. My bitcoind doesn't return any blocks. However if I put the hash at height 7 I'm getting 6 blocks
175 2015-08-07 08:14:33 <olalonde> Oops, incorrect
176 2015-08-07 08:14:38 <olalonde> Won't send me any blocks at all
177 2015-08-07 08:14:50 <olalonde> I guess I just misunderstood what the stop hash means
178 2015-08-07 08:15:58 <olalonde> Does bitcoind remember which block it sent me before?
179 2015-08-07 08:17:55 <olalonde> Oh, seems it is indeed doing that
180 2015-08-07 08:17:58 <olalonde> Is that correct?
181 2015-08-07 08:36:01 <wumpus> yes, it remembers which blocks a node already have for a connection to avoid sending/inving them again
182 2015-08-07 10:03:50 <olalonde> Right
183 2015-08-07 10:30:55 <wumpus> (so just disconnect and reconnect and it will forget)
184 2015-08-07 13:16:50 <morcos> sipa: petertodd: i wanted to update you guys with where sdaftuar and i are on limiting chains of unconfirmed txs in the mempool
185 2015-08-07 13:17:38 <morcos> primary focus of this was to be able to prevent cheap chains from clogging the bottom of the mempool and being too large to get evicted
186 2015-08-07 13:18:16 <morcos> we tried two approaches. putting every tx in a partition of all related txs and actually calculating the descendant package size for each tx
187 2015-08-07 13:19:27 <morcos> the partition approach is very fast, the slowest part is when a block is removed, but the overall slowdown was in the 1-2% range
188 2015-08-07 13:19:40 <morcos> thats with a limit of 1000 txs in a partition
189 2015-08-07 13:20:03 <morcos> the descendant package tracking can get quite slow, depending on how many ancestors you have
190 2015-08-07 13:20:51 <morcos> i'll let sdaftuar give you the actual numbers, but its about 2 us or so per ancestor, every time you add a tx.
191 2015-08-07 13:21:25 <morcos> we were toying around with a limit of 100 ancestors in addition to imposing the descendant package limit of 1000 txs
192 2015-08-07 13:21:57 <morcos> presumably all of these limits would have size limits as well, for instance no reason to have an ancestor package of >1MB bc then that tx isn't minable right away anyway
193 2015-08-07 13:22:42 <morcos> and the descendant package size limit is determined by how much you could evict during a surplus trim. (with a slightly modified surplus trim we were looking at 2.5M descendant packages)
194 2015-08-07 13:22:46 <morcos> anwyay, ok here's the question
195 2015-08-07 13:23:49 <morcos> i think sdaftuar and i would both prefer to go with tracking descendant information. its the information that is actually valuable to you and it also makes the mempool trimming much more effictive, bc you can now sort by package fee rate.
196 2015-08-07 13:24:23 <morcos> however... its slower, and it actually would have a bigger effect on how many recent txs would have been prevented
197 2015-08-07 13:24:52 <morcos> my gut is that's bc the stress tests were creating these very long single chains
198 2015-08-07 13:25:09 <morcos> i feel like 100 ancestor limit is more palatable as a policy change
199 2015-08-07 13:25:20 <morcos> than a 1000 partition limit
200 2015-08-07 13:26:09 <morcos> a 1000 partition limit means that you could be trying to link to a single recent unconfimred tx , but be prevented because some one else linked to another output from that tx with a tx that was related to lots of unconfirmed txs
201 2015-08-07 13:26:17 <morcos> the ancestor limit is something you control entirely
202 2015-08-07 13:27:10 <morcos> i looked at what percentage of txs entered the mempool with > 100 mempool ancestors
203 2015-08-07 13:27:20 <morcos> april: 1.1%
204 2015-08-07 13:27:25 <morcos> may: 1.5%
205 2015-08-07 13:27:36 <morcos> 7/6 - 7/14: 25.1%
206 2015-08-07 13:28:37 <morcos> anyway so i'm looking for some feedback on where to shoot for as a policy limit. then we'll write up some more detailed information and post it to the mailing list.
207 2015-08-07 13:29:24 <morcos> is 100 ancestors enough? are we willing to take some performance hit (the worst case for the descendant trackign is when a block is disconnected, and sdaftuar is looking at trying to optimize that behavior)
208 2015-08-07 13:30:04 <morcos> anyway, we'll have code and maybe even a PR by next week, but wanted to concentrate our efforts on one direction at this point
209 2015-08-07 16:26:54 <paveljanik> wumpus, please have a look at #6013 and #6346. Both rebased and ready for merge IMO. Thank you.
210 2015-08-07 17:28:12 <altcoinsfun> do you know guys what I'm thinking about this storm of shit coins called "alt-coins" - it is not a shit. Peoples just training. It is warm-up. We are staying in the start of Greatest BOOOOM of digital money!
211 2015-08-07 17:30:36 <altcoinsfun> As suggest every Dinosaur of Wall Street, such as Warren Edward Buffett, you should only BUY and try to never sell - this is your way to the success.
212 2015-08-07 17:34:52 <altcoinsfun> about "dev", as I'm commer here once per year. Who is this bastard who cutted-off IRC default from the bitcoind client? Every alt-coin now - disable this functionality, what can I say? NOBODY Can find NETWORK!
213 2015-08-07 17:35:16 <altcoinsfun> IT IS HELL
214 2015-08-07 17:35:21 <sipa> the code is out there in public
215 2015-08-07 17:35:33 <sipa> bitcoin develipers have no responsibility to help altcoins
216 2015-08-07 17:35:49 <altcoinsfun> I'm talking to you. You should look back
217 2015-08-07 17:36:11 <altcoinsfun> Behind your team - Thousands of youngsters who are just warming-up
218 2015-08-07 17:36:30 <altcoinsfun> they could make such mistake, you aren't
219 2015-08-07 17:36:38 <sipa> cool, i wish them all luck (seriously)
220 2015-08-07 17:36:48 <sipa> now take this elsewhere
221 2015-08-07 17:37:40 <nwilcox> Is there an "altcoin-dev" irc channel? Might be effective to have a known destination to point altcoin enthusiasts to.
222 2015-08-07 17:37:57 <sipa> i believe there is
223 2015-08-07 17:38:25 <sipa> ##altcoin-dev
224 2015-08-07 17:38:32 <nwilcox> Thanks.
225 2015-08-07 17:41:54 <altcoinsfun> I'll be back
226 2015-08-07 17:45:03 <altcoinsfun> .OCCo :cOCOcCOcco 8 c:c cO8@@@Oc
227 2015-08-07 17:45:07 <altcoinsfun> CoOOO8@@@ooC:: ::O:. .:O.:OCO .: :::O:::.
228 2015-08-07 17:45:12 <altcoinsfun> cOOOCO ococC@@@@@@@@@@@@@@@@@@@@@@@@@8@Oc ooc ooOOCOc:
229 2015-08-07 17:45:16 <altcoinsfun> :c8@@@O8c:: OoO:o8@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@8oc.
230 2015-08-07 17:45:20 <altcoinsfun> :c8@@@O8o:c 8COcC@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@8Cc.
231 2015-08-07 17:45:24 <altcoinsfun> 8O8@@@O8o oO88OOO8@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@Oo
232 2015-08-07 17:45:28 <altcoinsfun> :@@@8OOo 888CoC888888@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@8O:
233 2015-08-07 17:45:32 <altcoinsfun> OOOC :.::oCOo8O8@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@Oc
234 2015-08-07 17:45:36 <altcoinsfun> .CCCo ::::coOo8O8@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@88c
235 2015-08-07 17:45:40 <altcoinsfun> c cccOO88@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@8OC:
236 2015-08-07 17:45:44 <altcoinsfun> :cocc .cOO88O88OO88@@@@@@@@@@@@@@@@@@@@@@@@@@@@@8Oo
237 2015-08-07 17:45:48 <altcoinsfun> :8@@ooO@@8o o occcO8@@@@@@@@@@@@@@@@@@@@@@@@@@@88O@@8O@88@@@88Cc:
238 2015-08-07 17:45:52 <altcoinsfun> :8@@ocC8@8C c c:ccCO88@8@8@@@@@@@8@@@@@8@@@@@8@O8C88OC8OO@88OOo:.
239 2015-08-07 17:45:55 <altcoinsfun> ccoc8@Oc :O@@O :co. .o
240 2015-08-07 17:45:59 <altcoinsfun> :::o8@o: .C@@O :.
241 2015-08-07 17:46:03 <altcoinsfun> OccO8@C: CO@@O oO:.
242 2015-08-07 17:46:07 <altcoinsfun> Oooo8@o: oO@@O c:: .:::c:. oOOCoc :
243 2015-08-07 17:46:10 <altcoinsfun> @OOO8@O: O8@@O :o .coco8OOcoOOC8OC @@@@88 Oc
244 2015-08-07 17:46:13 <altcoinsfun> @@@@@@Oc oO@@O .c @@@@@@::
245 2015-08-07 17:46:17 <altcoinsfun> @@@@@@O: O@@@O CO c@@@@@@@@@oo .cc:
246 2015-08-07 17:46:21 <altcoinsfun> @@@@@@o: CO@@O : .c: .:O@@@@@@@@@@@o:: ..
247 2015-08-07 17:46:25 <altcoinsfun> @@@@@@Oc O8@@O o :co. :o8@@@@@@@@@@@Coo
248 2015-08-07 17:46:32 <altcoinsfun> @OO@@@o: oO@@O : :. :.::.:ooOoo@@@O@@@@@@@@@O8CC8OCOCoOoOC:
249 2015-08-07 17:46:36 <altcoinsfun> @88@@@O: O8@@O :c :cOO888@8@@@@@@@@O88:c8 :cc OOOOC88O8888OCc:
250 2015-08-07 17:46:40 <altcoinsfun> 8@@@88o: oO@8C ooO : . ..OCOOCOOOOOCOOCOO8@88... ..coC.. .o@COOooOCocccc:
251 2015-08-07 17:46:40 <Squidicuz> !ops
252 2015-08-07 17:46:44 <altcoinsfun> 8@@@88O: O8@8o 88@ c OCO8CO8O8OOOOO8O8@@@ :c :c:OO:: o@8OOcc8CCcccc:
253 2015-08-07 17:46:48 <altcoinsfun> O@@@OO.. oO@Oo CoO.. .: :::OoCOoO:.:.c@@@@@@O.:Coo.. oo8@@@@@@Cc ..:.:
254 2015-08-07 17:46:48 <rodarmor> Ohh, ooh, I know this one, it's the terminator.
255 2015-08-07 17:46:52 <altcoinsfun> @@@@8O:: C8@8C .O@@@:o oooocoocoooO@@@@@@@@@@@88@OOO@@@@@@@@@@Oo. o:
256 2015-08-07 17:55:20 <drazisil> cripe
257 2015-08-07 19:42:19 <cfields> wumpus: still around?
258 2015-08-07 19:43:37 <cfields> wumpus: i tracked down the weird travis bug from a few days ago, will PR a fix in a min. In looking, though, i noticed lots of weird locking problems. I'd really like to annotate the rpc functions for the clang tools, but that requires that each one has a function definition
259 2015-08-07 19:44:02 <cfields> as it is now, they're called via function pointers, which don't play nice
260 2015-08-07 19:44:34 <cfields> would you be up for moving away from the function-table for that purpose?
261 2015-08-07 19:45:33 <wumpus> cfields: not sure
262 2015-08-07 19:45:51 <wumpus> cfields: what would be the alternative?
263 2015-08-07 19:46:25 <cfields> wumpus: best i can come up with is a nasty switch/case to call the correct function :\
264 2015-08-07 19:46:40 <wumpus> no, I definitely would not like that
265 2015-08-07 19:47:30 <wumpus> the direction rpcserver would be progressing is that different modules register their own calls - so a big switch statement is a move in the wrong direction
266 2015-08-07 19:47:32 <cfields> it can be wrapped in a functor, but that would require each function having the same lock requirements
267 2015-08-07 19:47:42 <wumpus> they already do, right?
268 2015-08-07 19:47:50 <wumpus> allt he locks have been moved inside the functions
269 2015-08-07 19:48:21 <wumpus> RPC functions are called without any locks at all
270 2015-08-07 19:48:33 <cfields> wumpus: what i used to test: http://pastebin.com/raw.php?i=HeTZi58V
271 2015-08-07 19:48:33 <wumpus> then aquire the locks that they need themselves
272 2015-08-07 19:49:04 <cfields> wumpus: indeed they should, that was the problem here: https://github.com/bitcoin/bitcoin/blob/master/src/test/rpc_wallet_tests.cpp#L30
273 2015-08-07 19:49:04 <wumpus> so RPC functions can be regarded as top-level in that regard
274 2015-08-07 19:50:07 <cfields> ok, i'll just fixup the test and leave the rest alone
275 2015-08-07 19:51:21 <wumpus> right - so apparently it's a test-only issue, probably missed when the locks were moved inside the functions instead of externally aquired?
276 2015-08-07 19:51:59 <cfields> looks like it, i don't see any other offenders
277 2015-08-07 19:52:07 <wumpus> good
278 2015-08-07 19:52:58 <cfields> well, there's 1 LOCK2 that's probably unnecessary then, but i guess it preserves the lock order so things don't go bad
279 2015-08-07 19:54:13 <wumpus> heh the lock order is very important
280 2015-08-07 21:00:41 <drazisil> Why is it that when I enable rest my rpc connections fail, but cli still works?
281 2015-08-07 21:01:19 <drazisil> does the port change?
282 2015-08-07 21:03:20 <warren> drazisil: rpc connections from what?
283 2015-08-07 21:04:02 <drazisil> warren: insight
284 2015-08-07 21:09:06 <wumpus> enabling rest has no impact on other RPC usage
285 2015-08-07 21:09:41 <gmaxwell> wumpus: should have no. :)
286 2015-08-07 21:10:38 <gmaxwell> drazisil: could it be something like when you enable rest something that has been trying rest but failing instantly connects in and sucks up all the rpc threads with kept-alive idle rest connections?
287 2015-08-07 21:12:06 <wumpus> gmaxwell: well there are tests for rest/no-rest, and people are using it
288 2015-08-07 21:13:42 <gmaxwell> wumpus: See my example of how 'enabling rest' could impact RPC usage. :) I only made the correction because while it's possible drazisil made an error, generally you can tell if RPC stops working if you enable rest, so maybe there is an interaction.