1 2014-10-15 00:00:06 <sipa> but version having 300 kb of object code is crazy
  2 2014-10-15 00:00:39 <coryfields_> yea. i'd really like to nuke those util deps..
  3 2014-10-15 00:00:54 <coryfields_> i'll leave it up to you to fight the error/logging stuff :)
  4 2014-10-15 00:01:19 <sipa> deal
  5 2014-10-15 00:02:18 <coryfields_> i went the most hackish route for now and just deleted em as you suggested. I figured we'd discuss that separately when we got to it.
  6 2014-10-15 00:06:16 <coryfields_> libbitcoinconsensus.0.dylib
  7 2014-10-15 00:06:20 <coryfields_> woohoo :)
  8 2014-10-15 00:09:03 <sipa> nize
  9 2014-10-15 01:08:41 <FXOR> I have a pretty in depth knowledge of how the bitcoin protocol works, but I can't figure out one thing.
 10 2014-10-15 01:11:03 <FXOR> how does a wallet with a private address know that a utxo's receiving address belongs to it
 11 2014-10-15 01:12:12 <FXOR> wouldn't it have to sha256 and base58 encode every permutation to see if it belongs?
 12 2014-10-15 01:13:35 <gmaxwell> "every permutation" ? Its not clear to me what you're thinking of specifically.
 13 2014-10-15 01:15:00 <FXOR> I mean running sha256 and encoding the public key until getting the result in the transaction.
 14 2014-10-15 01:15:45 <gwillen> FXOR: the wallet knows all its own addresses, the transaction has the address in it, it just has to check if it's in the list
 15 2014-10-15 01:15:58 <gwillen> if it has a lot of keys, it has to check all of them, yeah, but that's really fast
 16 2014-10-15 01:16:18 <gwillen> unless I'm misunderstanding what you're asking
 17 2014-10-15 01:16:45 <gwillen> (and of course what I said only applies to standard one-recipient transaction outputs)
 18 2014-10-15 01:17:48 <FXOR> how many permutations for a single publicaddy are there?
 19 2014-10-15 01:20:49 <gwillen> what do you mean by 'permutations'?
 20 2014-10-15 01:21:36 <phantomcircuit> FXOR, there is exactly 1 canonical hash value per public key
 21 2014-10-15 01:22:04 <phantomcircuit> iirc you can pad the public key to get more but that will fail IsStandard and wont be relayed by most of the network
 22 2014-10-15 01:23:26 <phantomcircuit> gmaxwell, is the public key DER encoded?
 23 2014-10-15 01:23:32 <phantomcircuit> or is that just the signature
 24 2014-10-15 01:24:37 <Luke-Jr> FXOR: all addresses are encoded as exactly 1 scriptPubKey ever
 25 2014-10-15 01:24:41 <Luke-Jr> there are no permutations possible
 26 2014-10-15 01:25:14 <Luke-Jr> you don't need to look for weird stuff you don't expect, because that's simply not a valid payment to your address
 27 2014-10-15 01:25:18 <gwillen> Luke-Jr: but given a scriptpubkey, determining if you can redeem it is not necessarily trivial
 28 2014-10-15 01:25:27 <gmaxwell> FXOR:  I don't know what you're talking about "permutations" there are no permutations.
 29 2014-10-15 01:25:33 <Luke-Jr> gwillen: it is if you keep a list of scriptPubKeys you're expecting
 30 2014-10-15 01:25:43 <gwillen> (but presumably there's only one standard one-recipient scriptpubkey for a given pubkey in your wallet)
 31 2014-10-15 01:25:49 <Luke-Jr> gwillen: add to the list for every invoice/address you generate, optionally remove when paid
 32 2014-10-15 01:25:50 <gmaxwell> A single private key _uniquely_ defines a single public key.
 33 2014-10-15 01:26:46 <Luke-Jr> FXOR: also, addresses are inhernetly public - there is no "private address"
 34 2014-10-15 01:27:23 <phantomcircuit> gmaxwell, but does it also uniquely define a single public key encoding which is accepted by OP_CHECKSIG
 35 2014-10-15 01:27:28 <phantomcircuit> that im not so sure of
 36 2014-10-15 01:27:39 <Luke-Jr> phantomcircuit: not relevant, though
 37 2014-10-15 01:28:11 <Luke-Jr> if you ask for payment at 1xxxx and they send to something else, they didn't pay you
 38 2014-10-15 01:28:30 <gmaxwell> phantomcircuit: what luke said.  Simply because you only accept the public keys _you specified_ not any which you could conceivably redeem; doing the latter would get you robbed right quick.
 39 2014-10-15 01:28:58 <gmaxwell> (e.g. when some wiseass pays you with 1 of 2 {you, them} ... sure you can spend it, ... and so can they, bye bye funds.)
 40 2014-10-15 01:29:00 <FXOR> Luke-Jr: how can a single public key have multiple addresses?
 41 2014-10-15 01:29:17 <Luke-Jr> FXOR: it doesn't. you use a single key once ever.
 42 2014-10-15 01:29:18 <gmaxwell> FXOR: What do you mean by permutation?
 43 2014-10-15 01:29:28 <Luke-Jr> unless you mean a watch-only HD wallet, which you'd have to read the spec on
 44 2014-10-15 01:29:40 <phantomcircuit> gmaxwell, right
 45 2014-10-15 01:30:01 <phantomcircuit> i wasn't saying otherwise (for a number of reasons beyond 1/2)
 46 2014-10-15 01:30:18 <FXOR> by permutations I meant outputs of sha256. i had incorrectly assumed there was a "salt"  added of sorts similar to mining
 47 2014-10-15 01:32:53 <FXOR> e.g. timestamps or whatveer data the miner chooes
 48 2014-10-15 01:33:10 <phantomcircuit> FXOR, an address is simply base58(ripemd160(sha256(der(publickey))))
 49 2014-10-15 01:33:27 <jgarzik> compressed vs. uncompressed
 50 2014-10-15 01:35:57 <FXOR> yup, this : https://en.bitcoin.it/wiki/Technical_background_of_Bitcoin_addresses helped clear things up for me
 51 2014-10-15 01:36:44 <SomeoneWeird> phantomcircuit: 'simply' ;p
 52 2014-10-15 01:38:45 <jgarzik> ICYMI: SSL 3.0 bug http://googleonlinesecurity.blogspot.com/2014/10/this-poodle-bites-exploiting-ssl-30.html
 53 2014-10-15 01:39:34 <FXOR> phantomcircuit: What about the checksum or version byte, or is that defined in the ripemd160 function.
 54 2014-10-15 01:41:20 <phantomcircuit> FXOR, oh forgot about that
 55 2014-10-15 01:41:25 <phantomcircuit> shrug
 56 2014-10-15 01:41:32 <phantomcircuit> logically that's what it's doing
 57 2014-10-15 01:42:32 <phantomcircuit> FXOR, so there are actually multiple addresses per private key, but that doesn't really matter because you give someone one specific address to use
 58 2014-10-15 01:42:55 <phantomcircuit> since it's a hash they cannot generate another address which would be valid for the corresponding private key
 59 2014-10-15 01:42:58 <phantomcircuit> so it doesn't matter
 60 2014-10-15 01:43:10 <FXOR> how are the multiple addresses generated w/o a salt of some sort? ot is there?
 61 2014-10-15 01:43:12 <phantomcircuit> which is what Luke-Jr was telling you before
 62 2014-10-15 01:43:26 <BlueMatt> jgarzik: yup, ssl is hilariously broken...me and a friend hacked together a "if I can mitm you, I can tweet from your twitter account" PoC that worked surprisingly well a year ago
 63 2014-10-15 01:43:37 <phantomcircuit> FXOR, the binary representation of the public key has two forms, compressed and uncompessed
 64 2014-10-15 01:43:51 <phantomcircuit> but more so you can null pad various bits to get even more representations
 65 2014-10-15 01:43:58 <phantomcircuit> they're all logically an identical public key
 66 2014-10-15 01:44:03 <phantomcircuit> but have difference hash values
 67 2014-10-15 01:44:10 <phantomcircuit> does that make sense?
 68 2014-10-15 01:44:28 <Luke-Jr> FXOR: the checksum is a checksum, it is discarded before it become a scriptPubKey
 69 2014-10-15 01:44:50 <Luke-Jr> FXOR: the version tells the wallet what format the script is
 70 2014-10-15 01:45:28 <Luke-Jr> version 0 is OP_DUP OP_HASH160 <data> OP_EQUALVERIFY OP_CHECKSIG
 71 2014-10-15 01:45:46 <Luke-Jr> version 5 is OP_HASH160 <data> OP_EQUAL
 72 2014-10-15 01:46:20 <Luke-Jr> FXOR: you only look for the format you expect, nothing more
 73 2014-10-15 01:46:41 <Luke-Jr> if someone changes or ignores the version on the address, they didn't pay you
 74 2014-10-15 01:48:07 <FXOR> so its is or isnt possible to have multiple public addresses(with the same format) correspondong to a single privatekey
 75 2014-10-15 01:48:37 <Luke-Jr> FXOR: define "single private key"
 76 2014-10-15 01:49:11 <Luke-Jr> FXOR: a single ECDSA private key, it's possible, but you'd have to do it yourself and it'd be a stupid idea; a single wallet, it's however you want
 77 2014-10-15 01:49:36 <Luke-Jr> in no case can someone else make up new addresses that they can argue pay you
 78 2014-10-15 01:49:47 <Luke-Jr> unless you tell them explicitly how anyway
 79 2014-10-15 02:00:59 <FXOR> Yes, a single ecdsa private key is what I was referring to when I said private key
 80 2014-10-15 02:09:38 <Theonevortex> Hello all
 81 2014-10-15 02:11:56 <sipa> FXOR: a single ecdsa private key could be used to construct many possible addresses (through p2sh scripts involving it, ot by using the compressed or uncompressed form), but you'd still need to know the actuala script to be able to spend funds
 82 2014-10-15 02:12:17 <sipa> so typically it is still the receiver that determines which scripts he will accept payments to
 83 2014-10-15 02:12:42 <sipa> the fact that other scripts can be unlocked using the same private key is not actually relevant
 84 2014-10-15 03:32:19 <coryfields_> jtimon: ping. around?
 85 2014-10-15 03:33:09 <coryfields_> jtimon: (it's nowhere near ready to PR, but thought you might like to have a look): https://github.com/theuni/bitcoin/tree/libbitcoinconsensus
 86 2014-10-15 03:34:25 <coryfields_> rather, it won't be pr'd, but chunks at a time will
 87 2014-10-15 03:37:22 <jtimon> I'll take a look, is that already boost free ?
 88 2014-10-15 03:40:11 <coryfields_> yea. boost-free, builds standalone solib/dll/dylib, and passes all tests
 89 2014-10-15 03:40:39 <coryfields_> well, dll has stupid toolchain dependencies due to a libtool bug. that's on my list.
 90 2014-10-15 03:49:19 <BlueMatt> coryfields_: so it did magically work on osx?
 91 2014-10-15 03:49:34 <coryfields_> yep, got lucky for once :)
 92 2014-10-15 03:53:24 <coryfields_> sipa: i'm going to leave my macbook running tonight doing a fresh sync with headers-first. anything out of the ordinary you need tested?
 93 2014-10-15 03:54:27 <kanzure> what's the background on wanting to remove boost?
 94 2014-10-15 03:55:19 <coryfields_> kanzure: only for libconsensus, to simplify it on our side and users' side
 95 2014-10-15 03:55:30 <kanzure> more specifically: faster builds?
 96 2014-10-15 03:57:11 <coryfields_> others may have different motivations. For me, it greatly reduces the infrastructure needed to build/deploy, and simplifies things for anyone using the lib
 97 2014-10-15 03:57:24 <coryfields_> outside the scope of the lib (for bitcoind), nothing's changing
 98 2014-10-15 03:57:33 <kanzure> what are the other popular motivations? heh
 99 2014-10-15 03:58:37 <coryfields_> not sure. it was my requirement for working on the lib :p
100 2014-10-15 04:00:38 <coryfields_> kanzure: libbitcoinconsensus is meant to be as barebones as possible. low barrier for entry, so that others are encouraged to use it rather than try to reimplement. so, simpler is better.
101 2014-10-15 05:26:48 <BlueMatt> coryfields_: NICE!
102 2014-10-15 10:21:02 <fanquake> sipa tested HF on osx fairly extensively now, haven’t seen any issues.
103 2014-10-15 10:39:26 <Adlai> would running an electrum server on hf be a good idea / valuable test?
104 2014-10-15 10:40:30 <Luke-Jr> running an electrum server might not be a good idea in general. <.<
105 2014-10-15 10:41:13 <Adlai> why not?
106 2014-10-15 10:41:44 <Luke-Jr> Adlai: poor design? :p  (look at its protocol..)
107 2014-10-15 10:42:11 <Adlai> w/e, i'm looking for ways to help test HF
108 2014-10-15 10:42:29 <Adlai> and this seems like it'll put some real world stress on it, beyond just relaying between 17-22 connections
109 2014-10-15 10:42:31 <Luke-Jr> yes, it was mostly a joke :P
110 2014-10-15 10:45:07 <Adlai> 
ACTION treads the sarchasm cautiously around these parts

111 2014-10-15 10:47:16 <wumpus> Adlai: everything that uses the HF code is a valuable test
112 2014-10-15 10:51:31 <Adlai> can i tell electrum to use bitcoind's leveldb?
113 2014-10-15 10:52:38 <wumpus> I doubt that, but that's not a question for here
114 2014-10-15 10:53:10 <Luke-Jr> wumpus: sure it is O.o?
115 2014-10-15 10:53:19 <Luke-Jr> not that the relevant people are here
116 2014-10-15 10:53:34 <Adlai> 
ACTION asked in #electrum

117 2014-10-15 10:56:55 <wumpus> Luke-Jr: well I don't mean it's necessarily off-topic, but I doubt anyone here knows
118 2014-10-15 11:32:42 <Adlai> what can i do to better keep an eye on possible breakage in HF?
119 2014-10-15 11:33:15 <Adlai> meanwhile i just stare at -printtoconsole and -getconnectioncount
120 2014-10-15 11:36:14 <wumpus> the most serious problems would be if your client gets stuck, ie, doesn't get any new blocks, even though  it's connected to working peers... or if there are crashes/assertion errors
121 2014-10-15 12:06:01 <Adlai> why does the client retain many orphaned blocks, visible with getchaintips? is it juuust in case a stronger fork appears from one of them?
122 2014-10-15 12:10:08 <wumpus> chaintips doesn't show orphaned blocks
123 2014-10-15 12:10:39 <wumpus> orphan blocks are blocks that have no known parent, they used to be stored in memory but after headers-first these are not stored  at all anymore
124 2014-10-15 12:13:50 <wumpus> as for alternative chaintips, sometimes it happens that there is a small fork in the chain but these are quite rare in bitcoin and generally not longer than one block
125 2014-10-15 12:15:02 <wumpus> you can run linearize.py (part of the repository) to get a linear block history without forks, if you need that
126 2014-10-15 12:16:32 <Adlai> so without explicitly linearizing, the fork less taken gets ignored but still remembered on-disk?
127 2014-10-15 12:16:45 <wumpus> yes
128 2014-10-15 12:17:56 <wumpus> that's always been the case, though getchaintips is a relatively new addition that allows inspecting the alternative tips
129 2014-10-15 12:19:01 <Adlai> total_amount in gettxoutsetinfo is the sum of all potentially spendable outputs?
130 2014-10-15 12:19:26 <Adlai> eg it includes the +2Kbtc in the xcp arson address?
131 2014-10-15 12:20:55 <wumpus> yes
132 2014-10-15 12:21:33 <wumpus> unless you use OP_RETURN to explicitly mark an output as non-spendable, it will end up in there
133 2014-10-15 12:22:03 <Adlai> 
ACTION just checking his understanding

134 2014-10-15 14:08:07 <btcdrak> "arson address"
135 2014-10-15 14:10:17 <Mikuru> A lot more descriptive than burn address, and accurate too.
136 2014-10-15 14:19:14 <czzarr> hello, can someone please explain to me the serializing algorithm in coins.h? I don't get it from the doc and have a really hard time understanding c++
137 2014-10-15 14:19:53 <czzarr> associated SE question: https://bitcoin.stackexchange.com/questions/31978/how-does-serializing-in-coins-h-work
138 2014-10-15 14:30:46 <ligar> Odd question. Is it possible to seperate the function of block transmission and handeling from transaction\broadcasting handeling?
139 2014-10-15 14:31:12 <ligar> Idea being one node for one and another for the other.
140 2014-10-15 14:31:45 <ligar> question is basicly on a protocol lvl not bitcoind
141 2014-10-15 14:32:01 <wumpus> yes, those uses of the network are separate
142 2014-10-15 14:33:35 <justanotheruser> ligar: You want one node that validates blocks and one that validates transactions?
143 2014-10-15 14:33:35 <ligar> <- in the early brain stormimming of makinf a proff of consept of seperation the functions of bitcoind into conponate parts alowing seperate nodes to fill those functions (think internal cluster for a company)
144 2014-10-15 14:34:01 <ligar> justanotheruser: YES  using a comon but seperate DB
145 2014-10-15 14:34:40 <justanotheruser> ligar: and a user wouldn't be running both clients?
146 2014-10-15 14:34:50 <ligar> no
147 2014-10-15 14:35:12 <justanotheruser> ligar: that wouldn't work very well
148 2014-10-15 14:35:20 <wumpus> though you need the blocks (or at least, a utxo set) to verify transactions against
149 2014-10-15 14:35:33 <ligar> correct
150 2014-10-15 14:35:48 <ligar> the nodes would share a comon current unspent DB
151 2014-10-15 14:36:06 <justanotheruser> you could trust someone elses DB, but at that point, why even validate transactions?
152 2014-10-15 14:36:10 <ligar> the node(s) handeling injesting would up date this
153 2014-10-15 14:36:28 <ligar> justanotheruser: I would own all the boxes
154 2014-10-15 14:36:39 <ligar> was thinking more High avalibiility
155 2014-10-15 14:36:48 <justanotheruser> 10:34 < justanotheruser> ligar: and a user wouldn't be running both clients?
156 2014-10-15 14:36:51 <justanotheruser> 10:34 < ligar> no
157 2014-10-15 14:36:53 <justanotheruser> so you mean "yes"
158 2014-10-15 14:37:08 <ligar> but not on the same computer
159 2014-10-15 14:37:15 <ligar> or instance
160 2014-10-15 14:37:39 <ligar> I was wondering if the protocol allowed the handeling seperately
161 2014-10-15 14:37:59 <justanotheruser> ligar: okay, now what does this accomplish being between computers?
162 2014-10-15 14:38:25 <ligar> I would think it would scale better in the really long run
163 2014-10-15 14:38:34 <ligar> Just wondering if it is possible
164 2014-10-15 14:39:12 <ligar> Something like this wouldn't be needed anywhere in the near time frame but maybe in 10-15 years
165 2014-10-15 14:40:05 <justanotheruser> ligar: I'm confused. Why is 2 servers cheaper than one?
166 2014-10-15 14:40:26 <ligar> It would be nice to be able to split the full blockchain into smaller node for storage
167 2014-10-15 14:40:54 <ligar> I am looking at it when there is about 100K Tx per sec and the block chain hit 20TB
168 2014-10-15 14:41:38 <justanotheruser> ligar: do you know what SPV is?
169 2014-10-15 14:41:51 <ligar> nope
170 2014-10-15 14:41:55 <ligar> elighten me
171 2014-10-15 14:42:48 <ligar> NM i have the power! (google)
172 2014-10-15 14:46:59 <ligar> I not really sure SPV invalidates what I was thinking about
173 2014-10-15 16:04:56 <Arnavion> Damn
174 2014-10-15 16:05:26 <Arnavion> bitcoin-qt showed an error about needing to close because of a RunawayException or something, and to check debug.log
175 2014-10-15 16:05:41 <Arnavion> I didn't think to screencap the dialog box, and debug.log's doesn't have anything about the crash
176 2014-10-15 16:05:49 <Arnavion> debug.log doesn't*
177 2014-10-15 16:08:10 <Arnavion> Oh, it did print the dialog box's contents to stdout
178 2014-10-15 16:10:05 <Arnavion> http://privatepaste.com/ccdedc50f7   This is stdout and the last few bits from debug.log
179 2014-10-15 16:11:36 <Arnavion> Oh, haha
180 2014-10-15 16:11:51 <Arnavion> snapperd caused an error in btrfs and put the whole filesystem in read-only mode
181 2014-10-15 16:12:03 <Arnavion> That explains it
182 2014-10-15 16:15:51 <ThomasV> any core dev here?
183 2014-10-15 16:19:10 <blast_> most of them are, although seemingly afk atm
184 2014-10-15 16:34:06 <coryfields_> sipa: full sync on osx without issue w/ headers-first. very speedy :)
185 2014-10-15 16:42:13 <jgarzik> ThomasV, generally yes
186 2014-10-15 16:43:05 <ThomasV> jgarzik: I am listed as an author of bip39, but I actually disagree with its content. can you remove me from the list?
187 2014-10-15 16:44:10 <jgarzik> ThomasV, can you submit a pull request for that?
188 2014-10-15 16:44:18 <ThomasV> ok
189 2014-10-15 16:44:35 <jgarzik> ThomasV, it will look much better coming from you, than anyone else
190 2014-10-15 16:45:04 <ThomasV> sure
191 2014-10-15 16:50:13 <gmaxwell> ThomasV: you can open a pull req is what I think jeff is saying. :) Will happily merge asap.
192 2014-10-15 17:25:38 <ThomasV> gmaxwell: https://github.com/bitcoin/bips/pull/111
193 2014-10-15 17:26:02 <gmaxwell> merged.
194 2014-10-15 17:26:26 <ThomasV> thanks
195 2014-10-15 19:34:56 <kruug> attempting to set up gitian to build with, but I keep getting this error when attempting to setup the lxc vm's http://paste.debian.net/127015/
196 2014-10-15 19:35:26 <kruug> using these instructions
197 2014-10-15 19:35:27 <kruug> https://github.com/bitcoin/bitcoin/blob/master/doc/gitian-building.md#create-a-new-virtualbox-vm
198 2014-10-15 20:27:00 <kruug> Anyone able to offer assistance with gitian?
199 2014-10-15 20:27:18 <kruug> Since DevRandom doesn't offer much in the way of documentation/support channels :/
200 2014-10-15 21:55:42 <fanquake> coreyfields_ See my comment in #4993
201 2014-10-15 21:56:06 <fanquake> I can make a pull for that if you intend on merging that fix.
202 2014-10-15 22:45:53 <coryfields_> fanquake: sure
203 2014-10-15 22:46:47 <coryfields_> fanquake: please be sure to add it everywhere that's needed. i should think you'd need it for test-bitcoin_qt too
204 2014-10-15 22:53:34 <coryfields_> gmaxwell: ping
205 2014-10-15 22:53:37 <gmaxwell> pong
206 2014-10-15 22:54:13 <coryfields_> gmaxwell: are you familiar with our LockedPageManager ?
207 2014-10-15 22:55:25 <gmaxwell> yes, it's kinda broken... it really needs a different appraoch (an allocator pool). Not super familar anymore, long time since I looked at it.
208 2014-10-15 22:56:27 <coryfields_> ok. up for discussing a 3rd approach, then? or... maybe briefly explaining why it'd benefit from pooling?
209 2014-10-15 22:57:02 <coryfields_> (i'm looking to nuke boost there, for the sake of signing in a future lib)
210 2014-10-15 22:57:12 <sipa> because now if you allocate 1000 times 4 bytes, you may end up locking 1000 pages
211 2014-10-15 22:57:31 <sipa> pooling would reuse the already locked pages for multiple objects
212 2014-10-15 22:58:54 <sipa> coryfields_: there's no way you can avoid threading, right?
213 2014-10-15 22:59:02 <coryfields_> ok, maybe 3rd option is worth discussing, then..
214 2014-10-15 22:59:03 <sipa> and i don't want to go back to per-OS threading code
215 2014-10-15 22:59:28 <coryfields_> sipa: i reimplemented using posix_memalign on a page boundary
216 2014-10-15 23:00:00 <sipa> hmm?
217 2014-10-15 23:00:10 <coryfields_> so afaik, that means no need for bookkeeping, just lock the current pages and unlock when done
218 2014-10-15 23:00:21 <coryfields_> that's why i wanted to discuss, i fear i'm missing some major aspect :)
219 2014-10-15 23:00:51 <sipa> hmm
220 2014-10-15 23:01:22 <coryfields_> are there many (any) cases where we're allocating > $pagesize ?
221 2014-10-15 23:02:02 <coryfields_> where pagesize is assumed to be the lowest realistic size we'd encounter in the wild
222 2014-10-15 23:03:40 <sipa> coryfields_: no, the opposite
223 2014-10-15 23:04:12 <sipa> so i think your approach would work, actually, but it makes the average case equal to the current worst case
224 2014-10-15 23:04:24 <coryfields_> if no, then with posix_memalign aligned to page boundary, we can use our allocator to lock the current structure and know that it won't overlap with the next
225 2014-10-15 23:04:26 <sipa> (a separate memlock for every allocation)
226 2014-10-15 23:04:47 <sipa> but i think they are used so infrequently that it doesn't really matter
227 2014-10-15 23:05:05 <sipa> is posix_memalign available on window?
228 2014-10-15 23:05:12 <coryfields_> there's something similar, yes
229 2014-10-15 23:05:36 <coryfields_> _aligned_malloc
230 2014-10-15 23:06:07 <coryfields_> (fwiw, it's standard in c++11)
231 2014-10-15 23:06:31 <kruug> ugh, no gitian helpers yet?
232 2014-10-15 23:06:44 <sipa> kruug: sorry, haven't used gitian in ages
233 2014-10-15 23:07:02 <coryfields_> sipa: worth pushing up my POC as an RFC, then?
234 2014-10-15 23:07:07 <sipa> and the gitian descriptors in master aren't up t date with the dependency system
235 2014-10-15 23:07:12 <sipa> coryfields_: sure
236 2014-10-15 23:07:15 <coryfields_> kruug: i can help after this discussion
237 2014-10-15 23:07:47 <coryfields_> kruug: quick hint though, bail on lxc for now.
238 2014-10-15 23:08:12 <sipa> secure_allocator is really only used for private keys and encryption keys
239 2014-10-15 23:08:43 <coryfields_> yea, there seem to be only 2 users of it. i figured it'd be ok to drop the singleton and force those users to use a vector w/ secure_allocator
240 2014-10-15 23:08:47 <sipa> so you could as well just add $PAGESIZE to the block being malloced, and round up the returned pointer to be page-aligned, and lock that
241 2014-10-15 23:09:05 <sipa> CKey uses the locked page manager without using a vector
242 2014-10-15 23:09:12 <sipa> or anything allocatorable
243 2014-10-15 23:09:28 <sipa> as it also works on stack area
244 2014-10-15 23:09:33 <coryfields_> it has to live on stack?
245 2014-10-15 23:09:47 <sipa> i avoid heap memory wherver possible, yes
246 2014-10-15 23:10:01 <coryfields_> ok, fair enough
247 2014-10-15 23:10:31 <sipa> which sounds hard to combine with your suggested approach
248 2014-10-15 23:10:48 <sipa> but it's not a very strng requirement, i guess - CKey is not performance critical, mostly
249 2014-10-15 23:11:43 <kruug> coryfields_: so, install kvm and leave off the --lxc flags?
250 2014-10-15 23:12:07 <coryfields_> kruug: i would definitely start with that, yes
251 2014-10-15 23:12:21 <sipa> coryfields_: you may want impus from wumpus, who wrote LockedPageManager
252 2014-10-15 23:13:32 <coryfields_> sure
253 2014-10-15 23:13:52 <coryfields_> sipa: i'm not convinced of the approach yet, just wanted to make sure i understood the underlying issues
254 2014-10-15 23:14:20 <coryfields_> if we went forward with this, must of that movement for boost wouldn't be needed
255 2014-10-15 23:14:34 <coryfields_> (not that it's bad movement though, imo)
256 2014-10-15 23:14:38 <coryfields_> *most
257 2014-10-15 23:14:43 <sipa> i think it's fine to change CKey's vch back into a vector
258 2014-10-15 23:14:49 <sipa> so it can use an allocator
259 2014-10-15 23:15:11 <kruug> ugh, same issue... `The file /dev/mapper/loop0p1 does not exist and no size was specified.`
260 2014-10-15 23:15:39 <coryfields_> kruug: are you running in a vm, by chance?
261 2014-10-15 23:15:50 <kruug> coryfields_: esxi VM, yeah
262 2014-10-15 23:16:29 <kruug> do i not need to do this if I'm already in a VM?
263 2014-10-15 23:16:45 <sipa> not all VMs support a second level VM
264 2014-10-15 23:16:46 <coryfields_> kruug: knowing nothing of the /dev/mapper issues at play here, sounds like your guest can't be used to host
265 2014-10-15 23:17:35 <coryfields_> kruug: gitian creates a vm, then runs a build in it
266 2014-10-15 23:18:10 <gmaxwell> coryfields_: basically mlock is only really sane to use on a contigious hunk of memory. What happens with the current approach is that you quickly run out of the mlock ulimit and nothing further is locked.
267 2014-10-15 23:18:23 <coryfields_> sipa: i considered over-allocating as you suggested above, but i found a hole in it. now i can't remember what it was...
268 2014-10-15 23:18:24 <gmaxwell> (and its a silent failure)
269 2014-10-15 23:18:39 <sipa> gmaxwell: unconvinced; i doubt we often have many mlocked objects simultaneously
270 2014-10-15 23:19:40 <coryfields_> gmaxwell: ah, so that was the basis for the pooling idea i suppose?
271 2014-10-15 23:20:08 <sipa> coryfields_: the overallocation trick works, but you need a pointer back somewhere to find the start of the malloced region to free it
272 2014-10-15 23:20:22 <sipa> so you need $PAGESIE + $PTRSIZE overallocation i guess
273 2014-10-15 23:20:24 <gmaxwell> sipa: the default limit in fedora, for example, is 32k.
274 2014-10-15 23:20:26 <gmaxwell> (IIRC)
275 2014-10-15 23:20:40 <sipa> i doubt we ever have more than the encryption key + one CKey
276 2014-10-15 23:21:34 <gmaxwell> Another issue, is that lock lock unlock can potentially unlock both.
277 2014-10-15 23:21:47 <sipa> yes, that's why lockedpagemamanger exists
278 2014-10-15 23:21:52 <sipa> to keep refcounts on locked pages
279 2014-10-15 23:22:15 <coryfields_> why would there be a need for refcounting if aligned to page boundaries?
280 2014-10-15 23:22:25 <sipa> there is no need in that case
281 2014-10-15 23:22:32 <kruug> argh...
282 2014-10-15 23:22:40 <sipa> as you'll never allocate twice in the same page
283 2014-10-15 23:22:45 <kruug> coryfields_: nested virtualization is disabled >.<
284 2014-10-15 23:23:09 <coryfields_> sipa: ok good, that was my intention :)
285 2014-10-15 23:23:33 <sipa> but it does mean effectively overallocating mlocked pages (but they'll be usable still for other purposes)
286 2014-10-15 23:24:22 <gmaxwell> sipa: in any case, I tested this before and had confirmed that we were actually failing to mlock, but it might be n-fixes ago. You can test pretty easilly by making sure to notice failure.
287 2014-10-15 23:24:33 <sipa> coryfields_: perhaps you can put some debug code in lockedpagemanager to print out the number of locked pages in the map, and then do a wallet encryption and send using it
288 2014-10-15 23:24:40 <coryfields_> right, so other data will end up locked as well, which doesn't seem problematic
289 2014-10-15 23:25:19 <sipa> indeed, but 32k max mlocked area in your example means at most 8 locked objects
290 2014-10-15 23:25:24 <sipa> regardless of their size
291 2014-10-15 23:25:59 <sipa> here (ubuntu 14.04) is is 64 KiB
292 2014-10-15 23:27:36 <coryfields_> mm, yea. that's pushing it.
293 2014-10-15 23:27:49 <sipa> note that the worst case right now is not any better
294 2014-10-15 23:28:12 <sipa> it only makes a difference if for some reason the standard allocator allocates the different locked objects in the same page
295 2014-10-15 23:29:07 <gmaxwell> well with the pool approach you'd have success so long as the sum of locked data and overhead is <32k (or whatever the target is).
296 2014-10-15 23:29:23 <sipa> yes, that would be very good
297 2014-10-15 23:29:31 <coryfields_> the pool approach implies some sort of global state though, which i'm really hoping to avoid
298 2014-10-15 23:29:46 <sipa> allocators by definition have some global state
299 2014-10-15 23:29:56 <sipa> it's just that the standard one hides it
300 2014-10-15 23:30:17 <coryfields_> at the shipped-lib level, i mean
301 2014-10-15 23:30:18 <sipa> so by leveraging the state of the standard allocator, you avoid explicit threading code
302 2014-10-15 23:31:00 <gmaxwell> or we could perhaps just change things to allocate some object with all the locked data we'll ever have in it.. up front.
303 2014-10-15 23:31:11 <gmaxwell> and then we just need locking for that object.
304 2014-10-15 23:31:25 <sipa> ... and a custom allocator implementation for it
305 2014-10-15 23:31:34 <sipa> i think we first need to know what the sizes and counts are of objects we typically want locked
306 2014-10-15 23:31:38 <sipa> i think they're few
307 2014-10-15 23:33:52 <coryfields_> sipa: ok, any worst-case you can think of other than your proposed test above?
308 2014-10-15 23:34:13 <sipa> no - my guessis that we'd only have 2-3 objects locked at any time
309 2014-10-15 23:34:30 <gmaxwell> I mean, even forget the custom allocator. if we know what data we'll need locked, just make some object with fields for each part of it up front. lock that object at start.. then just use it for all the storage of that data, since we use it in so few a number of places.
310 2014-10-15 23:35:01 <sipa> but gmaxwell's experience shows otherwise apparently, so either his information is out of date, or my understanding of how CKey's and similar are used
311 2014-10-15 23:35:24 <sipa> gmaxwell: well you can be signing stuff from different threads etc simultaneously
312 2014-10-15 23:35:25 <coryfields_> gmaxwell: for a handful of uses, it is quite over-engineered
313 2014-10-15 23:38:16 <gmaxwell> my information could be out of date; as I said it can be tested by making sure the failures are not silent.
314 2014-10-15 23:38:46 <coryfields_> well, given its few uses, i suppose it'd be reasonable to hard-code the element size, then. that'd avoid much of the pooling complication.
315 2014-10-15 23:40:14 <coryfields_> i'll experiment some. thanks for the help.
316 2014-10-15 23:41:14 <gmaxwell> one complication is that the usage is really inadequate. E.g. the json parsing and qt allocate memory and can end up leaking your decryption keys.
317 2014-10-15 23:42:30 <coryfields_> how do you mean?
318 2014-10-15 23:43:26 <gmaxwell> coryfields_: you enter in wallet passphrases through json and through QT gui.
319 2014-10-15 23:43:43 <gmaxwell> both involve library code which we didn't write, which (I assume!) does its own allocations.
320 2014-10-15 23:44:04 <coryfields_> gmaxwell: oh, right. indeed, i've wondered about this odd line we've drawn in the sand
321 2014-10-15 23:44:33 <gmaxwell> I mean, it isn't unreasonable to say "we take care of our part, even though we know it isn't enough".
322 2014-10-15 23:44:48 <coryfields_> turtles all the way down though i suppose, like compiling your compiler ;)
323 2014-10-15 23:45:34 <gmaxwell> with the change to jeff's json stuff we could reasonably be sure that the json parts were covered too, I expect.