1 2014-09-15 00:46:04 <nullbyte> .
2 2014-09-15 03:46:20 <nickodell> Hi, I just submitted a pull request, and I'd like to hear any feedback you guys might have.
3 2014-09-15 03:46:24 <nickodell> https://github.com/bitcoin/bitcoin/pull/4916
4 2014-09-15 06:49:52 <wumpus> nickodell: I think it's overkill; I intentionally changed to use wildcards in 0.9
5 2014-09-15 06:49:56 <wumpus> nickodell: +not
6 2014-09-15 06:54:05 <wumpus> in the well-defined notation for subnets (/x.x.x.x and /y) it is clear what to expect, wildcards were always a bad choice
7 2014-09-15 07:13:43 <wumpus> I mean it's just a matter of taste whether you like 129.125.* or 129.125.0.0/16 better, but the latter is the same you'd use in firewall rules
8 2014-09-15 07:15:31 <gwillen> the former is more obvious to average people, to be fair
9 2014-09-15 07:15:39 <gwillen> (and it's what we use in IRC banmasks)
10 2014-09-15 07:18:42 <phantomcircuit> gwillen, irc banmasks are actually wildcards
11 2014-09-15 07:18:51 <phantomcircuit> ie
12 2014-09-15 07:18:56 <phantomcircuit> 192.168.*.0
13 2014-09-15 07:18:58 <wumpus> in IRC banmasks you can also do say *.google.com
14 2014-09-15 07:19:02 <gwillen> yes, that's true
15 2014-09-15 07:19:15 <gwillen> and if you want you can do 192.168.1*2.7, or something equally nuts
16 2014-09-15 07:19:18 <gwillen> since it's just string-matching
17 2014-09-15 07:19:34 <wumpus> in the case of *names*, wildcards make sense, but here we're pattern matching network addresses
18 2014-09-15 07:19:49 <gmaxwell> freenode's IRC servers can also do CIDR.
19 2014-09-15 07:19:54 <gwillen> but it is true that the wildcard syntax is much more obvious to anybody who doesn't work with networks
20 2014-09-15 07:20:12 <gwillen> foo.* is self-evident, whereas foo/24 is nonsense to someone who's never used CIDR before
21 2014-09-15 07:20:51 <gwillen> (and of course the name "CIDR" is a throwback which describes itself as the opposite of a thing that hasn't existed in a decade. But that's neither here nor there. ;-)
22 2014-09-15 07:20:53 <wumpus> but if you're setting up allow rules, *you are working with networks*
23 2014-09-15 07:21:20 <gmaxwell> gwillen: not just equally nuts, but underdefined. e.g. does your 192.168.1*2.7 match 192.168.12.7? :)
24 2014-09-15 07:21:23 <gwillen> wumpus: sure, but that doesn't mean your a network professional
25 2014-09-15 07:21:29 <gwillen> you're, argh
26 2014-09-15 07:21:37 <gwillen> it could just mean you want to open something to your LAN
27 2014-09-15 07:21:59 <gwillen> the tragic truth is that many people in this era are LAN administrators involuntarily ;-)
28 2014-09-15 07:22:02 <phantomcircuit> gwillen, it will however confuse people when 1*.0.0.* doesn't work
29 2014-09-15 07:22:05 <wumpus> the most obvious use of wildcards is '*', and I'd like to avoid that one to be used easily
30 2014-09-15 07:22:20 <gwillen> phantomcircuit: well, this particular CL avoids that, since it has a nice error message if you try that
31 2014-09-15 07:22:45 <gwillen> gmaxwell: in the case of banmasks I'm pretty sure it does
32 2014-09-15 07:22:46 <phantomcircuit> gwillen, still gonna confuse people i think
33 2014-09-15 07:22:49 <gwillen> ACTION shrug
34 2014-09-15 07:22:53 <gmaxwell> gwillen: normal network masking behavior is almost, though indeed, not quite universally used. There are bazillions of guides and calculators online.
35 2014-09-15 07:23:25 <gmaxwell> gwillen: it does, but thats my point, arguably it could go either way. 1*2 matching 12 is pretty surprising though!
36 2014-09-15 07:23:32 <gwillen> it just always seems unfortunate to me when there's clearly at least someone who really wants a feature, enough to spend a lot of time writing it
37 2014-09-15 07:23:58 <gwillen> and yet it does not make it in :-)
38 2014-09-15 07:24:08 <wumpus> gwillen: I agree, the problem is if we end up with a lot of 'would be nice' features, which cost time to maintain and test
39 2014-09-15 07:24:14 <gwillen> (maybe not a _lot_ of time. But it's a very nice CL.)
40 2014-09-15 07:24:31 <phantomcircuit> gmaxwell, iirc the correct way to do that is 192.168.1?*2.7
41 2014-09-15 07:24:32 <wumpus> if there is just *one true way* to provide the input it's harder for strange edge cases to come up
42 2014-09-15 07:25:44 <gmaxwell> or creates risks when you goof it up and accidentally allow the whole internet. (which people have done. :()
43 2014-09-15 07:30:07 <phantomcircuit> gmaxwell, or more hilariously ban the entire internet
44 2014-09-15 07:30:10 <phantomcircuit> which i have done
45 2014-09-15 07:30:51 <gmaxwell> on IRC thats the consequence. which is funny. in rpcallow in bitcoin, the consequence is you allow the whole internet to steal your bitcoins. :(
46 2014-09-15 07:35:33 <wumpus> it looks like nickodell is actually trying to do 'rpcallowip=*'... wonder why https://github.com/bitcoin/bitcoin/issues/4894
47 2014-09-15 07:38:40 <gmaxwell> erk. well if =0/0 we're almost always a misconfiguration, I'd say that the special case of =* might be not so crazy to handle.
48 2014-09-15 07:38:45 <gmaxwell> er weren't
49 2014-09-15 07:39:07 <Luke-Jr> ^
50 2014-09-15 07:39:18 <wumpus> handle, as in print a rant why it's a bad idea? :p
51 2014-09-15 07:39:58 <Luke-Jr> ACTION wonders how we behave if someone puts bits in the netmasked-off part
52 2014-09-15 07:40:15 <wumpus> they're ignored
53 2014-09-15 07:40:22 <Luke-Jr> maybe we should error in that case
54 2014-09-15 07:40:50 <phantomcircuit> yeah... if someone enters 0/0 as the netmask we should probably complain really loudly but have another flag to over ride it
55 2014-09-15 07:40:52 <phantomcircuit> something like
56 2014-09-15 07:40:59 <phantomcircuit> no-i-really-want-to-do-this
57 2014-09-15 07:41:05 <wumpus> Luke-Jr: that's a good point though
58 2014-09-15 07:41:56 <gmaxwell> I think there are valid applications of it, though they're specialized. For example, if you are completely delegating that control to an external firewall (which, perhaps you've automated its reconfiguration...)
59 2014-09-15 07:42:26 <wumpus> Luke-Jr: should at least be a test in subnet_test for that case
60 2014-09-15 07:43:47 <wumpus> gmaxwell: agreed, but for people with such advanced configurations, thinking of using ::/0 for 'everything' shouldn't be difficult :)
61 2014-09-15 07:44:36 <wumpus> or 0.0.0.0/0 but that allows only ipv4
62 2014-09-15 07:45:18 <wumpus> as bitcoin treats ipv4 as part of ipv6 there is actually no way to do the opposite...
63 2014-09-15 07:46:41 <gmaxwell> sure sure, just not breaking it completely would be nice.
64 2014-09-15 07:47:03 <Luke-Jr> sigh, what is the dev ML becoming..
65 2014-09-15 07:47:39 <wumpus> sure; at the moment just * (and other combinations with wildcards) don't work, but other ways of specifying 'everything' work. sp ot
66 2014-09-15 07:47:47 <wumpus> so it's possible but just a bit harder
67 2014-09-15 07:48:11 <wumpus> Luke-Jr: just ignore, or ban, please don't reply
68 2014-09-15 07:51:24 <gmaxwell> Luke-Jr: I assume it's not just mr eranig thats wtfing you?
69 2014-09-15 07:52:45 <wumpus> I supposed it's "ggprodukcija kig"
70 2014-09-15 07:53:14 <wumpus> oh right :)
71 2014-09-15 08:01:39 <Luke-Jr> gmaxwell: it just seems to get worse and worse :x
72 2014-09-15 08:03:36 <wumpus> does it? apart from lone posts such as this, I don't see a clear pattern of it becoming worse
73 2014-09-15 08:04:04 <Luke-Jr> wumpus: perhaps things are not quite as they seem :D
74 2014-09-15 08:07:16 <bardi_harborow> Is there a specific reason why the Bitcoin reference implementation was coded in C++? I understand that it is a way to get maximum performance, but by definition a reference client is designed to be for "reference", not for production use. It is extremely annoying having to dig through C++ code (many developers don't know C++, including myself). Maybe I'm just too lazy to learn C++ and I'm just bitching, but I would like an
75 2014-09-15 08:07:16 <wumpus> and even this occurence could be explained with language barriers
76 2014-09-15 08:08:10 <Luke-Jr> bardi_harborow: C++ is one of the most popular programming languages, and easily understood
77 2014-09-15 08:08:11 <lclc> bardi_harborow: look at an alternative implemenation if you don't like reading C++ code, e.g. btcd
78 2014-09-15 08:08:28 <Luke-Jr> lclc: something even more obscure than C++ :p
79 2014-09-15 08:09:04 <lclc> Luke-Jr: Go is easier to read for Python etc developers afaik
80 2014-09-15 08:09:05 <wumpus> because the prophet satoshi delivered his tablets with the initial bitcoin code from the mountain encoded in c++
81 2014-09-15 08:09:07 <brisque> Luke-Jr: want to see my bitcoin client written entirely in 8008 assembly?
82 2014-09-15 08:09:24 <Luke-Jr> brisque: no, I value my eyes
83 2014-09-15 08:09:29 <wumpus> brisque: yes! although I prefer Z80 assembly, but it's close enough
84 2014-09-15 08:09:44 <Luke-Jr> I much prefer MIPS assembly.
85 2014-09-15 08:10:32 <brisque> imagine how little people could actually audit a MIPS-bitcoind
86 2014-09-15 08:10:37 <bardi_harborow> Are you try to tell me that `print('Hello World!')` isn't better than `#include <iostream> int main() { std::cout << "hello world!\n"; return 0; }`
87 2014-09-15 08:11:24 <phantomcircuit> bardi_harborow, if you dont know c++
88 2014-09-15 08:11:31 <phantomcircuit> you're probably not really a developer
89 2014-09-15 08:11:35 <wumpus> anyhow, bitcoind's c++ is quite understandable, as far as c++ code goes, compared to say boost...
90 2014-09-15 08:11:40 <phantomcircuit> ACTION runs from the inevitable flame war
91 2014-09-15 08:12:43 <bardi_harborow> That Go implementation is music to my ears! Thank you. However I still don't know what a struct is. (Sounds like some Googling is in order.)
92 2014-09-15 08:13:36 <Luke-Jr> bardi_harborow: a class without code.
93 2014-09-15 08:14:10 <bardi_harborow> So should I actually learn C++, is that worthwhile? (phantomcircuit: I do at least agree that bitcoind's code is good as far as C++ goes)
94 2014-09-15 08:14:22 <bardi_harborow> Luke-Jr: Hmm... And the point?
95 2014-09-15 08:14:29 <wumpus> bardi_harborow: let's start at point 1 first... what are you trying to do?
96 2014-09-15 08:15:43 <wumpus> learning bitcoin deeply, from a high level, goes far beyond learning any programming language, there's also ECDSA math, theory about robust P2P networks, ... understanding source code is useful, but may not be the best starting point, unless your goal is to contribute to it
97 2014-09-15 08:15:55 <phantomcircuit> bardi_harborow, i have yet to meet a competent developer who didn't at least know the basics of c and c++
98 2014-09-15 08:17:19 <bardi_harborow> wumpus: I understand most of the Bitcoin components. (Though I'm a little rusty as to which endian things are in.) I just don't know EXACTLY how the code works.
99 2014-09-15 08:17:33 <wumpus> phantomcircuit: but the guy didn't even mention trying to develop anything, so let's not start with judging his skill as developer
100 2014-09-15 08:17:56 <wumpus> although granted, this is #bitcoin-dev :p
101 2014-09-15 08:18:19 <Luke-Jr> at least bardi_harborow knows what endian is :p
102 2014-09-15 08:18:20 <phantomcircuit> wumpus, many developers don't know C++, including myself
103 2014-09-15 08:18:49 <phantomcircuit> lots of schools dont teach anything about c or c++ now
104 2014-09-15 08:18:52 <phantomcircuit> which is unfortunate
105 2014-09-15 08:19:10 <bardi_harborow> The biggest thing I have a problem with is the lack or a proper API. The only way I can talk to Bitcoind is RPC and by keeping my keys on a server. We REALLY Watch only addresses.
106 2014-09-15 08:19:12 <bardi_harborow> need*
107 2014-09-15 08:19:32 <Luke-Jr> phantomcircuit: s/about.*//
108 2014-09-15 08:19:44 <wumpus> bardi_harborow: if you really want to read source code, you don't need to read the bitcoind code, you could also read bitcoinj, or one of the other full node or SPV implemenatations
109 2014-09-15 08:19:45 <lclc> bardi_harborow: use a blockchain service (or set up your own) for that, e.g. obelisk
110 2014-09-15 08:19:46 <phantomcircuit> Luke-Jr, heh
111 2014-09-15 08:20:03 <Luke-Jr> bardi_harborow: what we really need are watch-only *wallets*.
112 2014-09-15 08:20:49 <wumpus> bardi_harborow: the code in git master (which will be 0.10 at some point in the future) supports watch-only w/ importaddress
113 2014-09-15 08:21:22 <wumpus> bardi_harborow: there are also plenty of other projects which allow looking up addresses, such as bitpay's insight
114 2014-09-15 08:21:37 <bardi_harborow> lclc: I did try to con someone on BT into funding/helping develop a web blockchain explorer + API + wallet based on obelisk but no one took the bait.
115 2014-09-15 08:22:48 <bardi_harborow> wumpus: Oh, they got around to adding it finally? I was following the pull request at some stage.
116 2014-09-15 08:22:48 <brisque> not deceiving people might be a nice start.
117 2014-09-15 08:23:04 <bardi_harborow> brisque: Huh?
118 2014-09-15 08:23:08 <brisque> bardi_harborow: er, that's a very early feature.
119 2014-09-15 08:23:17 <lclc> there are already a few of those out there (blockchain explorer), so no point in writing your own IMHO. an Obelisk is the API. Wallets hm.. if you have an idea for a feature the others don't have maybe
120 2014-09-15 08:23:36 <brisque> > I did try to con someone
121 2014-09-15 08:23:38 <brisque> don't do that.
122 2014-09-15 08:25:19 <bardi_harborow> "con" is a metaphor indicating that people are reluctant to do so and therefore it is hard to convince them and you feel like a scammer because you have to try so hard. (Not sure if I'm explaining this properly, it made sense to me.)
123 2014-09-15 08:26:07 <wumpus> you meant it figuratively, but be careful with that word here, people are very sensitive to it :)
124 2014-09-15 08:26:15 <brisque> a "con" is a term used for defrauding people.
125 2014-09-15 08:26:16 <Luke-Jr> bardi_harborow: I think that is a niche aspect of your culture. Usually it indicates malice or deception.
126 2014-09-15 08:27:08 <bardi_harborow> ACTION hides from the inadvertent flamewar he has created.
127 2014-09-15 08:29:34 <brisque> unfortunate misunderstanding of English there I suspect.
128 2014-09-15 08:30:26 <wumpus> yes
129 2014-09-15 08:35:15 <wumpus> Luke-Jr: re: #4919, so we were actually compiling the tests without the glibc compatibility?
130 2014-09-15 08:35:31 <Luke-Jr> wumpus: that one test, I presume
131 2014-09-15 08:35:47 <Luke-Jr> so, just Linux ARM
132 2014-09-15 08:35:58 <wumpus> Luke-Jr: right, thanks
133 2014-09-15 08:36:41 <wumpus> it's not that important on ARM as we don't distribute executables for that, still, it makes sense to be consistent (and the = was clearly intended)
134 2014-09-15 08:40:07 <wumpus> I see this in the travis output: The command "if [ "$RUN_TESTS" = "true" ]; then make check; fi" exited with 0.
135 2014-09-15 08:40:15 <wumpus> does that mean the tests don't run at all?
136 2014-09-15 08:41:26 <wumpus> I'm still getting used to travis a bit
137 2014-09-15 08:44:04 <wumpus> ok, so HOST=arm-linux-gnueabihf and HOST=x86_64-w64-mingw32 only do a build, not testing
138 2014-09-15 08:45:09 <wumpus> or the other way around - only *linux* run the tests
139 2014-09-15 08:45:18 <wumpus> eh, x86/x86_64 linux
140 2014-09-15 08:54:28 <wumpus> ok, commented on #4907
141 2014-09-15 08:55:13 <phantomcircuit> brisque, no that is actually an accepted use of the term in certain geographic areas
142 2014-09-15 08:55:25 <phantomcircuit> it is however ill advised
143 2014-09-15 08:55:48 <wumpus> phantomcircuit: just avoid it in bitcoin circles, because we're so used to actually getting conned :p
144 2014-09-15 09:00:47 <phantomcircuit> wait what
145 2014-09-15 09:00:53 <phantomcircuit> tests are running on windows o.o
146 2014-09-15 09:01:47 <wumpus> yes, running the tests in wine has actually caught a few bugs already, wine is stricter in windows API use than windows itself
147 2014-09-15 09:01:59 <phantomcircuit> ha
148 2014-09-15 09:03:21 <wumpus> what travis could add is also running the w64 tests (the old pulltester uses an ancient mingw, which didn't support 64 bit yet)
149 2014-09-15 09:09:57 <phantomcircuit> so i noticed that the ckey bdb objects dont have a checksum
150 2014-09-15 09:10:15 <phantomcircuit> and as such are verified by calculating the public key from the private key when the wallet passphrase is supplied
151 2014-09-15 09:10:25 <phantomcircuit> this takes ages
152 2014-09-15 09:10:46 <wumpus> phantomcircuit: this only checks one key, except the first time
153 2014-09-15 09:11:04 <phantomcircuit> wumpus, the first time the wallet is loaded
154 2014-09-15 09:11:17 <wumpus> no, the first time a passphrase is specified
155 2014-09-15 09:11:18 <phantomcircuit> but for huge wallets calculating the public key is extremely expensive
156 2014-09-15 09:11:25 <phantomcircuit> oh really?
157 2014-09-15 09:11:26 <phantomcircuit> huh
158 2014-09-15 09:12:10 <wumpus> there is a recent commit changing that behavior; it used to check only one key every time the pass phrase was given, now it checks all keys the first time, and only one key after that
159 2014-09-15 09:14:09 <phantomcircuit> wumpus, i tried it yesterday and it seemed like it was generating the public key in ram for all of the keys the first time a passphrase was specified to check for corruption
160 2014-09-15 09:14:22 <phantomcircuit> am i wrong about the reason for generating the pub key?
161 2014-09-15 09:15:24 <wumpus> phantomcircuit: see https://github.com/bitcoin/bitcoin/pull/4670
162 2014-09-15 09:15:37 <wumpus> (and discussion in #4011)
163 2014-09-15 09:16:06 <wumpus> and yes, it does that indeed as integrity check
164 2014-09-15 09:16:42 <wumpus> note that the original proposal was to check all keys every time, I already got that watered down to just the first time
165 2014-09-15 09:16:46 <phantomcircuit> hmm didn't notice that
166 2014-09-15 09:16:50 <phantomcircuit> wish i had
167 2014-09-15 09:18:14 <wumpus> as I see it the problem is that no one developing bitcoin core is actually using the wallet (at least with any serioius workload)
168 2014-09-15 09:18:37 <phantomcircuit> wumpus, this should definitely be replaced with a simple checksum
169 2014-09-15 09:18:42 <wumpus> another one of my arguments for splitting it off
170 2014-09-15 09:18:53 <wumpus> we shouldn't be maintaining something we don't care about
171 2014-09-15 09:18:59 <phantomcircuit> GetPubKey takes about 250x longer than sha256d
172 2014-09-15 09:19:18 <wumpus> phantomcircuit: well there is an obvious reason to have verification slow...
173 2014-09-15 09:19:27 <phantomcircuit> wumpus, ?
174 2014-09-15 09:19:34 <wumpus> brute forcing
175 2014-09-15 09:19:42 <phantomcircuit> this has no effect on that
176 2014-09-15 09:20:02 <wumpus> the cheaper it is to check that the result of decryption is valid, the easier it is to brute force
177 2014-09-15 09:20:17 <wumpus> that's why a full ecdsa check is required
178 2014-09-15 09:22:05 <phantomcircuit> wumpus, that's covered by the function SetKeyFromPassphrase
179 2014-09-15 09:22:20 <wumpus> checking all keys is indeed not relevant for this, but replacing the full-ecdsa integrity check with a sha256 checksum would be
180 2014-09-15 09:22:23 <phantomcircuit> EVP_BytesToKey takes a parameter that specifies the number of rounds
181 2014-09-15 09:22:52 <wumpus> that's only for converting the passphrase to a binary key, it doesn't avoid brute forcing the binary keys theyselves
182 2014-09-15 09:23:02 <phantomcircuit> ???
183 2014-09-15 09:23:12 <phantomcircuit> it's a 256 bit key used for aes
184 2014-09-15 09:23:22 <phantomcircuit> you gotta be kidding
185 2014-09-15 09:23:53 <phantomcircuit> adjusting the time it takes to generate a key from a passphrase is the only place that the time to verify should be adjusted
186 2014-09-15 09:24:17 <wumpus> I'm sure defense in depth is appropriate here
187 2014-09-15 09:25:36 <phantomcircuit> :/
188 2014-09-15 09:25:50 <phantomcircuit> wumpus, if the intention is to provide for a delay
189 2014-09-15 09:25:58 <phantomcircuit> then this is broken from a security perspective
190 2014-09-15 09:26:09 <phantomcircuit> the implementation of GetPublicKey is about 4x away from being optimal
191 2014-09-15 09:26:19 <phantomcircuit> so an attacker has a 4x speed up relative to the normal user
192 2014-09-15 09:26:27 <wumpus> *only* 4x
193 2014-09-15 09:26:42 <phantomcircuit> for pass phrase strengthening that is an enormous improvement
194 2014-09-15 09:26:58 <wumpus> and probably the secp256k library speeds this up?
195 2014-09-15 09:27:17 <phantomcircuit> wumpus, secp256k lib plus not doing it TWICE which is what the code does
196 2014-09-15 09:27:49 <phantomcircuit> wumpus, this is why the only correct place to put the strengthening is in the nRounds parameter to EVP_BytesToKey
197 2014-09-15 09:27:51 <wumpus> ok, feel free to optimize it
198 2014-09-15 09:29:13 <brisque> phantomcircuit: incredibly ill advised. it's always going to cause problem for the rest of the english speaking world.
199 2014-09-15 09:29:56 <phantomcircuit> brisque, the cool parts of the english world know what it means
200 2014-09-15 09:29:58 <phantomcircuit> woot
201 2014-09-15 09:30:00 <phantomcircuit> ACTION runs
202 2014-09-15 09:30:39 <brisque> ACTION whacks phantomcircuit with a wet trout
203 2014-09-15 09:31:04 <wumpus> phantomcircuit: it would be acceptable to put the sha256sum on the encrypted data
204 2014-09-15 09:31:37 <wumpus> so integrity can be verified at loading time without having the passphrase, and it wouldn't affect decryption that way
205 2014-09-15 09:32:27 <phantomcircuit> wumpus, i was thinking hmac of the encrypted data using the passphrase as a key
206 2014-09-15 09:32:48 <phantomcircuit> doesn't introduce any new information
207 2014-09-15 09:33:04 <phantomcircuit> but it means the values cannot be verified at load time
208 2014-09-15 09:33:04 <wumpus> because the whole reason for the integrity check is to armor against database corruption, no need for the passphrase for that at all
209 2014-09-15 09:33:23 <phantomcircuit> gmaxwell, sipa do either of you have an opinion here?
210 2014-09-15 09:33:24 <wumpus> yeah, just like now
211 2014-09-15 09:34:16 <phantomcircuit> wumpus, a simple hash leaks whether the db is corrupt to someone without the passphrase
212 2014-09-15 09:34:23 <phantomcircuit> im not sure we actually care about that though
213 2014-09-15 09:34:24 <wumpus> phantomcircuit: ... so?
214 2014-09-15 09:34:33 <phantomcircuit> it's worth considering though
215 2014-09-15 09:34:36 <wumpus> other databases such as leveldb provide internal hashes for that
216 2014-09-15 09:34:53 <phantomcircuit> that's the only thing we'd be meaningfully changing by adding a hash
217 2014-09-15 09:34:54 <brisque> phantomcircuit: it's sort of far fetched, but couldn't you use knowledge of one wallet encrypted with two passwords, one known, to easily recover the password without the KDF to use on a different wallet? :P
218 2014-09-15 09:35:04 <phantomcircuit> so it is worth considering (at least for a moment :P)
219 2014-09-15 09:35:35 <phantomcircuit> brisque, code only supports a single passphrase
220 2014-09-15 09:35:36 <wumpus> for checking whether the passphrase is valid, checking one key suffices
221 2014-09-15 09:35:45 <wumpus> *given* the database is uncorrupted
222 2014-09-15 09:35:52 <brisque> ie we know A and B are the same contents encrypted with different keys, and B and C probably have the same key but it's unknown, you can attack B very quickly.
223 2014-09-15 09:36:00 <phantomcircuit> wumpus, oh right, that's the other thing
224 2014-09-15 09:36:13 <phantomcircuit> with the current setup there is no way to distinguish corruption from incorrect keys
225 2014-09-15 09:36:28 <wumpus> phantomcircuit: there is -- some keys decode fine, others don't
226 2014-09-15 09:36:32 <phantomcircuit> so theoretically someone could resist giving up the key
227 2014-09-15 09:36:43 <phantomcircuit> wumpus, the master key could be corrupt
228 2014-09-15 09:36:50 <phantomcircuit> all keys would fail
229 2014-09-15 09:36:56 <phantomcircuit> a single bit flip
230 2014-09-15 09:37:41 <phantomcircuit> that's a fairly strong argument for using an hmac
231 2014-09-15 09:37:43 <wumpus> we could armor against bitflips, with a custom wallet format
232 2014-09-15 09:38:18 <brisque> oh that would be incredibly amusing. imagine you have a key that's AES encrypted with a password. you decrypt it with the wrong password and use the resulting junk data as a private key. you now have the ability to give up one password and not expose the true funds.
233 2014-09-15 09:39:03 <phantomcircuit> wumpus, sure, but im saying that the possibility of a bitflip gives someone under duress the ability to prove a false pass phrase without the attacker being certain they're lying
234 2014-09-15 09:39:05 <wumpus> then again, it's better to just keep proper backups... or use a deterministic wallet, so that the smalll seed is the only thing you need to store and protect, not every key
235 2014-09-15 09:39:13 <phantomcircuit> which is itself a security feature potentially
236 2014-09-15 09:39:29 <phantomcircuit> (of course no real world attacker is going to believe you... but still)
237 2014-09-15 09:39:44 <brisque> wumpus: people will insist on adding "watch only" addresses though. having HD support in bitcoin core won't make the need for key storage go away.
238 2014-09-15 09:40:06 <brisque> well, I suppose we don't care about watch only integrity.
239 2014-09-15 09:40:12 <brisque> scratch that.
240 2014-09-15 09:40:28 <wumpus> phantomcircuit: that gets a bit obscure, but I agree it's amusing to think about, you could have multiple passphrases each 'containing coins' hehe
241 2014-09-15 09:41:16 <wumpus> then again, it also means that someone could make a typo (a very rare one) and end up with an empty wallet
242 2014-09-15 09:41:40 <phantomcircuit> wumpus, huh
243 2014-09-15 09:41:58 <wumpus> also the bitcoin core wallet format is completely unsuited to that kind of deniability... it friggin contains a copy of all of your transactions, unencrypted.
244 2014-09-15 09:42:10 <phantomcircuit> actually with hmac support this could be trivially modified such that each pass phrase was effectively it's own wallet
245 2014-09-15 09:42:30 <brisque> wumpus: did you know scammers use that?
246 2014-09-15 09:42:45 <wumpus> brisque: I'm not surprised
247 2014-09-15 09:42:59 <phantomcircuit> wumpus, well yeah it also contains a pubkey, what im saying is you could provide a false pass phrase and claim the encrypted master key is corrupt
248 2014-09-15 09:43:05 <phantomcircuit> all the transactions would be there
249 2014-09-15 09:43:08 <phantomcircuit> all the pubkeys
250 2014-09-15 09:43:20 <wumpus> brisque: it's a trade-off between security and convenience, those are always abused, but they're necessary nevertheless
251 2014-09-15 09:43:25 <phantomcircuit> everything would look normal
252 2014-09-15 09:43:33 <phantomcircuit> but the private keys wouldn't decrypt
253 2014-09-15 09:43:50 <phantomcircuit> there's no way to prove you're lying because literally a single bit flip could cause that
254 2014-09-15 09:44:00 <brisque> that wouldn't fly though
255 2014-09-15 09:44:15 <phantomcircuit> brisque, depends entirely on who your making the argument to
256 2014-09-15 09:44:19 <brisque> nobody would believe it, especially if you have multiple backups all with the same "flip"
257 2014-09-15 09:44:28 <wumpus> brisque: a part of the problem is that bitcoin core is a full-node wallet, so people are supposed to keep it running even if they're not directly using the wallet... so in the case that the wallet is locked, it still needs to display something
258 2014-09-15 09:44:53 <phantomcircuit> wumpus, what happens to the ui if you build with --disable-wallet ?
259 2014-09-15 09:44:58 <brisque> wumpus: hopefully one day that need goes away.
260 2014-09-15 09:45:10 <brisque> phantomcircuit: you get a sort of boring bitcoin core with statistics, I think.
261 2014-09-15 09:45:34 <wumpus> phantomcircuit: you get just the debug window
262 2014-09-15 09:46:10 <wumpus> which is as close to a proper 'node UI' that we've got
263 2014-09-15 09:46:23 <brisque> wasn't there a pull which changed the UI to the "bandwidth" view?
264 2014-09-15 09:46:40 <wumpus> the debug window *has* the bandwidth view
265 2014-09-15 09:46:47 <wumpus> that's just one of the tabs...
266 2014-09-15 09:46:50 <brisque> oh, sorry.
267 2014-09-15 09:47:02 <brisque> you can tell I don't use the GUI much.
268 2014-09-15 09:47:20 <wumpus> sometimes I'm afraid I'm the only one that does... sometimes
269 2014-09-15 09:47:37 <brisque> bitcoind-ncurses does the job though
270 2014-09-15 09:48:58 <phantomcircuit> wumpus, what are the block generation rules for regtest?
271 2014-09-15 09:49:01 <wumpus> in those dark times I think that just dropping all the GUI code, all the wallet code, would be the best way forward
272 2014-09-15 09:49:28 <brisque> please don't. core is the only one that gets things right.
273 2014-09-15 09:49:44 <phantomcircuit> ah right setgenerate
274 2014-09-15 09:49:52 <brisque> most clients don't use compressed point keys, don't use KDF properly, don't do a single thing right.
275 2014-09-15 09:50:30 <Arnavion> Then there's people like me who use the GUI with --disable-wallet
276 2014-09-15 09:50:31 <hearn> i think bitcoinj based wallets should get those things right, at least. well how strong to make the kdf is a wallet-specific thing
277 2014-09-15 09:51:02 <hearn> lighthouse auto-tunes it to always take about 2 seconds whatever hardware you have. other wallets don't bother, i think. need to push them to incorporate that code.
278 2014-09-15 09:51:13 <brisque> is there a bitcoinj wallet that allows connecting only to a local node, is HD, and uses compressed point keys?
279 2014-09-15 09:52:03 <hearn> yes
280 2014-09-15 09:52:04 <wumpus> I should really appoing a 'wallet maintainer' and 'GUI maintainer' so that I don't have to worry about those anymore
281 2014-09-15 09:52:07 <wumpus> appoint*
282 2014-09-15 09:52:24 <hearn> well, at least the latest version of the library will (a) automatically use a local node when found, (b) can be HD and (c) it's used compressed keys for a while
283 2014-09-15 09:52:35 <hearn> however not all wallets have upgraded to the latest code yet, they're mostly still in beta
284 2014-09-15 09:52:54 <phantomcircuit> wumpus, er just a thought.. possibly regtest stuff should be a build time option
285 2014-09-15 09:53:06 <hearn> e.g. the Bitcoin Wallet for Android beta has all those points (you can specify a node to connect to). but it's still in beta
286 2014-09-15 09:53:14 <wumpus> phantomcircuit: why?
287 2014-09-15 09:53:41 <phantomcircuit> wumpus, with testnet/regtest an attack that can modify a single pointer can compromise a node
288 2014-09-15 09:53:57 <phantomcircuit> (in such a way that the node will still seem to work correctly)
289 2014-09-15 09:54:06 <wumpus> well, if you can modify a pointer, you can do much worse
290 2014-09-15 09:54:08 <phantomcircuit> seems prudent
291 2014-09-15 09:54:38 <phantomcircuit> wumpus, cant think of anything else which would be quite as easy
292 2014-09-15 09:54:57 <phantomcircuit> most everything else that's truly concensus critical is constant and probably in read only pges
293 2014-09-15 09:54:59 <phantomcircuit> pageS*
294 2014-09-15 09:55:14 <wumpus> ELF plt tables
295 2014-09-15 09:55:17 <brisque> hearn: as far as I can tell there's little that's of use to me now, today though
296 2014-09-15 09:55:46 <hearn> right. but i guess within a couple of months there will be quite a few wallets that meet your criteria
297 2014-09-15 09:56:02 <hearn> although i'd note that the Core wallet is not HD
298 2014-09-15 09:56:13 <hearn> if all you want is "has full mode security and uses compressed keys" then existing bitcoinj wallets can do that.
299 2014-09-15 09:56:18 <hearn> if you connect them to localhost
300 2014-09-15 09:56:40 <brisque> they don't use change though, right?
301 2014-09-15 09:56:44 <brisque> multibit at least does not
302 2014-09-15 09:57:06 <hearn> mostly not, no. that turned out to be incompatible with mass adoption.
303 2014-09-15 09:57:11 <phantomcircuit> wumpus, elf plt table is at a stupidly random address on modern systems
304 2014-09-15 09:57:16 <hearn> i don't think there's any spv wallet that exactly matches the behaviour of core, today
305 2014-09-15 09:57:25 <hearn> making one wouldn't be hard. it's just niche though.
306 2014-09-15 09:57:32 <brisque> what? how is change addresses incompatible with people using it?
307 2014-09-15 09:57:34 <hearn> or just wait for the HD wallets
308 2014-09-15 09:57:45 <phantomcircuit> trying to hit that from remote would be like trying to hit a penny on the moon from alpha centauri
309 2014-09-15 09:57:47 <wumpus> phantomcircuit: so is everything, we compile bitcoin with address space randomization with good reason
310 2014-09-15 09:58:06 <hearn> because people lose their backups and clone wallets all the time. random keys causes people who aren't specialists to lose money.
311 2014-09-15 09:59:33 <brisque> so have a system that makes them in the future like bitcoin core? compromising people's privacy for minor usability is one insane tradeoff.
312 2014-09-15 09:59:51 <wumpus> phantomcircuit: anyhow - always compiling with testing support makes sure that tests are run... making testing an obscure build-time feature makes sure that no one runs them
313 2014-09-15 10:00:18 <wumpus> and also makes it unclear what you're really testing, are you testing the original code or code especially built for testing?
314 2014-09-15 10:00:25 <hearn> no, it's exactly the right tradeoff that all wallet devs who have large userbases have arrived at. believe me, if you had to support hundreds of thousands of ordinary people who just heard about bitcoin on CNN, you'd do the same thing. anyway, like i said, it'll all be HD soon enough so then we can have our cake and eat it.
315 2014-09-15 10:00:28 <phantomcircuit> wumpus, hmm
316 2014-09-15 10:00:34 <brisque> I mean between bitcoinj's insane bloom filters and not using change address, there's no privacy to be had.
317 2014-09-15 10:00:43 <phantomcircuit> wumpus, well pCurrentBaseParams is a static so it's allocated in.. bss? i think
318 2014-09-15 10:00:46 <brisque> fine.
319 2014-09-15 10:01:03 <phantomcircuit> anyways that should have a large amount of entropy in the base offset
320 2014-09-15 10:01:06 <wumpus> #ifndef run_tests dosomethingsneaky() #endif :p
321 2014-09-15 10:01:16 <hearn> the bloom filters are not "insane", they reflect that building a consistently false story to remote peers whilst staying within bandwidth limits is rather challenging and currently nobody is working on it
322 2014-09-15 10:01:17 <SomeoneWeird> :(
323 2014-09-15 10:01:19 <SomeoneWeird> you make me sad
324 2014-09-15 10:01:20 <phantomcircuit> i thought it was a heap var
325 2014-09-15 10:01:27 <phantomcircuit> er
326 2014-09-15 10:01:28 <phantomcircuit> stack
327 2014-09-15 10:01:34 <phantomcircuit> in which case it probably would be bad
328 2014-09-15 10:01:43 <hearn> turning up the FP rate at the moment would only raise the privacy bar slightly, but waste a lot of bandwidth and users time. using them most effectively needs more R&D
329 2014-09-15 10:02:19 <brisque> I don't think people are aware of just how revealing the filters are.
330 2014-09-15 10:02:33 <phantomcircuit> hearn, with HD wallets, how far do you walk down possible trees to identify IsMine() ?
331 2014-09-15 10:02:44 <brisque> hearn: if you know, someone, was to modify their nodes to dump the filters, they'd have lots of them by now.
332 2014-09-15 10:03:00 <hearn> the lookahead region is by default 100 keys
333 2014-09-15 10:03:14 <brisque> phantomcircuit: Electrum walks 5 empty addresses ahead, and then gives up, as a comparison.
334 2014-09-15 10:03:17 <hearn> brisque: i am quite aware thanks. i would worry more about passive interception by intelligence agencies, myself.
335 2014-09-15 10:03:50 <brisque> hearn: I'm doing quite alright on my own.
336 2014-09-15 10:04:34 <hearn> hm?
337 2014-09-15 10:04:38 <wumpus> well, at least usually intelligence agencies don't try to rob you
338 2014-09-15 10:04:51 <moa> not directly ...
339 2014-09-15 10:05:08 <brisque> wumpus: excuse me? I have no intention of robbing anybody.
340 2014-09-15 10:05:08 <phantomcircuit> wumpus, no but they do like recording things
341 2014-09-15 10:05:18 <wumpus> if you leak some information to a random peer, that person may be planning on robbing you
342 2014-09-15 10:05:28 <hearn> i'd give it an 80% chance that somewhere inside the nsa is a database mapping ip addresses to identities, and bitcoin addresses to ips
343 2014-09-15 10:05:28 <phantomcircuit> most of the current clients use of bloom filters basically announces to the world almost exactly which addresses are theirs
344 2014-09-15 10:05:43 <wumpus> I'm much more worried about that personally
345 2014-09-15 10:05:55 <brisque> wumpus: nothing to hide, nothing to fear.
346 2014-09-15 10:06:04 <phantomcircuit> as currently used by most spv clients
347 2014-09-15 10:06:20 <phantomcircuit> bloom filters are just an optimization over transmitting a plaintext list of addresses
348 2014-09-15 10:06:26 <hearn> wumpus: "that person" is most likely the other side of the world, and sees only an ip address. if you're worried about physically local attackers, i'd be more concerned about someone looking over your shoulder ...
349 2014-09-15 10:06:32 <phantomcircuit> the only meaningful difference is for addresses that never get used
350 2014-09-15 10:06:32 <wumpus> brisque: I'm not saying that... I'm not saying that you don't have to worry about intelligence agencies *at all*
351 2014-09-15 10:06:44 <wumpus> but in practice, crime and organized crime is much more likely to happen to you
352 2014-09-15 10:06:49 <brisque> phantomcircuit: it's surprising how many filters you can capture. even if they are not using bloom filters, Electrum servers are extremely noisy with the keys they are interested in.
353 2014-09-15 10:07:35 <hearn> yeah, but what are you going to do with them? physically go mug someone? you would have better luck just goingt to a bitcoin conference and being a traditional mugger. watch out for the guys with the nice gold watches.
354 2014-09-15 10:07:43 <brisque> phantomcircuit: Electrum does this thing where it randomly selects a new server to query every time, so given enough reconnections they're bound to eventually hit a peer who cares to record it.
355 2014-09-15 10:07:45 <wumpus> and for that reason I encourage people to be wary of privacy in bitcoin
356 2014-09-15 10:07:48 <hearn> intelligence agencies are a "known unknown" - we know they exist, what they do, what they're interested in
357 2014-09-15 10:08:05 <phantomcircuit> brisque, yeah there's a reason bitcoin doesn't randomly reconnect to new nodes
358 2014-09-15 10:08:16 <hearn> random people collecting filters and linking addresses ..... ok. what motivates these people? what is their plan? how much money can they make by doing this?
359 2014-09-15 10:08:20 <wumpus> hearn: it could be physical, it could be to make you a target for a spear fishing expedition, or social manipulation
360 2014-09-15 10:08:41 <phantomcircuit> ha spear phishing
361 2014-09-15 10:08:44 <wumpus> basically you don't want to adverize LOT OF COINS BEHIND THIS IP
362 2014-09-15 10:08:47 <moa> honey pot?
363 2014-09-15 10:09:00 <phantomcircuit> that reminds me i need to write some simple program to remove that stupid .jar spear phishing malware
364 2014-09-15 10:09:09 <phantomcircuit> av companies are at least 2 weeks behind on signatures for it
365 2014-09-15 10:09:18 <hearn> for end users just getting an IP address isn't that helpful. for businesses, sure, but then again, they can just run a full node
366 2014-09-15 10:10:04 <brisque> phantomcircuit: I was very amused when I found that feature. people also have a very bad habbit of making a connection through an anonymiser, making a transaction, and then connecing back with their real IP address.
367 2014-09-15 10:10:31 <wumpus> well I'm talking about people that have significant capital in bitcoin, sure, if you have 20 mBTC to buy a pizza, by all means go ahead broadcast your filters far and wide :-)
368 2014-09-15 10:10:36 <hearn> probably, encrypting p2p connections is the next big privacy win after HD. there are some research papers coming out soon on bloom filter privacy, but they only help a little bit. most of the issues are quite well understood, there's just nobody around to write all the difficult code required.
369 2014-09-15 10:11:01 <hearn> brisque: you run an electrum server?
370 2014-09-15 10:11:22 <phantomcircuit> hearn, err for bloom filters basically some constants need to be changed
371 2014-09-15 10:11:32 <hearn> phantomcircuit: no, sadly it's a lot more complicated than that
372 2014-09-15 10:11:33 <phantomcircuit> iirc the math for what they should be isn't super complicated
373 2014-09-15 10:11:56 <hearn> e.g. see the first and last comments on this bug https://code.google.com/p/bitcoinj/issues/detail?id=510
374 2014-09-15 10:11:59 <brisque> hearn: yes.
375 2014-09-15 10:12:38 <hearn> phantomcircuit: bumping up the FP rate is easy, it's one line of code for the wallet dev. but as anyone who has seen a detective show knows, there's a big difference between lying and lying convincingly
376 2014-09-15 10:12:44 <hearn> today bitcoinj can be configured to lie very easily.
377 2014-09-15 10:12:57 <hearn> making it lie convincingly, even when put up against a skilled interrogator, is much harder.
378 2014-09-15 10:13:24 <hearn> i don't know of any other clients/servers that do this sort of thing. TextSecure used to use bloom filters to retrieve phone book hits but they had to abandon it as the product got more popular due to scaling issues.
379 2014-09-15 10:13:51 <hearn> and i don't know of any products at all except bitcoinj where there has been real, academic research into how they do PIR
380 2014-09-15 10:14:06 <hearn> so this is fairly cutting edge stuff and it'll take a long time to get it totally right
381 2014-09-15 10:14:17 <wumpus> indeed
382 2014-09-15 10:14:20 <brisque> hearn: to be clear I don't have any intention of using or selling anybodies information. finding somebody who would actually be interested in it would be impossible, they'd likely be the sort of people who would have it anyway.
383 2014-09-15 10:14:27 <hearn> brisque: right
384 2014-09-15 10:15:25 <brisque> and it's sort of brainlessly easy to capture information, electrum in particular already has most of the logging stuff in place. by default it has commands for printing say, every IP address connected to you.
385 2014-09-15 10:16:31 <wumpus> it's easy to capture information; processing information usefully is much more involved
386 2014-09-15 10:17:12 <wumpus> ie, converting it to actionable information that one could sell
387 2014-09-15 10:17:57 <brisque> I don't think there's anything of value in a bag filled with bloom filters. if anything, you could make it all up and just sell junk, and nobody would be any the wiser.
388 2014-09-15 10:18:39 <wumpus> one bag of bloom filters please, thank you
389 2014-09-15 10:18:45 <phantomcircuit> hearn, basically the problem is that the bloom filter is updated when a positive result is transmitted
390 2014-09-15 10:18:56 <phantomcircuit> false positives dont result in that update
391 2014-09-15 10:19:06 <hearn> i think there's value to the NSA, sorta. according to some weak definition of "value". because they want to have a map of everyone's financial transactions, and unlike other actors, they already have databases of IPs to facebook profiles, IPs to email addresses, email addresses to real world identities etc
392 2014-09-15 10:19:09 <phantomcircuit> or they do and you get more crap as the filter tends towards 1
393 2014-09-15 10:19:15 <hearn> phantomcircuit: well there's a bunch of problems
394 2014-09-15 10:19:37 <hearn> phantomcircuit: e.g. the remote node can currently deviate from the protocol and send you transactions that were not actually matching the filter to drive up the apparent FP rate and thus drive down the noisyness of the filter. that one is easy to fix
395 2014-09-15 10:19:57 <hearn> phantomcircuit: but there are others. yes the core problem is that if you want false positives, you have to keep consistently wanting them, or else it becomes apparent they were FPs
396 2014-09-15 10:20:03 <hearn> now what happens if your filter hits satoshidice addresses?
397 2014-09-15 10:20:12 <phantomcircuit> you cry and go home
398 2014-09-15 10:20:18 <brisque> hearn: there's little to no value in the quantity of information I have. somebody with the power to make hundreds of nodes and make them all sniff and cross report? sure then there's a problem.
399 2014-09-15 10:20:23 <phantomcircuit> hearn,
400 2014-09-15 10:20:25 <wumpus> it comes to a full stop
401 2014-09-15 10:20:34 <hearn> you need to watch the FPs coming back, understand which ones are using most of your bandwidth quota, then re-generate the bloom filter including the FPs that didn't contribute much and dropping those that did, in an attempt to get back under your bandwidth limit
402 2014-09-15 10:20:52 <hearn> currently bitcoinj doesn't even *have* a bandwidth targeting API
403 2014-09-15 10:21:01 <hearn> the wallets don't even tell it, you're on DSL, you're on 3G, etc
404 2014-09-15 10:21:29 <hearn> the only API wallets have is "target an FP rate of X%" but that's totally useless, because X% FP rate results in dramatically more bandwidth usage in later parts of the chain to earlier parts, etc
405 2014-09-15 10:21:43 <hearn> the actual resource you want to target is bytes/second. but this is all a lot of complex code.
406 2014-09-15 10:21:52 <brisque> hearn: you also need to retain your bloom filters. if you add different junk on every launch, it's easy to just intersect them and find the real filter.
407 2014-09-15 10:21:55 <hearn> and HD wallets were a much bigger and more important privacy win, so that (+lighthouse) is what i worked on this summer
408 2014-09-15 10:22:12 <hearn> brisque: right. they need to be persisted to disk, or you need to hope you just pick different peers each time.
409 2014-09-15 10:22:19 <hearn> there's lots that must be done
410 2014-09-15 10:22:30 <brisque> deterministic filter junk?
411 2014-09-15 10:22:46 <hearn> now from my POV encryption is a lower hanging fruit. means the NSA can't just suck up and intersect every filter they see with a few MapReduces, which they are easily capable of.
412 2014-09-15 10:23:07 <hearn> they'd have to run nodes if they wanted to get that data back, and then it's costing them money (and that money they spend benefits us because they're running nodes), and they still can't get as much visibility
413 2014-09-15 10:23:11 <hearn> plus they run the risk of detection
414 2014-09-15 10:23:34 <brisque> with what exactly though. if openssl is going away you can't use it for encryption.
415 2014-09-15 10:23:43 <hearn> once the wires are encrypted it makes sense to go back and try to scramble filters better.
416 2014-09-15 10:23:53 <hearn> probably just AES + HMAC + ECDH at the start.
417 2014-09-15 10:24:05 <brisque> and plus, detection means nothing, what about that fake tor node that connected to every single node on the network? sure that was detected, but we can't do anything but sit back and say "weird"
418 2014-09-15 10:24:23 <hearn> right, value of detection is low if only because their response would be "yeah, we're spying on you, suck it up"
419 2014-09-15 10:24:30 <hearn> but those guys lurrrve secrecy.
420 2014-09-15 10:24:48 <hearn> it's sort of hard coded into their culture. so forcing them to weaken it at least tips the playing field a little, which is all we can do ultimately
421 2014-09-15 10:24:53 <hearn> anyway i have to go now. have a nice afternoon chaps.
422 2014-09-15 10:27:10 <phantomcircuit> how are hd wallets a privacy win?
423 2014-09-15 10:27:15 <brisque> speaking of secp256k1, the benchmark tool sort of fails for me.
424 2014-09-15 10:27:53 <brisque> phantomcircuit: most people just use the one address over and over. all it takes is a good scrape of bitcointalk to identify a very large portion of all addresses in use.
425 2014-09-15 10:27:57 <wumpus> phantomcircuit: shouldn't matter for privcacy at all
426 2014-09-15 10:29:56 <wumpus> if someone on the outside can see how you generate your private keys, there is much more wrong than privacy
427 2014-09-15 10:30:40 <brisque> in the case of Multibit, HD means it'll actually use more than one key though
428 2014-09-15 10:31:13 <wumpus> lol
429 2014-09-15 10:31:26 <phantomcircuit> wumpus, that's what i was thinking which is why im confused by mikes statement
430 2014-09-15 10:31:44 <phantomcircuit> if anything they're potentially worse than random keys
431 2014-09-15 10:32:17 <brisque> what he means is "use more than one address for the whole wallet".
432 2014-09-15 10:32:27 <brisque> because that's a new, novel thing.
433 2014-09-15 10:32:56 <phantomcircuit> oh right
434 2014-09-15 10:33:14 <phantomcircuit> he means it's better than what is happening with bitcoinj based wallet currently
435 2014-09-15 10:33:16 <phantomcircuit> shrug
436 2014-09-15 10:33:52 <brisque> adding encryption to the p2p protocol is unlikely though, right?
437 2014-09-15 10:34:05 <brisque> I mean without openssl that's a fair chunk of code to write, and get right.
438 2014-09-15 10:34:38 <phantomcircuit> brisque, actually it could be done fairly trivially today
439 2014-09-15 10:34:45 <phantomcircuit> it wouldn't be secure against mitm at all
440 2014-09-15 10:34:51 <phantomcircuit> but it's never going to be
441 2014-09-15 10:34:52 <phantomcircuit> so who cares
442 2014-09-15 10:35:00 <brisque> how would it be done?
443 2014-09-15 10:35:27 <phantomcircuit> just slap unauthenticated tls on there
444 2014-09-15 10:35:31 <brisque> basic AES, key in the addr message?
445 2014-09-15 10:35:49 <phantomcircuit> version/version both peers advertise tls flag? cool we start talking tls
446 2014-09-15 10:36:40 <phantomcircuit> only issue is the message handling is in a separate thread from the networking code
447 2014-09-15 10:36:45 <phantomcircuit> so that could have some timing issues
448 2014-09-15 10:37:01 <phantomcircuit> actually no that should be pretty easy to do
449 2014-09-15 10:37:31 <wumpus> curvecp would be a possible alternative to tls
450 2014-09-15 10:37:47 <brisque> what cipher implementation would it use? can't have too much baggage.
451 2014-09-15 10:38:10 <brisque> assume openssl is dodo.
452 2014-09-15 10:38:19 <moa> zmq ?
453 2014-09-15 10:38:33 <phantomcircuit> moa, no
454 2014-09-15 10:38:35 <phantomcircuit> just
455 2014-09-15 10:38:36 <phantomcircuit> no
456 2014-09-15 10:38:38 <wumpus> zmq uses curvecp
457 2014-09-15 10:38:42 <wumpus> but it's not the same thing.
458 2014-09-15 10:38:45 <moa> :)
459 2014-09-15 10:39:21 <phantomcircuit> er
460 2014-09-15 10:39:35 <phantomcircuit> wumpus, if stdout stalls and printtoconsole is specified
461 2014-09-15 10:39:41 <phantomcircuit> the thread stalls
462 2014-09-15 10:39:46 <phantomcircuit> that's nasty
463 2014-09-15 10:40:23 <wumpus> anyhow - optionally encrypting the P2P protocol is a possible future direction, it avoids passive interception at a large scale
464 2014-09-15 10:40:50 <brisque> phantomcircuit: you also get weird happening if your -blocknotify gets backlogged
465 2014-09-15 10:41:38 <wumpus> in itself it doesn't do anything against targeted attacks, although a step further nodes could be told to trust specific other nodes with certain public keys
466 2014-09-15 10:42:42 <wumpus> phantomcircuit: I much prefer tail debug.log
467 2014-09-15 10:43:04 <phantomcircuit> normally that's what i do
468 2014-09-15 10:43:13 <phantomcircuit> but im adding/removing print statements fairly rapidly
469 2014-09-15 10:43:19 <phantomcircuit> so killall etc gets annoying fast
470 2014-09-15 10:43:28 <wumpus> the valid use of printtoconsole is to log to systemd, for example, which doesn't block
471 2014-09-15 10:44:33 <wumpus> if you use it to log to a terminal... well, only do that for troubleshooting
472 2014-09-15 10:45:51 <phantomcircuit> 2014-09-15 10:45:18 valRequest.type() == obj_type 1
473 2014-09-15 10:45:51 <phantomcircuit> 2014-09-15 10:45:33 keypool added key 123270, size=100
474 2014-09-15 10:46:06 <phantomcircuit> 2014-09-15 10:44:49 1...2...3...SetCrypted()
475 2014-09-15 10:46:06 <phantomcircuit> er
476 2014-09-15 10:46:07 <phantomcircuit> 2014-09-15 10:45:18 valRequest.type() == obj_type 1
477 2014-09-15 10:46:07 <phantomcircuit> 2014-09-15 10:45:33 keypool added key 123270, size=100
478 2014-09-15 10:46:49 <phantomcircuit> 0.35694005
479 2014-09-15 10:47:04 <phantomcircuit> so that's a single point multi roughly
480 2014-09-15 10:47:23 <phantomcircuit> (that's ms per key)
481 2014-09-15 10:47:37 <wumpus> you've added a -logtoirc option now? ;)
482 2014-09-15 10:48:38 <phantomcircuit> wumpus, not yet
483 2014-09-15 10:48:46 <phantomcircuit> (it helps in logs to have the actual info)
484 2014-09-15 10:48:58 <phantomcircuit> (yes you reading these logs.. that was for you!)
485 2014-09-15 10:50:08 <phantomcircuit> wumpus, im trying to break performance in the wallet code
486 2014-09-15 10:50:20 <phantomcircuit> so far the only thing that's an issue is the initial decryption
487 2014-09-15 10:51:19 <wumpus> yes, as said, using ecdsa to verify database integrity is probably not the best idea
488 2014-09-15 10:51:53 <phantomcircuit> my point was that so far all of the rest of the code is doing very well
489 2014-09-15 10:52:07 <wumpus> ok that's nice!
490 2014-09-15 10:52:24 <phantomcircuit> im generating 100 transactions with 1000 outputs of random sizes then a block goto 1
491 2014-09-15 10:53:21 <phantomcircuit> once the keys are decrypted the wallet flush is the slowest part
492 2014-09-15 10:53:54 <wumpus> that makes sense
493 2014-09-15 10:54:31 <wumpus> the wallet flush forces berkeleydb to clean up its intermediate data files and make the wallet.dat self-contained, that's probably not a fast operation
494 2014-09-15 10:56:06 <phantomcircuit> wumpus, with 250k active keys it's only taking about 100ms (im cheating and using a tmpfs though so it's probably 1000x faster than hdd and 100x faster than ssd)
495 2014-09-15 10:56:07 <wumpus> it's more than just 'flush to disk'
496 2014-09-15 10:56:44 <wumpus> wow that's pretty fast
497 2014-09-15 10:57:14 <michagogo> 13:46:34 <phantomcircuit> so that's a single point multi roughly
498 2014-09-15 10:57:18 <michagogo> Erm, what is?
499 2014-09-15 10:57:27 <michagogo> Oh, wait a minute
500 2014-09-15 10:57:45 <phantomcircuit> + 11.01% bitcoind bitcoind [.] (anonymous namespace)::sha256::Transform(unsigned int*, unsigned char const*)
501 2014-09-15 10:57:52 <phantomcircuit> any idea what that is
502 2014-09-15 10:58:02 <phantomcircuit> michagogo, 0.354ms
503 2014-09-15 10:58:30 <michagogo> phantomcircuit: did you say that shortly before 13:46:34 <phantomcircuit> so that's a single point multi roughly ?
504 2014-09-15 10:58:35 <wumpus> phantomcircuit: that's the sha256 compression function
505 2014-09-15 10:58:36 <phantomcircuit> 2014-09-15 10:58:22 Flushed wallet.dat 70ms
506 2014-09-15 10:58:42 <phantomcircuit> michagogo, yeah
507 2014-09-15 10:58:46 <michagogo> Gah, it's that bug again
508 2014-09-15 10:58:56 <phantomcircuit> michagogo, what bug?
509 2014-09-15 10:59:11 <michagogo> For some reason my client is not showing parts of history
510 2014-09-15 10:59:16 <michagogo> At random, it seems
511 2014-09-15 10:59:39 <michagogo> Usually killing the app and reopening it fixes it, but apparently not in this case
512 2014-09-15 10:59:45 <michagogo> :-/
513 2014-09-15 11:00:46 <phantomcircuit> huh
514 2014-09-15 11:00:48 <phantomcircuit> it's 4m
515 2014-09-15 11:00:50 <phantomcircuit> 4am
516 2014-09-15 11:00:51 <phantomcircuit> derp
517 2014-09-15 11:00:56 <phantomcircuit> oh well
518 2014-09-15 11:06:28 <phantomcircuit> wumpus, i think the wallet flush is actually cpu limited
519 2014-09-15 11:06:36 <phantomcircuit> which is odd
520 2014-09-15 11:06:45 <wumpus> phantomcircuit: maybe because you took out all I/O factors?
521 2014-09-15 11:06:53 <phantomcircuit> wumpus, yeah
522 2014-09-15 11:06:57 <wumpus> everything is CPU limited if you only work with tmpfs :-)
523 2014-09-15 11:06:58 <phantomcircuit> and then i added a bunch of cpu load
524 2014-09-15 11:07:09 <phantomcircuit> well
525 2014-09-15 11:07:14 <phantomcircuit> i am looking for cpu time issues
526 2014-09-15 11:07:15 <phantomcircuit> :P
527 2014-09-15 11:11:05 <phantomcircuit> huh
528 2014-09-15 11:11:06 <phantomcircuit> so
529 2014-09-15 11:11:21 <phantomcircuit> SelectCoins is called multiple times and the wallet is flushed in each
530 2014-09-15 11:11:24 <phantomcircuit> for the same tx
531 2014-09-15 11:12:08 <phantomcircuit> nvm
532 2014-09-15 11:13:45 <wumpus> the wallet is flushed in selectcoins?!
533 2014-09-15 11:14:22 <wumpus> hmm I remember there was a pull a while ago that reduced the number of wallet flushes, maybe this one too?
534 2014-09-15 11:14:27 <phantomcircuit> wumpus, no it's marked as dirty or something and the background thread flushes it
535 2014-09-15 11:14:40 <phantomcircuit> im just looking at debug logs now
536 2014-09-15 11:14:52 <wumpus> here https://github.com/bitcoin/bitcoin/pull/4805
537 2014-09-15 11:15:29 <michagogo>
538 2014-09-15 11:15:29 <wumpus> I suggest looking at cozz's pulls in general, he is doing a lot of work on the wallet code https://github.com/bitcoin/bitcoin/pulls/cozz
539 2014-09-15 11:17:34 <phantomcircuit> wumpus, some of those commits have side effects :/
540 2014-09-15 11:18:00 <wumpus> phantomcircuit: so, review them and comment
541 2014-09-15 11:18:38 <wumpus> that's why pull requests are not immediately merged, to give people some time to review them and come up with cases that they break
542 2014-09-15 11:21:58 <phantomcircuit> wumpus, can you review my sanity? it seems like 4805 is adding &walletdb in a bunch of places that it was implicit before
543 2014-09-15 11:22:12 <phantomcircuit> i dont see a problem with this, but it seems unnecessary for the pr
544 2014-09-15 11:23:31 <wumpus> phantomcircuit: yes, it passes the walletdb explicitly now
545 2014-09-15 11:23:34 <wumpus> that's on purpose
546 2014-09-15 11:24:57 <wumpus> it gives the caller control over when, and if, flushing happens
547 2014-09-15 11:26:39 <phantomcircuit> wumpus, ah so i should go to sleep
548 2014-09-15 11:26:40 <phantomcircuit> :P
549 2014-09-15 11:27:13 <wumpus> then again it *does* change flushing behavior in quite a few places, and I've not audited every one of them
550 2014-09-15 11:27:42 <wumpus> which is why I've only ACKed the concept, not the code
551 2014-09-15 11:28:11 <wumpus> in general we have a complete lack of reviewers for anything wallet related
552 2014-09-15 11:29:15 <wumpus> I've even sent a mail to the mailing list to ask at some point, and I've encouraged cozz to go looking for people himself, but it seems in vain...
553 2014-09-15 11:30:40 <wumpus> so, at some point it's better to freeze the code as-is, apart from serious bugfixes
554 2014-09-15 11:31:54 <phantomcircuit> 4702 is scary
555 2014-09-15 11:47:59 <wumpus> it is
556 2014-09-15 12:14:01 <phantomcircuit> wumpus, wallet with 41.5k outputs in 1k transactions can generate a tx with 100 random sized outputs in ~20 seconds
557 2014-09-15 13:11:40 <phantomcircuit> wumpus, neat
558 2014-09-15 13:12:06 <phantomcircuit> approximatesubset appears to be a good candidate for optimization
559 2014-09-15 13:12:23 <phantomcircuit> looks like it's causing a ton of branch prediction misses
560 2014-09-15 14:22:21 <sipa> Luke-Jr: i've been receiving spammy mails such as that privately with my address on bitcoin.org; i didn't notice that this one was to the dev list :)
561 2014-09-15 14:33:43 <chichov> if the current tx fee is 0.0001 BTC per 1000 bytes, then how come GetMinFee calculates it using the GMF_RELAY option and thus at a rate of 0.00001 BTC per 1000 bytes?
562 2014-09-15 14:38:04 <sipa> brisque: how does the benchmark tool fail for you?
563 2014-09-15 14:38:23 <sipa> brisque: its output is a bit weird, but the runtime should be a good indication
564 2014-09-15 14:39:08 <chichov> and furthermore, what is exact difference between minTxFee and minRelayTxFee?
565 2014-09-15 14:39:59 <sipa> phantomcircuit: addingna checksum to encrypted keys probably makes sense; part of the original reasoning was that requiring an attacker to perform very different technologies (aes, ec multiply) woukd be harder, and this automatically gave us a sortof chdcksum
566 2014-09-15 14:40:24 <brisque> sipa: I'm not even sure if I'm parsing the output right. it's meant to print valid/total number of keys right? the output for me is 0/1000000, so I'm assuming that's not correct. I'm not sure how to read the output of it otherwise.
567 2014-09-15 14:42:11 <brisque> so secp256k1_ge_is_valid is passing, but secp256k1_ecdsa_sig_verify is failing. not sure why.
568 2014-09-15 14:45:21 <sipa> chichov: for relay the fee setting is 1000 satoshi; that's just the assumed value necessary (_and required) for relay; as a wallet, when creating a transaction, you don't only want them relayed but also mined
569 2014-09-15 14:45:59 <sipa> brisque: perfectly intentional; i shouls change the output
570 2014-09-15 14:46:14 <sipa> brisque: the bwenchmark is trying to verify randomly generated signatures
571 2014-09-15 14:46:40 <sipa> brisque: anything else requires including signing (and thus signing perfmance) in the test
572 2014-09-15 14:47:09 <chichov> sipa: so at the end of the day, if you want it mined, you should orient on the nMinTxFee, which is 10000 Satoshis?
573 2014-09-15 14:48:03 <sipa> chichov: you should orient on what is necessary to get it mined; fee estimation will give much better guesses than a constant in the source code
574 2014-09-15 14:48:24 <chichov> sipa: how do I get my hands on that?
575 2014-09-15 14:48:50 <brisque> sipa: ah, makes sense
576 2014-09-15 14:48:50 <chichov> sipa: I only know that you have ceiling(TxSize/1000) * nMinTxFee
577 2014-09-15 14:51:59 <sipa> chichov: use git master
578 2014-09-15 14:56:55 <chichov> sipa: this whole thing is rather confusing. In the code comments it states that 1000 Satoshi / kB are for relaying and mining and 10000 Satoshi / kB are for sending. Which one is now used for calculating the fees when it should be mined?
579 2014-09-15 14:57:28 <sipa> chichov: 10000 is used when creating a transaction, 1000 when relaying/mining
580 2014-09-15 14:57:44 <chichov> sipa: and why does this distinction exist?
581 2014-09-15 14:58:08 <sipa> but to get a transaction you created mined, it doesn't just have to get be above the minimum, it competes with all other transactions for block space
582 2014-09-15 14:58:15 <brisque> who controls bitcoin.org? 'cos it's down.
583 2014-09-15 14:59:23 <sipa> chichov: also, compatibiloty; not every node in the network uses the same relay fee setting
584 2014-09-15 14:59:43 <sipa> chichov: you need a reasonable chance of being relayed by a significant amount of nodes
585 2014-09-15 15:00:01 <chichov> sipa: so you overestimate the costs rather than underestimate them?
586 2014-09-15 15:00:06 <sipa> yes
587 2014-09-15 15:00:19 <sipa> which is silly and leads to suboptimal fee prices
588 2014-09-15 15:00:25 <sipa> so we need a free market there
589 2014-09-15 15:00:37 <chichov> sipa: is the dust rule calculated with 1000 Satoshis then?
590 2014-09-15 15:00:38 <sipa> which is what fee estimation enables
591 2014-09-15 15:00:45 <sipa> yes
592 2014-09-15 15:00:57 <sipa> iirc at least
593 2014-09-15 15:01:01 <chichov> hm, and this fee estimation is some new feature? I haven't seen it around yet
594 2014-09-15 15:01:10 <sipa> it is in git master
595 2014-09-15 15:01:32 <chichov> can you give me a hint where to look for it?
596 2014-09-15 15:02:12 <sipa> i'm not very familiar with that code
597 2014-09-15 15:02:23 <chichov> do you know what it more or less does?
598 2014-09-15 15:02:26 <gavinandresen> chichov : txmempool.cpp, see the CMinerPolicyEstimator class
599 2014-09-15 15:02:38 <chichov> gavinandresen: thanks
600 2014-09-15 15:02:57 <sipa> chichov: yes, it tracks transactions in the mempool and observes how long it takes for them to be mined
601 2014-09-15 15:03:21 <gavinandresen> See http://bitcoincore.org/smartfee/ for output from its estimation algorithm
602 2014-09-15 15:03:28 <chichov> sipa: sounds like adaptive fee calculation, cool
603 2014-09-15 15:05:58 <chichov> gavinandresen: the estimates seem a bit high, for ~0.0004 BTC you wait 6 blocks? seems high
604 2014-09-15 15:06:35 <chichov> I've seen transactions with 0.0001 BTC per kB being confirmed right away
605 2014-09-15 15:06:59 <gavinandresen> chichov: zero-priority? They might get confirmed because they're high priority
606 2014-09-15 15:08:17 <gavinandresen> ⦠and if you're lucky and you send a transaction when blocks are randomly found close together, the second miner will accept a lot more low-fee transactions than usual.
607 2014-09-15 15:08:34 <gavinandresen> If you have a magic ball that can predict that the next block is found quickly, that'd be a spiffy addition to the fee estimation code.
608 2014-09-15 15:09:30 <gavinandresen> all that said: I don't claim CMinerPolicyEstimator is perfect. Just that it is much better than the fixed-fees we have now.