1 2012-11-27 00:03:20 <sipa> Luke-Jr: test_bitcoin indeed writes some blocks to blk00000.dat
2 2012-11-27 00:03:21 <sipa> Luke-Jr: test_bitcoin indeed writes some blocks to blk00000.dat
3 2012-11-27 00:03:43 <sipa> i wonder whether 0.7.x did that as well
4 2012-11-27 00:14:02 <sipa> Luke-Jr: 0.7.1 does also write blocks to blk0001.dat :)
5 2012-11-27 00:14:42 <sipa> however, pre-ultraprune it was always just appending; now it writes at the end of the used section of the file, which in case of a fresh memenv-backed db is nothing
6 2012-11-27 00:14:56 <sipa> so running test_bitcoin really corrupts your block database
7 2012-11-27 00:37:24 <Luke-Jr> sipa: eh, but master just worked :/
8 2012-11-27 00:37:25 <Luke-Jr> sipa: eh, but master just worked :/
9 2012-11-27 00:40:17 <sipa> normal sync; test_bitcoin; bitcoind -reindex <-- should fail
10 2012-11-27 00:40:18 <sipa> normal sync; test_bitcoin; bitcoind -reindex <-- should fail
11 2012-11-27 00:40:51 <sipa> (well, it should work, but the reindex will fail after the genesis block, and normal sync should recover)
12 2012-11-27 00:40:52 <sipa> (well, it should work, but the reindex will fail after the genesis block, and normal sync should recover)
13 2012-11-27 00:46:17 <Luke-Jr> pretty sure I've done that a number of times <.<
14 2012-11-27 00:46:18 <Luke-Jr> pretty sure I've done that a number of times <.<
15 2012-11-27 00:49:19 <sipa> hmm, strange!
16 2012-11-27 00:49:20 <sipa> hmm, strange!
17 2012-11-27 00:49:22 <sipa> afk
18 2012-11-27 00:49:23 <sipa> afk
19 2012-11-27 00:57:52 <weex> is there a way to convert a compressed private key into an uncompressed one?
20 2012-11-27 01:01:37 <sipa> private keys are not compressed or uncompressed
21 2012-11-27 01:01:38 <sipa> private keys are not compressed or uncompressed
22 2012-11-27 01:02:01 <sipa> each private key has a public key, and that public key can be serialized in compressed or uncompressed form
23 2012-11-27 01:02:02 <sipa> each private key has a public key, and that public key can be serialized in compressed or uncompressed form
24 2012-11-27 01:03:18 <weex> so the first letter of the privkey is just to tell how the public key should be handled?
25 2012-11-27 01:03:22 <sipa> yes
26 2012-11-27 01:03:23 <sipa> yes
27 2012-11-27 01:03:31 <sipa> actually, no
28 2012-11-27 01:03:34 <sipa> the last base
29 2012-11-27 01:03:50 <sipa> but adding a byte changes every character (including the first one) in base58 encoding
30 2012-11-27 01:03:51 <sipa> but adding a byte changes every character (including the first one) in base58 encoding
31 2012-11-27 01:04:41 <weex> if i'm trying to import a compressed key and getting invalid key, i'm just wondering if there is a transformation i can do to make it importable
32 2012-11-27 01:04:54 <sipa> into what software?
33 2012-11-27 01:04:55 <sipa> into what software?
34 2012-11-27 01:04:58 <weex> litcoind
35 2012-11-27 01:05:09 <sipa> there's no point
36 2012-11-27 01:05:22 <weex> litecoind (i have modified addrgen to create litecoin addresses)
37 2012-11-27 01:05:32 <sipa> if the software doesn't support compressed keys, there's no point trying to import it
38 2012-11-27 01:05:33 <sipa> if the software doesn't support compressed keys, there's no point trying to import it
39 2012-11-27 01:05:40 <sipa> as the address will not match anyway
40 2012-11-27 01:05:41 <sipa> as the address will not match anyway
41 2012-11-27 01:05:41 <weex> that's the answer i was looking for
42 2012-11-27 01:05:42 <weex> that's the answer i was looking for
43 2012-11-27 01:06:33 <weex> ok so basically unspendable until more code is working
44 2012-11-27 01:13:36 <weex> does bitcoind support importing compressed keys?
45 2012-11-27 01:14:40 <sipa> yes
46 2012-11-27 01:14:41 <sipa> yes
47 2012-11-27 01:14:46 <sipa> since 0.5, iirc
48 2012-11-27 01:14:47 <sipa> since 0.5, iirc
49 2012-11-27 01:15:36 <sipa> since 0.6
50 2012-11-27 03:17:26 <phantomcircuit> am i reading this invoicing proposal correctly that it's wanting to use the X.509 (ie https) infrastructure for bitcoin invoices?
51 2012-11-27 03:17:27 <phantomcircuit> am i reading this invoicing proposal correctly that it's wanting to use the X.509 (ie https) infrastructure for bitcoin invoices?
52 2012-11-27 03:18:46 <gmaxwell> X.509 is _not_ https.
53 2012-11-27 03:19:13 <phantomcircuit> gmaxwell, find me a cert issuer that isn't entirely based around https :|
54 2012-11-27 03:19:14 <phantomcircuit> gmaxwell, find me a cert issuer that isn't entirely based around https :|
55 2012-11-27 03:19:17 <gmaxwell> But yes, it's talking about using the x.509 certificate infrastructure at least as an initial option for persistant identities.
56 2012-11-27 03:19:18 <gmaxwell> But yes, it's talking about using the x.509 certificate infrastructure at least as an initial option for persistant identities.
57 2012-11-27 07:23:20 <Jouke> helo: yesterday you tested that backup wallet didn't overwrite an existing file, could you give more detail: https://github.com/bitcoin/bitcoin/issues/2035
58 2012-11-27 07:23:21 <Jouke> helo: yesterday you tested that backup wallet didn't overwrite an existing file, could you give more detail: https://github.com/bitcoin/bitcoin/issues/2035
59 2012-11-27 09:58:12 <gribble> 209812
60 2012-11-27 09:58:12 <sipa> ;;bc,blocks
61 2012-11-27 09:58:13 <gribble> 209812
62 2012-11-27 11:59:12 <slush1> Where the proposal of Payment Protocol which is currently discussed on mailing list can be found?
63 2012-11-27 11:59:13 <slush1> Where the proposal of Payment Protocol which is currently discussed on mailing list can be found?
64 2012-11-27 12:00:50 <Luke-Jr> slush1: https://gist.github.com/4120476
65 2012-11-27 12:00:51 <Luke-Jr> slush1: https://gist.github.com/4120476
66 2012-11-27 12:01:27 <slush1> thanks
67 2012-11-27 12:01:28 <slush1> thanks
68 2012-11-27 12:13:03 <Luke-Jr> slush1: btw, to help clarify the stratum discussion: the problem isn't across new blocks as much as it is updating work within a block
69 2012-11-27 12:13:04 <Luke-Jr> slush1: btw, to help clarify the stratum discussion: the problem isn't across new blocks as much as it is updating work within a block
70 2012-11-27 12:14:16 <slush1> luke-jr is there any reason why miners should not accept new job (almost) immediately?
71 2012-11-27 12:14:17 <slush1> luke-jr is there any reason why miners should not accept new job (almost) immediately?
72 2012-11-27 12:14:37 <Luke-Jr> slush1: no, that's the problem; cgminer insists on using the oldest work it has until clean_jobs=true
73 2012-11-27 12:14:56 <Luke-Jr> because it's still "valid" according to stratum
74 2012-11-27 12:15:06 <slush1> For example, stratum proxy don't send long polling broadcast on clean_jobs=False flags, so getwork miners may be a bit late on new jobs. But as far as they should (by getwork definition) ask for jobs every minute, this is almost no problem. And stratum subminers know about new jobs immediately
75 2012-11-27 12:15:40 <slush1> luke-jr then it is just implementation bug in the cgminer and should be fixed. The fact there's some flag "you should use this job" won't fix what is broken in the code.
76 2012-11-27 12:15:51 <Luke-Jr> slush1: it's intentional
77 2012-11-27 12:15:52 <Luke-Jr> slush1: it's intentional
78 2012-11-27 12:15:57 <slush1> ckolivas told me that cgminer is trying the newest job, so it looks it wasn't his purpose
79 2012-11-27 12:15:59 <slush1> why?
80 2012-11-27 12:16:00 <slush1> why?
81 2012-11-27 12:16:06 <Luke-Jr> oh?
82 2012-11-27 12:16:23 <Luke-Jr> no idea then, as it's clearly re-using the old job after it's had the new one for a while
83 2012-11-27 12:16:24 <Luke-Jr> no idea then, as it's clearly re-using the old job after it's had the new one for a while
84 2012-11-27 12:16:39 <slush1> as I said, I think it is more the bug than the feature
85 2012-11-27 12:17:03 <Luke-Jr> but if you're assuming 60 seconds of grace time, maybe I need to adjust how often I send new jobs
86 2012-11-27 12:17:04 <Luke-Jr> but if you're assuming 60 seconds of grace time, maybe I need to adjust how often I send new jobs
87 2012-11-27 12:17:05 <slush1> we discussed this with ckolivas months before and he confirmed that cgminer should always use the latest job
88 2012-11-27 12:17:06 <slush1> we discussed this with ckolivas months before and he confirmed that cgminer should always use the latest job
89 2012-11-27 12:17:19 <Luke-Jr> it certainly doesn't in practice.
90 2012-11-27 12:17:30 <slush1> is ckolivas aware of it?
91 2012-11-27 12:17:37 <slush1> I must say that I'm a bit lost in cgminer's discussion..
92 2012-11-27 12:17:45 <Luke-Jr> who knows, he ignores me
93 2012-11-27 12:17:50 <slush1> lol
94 2012-11-27 12:17:51 <slush1> lol
95 2012-11-27 12:18:11 <slush1> is there any evidence/source code which makes the problem? I can report it to him
96 2012-11-27 12:18:17 <Luke-Jr> 120 - 5 seconds latency - 60 seconds grace time = 55 seconds
97 2012-11-27 12:18:34 <slush1> what's that ? ^
98 2012-11-27 12:18:35 <slush1> what's that ? ^
99 2012-11-27 12:18:47 <Luke-Jr> 120 seconds is how long jobs are valid from eloipool
100 2012-11-27 12:18:48 <Luke-Jr> 120 seconds is how long jobs are valid from eloipool
101 2012-11-27 12:18:55 <Luke-Jr> so 55 seconds between job refreshes I guess
102 2012-11-27 12:18:56 <Luke-Jr> so 55 seconds between job refreshes I guess
103 2012-11-27 12:18:59 <Luke-Jr> currently using 96 seconds
104 2012-11-27 12:19:02 <slush1> My pool is updating jobs every 30 seconds, just because it is cheap enough and there's no reason why make windows bigger
105 2012-11-27 12:19:03 <slush1> My pool is updating jobs every 30 seconds, just because it is cheap enough and there's no reason why make windows bigger
106 2012-11-27 12:19:30 <slush1> are you dropping jobs older than 120 seconds?
107 2012-11-27 12:19:31 <slush1> are you dropping jobs older than 120 seconds?
108 2012-11-27 12:19:35 <Luke-Jr> yes
109 2012-11-27 12:19:36 <Luke-Jr> yes
110 2012-11-27 12:19:46 <Luke-Jr> and even before dropping, I reject shares on jobs older than 120 seconds
111 2012-11-27 12:19:47 <Luke-Jr> and even before dropping, I reject shares on jobs older than 120 seconds
112 2012-11-27 12:20:04 <slush1> does eloipool already implements stratum?
113 2012-11-27 12:20:05 <slush1> does eloipool already implements stratum?
114 2012-11-27 12:20:10 <slush1> I'm not following the latest development
115 2012-11-27 12:20:11 <slush1> I'm not following the latest development
116 2012-11-27 12:20:49 <Luke-Jr> yes
117 2012-11-27 12:20:50 <Luke-Jr> yes
118 2012-11-27 12:20:56 <slush1> well, with correct implementation of miners, 120 seconds is far enough.
119 2012-11-27 12:20:57 <slush1> well, with correct implementation of miners, 120 seconds is far enough.
120 2012-11-27 12:21:10 <Luke-Jr> but I always send clean_work=false and 96 seconds between updates right now
121 2012-11-27 12:21:11 <Luke-Jr> but I always send clean_work=false and 96 seconds between updates right now
122 2012-11-27 12:21:23 <slush1> However it's not too safe to drop jobs before you send clean_jobs flag
123 2012-11-27 12:22:00 <slush1> ok, so can we agree that the main problem is that bug in cgminer?
124 2012-11-27 12:22:03 <Luke-Jr> slush1: BFGMiner is using the new job immediately for all except BFL devices (and BFL devices also if clean_jobs=true)
125 2012-11-27 12:22:10 <Luke-Jr> slush1: in this case, yes
126 2012-11-27 12:22:16 <slush1> and it should use latest known job
127 2012-11-27 12:22:23 <Luke-Jr> slush1: but if your proxy needs 60 second grace window, I need to reduce my update duration
128 2012-11-27 12:22:24 <Luke-Jr> slush1: but if your proxy needs 60 second grace window, I need to reduce my update duration
129 2012-11-27 12:22:42 <slush1> there's no 60 second timeout in the proxy
130 2012-11-27 12:22:51 <slush1> it just send LP broadcasts on clean_jobs
131 2012-11-27 12:22:52 <Luke-Jr> but 60 seconds before miners get new work from proxy
132 2012-11-27 12:22:53 <Luke-Jr> but 60 seconds before miners get new work from proxy
133 2012-11-27 12:23:15 <Luke-Jr> up to*
134 2012-11-27 12:23:23 <slush1> so it is up to miner implementation how often it calls getwork, instead of reusing the same with rollntime
135 2012-11-27 12:24:00 <slush1> shortly said - if getwork miners worked with eloipool without problem before, there's probably no need to change eloipool codebase because of stratum proxy
136 2012-11-27 12:24:01 <slush1> shortly said - if getwork miners worked with eloipool without problem before, there's probably no need to change eloipool codebase because of stratum proxy
137 2012-11-27 12:24:57 <Luke-Jr> it's not up to miner implementation with getwork; pool tells the miner how long a job is valid
138 2012-11-27 12:24:58 <Luke-Jr> it's not up to miner implementation with getwork; pool tells the miner how long a job is valid
139 2012-11-27 12:25:16 <slush1> oh, I forgot that your pool implements these getwork extensions
140 2012-11-27 12:25:17 <slush1> oh, I forgot that your pool implements these getwork extensions
141 2012-11-27 12:25:21 <slush1> I never used them :-)
142 2012-11-27 12:25:22 <slush1> I never used them :-)
143 2012-11-27 12:25:30 <Luke-Jr> explains why they're all missing in stratum :/
144 2012-11-27 12:25:31 <Luke-Jr> explains why they're all missing in stratum :/
145 2012-11-27 12:26:19 <slush1> Although I understand this is the difference between getwork miner->pool and getwork miner->proxy->pool, I definitely don't see the reason why this should be available for stratum miner
146 2012-11-27 12:26:20 <slush1> Although I understand this is the difference between getwork miner->pool and getwork miner->proxy->pool, I definitely don't see the reason why this should be available for stratum miner
147 2012-11-27 12:26:29 <Luke-Jr> ugh, I really don't have any way for StratumServer to see if the previous job is still valid or not :/
148 2012-11-27 12:26:37 <slush1> I mean - defining some refresh windows may be required by getwork miners, because getwork protocol is fucked up
149 2012-11-27 12:26:38 <slush1> I mean - defining some refresh windows may be required by getwork miners, because getwork protocol is fucked up
150 2012-11-27 12:26:58 <slush1> hm, can you elaborate a bit more?
151 2012-11-27 12:26:59 <slush1> hm, can you elaborate a bit more?
152 2012-11-27 12:27:10 <slush1> What previous job?
153 2012-11-27 12:27:11 <slush1> What previous job?
154 2012-11-27 12:27:22 <slush1> you mean - for statistical reasons?
155 2012-11-27 12:27:23 <slush1> you mean - for statistical reasons?
156 2012-11-27 12:27:47 <Luke-Jr> slush1: I mean to decide if clean_work should be true
157 2012-11-27 12:27:48 <Luke-Jr> slush1: I mean to decide if clean_work should be true
158 2012-11-27 12:28:05 <slush1> how so? You - as pool - has everything necessary to decide
159 2012-11-27 12:28:12 <Luke-Jr> at a different layer
160 2012-11-27 12:28:17 <Luke-Jr> not exposed to the network layer atm
161 2012-11-27 12:28:18 <Luke-Jr> not exposed to the network layer atm
162 2012-11-27 12:28:26 <slush1> well, that's implementation detial
163 2012-11-27 12:28:28 <slush1> detail
164 2012-11-27 12:28:29 <slush1> detail
165 2012-11-27 12:28:41 <Luke-Jr> ???
166 2012-11-27 12:28:42 <Luke-Jr> ???
167 2012-11-27 12:29:05 <slush1> and it's even not true. You can detect if prevhash has changed, and set clean job flag at any level :-)
168 2012-11-27 12:29:06 <slush1> and it's even not true. You can detect if prevhash has changed, and set clean job flag at any level :-)
169 2012-11-27 12:29:24 <slush1> although this is more like a workaround for some architectural issues in the pool
170 2012-11-27 12:29:25 <slush1> although this is more like a workaround for some architectural issues in the pool
171 2012-11-27 12:30:04 <Luke-Jr> ACTION is adding a IsJobValid :P
172 2012-11-27 12:30:10 <slush1> :-)
173 2012-11-27 12:30:11 <slush1> :-)
174 2012-11-27 12:37:57 <Luke-Jr> seems to be working
175 2012-11-27 12:43:05 <Luke-Jr> slush1: my (just-posted) post look correct?
176 2012-11-27 12:43:53 <slush1> luke-jr I would be surprised :-P
177 2012-11-27 12:43:54 <slush1> luke-jr I would be surprised :-P
178 2012-11-27 12:43:59 <Luke-Jr> ?
179 2012-11-27 12:44:05 <slush1> I'm kidding :)
180 2012-11-27 12:44:06 <slush1> I'm kidding :)
181 2012-11-27 12:45:44 <slush1> luke-jr I don't understand that "idle job updates" part, I'm probably missing something
182 2012-11-27 12:45:45 <slush1> luke-jr I don't understand that "idle job updates" part, I'm probably missing something
183 2012-11-27 12:46:27 <slush1> there's also nothing like "Stratum (protocol) expects a 60 second window". Is that something related to your implementation?
184 2012-11-27 12:46:53 <Luke-Jr> slush1: if the proxy sends work based on JobA immediately before I send JobB with clean_jobs=false, and the miner can process that work for 60 seconds, I need to guarantee JobA remains valid for 60 seconds
185 2012-11-27 12:47:09 <Luke-Jr> slush1: that's a protocol rule you implied with how you described the proxy behaviour to me :p
186 2012-11-27 12:48:01 <slush1> well, but this is not a part of stratum, it is part of getwork and proxy don't enforce that; it simply depends on >getwork< miners how they implement it.
187 2012-11-27 12:48:02 <slush1> well, but this is not a part of stratum, it is part of getwork and proxy don't enforce that; it simply depends on >getwork< miners how they implement it.
188 2012-11-27 12:48:18 <Luke-Jr> getwork has a specification
189 2012-11-27 12:48:19 <Luke-Jr> getwork has a specification
190 2012-11-27 12:48:33 <slush1> simply said, you can safely drop old jobs when you send out clean_jobs=True
191 2012-11-27 12:48:38 <slush1> as I'm doing it
192 2012-11-27 12:48:39 <slush1> as I'm doing it
193 2012-11-27 12:48:48 <Luke-Jr> that doesn't explain when to drop them with clean_jobs=false
194 2012-11-27 12:48:49 <Luke-Jr> that doesn't explain when to drop them with clean_jobs=false
195 2012-11-27 12:49:06 <slush1> there's not any limitation in this case
196 2012-11-27 12:49:13 <slush1> as miner can still submit old jobs
197 2012-11-27 12:49:14 <Luke-Jr> there must be, or stratum is useless
198 2012-11-27 12:49:14 <slush1> as miner can still submit old jobs
199 2012-11-27 12:49:15 <Luke-Jr> there must be, or stratum is useless
200 2012-11-27 12:49:18 <slush1> why?
201 2012-11-27 12:49:35 <Luke-Jr> because jobs cannot remain valid forever, and sending clean_jobs=true for every job is wasteful
202 2012-11-27 12:49:47 <slush1> it is not valid forever
203 2012-11-27 12:49:52 <slush1> it is valid until next clean_jobs
204 2012-11-27 12:49:55 <Luke-Jr> that's potentially forever
205 2012-11-27 12:49:56 <Luke-Jr> that's potentially forever
206 2012-11-27 12:49:59 <slush1> I'm not saying anywhere you must say clean_jobs with every job
207 2012-11-27 12:50:00 <slush1> I'm not saying anywhere you must say clean_jobs with every job
208 2012-11-27 12:50:03 <slush1> potentially not
209 2012-11-27 12:50:12 <Luke-Jr> I cannot allow jobs to remain valid longer than 2 minutes period.
210 2012-11-27 12:50:13 <Luke-Jr> I cannot allow jobs to remain valid longer than 2 minutes period.
211 2012-11-27 12:50:16 <slush1> why?
212 2012-11-27 12:50:17 <slush1> why?
213 2012-11-27 12:50:22 <Luke-Jr> the generation would be stale
214 2012-11-27 12:50:42 <slush1> Can you elaborate?
215 2012-11-27 12:50:43 <slush1> Can you elaborate?
216 2012-11-27 12:50:47 <slush1> Is that part of bitcoin specs?
217 2012-11-27 12:50:48 <slush1> Is that part of bitcoin specs?
218 2012-11-27 12:50:52 <Luke-Jr> the payouts in the coinbase would no longer be the correct ones
219 2012-11-27 12:50:57 <Luke-Jr> miners would get overpaid
220 2012-11-27 12:50:58 <Luke-Jr> miners would get overpaid
221 2012-11-27 12:51:00 <slush1> oh
222 2012-11-27 12:51:02 <slush1> oh
223 2012-11-27 12:51:08 <slush1> then you must send clean_jobs every few minutes
224 2012-11-27 12:51:16 <Luke-Jr> that is stupid
225 2012-11-27 12:51:17 <Luke-Jr> that is stupid
226 2012-11-27 12:51:28 <slush1> well, your requiest is pretty special :-)
227 2012-11-27 12:51:29 <slush1> well, your requiest is pretty special :-)
228 2012-11-27 12:51:35 <Luke-Jr> no, it's really not
229 2012-11-27 12:51:36 <Luke-Jr> no, it's really not
230 2012-11-27 12:51:38 <slush1> so you can expect that two minutes are fine
231 2012-11-27 12:51:39 <slush1> so you can expect that two minutes are fine
232 2012-11-27 12:51:58 <slush1> although I don't see _any_ reason why dropping jobs after two minutes should give you any issue
233 2012-11-27 12:51:59 <Luke-Jr> clean_jobs is going to invalidate the previous work in addition to the one 2 minutes ago
234 2012-11-27 12:52:00 <Luke-Jr> clean_jobs is going to invalidate the previous work in addition to the one 2 minutes ago
235 2012-11-27 12:52:19 <Luke-Jr> losing shares
236 2012-11-27 12:52:59 <slush1> yes. So just drop old jobs. There's no technical reason why miners should submit such old jobs
237 2012-11-27 12:53:00 <slush1> yes. So just drop old jobs. There's no technical reason why miners should submit such old jobs
238 2012-11-27 12:53:06 <Luke-Jr> your attempt to oversimplify things seems to make it only usable for a few use cases
239 2012-11-27 12:53:07 <Luke-Jr> your attempt to oversimplify things seems to make it only usable for a few use cases
240 2012-11-27 12:53:18 <slush1> you mean - for mining? ;)
241 2012-11-27 12:53:19 <slush1> you mean - for mining? ;)
242 2012-11-27 12:53:21 <Luke-Jr> how can I drop the job from 2 minutes ago without dropping the one from 1 minute ago?
243 2012-11-27 12:53:47 <Luke-Jr> I mean, it only works for your pool and those sufficiently similar to it
244 2012-11-27 12:53:48 <slush1> luke-jr as you're doing now; just drop the job on pool, so older submits won't be accepted
245 2012-11-27 12:53:49 <slush1> luke-jr as you're doing now; just drop the job on pool, so older submits won't be accepted
246 2012-11-27 12:53:56 <slush1> that's what you basically need
247 2012-11-27 12:53:57 <slush1> that's what you basically need
248 2012-11-27 12:54:06 <Luke-Jr> but stratum says I have to accept it
249 2012-11-27 12:54:07 <Luke-Jr> ?
250 2012-11-27 12:54:08 <Luke-Jr> ?
251 2012-11-27 12:54:31 <slush1> well, stratum also says "miners should use latest jobs"
252 2012-11-27 12:54:43 <slush1> so this is no brainer for me. Drop old jobs. problem solved
253 2012-11-27 12:54:45 <Luke-Jr> ok
254 2012-11-27 12:54:46 <Luke-Jr> ok
255 2012-11-27 12:55:16 <Luke-Jr> if you add an explicit "old jobs are invalid after 60 seconds in any case" that would make it clearest :P
256 2012-11-27 12:56:46 <slush1> what if pool wants to broadcast jobs in longer timeframes then? ;-)
257 2012-11-27 12:56:47 <slush1> what if pool wants to broadcast jobs in longer timeframes then? ;-)
258 2012-11-27 12:57:06 <slush1> There's really no need for hardcoded timeouts. Stating "miners should use latest jobs" is good enough
259 2012-11-27 12:57:12 <Luke-Jr> then it just needs to delay the expiration 60 seconds longer than that timeframe?
260 2012-11-27 12:57:13 <Luke-Jr> then it just needs to delay the expiration 60 seconds longer than that timeframe?
261 2012-11-27 12:57:18 <slush1> and if miner code is broken, no additional specification will fix it
262 2012-11-27 12:57:19 <slush1> and if miner code is broken, no additional specification will fix it
263 2012-11-27 12:57:38 <Luke-Jr> well, unless the proxy LPs for every job, the miner won't be usign the latest job in that case technically
264 2012-11-27 12:57:51 <Luke-Jr> ??? unless you're not enabling rollntime
265 2012-11-27 12:57:52 <Luke-Jr> ??? unless you're not enabling rollntime
266 2012-11-27 12:58:14 <slush1> well, we both know that getwork miners are going to die
267 2012-11-27 12:58:15 <slush1> well, we both know that getwork miners are going to die
268 2012-11-27 12:58:26 <Luke-Jr> hopefully stratum miners too
269 2012-11-27 12:58:27 <Luke-Jr> hopefully stratum miners too
270 2012-11-27 12:58:29 <Luke-Jr> ACTION ducks, jk
271 2012-11-27 12:58:29 <slush1> lol
272 2012-11-27 12:58:30 <slush1> lol
273 2012-11-27 12:58:51 <slush1> in some future versions of proxy, I can add additional LP broadcasts, which will enforce miners to use latest job
274 2012-11-27 12:58:54 <slush1> if it will be an issue
275 2012-11-27 12:59:11 <Luke-Jr> no, I'm good with current behaviour now that it's clear
276 2012-11-27 12:59:12 <Luke-Jr> no, I'm good with current behaviour now that it's clear
277 2012-11-27 12:59:17 <Luke-Jr> also, LPs will break cgminer
278 2012-11-27 12:59:18 <Luke-Jr> also, LPs will break cgminer
279 2012-11-27 12:59:30 <Luke-Jr> *sigh*
280 2012-11-27 13:00:33 <Luke-Jr> actually, if I were you, I'd personally LP on every new job just to force miners to fix their LP handling :D
281 2012-11-27 13:00:34 <slush1> that's hidden cost of overoptimized C code
282 2012-11-27 13:02:20 <slush1> I have one guy on pool, he has around 10% of stales at ~30 GHash/s. I sent him an email, he responded he's aware of it, but he doesn't care. So I think that enforcing LP and breaking miners will only result in lower pool hashrate :-)
283 2012-11-27 13:02:21 <slush1> I have one guy on pool, he has around 10% of stales at ~30 GHash/s. I sent him an email, he responded he's aware of it, but he doesn't care. So I think that enforcing LP and breaking miners will only result in lower pool hashrate :-)
284 2012-11-27 13:02:22 <slush1> I have one guy on pool, he has around 10% of stales at ~30 GHash/s. I sent him an email, he responded he's aware of it, but he doesn't care. So I think that enforcing LP and breaking miners will only result in lower pool hashrate :-)
285 2012-11-27 13:03:29 <Luke-Jr> I saw at least one miner that submits every share twice to guarantee it got through -.-
286 2012-11-27 13:03:30 <Luke-Jr> even when it gets a valid reply
287 2012-11-27 13:03:31 <Luke-Jr> even when it gets a valid reply
288 2012-11-27 13:04:46 <kinlo> wasn't that diablo's miner?
289 2012-11-27 13:05:17 <Luke-Jr> IIRC bitfury
290 2012-11-27 13:35:09 <BlueMatt> Luke-Jr: it doesnt (yet) make blocks, but if you want to use it to check blocks, yes, I would recommend it
291 2012-11-27 13:35:28 <BlueMatt> (ofc if every other node accepts a given block but only it rejects it, ignore it, but...)
292 2012-11-27 13:35:33 <Luke-Jr> BlueMatt: does it support BIP 23 Proposals yet?
293 2012-11-27 13:35:34 <Luke-Jr> BlueMatt: does it support BIP 23 Proposals yet?
294 2012-11-27 13:35:54 <BlueMatt> no
295 2012-11-27 13:35:55 <BlueMatt> no
296 2012-11-27 13:36:07 <BlueMatt> it doesnt support mining (do any non bitcoind clients?)
297 2012-11-27 13:37:20 <BlueMatt> Luke-Jr: a good miner should check blocks that are being created by multiple clients, not necessarily make them with multiple clients (only one client can atm, no?)
298 2012-11-27 13:37:55 <sipa> BlueMatt: BIP23 proposal == send a block for verification to a node, skipping PoW check
299 2012-11-27 13:38:26 <BlueMatt> mmm, well maybe I should read more than the title+abstract next time then...
300 2012-11-27 13:38:27 <BlueMatt> mmm, well maybe I should read more than the title+abstract next time then...
301 2012-11-27 13:38:49 <BlueMatt> Luke-Jr: anyway, it may be a while before bitcoinj handles anything like that (its a library, not a json-rpc server)
302 2012-11-27 13:45:41 <gavinandresen> gotta love kitchen-sink international standards: http://www.edifactory.de/msglist.D05A?m=INVOIC
303 2012-11-27 13:45:42 <gavinandresen> gotta love kitchen-sink international standards: http://www.edifactory.de/msglist.D05A?m=INVOIC
304 2012-11-27 13:51:29 <sipa> ha, "Dangerous goods"
305 2012-11-27 13:51:30 <sipa> ha, "Dangerous goods"
306 2012-11-27 13:53:45 <helo> Jouke: commented
307 2012-11-27 13:53:46 <helo> Jouke: commented
308 2012-11-27 14:00:12 <BlueMatt> gavinandresen: heh, lets implement the whole thing in bitcoin!
309 2012-11-27 14:24:40 <sipa> BlueMatt: surely transactions require a "Dangerous goods" flag!
310 2012-11-27 14:24:41 <sipa> BlueMatt: surely transactions require a "Dangerous goods" flag!
311 2012-11-27 14:25:35 <BlueMatt> sipa: ofc, that and the option for any one of a million other flags. I mean why add arbitrary text when you can just have a huge set of bits for any possible use?
312 2012-11-27 14:25:36 <BlueMatt> sipa: ofc, that and the option for any one of a million other flags. I mean why add arbitrary text when you can just have a huge set of bits for any possible use?
313 2012-11-27 14:28:20 <phantomcircuit> BlueMatt, perfectly logical
314 2012-11-27 14:28:21 <phantomcircuit> BlueMatt, perfectly logical
315 2012-11-27 14:29:57 <phantomcircuit> also the node at metaexch.com:8333 is running git master hardened against sybil attack
316 2012-11-27 14:29:58 <phantomcircuit> also the node at metaexch.com:8333 is running git master hardened against sybil attack
317 2012-11-27 14:30:17 <BlueMatt> wat?
318 2012-11-27 14:30:18 <BlueMatt> wat?
319 2012-11-27 14:30:40 <BlueMatt> so...they properly use addnode?
320 2012-11-27 14:30:41 <phantomcircuit> BlueMatt, it's hard to fill all of it's connection slots
321 2012-11-27 14:30:42 <phantomcircuit> BlueMatt, it's hard to fill all of it's connection slots
322 2012-11-27 14:30:50 <BlueMatt> so...they properly use addnode?
323 2012-11-27 14:31:13 <phantomcircuit> addnode isn't going to help against what im trying to prevent :|
324 2012-11-27 14:31:51 <BitDev> hi all
325 2012-11-27 14:31:59 <BitDev> its me again :)
326 2012-11-27 14:32:01 <BlueMatt> addnode makes a connection even when you fill the node's connection slots, so its not an attack against an individual
327 2012-11-27 14:32:15 <BlueMatt> though Im assuming you are still on about filling the connection slots of every node on the network
328 2012-11-27 14:32:16 <BlueMatt> though Im assuming you are still on about filling the connection slots of every node on the network
329 2012-11-27 14:32:25 <phantomcircuit> yup
330 2012-11-27 14:32:52 <phantomcircuit> it would be a lot easier to do than i previously calculated
331 2012-11-27 14:33:06 <BitDev> and i have new question, if my software receive inv message - is all block hashes come like in branch?
332 2012-11-27 14:33:07 <BitDev> and i have new question, if my software receive inv message - is all block hashes come like in branch?
333 2012-11-27 14:33:29 <BlueMatt> yea...have fun, even given unlimited bw, there is enough turn over all you would accomplish (for the most part) is make connections take a while and maybe sybil some unimportant nodes (which is why merchants, etc should be using addnode properly...)
334 2012-11-27 14:33:40 <BitDev> or its random position and i must myself put in right position of branch?
335 2012-11-27 14:33:57 <phantomcircuit> BlueMatt, that's the thing
336 2012-11-27 14:33:58 <phantomcircuit> BlueMatt, that's the thing
337 2012-11-27 14:34:10 <phantomcircuit> those unimportant people would be anybody new pretty much
338 2012-11-27 14:34:18 <BitDev> sipa help me please again
339 2012-11-27 14:34:29 <BlueMatt> phantomcircuit: uhh...no
340 2012-11-27 14:34:30 <BitDev> i can find answer of this
341 2012-11-27 14:34:39 <BlueMatt> phantomcircuit: unless you can magically be the only node on the dnsseeds
342 2012-11-27 14:35:04 <phantomcircuit> BlueMatt, uh i could be the only connectable node right now
343 2012-11-27 14:35:05 <phantomcircuit> BlueMatt, uh i could be the only connectable node right now
344 2012-11-27 14:35:07 <gmaxwell> BlueMatt: uh, he can and thats irrelevant.
345 2012-11-27 14:35:18 <BitDev> can anyone help me?
346 2012-11-27 14:35:29 <gmaxwell> BitDev: you need to ask a better question.
347 2012-11-27 14:35:30 <gmaxwell> BitDev: you need to ask a better question.
348 2012-11-27 14:35:53 <BlueMatt> phantomcircuit: again, there is enough turnover that I doubt you could be successfully the only one, though obv you could be one of a few
349 2012-11-27 14:35:55 <BitDev> i receive inv packet - there are hashes
350 2012-11-27 14:36:18 <BlueMatt> gmaxwell: Im assuming you are just agreeing with his only connectable node thing, or is there some other magic he can do?
351 2012-11-27 14:36:27 <BitDev> i just wander if hashes are in order or its position is random
352 2012-11-27 14:36:30 <phantomcircuit> BlueMatt, i'm (very) sure that i could be the only one that is available for the majority of the time
353 2012-11-27 14:37:06 <BlueMatt> (I also assume you mean the only 8/9 available, because you would need that, not just one)
354 2012-11-27 14:37:10 <BlueMatt> (well, 8/9 ips)
355 2012-11-27 14:37:23 <BitDev> if i just connected and have no hashes - i send genesis hash by getheaders packet and i receive inv packet
356 2012-11-27 14:37:28 <phantomcircuit> yeah i was about to say it would be trivial to have 8 ip's in different groups
357 2012-11-27 14:37:46 <BitDev> this mean that all hashes in packet goes 1,2,3,4,5,6,... or can be 1,4.2,3,5,6... ?
358 2012-11-27 14:37:47 <BitDev> this mean that all hashes in packet goes 1,2,3,4,5,6,... or can be 1,4.2,3,5,6... ?
359 2012-11-27 14:37:49 <BlueMatt> phantomcircuit: you could probably get close, but Im not so sure there wouldnt be enough turnover to keep quite a few nodes connected
360 2012-11-27 14:37:59 <phantomcircuit> although practically it would require more just because those 8 nodes would get absolutely hammered
361 2012-11-27 14:38:00 <phantomcircuit> although practically it would require more just because those 8 nodes would get absolutely hammered
362 2012-11-27 14:38:02 <BlueMatt> but, yea, "important" nodes should be using addnode
363 2012-11-27 14:38:04 <BitDev> now you understand my question?
364 2012-11-27 14:38:05 <BitDev> now you understand my question?
365 2012-11-27 14:38:24 <BlueMatt> phantomcircuit: ofc
366 2012-11-27 14:39:19 <BlueMatt> also, bitcoin backbond
367 2012-11-27 14:39:20 <BlueMatt> e
368 2012-11-27 14:39:47 <phantomcircuit> BlueMatt, i doubt a practical doublespend could be pulled off simply because correlating node with service isn't really that easy
369 2012-11-27 14:39:48 <phantomcircuit> BlueMatt, i doubt a practical doublespend could be pulled off simply because correlating node with service isn't really that easy
370 2012-11-27 14:40:06 <phantomcircuit> however if you were just intent on breaking things...
371 2012-11-27 14:40:07 <phantomcircuit> however if you were just intent on breaking things...
372 2012-11-27 14:40:12 <gmaxwell> BlueMatt: he can fill up slots on all the public nodes he can find... but that doesn't harm nodes that have private addnode peering.
373 2012-11-27 14:40:13 <gmaxwell> BlueMatt: he can fill up slots on all the public nodes he can find... but that doesn't harm nodes that have private addnode peering.
374 2012-11-27 14:40:25 <BlueMatt> gmaxwell: ok, so yes you are just agreeing
375 2012-11-27 14:40:26 <BlueMatt> gmaxwell: ok, so yes you are just agreeing
376 2012-11-27 14:40:28 <BitDev> some one knows the answer?
377 2012-11-27 14:40:43 <gmaxwell> BlueMatt: well disagreeing that he couldn't be the only connectable dns seed.
378 2012-11-27 14:40:48 <phantomcircuit> i think the disagreement here is about the impact
379 2012-11-27 14:41:10 <phantomcircuit> gmaxwell, private addnode peering doesn't count as a connectable dns seed :)
380 2012-11-27 14:41:24 <BlueMatt> phantomcircuit: oh, I have no doubt you could cause issues for many people, but, if "important" nodes are doing things right, impact should be (largely) low
381 2012-11-27 14:41:33 <gmaxwell> BitDev: the inv is just telling you what is available. I don't think you can depend on it being in any particular order.
382 2012-11-27 14:41:34 <gmaxwell> BitDev: the inv is just telling you what is available. I don't think you can depend on it being in any particular order.
383 2012-11-27 14:42:05 <BitDev> hm, then i must send getdata and check all thing all by myself?
384 2012-11-27 14:42:06 <BitDev> hm, then i must send getdata and check all thing all by myself?
385 2012-11-27 14:42:14 <gmaxwell> phantomcircuit: It doesn't but bluematt said "maybe sybil some unimportant nodes (which is why merchants, etc should be using addnode properly...)" and I agree with that.
386 2012-11-27 14:42:15 <gmaxwell> phantomcircuit: It doesn't but bluematt said "maybe sybil some unimportant nodes (which is why merchants, etc should be using addnode properly...)" and I agree with that.
387 2012-11-27 14:42:45 <BlueMatt> gmaxwell: my point is less that he couldnt be for a while, and more that there is quite a bit of turnover and I would think many nodes would be able to connect regardless
388 2012-11-27 14:42:46 <BlueMatt> gmaxwell: my point is less that he couldnt be for a while, and more that there is quite a bit of turnover and I would think many nodes would be able to connect regardless
389 2012-11-27 14:42:48 <phantomcircuit> gmaxwell, even with addnode eventually it's likely the slots would be taken by others
390 2012-11-27 14:42:52 <gmaxwell> BitDev: if you dont a malicious node could just lie to you. You can't trust your peers not to lie.
391 2012-11-27 14:42:53 <gmaxwell> BitDev: if you dont a malicious node could just lie to you. You can't trust your peers not to lie.
392 2012-11-27 14:42:58 <gmaxwell> phantomcircuit: only if they are public nodes.
393 2012-11-27 14:42:59 <gmaxwell> phantomcircuit: only if they are public nodes.
394 2012-11-27 14:43:24 <BlueMatt> phantomcircuit: if you addnode, it will accept a connection from an addnode peer even if your connection slots are full
395 2012-11-27 14:43:28 <BlueMatt> (and will make them)
396 2012-11-27 14:43:28 <gmaxwell> BlueMatt: well there I don't agree. he can keep a node so stuffed full that he'll likely end up with all slots which he won't turnover.
397 2012-11-27 14:43:34 <BitDev> hm thats true
398 2012-11-27 14:43:51 <BitDev> ok, thnx i will query data )
399 2012-11-27 14:43:51 <BlueMatt> gmaxwell: no, my point is that new nodes come online quick enough he would lose some
400 2012-11-27 14:43:52 <BlueMatt> gmaxwell: no, my point is that new nodes come online quick enough he would lose some
401 2012-11-27 14:44:23 <gmaxwell> BlueMatt: prior to dnsseed we had slow initial connectivity even absent an attack. I'm not convinced.
402 2012-11-27 14:44:24 <gmaxwell> BlueMatt: prior to dnsseed we had slow initial connectivity even absent an attack. I'm not convinced.
403 2012-11-27 14:45:47 <phantomcircuit> BlueMatt, that's actually very unlikely given that the attacker would be trying to connect far more rapidly than any "real" node
404 2012-11-27 14:45:49 <phantomcircuit> bbl
405 2012-11-27 14:45:50 <phantomcircuit> bbl
406 2012-11-27 14:46:38 <BlueMatt> phantomcircuit: an attacker is trying to connect more rapidly, yes, and you would shut out many nodes, but the total volume of connection attempts of non-attacker nodes vs the total volume of connection attempts from good nodes (I would think) is such that some good nodes would connect
407 2012-11-27 14:46:39 <BlueMatt> phantomcircuit: an attacker is trying to connect more rapidly, yes, and you would shut out many nodes, but the total volume of connection attempts of non-attacker nodes vs the total volume of connection attempts from good nodes (I would think) is such that some good nodes would connect
408 2012-11-27 14:46:48 <BlueMatt> not to say partitions wouldnt form quite rapidly
409 2012-11-27 15:02:16 <kjj> hmm. I wish the keypool had configurable high and low water marks...
410 2012-11-27 15:02:17 <kjj> hmm. I wish the keypool had configurable high and low water marks...
411 2012-11-27 15:09:31 <gmaxwell> kjj: ... why?
412 2012-11-27 15:09:32 <gmaxwell> kjj: ... why?
413 2012-11-27 15:09:59 <kjj> intelligent backup management.
414 2012-11-27 15:10:00 <kjj> intelligent backup management.
415 2012-11-27 15:11:15 <kjj> also, it would help a lot on the forums
416 2012-11-27 15:11:16 <kjj> also, it would help a lot on the forums
417 2012-11-27 15:11:25 <sipa> deterministic wallets would help more :)
418 2012-11-27 15:11:26 <sipa> deterministic wallets would help more :)
419 2012-11-27 15:13:07 <kjj> heh. EC wallets have new and different backup problems
420 2012-11-27 15:13:08 <kjj> heh. EC wallets have new and different backup problems
421 2012-11-27 15:13:59 <sipa> you could do a "renew seed" action right before every backup
422 2012-11-27 15:14:00 <sipa> you could do a "renew seed" action right before every backup
423 2012-11-27 15:14:07 <sipa> if you don't like long-lived seeds
424 2012-11-27 15:15:26 <kjj> that just combines the problem sets of both systems
425 2012-11-27 15:15:27 <kjj> that just combines the problem sets of both systems
426 2012-11-27 15:15:29 <gmaxwell> sipa: On that subject??? I'm concerned that we need to have a boring determinstic wallet option in addition to the type-2 stuff simply because we haven't managed to wrangle serious cryptoanalytic review of the type-2 stuff yet.
427 2012-11-27 15:15:30 <gmaxwell> sipa: On that subject??? I'm concerned that we need to have a boring determinstic wallet option in addition to the type-2 stuff simply because we haven't managed to wrangle serious cryptoanalytic review of the type-2 stuff yet.
428 2012-11-27 15:16:32 <sipa> mhem
429 2012-11-27 15:16:33 <sipa> mhem
430 2012-11-27 15:17:30 <sipa> gmaxwell: well, if you consider the public extended key secret, that's exactly what you get, no?
431 2012-11-27 15:17:31 <sipa> gmaxwell: well, if you consider the public extended key secret, that's exactly what you get, no?
432 2012-11-27 15:18:00 <sipa> the resulting secrets are HMAC -> split -> EC multiply
433 2012-11-27 15:18:01 <sipa> the resulting secrets are HMAC -> split -> EC multiply
434 2012-11-27 15:18:29 <sipa> that's an EC multiplication of two random (from the point of an attacker) unknown and random numbers
435 2012-11-27 15:18:30 <sipa> that's an EC multiplication of two random (from the point of an attacker) unknown and random numbers
436 2012-11-27 15:20:37 <sipa> oh, not EC multiply... just modP multiply
437 2012-11-27 15:21:43 <gmaxwell> sipa: Right, I prefer that design because it is more conservative e.g. than what etotheipi had. And yes, _I_ have no idea how to attack it and don't even see a place to start because the multiply with a secret should effectively make the private keys 'unrelated'. And if this were for _any_ other part of the system I'd be completely comfortable with our bit of seminovel cryptography.
438 2012-11-27 15:23:26 <sipa> well, what would serious cryptoanalytic review mean? pay a cryptoanalist to try to break it?
439 2012-11-27 15:23:34 <kjj> in BIP32, is there a reason to use fixed sizes for depth and child number instead of VI?
440 2012-11-27 15:25:12 <sipa> the reason was that it needed to be limited anyway
441 2012-11-27 15:25:18 <sipa> but i can't remember why
442 2012-11-27 15:25:19 <sipa> but i can't remember why
443 2012-11-27 15:26:13 <gmaxwell> sipa: Well. Degrees of seriousness. Getting someone like DJB to at least _comment_ on it would be nice. As of right now I don't believe that anyone outside of this room who understands this stuff has seen it, except maybe bytecoin.
444 2012-11-27 15:26:33 <sipa> bytecoin has seen it, yes
445 2012-11-27 15:26:51 <sipa> he just made some comments on the formulation of the document
446 2012-11-27 15:26:59 <sipa> ha, DJB would be nice indeed :)
447 2012-11-27 15:28:07 <gmaxwell> I feel vaguely confident that nothing anyone would raise would be a reason not to use it in cases where its a big win (e.g. the webserver case), but I expect most usage won't be doing that and could use an alternative formulation where the private keys were just straight hmac output.
448 2012-11-27 15:28:10 <kjj> sipa: ok, I'll take your word for it, but I don't see any place where either of them need an obvious limit, nor any place where the representation matters
449 2012-11-27 15:29:18 <sipa> gmaxwell: well, we could decide not to expose an interface for importing extended public key chains initially
450 2012-11-27 15:29:33 <sipa> gmaxwell: if there is no way to export them either, it's clearly experimental
451 2012-11-27 15:29:34 <sipa> gmaxwell: if there is no way to export them either, it's clearly experimental
452 2012-11-27 15:30:33 <BlueMatt> the jenkins vm is getting kinda lonely...anyone core devs wanna sign up to manage an https download server/testnet node/etc?
453 2012-11-27 15:30:34 <BlueMatt> the jenkins vm is getting kinda lonely...anyone core devs wanna sign up to manage an https download server/testnet node/etc?
454 2012-11-27 15:31:21 <sipa> gmaxwell: i somehow prefer not to have two standards, especially if at least a restricted form of one is safe
455 2012-11-27 15:31:30 <gmaxwell> Yea.... :(
456 2012-11-27 15:31:57 <gmaxwell> Well an alternative would be to deploy hd-wallet determinstic support but continue using non-determinstic by default.
457 2012-11-27 15:32:48 <sipa> gmaxwell: how about trying to get dan kaminsky to look at it?
458 2012-11-27 15:33:07 <sipa> he's commented on bitcoin before
459 2012-11-27 15:33:23 <BlueMatt> is he a cryptographer or just a security researcher?
460 2012-11-27 15:33:35 <sipa> good point, i'm not sure
461 2012-11-27 15:33:40 <gmaxwell> I think his commentary would be more useful on the Bip32 structure than on the underlying cryptographic construct.
462 2012-11-27 15:35:47 <sipa> gmaxwell: given the fact that parent-extended-pubkey + child-privkey is enough to derive parent-extended-privkey, i think extended pubkeys should in any case be considered secret to a certain degree
463 2012-11-27 15:40:06 <phantomcircuit> kjj, personally i specify a very large keypool once and wait for all the new keypairs to be generated
464 2012-11-27 15:40:23 <phantomcircuit> from there on i run usign the normal keypool size
465 2012-11-27 15:40:24 <phantomcircuit> from there on i run usign the normal keypool size
466 2012-11-27 15:40:41 <sipa> don't share them at all, and you get a traditional derivation scheme (i'm very sure that multiplying two numbers about which nothing is known mod p, will give a number about which nothing is known)
467 2012-11-27 15:41:02 <sipa> unless one is 0, of course
468 2012-11-27 15:41:04 <phantomcircuit> the once huge value will however cause the keypool to remain ballooned and since the keypool parameter is smaller it will slowly deplete
469 2012-11-27 15:41:05 <phantomcircuit> the once huge value will however cause the keypool to remain ballooned and since the keypool parameter is smaller it will slowly deplete
470 2012-11-27 15:41:18 <phantomcircuit> as long as you backup before the pool starts to refill
471 2012-11-27 15:41:19 <phantomcircuit> as long as you backup before the pool starts to refill
472 2012-11-27 15:41:29 <phantomcircuit> you always have a good backup
473 2012-11-27 15:41:30 <phantomcircuit> you always have a good backup
474 2012-11-27 15:42:04 <kjj> phantomcircuit: yeah, that's what I'm suggesting should be the default behavior, rather than a silly trick that you have to understand and perform yourself
475 2012-11-27 15:42:05 <kjj> phantomcircuit: yeah, that's what I'm suggesting should be the default behavior, rather than a silly trick that you have to understand and perform yourself
476 2012-11-27 15:42:13 <gmaxwell> sipa: they aren't _strictly_ nothing is known, as you do sign with them.
477 2012-11-27 15:42:14 <gmaxwell> sipa: they aren't _strictly_ nothing is known, as you do sign with them.
478 2012-11-27 15:42:23 <phantomcircuit> kjj, true
479 2012-11-27 15:42:24 <phantomcircuit> kjj, true
480 2012-11-27 15:42:35 <phantomcircuit> kjj, how would you implement that though?
481 2012-11-27 15:42:45 <sipa> gmaxwell: that's a good p9oint
482 2012-11-27 15:42:46 <sipa> gmaxwell: that's a good p9oint
483 2012-11-27 15:42:49 <sipa> gmaxwell: i didn't consider that
484 2012-11-27 15:42:50 <sipa> gmaxwell: i didn't consider that
485 2012-11-27 15:43:05 <phantomcircuit> the problem is that something like this requires a way for there to be a signal to backup the wallet.dat again
486 2012-11-27 15:43:06 <sipa> i doubt it changes anything, but i wouldn't bet my life on it
487 2012-11-27 15:43:07 <sipa> i doubt it changes anything, but i wouldn't bet my life on it
488 2012-11-27 15:43:20 <phantomcircuit> i do it using polling of the getinfo api call but that's not something most users can do
489 2012-11-27 15:43:21 <phantomcircuit> i do it using polling of the getinfo api call but that's not something most users can do
490 2012-11-27 15:43:43 <kjj> phantomcircuit: start spewing warnings at the low water mark, and have backupwallet refill the pool just before the backup
491 2012-11-27 15:43:44 <kjj> phantomcircuit: start spewing warnings at the low water mark, and have backupwallet refill the pool just before the backup
492 2012-11-27 15:44:31 <gmaxwell> sipa: right??? obviously I think this construct is secure. But I do not have a proof of it nor do I have the number theory chops to attempt one. In the case that no one ever sees the signatures there is a someone intutive explination that it is secure but that one doesn't hold under signatures.
493 2012-11-27 15:44:32 <gmaxwell> sipa: right??? obviously I think this construct is secure. But I do not have a proof of it nor do I have the number theory chops to attempt one. In the case that no one ever sees the signatures there is a someone intutive explination that it is secure but that one doesn't hold under signatures.
494 2012-11-27 15:45:07 <phantomcircuit> kjj, ah so instead of refilling the pool automatically only do it when a backup is generated
495 2012-11-27 15:45:08 <phantomcircuit> kjj, ah so instead of refilling the pool automatically only do it when a backup is generated
496 2012-11-27 15:45:20 <gmaxwell> And I can give you an example where it fails too... a child key signs using an insecure nonce, and you can then shimmy up the chain and get other private keys.
497 2012-11-27 15:45:21 <gmaxwell> And I can give you an example where it fails too... a child key signs using an insecure nonce, and you can then shimmy up the chain and get other private keys.
498 2012-11-27 15:45:24 <phantomcircuit> that's actually a very good way to do it (optionally of course)
499 2012-11-27 15:45:27 <kjj> right.
500 2012-11-27 15:45:59 <phantomcircuit> i have no idea how you'd make that work in the ui but in the bitcoind i believe it would be (relatively) easy to change
501 2012-11-27 15:46:00 <kjj> I would even have the wallet start with NO keys, and require the first backupwallet command to fill it. but that would be more work
502 2012-11-27 15:46:00 <phantomcircuit> i have no idea how you'd make that work in the ui but in the bitcoind i believe it would be (relatively) easy to change
503 2012-11-27 15:46:01 <kjj> I would even have the wallet start with NO keys, and require the first backupwallet command to fill it. but that would be more work
504 2012-11-27 15:46:34 <kjj> yeah, I don't mess with the UI at all. no idea how to handle it there
505 2012-11-27 15:52:07 <abrkn> hmm when i try this one https://launchpad.net/~bitcoin/+archive/bitcoin it installs 0.6 when using apt-get. is there any issue with the priority of my sources?
506 2012-11-27 15:52:08 <abrkn> hmm when i try this one https://launchpad.net/~bitcoin/+archive/bitcoin it installs 0.6 when using apt-get. is there any issue with the priority of my sources?
507 2012-11-27 15:53:20 <sipa> BlueMatt: wasn't there a plan to put a crawler/dnsseed of some kind on the server?
508 2012-11-27 15:53:21 <sipa> BlueMatt: wasn't there a plan to put a crawler/dnsseed of some kind on the server?
509 2012-11-27 15:53:55 <BlueMatt> sipa: yea, that was another idea
510 2012-11-27 15:54:01 <BlueMatt> sipa: wanna run it?
511 2012-11-27 15:54:02 <BlueMatt> sipa: wanna run it?
512 2012-11-27 15:57:26 <sipa> BlueMatt: i suppose
513 2012-11-27 15:58:13 <BlueMatt> sipa: os of choice?
514 2012-11-27 15:58:14 <BlueMatt> sipa: os of choice?
515 2012-11-27 15:59:19 <sipa> something debian or ubuntu based?
516 2012-11-27 15:59:20 <sipa> something debian or ubuntu based?
517 2012-11-27 15:59:28 <BlueMatt> debian it is
518 2012-11-27 15:59:49 <phantomcircuit> http://imgur.com/gallery/t8D7l
519 2012-11-27 15:59:50 <phantomcircuit> hahah
520 2012-11-27 16:00:13 <abrkn> oh jesus fuck now i have to download the chain again :(
521 2012-11-27 16:00:14 <abrkn> oh jesus fuck now i have to download the chain again :(
522 2012-11-27 16:00:20 <abrkn> 48h+ last time
523 2012-11-27 16:00:21 <abrkn> 48h+ last time
524 2012-11-27 16:00:26 <sipa> abrkn: why?
525 2012-11-27 16:00:43 <abrkn> sipa: didnt like my index going from 0.6 to 0.7.1
526 2012-11-27 16:00:44 <abrkn> sipa: didnt like my index going from 0.6 to 0.7.1
527 2012-11-27 16:01:11 <abrkn> sipa: well, it's going pretty fast, maybe it's using the blkindex files i copied in
528 2012-11-27 16:01:25 <sipa> no, it will not
529 2012-11-27 16:01:26 <sipa> no, it will not
530 2012-11-27 16:01:27 <abrkn> err, the blk000.dat files
531 2012-11-27 16:01:33 <sipa> it will just append to them
532 2012-11-27 16:01:37 <sipa> and you'll use twice as much storage
533 2012-11-27 16:01:38 <sipa> and you'll use twice as much storage
534 2012-11-27 16:01:54 <abrkn> ok, because im getting like 1k/sec now
535 2012-11-27 16:02:09 <sipa> the first 130k blocks are very fast
536 2012-11-27 16:02:10 <sipa> the first 130k blocks are very fast
537 2012-11-27 16:02:16 <Eliel> abrkn: if you have lots of RAM, you could setup a ramdisk and keep blkindex.dat on the ramdisk until done and then copy it to the hdd.
538 2012-11-27 16:02:17 <Eliel> abrkn: if you have lots of RAM, you could setup a ramdisk and keep blkindex.dat on the ramdisk until done and then copy it to the hdd.
539 2012-11-27 16:02:31 <abrkn> ok, so i should stop bitcoind, remove the blk files, start over?
540 2012-11-27 16:02:32 <abrkn> ok, so i should stop bitcoind, remove the blk files, start over?
541 2012-11-27 16:02:53 <Eliel> abrkn: that'd be a good idea IMO.
542 2012-11-27 16:02:54 <Eliel> abrkn: that'd be a good idea IMO.
543 2012-11-27 16:03:12 <sipa> abrkn: you should put those blk files somewhere else, clear the actual datadir, and run with -loadblock=/path/to/blk0001.dat -loadblock=/path/to/blk0002.dat
544 2012-11-27 16:03:23 <sipa> that will import them from disk instead of from network
545 2012-11-27 16:05:04 <gmaxwell> note that you _must_ heed the "somewhere else" above if you go that way.
546 2012-11-27 16:05:31 <abrkn> ok, doing as you said now
547 2012-11-27 16:05:32 <abrkn> ok, doing as you said now
548 2012-11-27 16:06:25 <weex> for some reason, this minimal python address generator of mine/Joric's creates invalid compressed privkeys
549 2012-11-27 16:06:26 <weex> for some reason, this minimal python address generator of mine/Joric's creates invalid compressed privkeys
550 2012-11-27 16:06:42 <Eliel> by the way, why does the client ignore the earlier contents of blk0*.dat if blkindex.dat is missing and they're not empty or nonexistent?
551 2012-11-27 16:06:43 <Eliel> by the way, why does the client ignore the earlier contents of blk0*.dat if blkindex.dat is missing and they're not empty or nonexistent?
552 2012-11-27 16:06:50 <weex> i've been trying to compare what it does vs. bitcoin and that's quite a task
553 2012-11-27 16:07:05 <kjj> Eliel: it's complicated
554 2012-11-27 16:07:39 <kjj> weex: invalid in what way? is it just calculating the parity wrong?
555 2012-11-27 16:07:40 <sipa> Eliel: its write policy is "new blocks are appended", and without blkindex.dat, it doesn't know about any block, so every block is new
556 2012-11-27 16:07:41 <sipa> Eliel: its write policy is "new blocks are appended", and without blkindex.dat, it doesn't know about any block, so every block is new
557 2012-11-27 16:07:56 <weex> bitaddress.org says it's invalid and so does bitcoind
558 2012-11-27 16:08:02 <weex> https://github.com/weex/addrgen/blob/master/addrgen.py is where the code is
559 2012-11-27 16:08:03 <weex> https://github.com/weex/addrgen/blob/master/addrgen.py is where the code is
560 2012-11-27 16:08:06 <sipa> Eliel: 0.8 will work differently (it stores which part of the file is still available in the index)
561 2012-11-27 16:08:07 <sipa> Eliel: 0.8 will work differently (it stores which part of the file is still available in the index)
562 2012-11-27 16:08:18 <weex> i had it using compressed by default too (yikes but uncompressed seems valid)
563 2012-11-27 16:10:18 <sipa> Eliel: also 0.8 will have -reindex, which is like -loadblock, but uses the existing block files and reindexes and validates them in-place
564 2012-11-27 16:10:19 <sipa> Eliel: also 0.8 will have -reindex, which is like -loadblock, but uses the existing block files and reindexes and validates them in-place
565 2012-11-27 16:10:25 <kjj> ugh. python, and a just a wrapper for ssl libs
566 2012-11-27 16:10:50 <Eliel> sipa: will it trigger -reindex if the indexes go missing in 0.8?
567 2012-11-27 16:12:05 <sipa> Eliel: i think it's safe logic to have: if (block_files_present && !block_index_present) { reindex=1 }
568 2012-11-27 16:13:56 <Eliel> sipa: will it start in a lightweight client mode?
569 2012-11-27 16:13:57 <Eliel> sipa: will it start in a lightweight client mode?
570 2012-11-27 16:14:20 <kjj> weex: I found the bug. line 151, should be: payload = secret + chr(1)
571 2012-11-27 16:14:31 <sipa> Eliel: no
572 2012-11-27 16:14:47 <weex> k lemme try
573 2012-11-27 16:15:00 <sipa> what was the line?
574 2012-11-27 16:15:01 <sipa> what was the line?
575 2012-11-27 16:15:22 <kjj> they are adding 0x00 as the flag for a compressed key, should be 0x01
576 2012-11-27 16:16:10 <sipa> ha
577 2012-11-27 16:16:11 <sipa> ha
578 2012-11-27 16:16:13 <weex> kjj, nice! now out of curiosity should it be possible to take a private key generated the old way and turn it into a correct one?
579 2012-11-27 16:16:28 <kjj> weex: sure
580 2012-11-27 16:16:56 <kjj> decode back to the secret, attach the correct flag, re-encode
581 2012-11-27 16:18:36 <weex> ok i'll see if i can do that
582 2012-11-27 16:21:28 <kjj> I didn't look closely at your decode function, so I don't know if it is right or not. but at the very least, it appears to just return all of the data, including the incorrect compression flag
583 2012-11-27 16:21:29 <kjj> I didn't look closely at your decode function, so I don't know if it is right or not. but at the very least, it appears to just return all of the data, including the incorrect compression flag
584 2012-11-27 16:41:41 <helo> when blocks start filling up consistently, what will happen to all of the transactions that can't make it into blocks for extended periods of time?
585 2012-11-27 16:41:42 <helo> when blocks start filling up consistently, what will happen to all of the transactions that can't make it into blocks for extended periods of time?
586 2012-11-27 16:41:47 <helo> will they just keep accumulating?
587 2012-11-27 16:41:48 <helo> will they just keep accumulating?
588 2012-11-27 16:45:34 <kjj> we'll probably have the client recognize those and pop up a warning, asking the user if he wants the node to keep re-transmitting it, or if he wants to let it fall out of the network's collective memory pool
589 2012-11-27 16:45:35 <kjj> we'll probably have the client recognize those and pop up a warning, asking the user if he wants the node to keep re-transmitting it, or if he wants to let it fall out of the network's collective memory pool
590 2012-11-27 16:50:25 <weex> thanks kjj, looks like the re-encoding is working
591 2012-11-27 16:50:39 <phantomcircuit> at somepoint you want to doublespend yourself basically with a larger txfee to overrule the previous one
592 2012-11-27 16:50:40 <phantomcircuit> at somepoint you want to doublespend yourself basically with a larger txfee to overrule the previous one
593 2012-11-27 16:50:58 <phantomcircuit> but iirc the current memory pool wont evict a confirmed tx in favor of one with a larger txfee will it?
594 2012-11-27 16:54:18 <sipa> phantomcircuit: no
595 2012-11-27 16:54:19 <sipa> phantomcircuit: no
596 2012-11-27 16:54:41 <phantomcircuit> sipa, sorry is that yes it will evict or no it wont evict
597 2012-11-27 16:54:48 <sipa> phantomcircuit: it will not evict
598 2012-11-27 16:55:06 <phantomcircuit> yeah so you have to find a miner who is willing to replace your previous transaction with a new one
599 2012-11-27 16:55:16 <phantomcircuit> or to mine if
600 2012-11-27 16:55:17 <phantomcircuit> or to mine if
601 2012-11-27 16:55:18 <phantomcircuit> it*
602 2012-11-27 16:55:19 <phantomcircuit> it*
603 2012-11-27 16:55:20 <phantomcircuit> the old one
604 2012-11-27 16:55:21 <phantomcircuit> the old one
605 2012-11-27 16:55:21 <sipa> phantomcircuit: even if tx replacement would be re-enabled, the txins need to remain the same
606 2012-11-27 16:55:33 <sipa> so the only way to do it would be to reduce the change to yourself
607 2012-11-27 16:56:03 <helo> to keep it from being useful for double-spending?
608 2012-11-27 16:56:04 <helo> to keep it from being useful for double-spending?
609 2012-11-27 16:56:31 <sipa> i believe that's the idea yes - once in the collective network's memory pool, double spending is prevented
610 2012-11-27 17:20:58 <BlueMatt> well...I was gonna comment a bit on the payment protocol stuff...now it turned into a thread 41 emails deep...now there is no way Im gonna read through that many emails and respond...
611 2012-11-27 17:23:11 <jgarzik> BlueMatt: most of the recent stuff is rather off-track
612 2012-11-27 17:25:07 <kjj> sipa: but old unconfirmed transactions are eventually forgotten if the originating node stops sending them.
613 2012-11-27 17:25:28 <sipa> kjj: yes, but in an unpredictable fashion
614 2012-11-27 17:26:05 <kjj> agreed. and there probably isn't much desire to make it more predictable
615 2012-11-27 17:26:06 <kjj> agreed. and there probably isn't much desire to make it more predictable
616 2012-11-27 17:27:33 <kjj> but after 2 weeks, or whatever, we *could* have the local node forget the old transaction, and make a new one sending all back to the change address, allowing them to (eventually) be reclaimed
617 2012-11-27 17:28:17 <kjj> might not be a bad idea to do that now, actually. but it won't be a big deal until the block max is hit most of the time
618 2012-11-27 17:29:51 <kjj> Right now, the solution to "Help, I have a transaction that hasn't confirmed for a month" is to attack your wallet.dat with a hex editor, which is, erm, other than ideal.
619 2012-11-27 17:29:52 <kjj> Right now, the solution to "Help, I have a transaction that hasn't confirmed for a month" is to attack your wallet.dat with a hex editor, which is, erm, other than ideal.
620 2012-11-27 17:39:40 <abrkn> is this a reasonable way to keep track of btc sent to own addresses by using bitcoind? https://gist.github.com/4156125
621 2012-11-27 17:39:41 <abrkn> is this a reasonable way to keep track of btc sent to own addresses by using bitcoind? https://gist.github.com/4156125
622 2012-11-27 17:39:55 <gavinandresen> BlueMatt: feel free to comment on payment protocol stuff here. Most of the thread is "We don't like SSL/X.509 certificates, so {let's not bother / let's wait for something better / let's use something 0.0001% of the Internet world uses}"
623 2012-11-27 17:39:56 <gavinandresen> BlueMatt: feel free to comment on payment protocol stuff here. Most of the thread is "We don't like SSL/X.509 certificates, so {let's not bother / let's wait for something better / let's use something 0.0001% of the Internet world uses}"
624 2012-11-27 17:45:51 <phantomcircuit> gavinandresen, the basic concept is to piggy back on the existing certificates infrastructure right?
625 2012-11-27 17:45:52 <phantomcircuit> gavinandresen, the basic concept is to piggy back on the existing certificates infrastructure right?
626 2012-11-27 17:46:11 <gavinandresen> phantomcircuit: yes.
627 2012-11-27 17:46:12 <gavinandresen> phantomcircuit: yes.
628 2012-11-27 17:46:41 <BlueMatt> I actually find the use x509 pki solution to be quite an elegant solution to the problems we face (not that I like pki, as with all other sane people, I think it has serious problems)
629 2012-11-27 17:46:42 <BlueMatt> I actually find the use x509 pki solution to be quite an elegant solution to the problems we face (not that I like pki, as with all other sane people, I think it has serious problems)
630 2012-11-27 17:47:33 <kjj> I find protocol buffers to be more objectionable than SSL
631 2012-11-27 17:47:34 <kjj> I find protocol buffers to be more objectionable than SSL
632 2012-11-27 17:47:54 <gavinandresen> good, something in it for EVERYBODY to hate.
633 2012-11-27 17:47:57 <kjj> heh
634 2012-11-27 17:47:58 <kjj> heh
635 2012-11-27 17:47:58 <phantomcircuit> protobuf is a very efficiently packed encoding
636 2012-11-27 17:47:59 <phantomcircuit> protobuf is a very efficiently packed encoding
637 2012-11-27 17:48:11 <kjj> I found the segment on "why PB" to be less than convincing.
638 2012-11-27 17:48:12 <kjj> I found the segment on "why PB" to be less than convincing.
639 2012-11-27 17:48:19 <phantomcircuit> that being said im not really sure the libraries have been extensively tested for security issues
640 2012-11-27 17:48:20 <BlueMatt> kjj: and you suggest...?
641 2012-11-27 17:48:20 <phantomcircuit> that being said im not really sure the libraries have been extensively tested for security issues
642 2012-11-27 17:48:21 <kjj> phantomcircuit: yes. ANOTHER efficient packed encoding
643 2012-11-27 17:48:22 <kjj> phantomcircuit: yes. ANOTHER efficient packed encoding
644 2012-11-27 17:48:54 <gavinandresen> kjj: you read the JOSE specs? Or do you have some other favorite binary encoding? (don't say XML or we'll lynch you)
645 2012-11-27 17:48:55 <gavinandresen> kjj: you read the JOSE specs? Or do you have some other favorite binary encoding? (don't say XML or we'll lynch you)
646 2012-11-27 17:48:55 <kjj> BlueMatt: I would do JSON. I didn't find the reasons stated for not using JSON to be unconvincing
647 2012-11-27 17:48:56 <kjj> BlueMatt: I would do JSON. I didn't find the reasons stated for not using JSON to be unconvincing
648 2012-11-27 17:49:23 <phantomcircuit> json is hopelessly inefficient for something like this :|
649 2012-11-27 17:49:24 <phantomcircuit> json is hopelessly inefficient for something like this :|
650 2012-11-27 17:49:26 <drizztbsd> no, json is overrated
651 2012-11-27 17:49:30 <gavinandresen> phantomcircuit: I asked TD about protocol buffers and security, and he assures me they're solid.
652 2012-11-27 17:49:39 <drizztbsd> also yaml is better
653 2012-11-27 17:49:40 <drizztbsd> also yaml is better
654 2012-11-27 17:49:42 <gavinandresen> (if they weren't, Google would have major problems)
655 2012-11-27 17:49:45 <phantomcircuit> gavinandresen, k
656 2012-11-27 17:49:46 <phantomcircuit> gavinandresen, k
657 2012-11-27 17:50:05 <phantomcircuit> i wasn't aware google had any exposed api's using protobuf
658 2012-11-27 17:50:06 <phantomcircuit> i wasn't aware google had any exposed api's using protobuf
659 2012-11-27 17:50:16 <BlueMatt> kjj: if nothing else, the section on why not json is pretty convincing that a binary format is better
660 2012-11-27 17:50:33 <phantomcircuit> i got the opposite impression when i was looking at them, but it was just an impression
661 2012-11-27 17:50:34 <phantomcircuit> i got the opposite impression when i was looking at them, but it was just an impression
662 2012-11-27 17:50:37 <kjj> BlueMatt: I read it, but was unconvinced
663 2012-11-27 17:51:26 <BlueMatt> kjj: in any case, stating that we should use JSON without making an argument (aside from we already use it) when there is an argument against json doesnt do much in the way of convincing anyway
664 2012-11-27 17:51:26 <kjj> first of all, we trust the certificate, so it doesn't matter if someone can create multiple different encodings of the same data. we care about the data we got that was signed, not all of the possible other ways it *could* have been encoded
665 2012-11-27 17:52:10 <kjj> and it that really was important for some reason, we could demand that the JSON be ordered in a particular way prior to signing.
666 2012-11-27 17:52:11 <kjj> and it that really was important for some reason, we could demand that the JSON be ordered in a particular way prior to signing.
667 2012-11-27 17:52:22 <kjj> a canonical form, if you will
668 2012-11-27 17:52:23 <kjj> a canonical form, if you will
669 2012-11-27 17:52:36 <gavinandresen> kjj: complexity like that is the enemy of security.
670 2012-11-27 17:52:53 <BlueMatt> ^ this
671 2012-11-27 17:53:24 <kjj> if we assume that the attacker can sign different forms of the same data, why can't we also assume that they can just sign their own stuff instead?
672 2012-11-27 17:53:55 <kjj> either we trust the signature scheme, or we shouldn't be using it in the first place.
673 2012-11-27 17:54:02 <BlueMatt> do you have an argument for json (aside from its already used in some places in bitcoin?)
674 2012-11-27 17:54:15 <BlueMatt> which, btw, its only used in bitcoind, not in any place that is standard bitcoin stuff
675 2012-11-27 17:54:15 <gavinandresen> The risk would be that the attacker inserts some extra data that the check-canonical-signature code throws away but (maybe due to a bug) the display-transaction code displays. For example.
676 2012-11-27 17:54:16 <BlueMatt> which, btw, its only used in bitcoind, not in any place that is standard bitcoin stuff
677 2012-11-27 17:55:58 <kjj> gavinandresen: reject the whole thing then instead of silently discarding part of it. there is plenty of precedent. and it still doesn't matter unless you think that the attacker can control the data to be signed, which is already game over
678 2012-11-27 17:55:59 <kjj> gavinandresen: reject the whole thing then instead of silently discarding part of it. there is plenty of precedent. and it still doesn't matter unless you think that the attacker can control the data to be signed, which is already game over
679 2012-11-27 17:56:44 <kjj> BlueMatt: my argument for JSON is that it is already all over the code, and the people working on it know it very well, and the people that don't work on it can figure it out in like 30 seconds.
680 2012-11-27 17:56:58 <BlueMatt> all over what code?
681 2012-11-27 17:57:02 <BlueMatt> bitcoind's, yes
682 2012-11-27 17:57:46 <BlueMatt> the protocol buffer libs are also easy to understand, even if the data isnt as much
683 2012-11-27 17:58:03 <kjj> what is the argument in favor of PB?
684 2012-11-27 17:58:21 <BlueMatt> its better than the alternatives
685 2012-11-27 17:58:22 <kjj> saving a few bytes per payment?
686 2012-11-27 17:58:31 <kjj> heh. better how?
687 2012-11-27 17:58:32 <kjj> heh. better how?
688 2012-11-27 17:59:01 <BlueMatt> though maybe not significant, there are potential issues (if nothing else, complications in implementation) due to the multiple encodings issue
689 2012-11-27 17:59:05 <gavinandresen> kjj: Mike Hearn whipped up code for implementing signed invoices in about an hour. Feel free to do the same for JSON and send me the code, maybe you'll convince me it is easy.
690 2012-11-27 17:59:06 <gavinandresen> kjj: Mike Hearn whipped up code for implementing signed invoices in about an hour. Feel free to do the same for JSON and send me the code, maybe you'll convince me it is easy.
691 2012-11-27 17:59:37 <gavinandresen> (just be sure to handle the gotchas pointed out in the JOSE specs properly)
692 2012-11-27 18:02:32 <kjj> heh, that's the best argument in favor of PB that I've seen yet: "Someone else has already implemented it"
693 2012-11-27 18:03:50 <gavinandresen> Again, complexity is the enemy of security, and that's why the spec went from being JSON-based to PB-based. PB give zero "wiggle-room" to potential attackers.
694 2012-11-27 18:03:51 <gavinandresen> Again, complexity is the enemy of security, and that's why the spec went from being JSON-based to PB-based. PB give zero "wiggle-room" to potential attackers.
695 2012-11-27 18:04:15 <sipa> gavinandresen: actually, it does; the fields in a PB encoding can be reordered :)
696 2012-11-27 18:04:41 <phantomcircuit> and integers have something like the var_int weirdness
697 2012-11-27 18:04:50 <phantomcircuit> but in general there is less wiggleroom
698 2012-11-27 18:04:51 <phantomcircuit> but in general there is less wiggleroom
699 2012-11-27 18:04:58 <gavinandresen> ok, fine, much less wiggle room....
700 2012-11-27 18:04:59 <gavinandresen> ok, fine, much less wiggle room....
701 2012-11-27 18:05:21 <kjj> gavinandresen: I don't disagree with that principle, but the wiggle room in question was pretty damn small, while the cost of inflicting yet another incompatible binary packing standard on the bitcoin world is pretty high.
702 2012-11-27 18:05:44 <sipa> bitcoinj is already entirely PB based
703 2012-11-27 18:05:45 <sipa> bitcoinj is already entirely PB based
704 2012-11-27 18:05:59 <sipa> and JSON is only used in bitcoind RPC
705 2012-11-27 18:06:00 <sipa> and JSON is only used in bitcoind RPC
706 2012-11-27 18:06:44 <sipa> if anything, i wished that satoshi had used PB from the start, but at least for all core protocol stuff (blocks and transactions) we're stuck with it
707 2012-11-27 18:06:47 <kjj> and now EVERYONE that wants to deal with these invoices will be using PB too. see the difference?
708 2012-11-27 18:06:48 <kjj> and now EVERYONE that wants to deal with these invoices will be using PB too. see the difference?
709 2012-11-27 18:07:34 <sipa> yes, and otherwise EVERYONE that wants to deal with these invoices will be using JSON too. see the difference?
710 2012-11-27 18:07:59 <sipa> both are used in some places already, and we got to pick something
711 2012-11-27 18:08:08 <kjj> oh, totally. but JSON is a less icky than PB
712 2012-11-27 18:08:09 <kjj> oh, totally. but JSON is a less icky than PB
713 2012-11-27 18:08:45 <sipa> i love PB's simplicity far more than JSON's ambiguity (it doesn't even specify what kind of precision a "number" has)
714 2012-11-27 18:08:46 <sipa> i love PB's simplicity far more than JSON's ambiguity (it doesn't even specify what kind of precision a "number" has)
715 2012-11-27 18:08:51 <gavinandresen> http://code.google.com/p/protobuf/wiki/ThirdPartyAddOns has a nice list of the languages supported.... and they're not icky.
716 2012-11-27 18:10:33 <jgarzik> sipa: +1
717 2012-11-27 18:11:03 <jgarzik> JSON is runtime flexible, but that's not the best when you are hashing and need tight specificity
718 2012-11-27 18:11:04 <jgarzik> JSON is runtime flexible, but that's not the best when you are hashing and need tight specificity
719 2012-11-27 18:11:40 <jgarzik> thus JSON inevitably requires some layer on top, making things more strict
720 2012-11-27 18:11:41 <jgarzik> thus JSON inevitably requires some layer on top, making things more strict
721 2012-11-27 18:12:06 <jgarzik> RE <sipa> if anything, i wished that satoshi had used PB from the start
722 2012-11-27 18:12:08 <kjj> except that for this use, we don't need to be tight at all.
723 2012-11-27 18:12:09 <kjj> except that for this use, we don't need to be tight at all.
724 2012-11-27 18:12:11 <jgarzik> Satoshi actually said same
725 2012-11-27 18:12:20 <jgarzik> Satoshi said he would have used PB, if he had known about it at the time
726 2012-11-27 18:12:21 <jgarzik> Satoshi said he would have used PB, if he had known about it at the time
727 2012-11-27 18:12:30 <jgarzik> rather than a custom serialization
728 2012-11-27 18:12:57 <jgarzik> but even when he said that, it was obviously far too late to change
729 2012-11-27 18:13:26 <jgarzik> what was that other serialization.... minpack?
730 2012-11-27 18:13:28 <jgarzik> ACTION googles
731 2012-11-27 18:13:49 <jgarzik> msgpack
732 2012-11-27 18:13:51 <jgarzik> http://wiki.msgpack.org/display/MSGPACK/Overview
733 2012-11-27 18:14:09 <jgarzik> more compact than PB. less well known, fewer languages support, ordering might be more strict.
734 2012-11-27 18:14:18 <sipa> initially i thought PB was just something like ASN.1, trying to encode the structure of the data along with it
735 2012-11-27 18:14:25 <jgarzik> another, more exotic option, is to create a PB compiler and lib for bitcoin's custom serialization!
736 2012-11-27 18:14:26 <jgarzik> another, more exotic option, is to create a PB compiler and lib for bitcoin's custom serialization!
737 2012-11-27 18:14:34 <jgarzik> you get strict, defined order, re-use existing code.
738 2012-11-27 18:14:41 <sipa> but it's not; it really just adds exactly as much overhead to keep it infinitely extensible
739 2012-11-27 18:14:44 <jgarzik> no langs supported
740 2012-11-27 18:15:18 <jgarzik> if anybody wants that option, I could whip up a BB (bitcoinbufs) compiler
741 2012-11-27 18:15:43 <kjj> sipa: that's what I call the XML problem. when XML started making the rounds, people were making all sorts of crazy claims about not needing to know the format of data before interchange
742 2012-11-27 18:15:44 <kjj> sipa: that's what I call the XML problem. when XML started making the rounds, people were making all sorts of crazy claims about not needing to know the format of data before interchange
743 2012-11-27 18:15:58 <sipa> kjj: but that's it - PB doesn't do that
744 2012-11-27 18:16:10 <sipa> it does require you to know the data type being encoded
745 2012-11-27 18:16:11 <sipa> it does require you to know the data type being encoded
746 2012-11-27 18:16:14 <jgarzik> protobufs notably does not do as good a job of supporting vectors (repeated) as XDR-RPC, either. PB does not support defined sizes/limit for vectors.
747 2012-11-27 18:16:23 <sipa> which you in practice always know anyway
748 2012-11-27 18:16:24 <sipa> which you in practice always know anyway
749 2012-11-27 18:16:25 <kjj> sipa: and the structure
750 2012-11-27 18:16:26 <kjj> sipa: and the structure
751 2012-11-27 18:17:28 <sipa> jgarzik: yes, i agree there are some missed chances with PB (like tagged unions), but it is very simple
752 2012-11-27 18:17:29 <sipa> jgarzik: yes, i agree there are some missed chances with PB (like tagged unions), but it is very simple
753 2012-11-27 18:17:43 <jgarzik> yes. nanopb is cute. didn't know about that until recently.
754 2012-11-27 18:17:59 <gavinandresen> speaking of limits... I'm tempted to specify size limits for Invoice/Payment/Receipt messages, to try to head-off DoS attacks like "I'll send you a 10,000-long X.509 chain and hope you spend CPU minutes verifying it...."
755 2012-11-27 18:18:13 <jgarzik> gavinandresen: is there a version field in there?
756 2012-11-27 18:18:14 <jgarzik> gavinandresen: is there a version field in there?
757 2012-11-27 18:18:21 <jgarzik> gavinandresen: might make a mistake on limits
758 2012-11-27 18:18:22 <jgarzik> gavinandresen: might make a mistake on limits
759 2012-11-27 18:18:47 <gavinandresen> jgarzik: no version field. Probably a good idea to add one...
760 2012-11-27 18:18:48 <gavinandresen> jgarzik: no version field. Probably a good idea to add one...
761 2012-11-27 18:19:13 <jgarzik> gavinandresen: I agree that "cert type / cert bytes" is nicer than "x.509 cert bytes"
762 2012-11-27 18:19:27 <jgarzik> TD is right that the format is upgradeable... but why create a new proto field for each type?
763 2012-11-27 18:19:28 <jgarzik> TD is right that the format is upgradeable... but why create a new proto field for each type?
764 2012-11-27 18:19:35 <gavinandresen> jgarzik: meh. version=2 could do that....
765 2012-11-27 18:19:36 <gavinandresen> jgarzik: meh. version=2 could do that....
766 2012-11-27 18:20:01 <sipa> the cert data is likely to be black box bytes anyway
767 2012-11-27 18:20:13 <jgarzik> x509_cert bytes. next week, add jg_cert_format to foo.proto. next month, add jg_cert_2_format.
768 2012-11-27 18:20:15 <sipa> i prefer cert type/data as well
769 2012-11-27 18:20:16 <sipa> i prefer cert type/data as well
770 2012-11-27 18:20:32 <jgarzik> "x509_cert bytes" is just too hardcoded and inflexible
771 2012-11-27 18:20:33 <jgarzik> "x509_cert bytes" is just too hardcoded and inflexible
772 2012-11-27 18:21:00 <gavinandresen> will we have multiple types in one Invoice?
773 2012-11-27 18:21:11 <jgarzik> gavinandresen: doubtful
774 2012-11-27 18:21:13 <gavinandresen> (e.g. for extra security or backwards compatibility) ?
775 2012-11-27 18:21:31 <jgarzik> gavinandresen: but it's droll to forever have unused x509_cert field in there, even if technically optional by PB standards
776 2012-11-27 18:21:37 <kjj> oh, ha!
777 2012-11-27 18:21:38 <kjj> oh, ha!
778 2012-11-27 18:22:11 <gavinandresen> It'll also be droll to forever have to specify type="x509" if we never have anything else
779 2012-11-27 18:22:13 <kjj> we could just make an optional cert_type and assume that it is X509 if not present
780 2012-11-27 18:22:27 <jgarzik> kjj: sure, that's doable within PB easily
781 2012-11-27 18:22:29 <sipa> gavinandresen: i really don't care about that
782 2012-11-27 18:22:30 <sipa> gavinandresen: i really don't care about that
783 2012-11-27 18:22:31 <jgarzik> gavinandresen: ^
784 2012-11-27 18:22:43 <sipa> also fine
785 2012-11-27 18:22:44 <sipa> also fine
786 2012-11-27 18:22:51 <jgarzik> PB also permits "default = x509"
787 2012-11-27 18:22:56 <jgarzik> if not specified
788 2012-11-27 18:23:01 <gavinandresen> ok. I'll bend to consensus, cert_bytes and cert_type default=x509
789 2012-11-27 18:23:01 <kjj> but then whenever someone wants to use a different type of cert, they have to pick id numbers for whatever extra data that scheme might use, and hope that those choices don't overlap anyone else's ids
790 2012-11-27 18:23:02 <gavinandresen> ok. I'll bend to consensus, cert_bytes and cert_type default=x509
791 2012-11-27 18:23:07 <jgarzik> ACK
792 2012-11-27 18:23:08 <jgarzik> ACK
793 2012-11-27 18:23:42 <sipa> use a string for cert_type
794 2012-11-27 18:24:01 <kjj> and that looks like a general problem with PB
795 2012-11-27 18:24:02 <kjj> and that looks like a general problem with PB
796 2012-11-27 18:24:09 <phantomcircuit> gavinandresen, iirc protobuf you shouldn't need a version field
797 2012-11-27 18:24:10 <phantomcircuit> gavinandresen, iirc protobuf you shouldn't need a version field
798 2012-11-27 18:24:26 <phantomcircuit> which fields are present is all the info you should need
799 2012-11-27 18:24:27 <gavinandresen> string? what? No, we need to form a Working Group and register with IANA! (kidding, string it is)
800 2012-11-27 18:24:33 <phantomcircuit> maybe TD[gone] can correct me here
801 2012-11-27 18:24:34 <jgarzik> hehehe
802 2012-11-27 18:24:34 <phantomcircuit> maybe TD[gone] can correct me here
803 2012-11-27 18:25:01 <sipa> kjj: that's a general problem with any system trying to map a not-known-in-advance list of features into an integer
804 2012-11-27 18:25:15 <jgarzik> phantomcircuit: generally true, but again, as your data structure gets more flexible, the internal structure of the data may change. Something not directly expressed by .proto definition.
805 2012-11-27 18:25:30 <jgarzik> phantomcircuit: c.f. JSON/XML problems just discussed
806 2012-11-27 18:25:40 <kjj> sipa: heh. at risk of resurrecting the JSON vs. PB argument... there is no such problem mapping a string to a string. :)
807 2012-11-27 18:25:41 <kjj> sipa: heh. at risk of resurrecting the JSON vs. PB argument... there is no such problem mapping a string to a string. :)
808 2012-11-27 18:26:09 <jgarzik> phantomcircuit: A business rule might not permit more than 10,000 'repeated' for DoS reasons, like gavinandresen mentioned. But, you might want to increase those limits later.
809 2012-11-27 18:26:13 <sipa> kjj: that's exactly why a propose cert_type being a string
810 2012-11-27 18:26:14 <sipa> kjj: that's exactly why a propose cert_type being a string
811 2012-11-27 18:26:16 <gavinandresen> kjj: there's a whole section of the JOSE/JWS spec on problems mapping strings to strings....
812 2012-11-27 18:26:17 <gavinandresen> kjj: there's a whole section of the JOSE/JWS spec on problems mapping strings to strings....
813 2012-11-27 18:26:28 <jgarzik> phantomcircuit: thus, 'version' is occasionally needed
814 2012-11-27 18:26:29 <jgarzik> phantomcircuit: thus, 'version' is occasionally needed
815 2012-11-27 18:26:59 <jgarzik> kjj: two words: text encoding
816 2012-11-27 18:27:00 <phantomcircuit> actually is there anyway to limit the repeated count with the protobuf libs?
817 2012-11-27 18:27:05 <phantomcircuit> i never noticed one
818 2012-11-27 18:27:11 <jgarzik> phantomcircuit: no (also just discussed)
819 2012-11-27 18:27:17 <phantomcircuit> lol
820 2012-11-27 18:27:20 <gavinandresen> jgarzik: I was researching how to properly do MIME-type versioning this morning, by the way....
821 2012-11-27 18:27:21 <gavinandresen> jgarzik: I was researching how to properly do MIME-type versioning this morning, by the way....
822 2012-11-27 18:27:21 <kjj> sipa: not just for certs. if I want to add custom fields to my invoices, I can't just start using "kjj_myfield1", I have to pick an integer, and woe to anyone with a client that reads that number as something else
823 2012-11-27 18:27:21 <phantomcircuit> im a bit tired :|
824 2012-11-27 18:27:38 <jgarzik> phantomcircuit: ancient XDR permits this of course.
825 2012-11-27 18:27:46 <jgarzik> ACTION wrote an NFSv4 server, and deals with the XDR compiler
826 2012-11-27 18:27:47 <jgarzik> ACTION wrote an NFSv4 server, and deals with the XDR compiler
827 2012-11-27 18:28:17 <phantomcircuit> jgarzik, heh of course it can
828 2012-11-27 18:28:18 <gavinandresen> jgarzik: not yet clear to me if we need version numbers both in the protobuf format AND the request/response or just in the request/response
829 2012-11-27 18:28:20 <jgarzik> XDR has a very C-like definition language, where you may specify a variable-length array, varlen array w/ limit, varlen array w/ specific size
830 2012-11-27 18:28:44 <jgarzik> gavinandresen: oh good point
831 2012-11-27 18:29:15 <jgarzik> gavinandresen: it might indeed be applicable to that lower, packetizing layer that PB requires
832 2012-11-27 18:34:36 <gavinandresen> After reading http://www.informit.com/articles/article.aspx?p=1566460 I think I'm against a version field in the structures-- I think knowing what version you're expecting BEFORE you start parsing is the right way to go, and I can't think of situations where an application can't know that.
833 2012-11-27 18:34:37 <gavinandresen> After reading http://www.informit.com/articles/article.aspx?p=1566460 I think I'm against a version field in the structures-- I think knowing what version you're expecting BEFORE you start parsing is the right way to go, and I can't think of situations where an application can't know that.
834 2012-11-27 18:38:44 <jgarzik> gavinandresen: +1
835 2012-11-27 18:39:09 <jgarzik> PB apps _must_ know it, because they must packetize PB (store a message length, and possibly a message checksum)
836 2012-11-27 18:39:34 <jgarzik> PB does not define the transport format, just the message itself.
837 2012-11-27 18:41:15 <kjj> wget cracks me up sometimes. it can ignore whole directories, but to ignore a file, you have to download it, and then delete it
838 2012-11-27 18:44:58 <kjj> anyhow, I have to run to a board meeting. I'll make this one last pitch to avoid PB, and then I won't bring it up again. because of the field IDs, PB requires a central definition. The community can't experiment with different things and then come to their own consensus
839 2012-11-27 18:46:10 <kjj> (unless someone wants to start BANA, but that is just a different central repository of IDs)
840 2012-11-27 18:46:55 <kjj> actually, shit, at the rate we are using version bytes for keys, we are going to need BANA in a few years anyway
841 2012-11-27 18:46:56 <kjj> actually, shit, at the rate we are using version bytes for keys, we are going to need BANA in a few years anyway
842 2012-11-27 18:50:34 <jgarzik> kjj: disagree
843 2012-11-27 18:50:35 <jgarzik> kjj: disagree
844 2012-11-27 18:50:42 <jgarzik> kjj: That is what extensions are for: https://developers.google.com/protocol-buffers/docs/reference/python-generated#extension
845 2012-11-27 18:50:43 <jgarzik> kjj: That is what extensions are for: https://developers.google.com/protocol-buffers/docs/reference/python-generated#extension
846 2012-11-27 18:51:38 <jgarzik> Disagree with the "community can't experiment" bit, I mean. Obviously, it does require some method of centralized definition -- and that's a good thing, IMO.
847 2012-11-27 18:52:25 <gavinandresen> yeah, if it didn't require a centralized definition I would have just written the code and submitted a patch request.
848 2012-11-27 19:25:57 <jgarzik> picocoin, libccoin announcement: https://bitcointalk.org/index.php?topic=128055.0
849 2012-11-27 19:29:03 <gmaxwell> jgarzik: wrt picocoin security, ??? you'd be an ideal usecase for seccomp 2.0 security. (the BPF filtering of syscalls stuff)
850 2012-11-27 19:29:33 <jgarzik> gmaxwell: yep
851 2012-11-27 19:30:40 <jgarzik> gmaxwell: Trying to make the network client as dumb and secure as possible. Sort of an internal bitcoin server, to which the wallet process connects. Going to use bloom filters to keep keys out of the network process. chroot (or selinux if present), ...
852 2012-11-27 19:30:41 <jgarzik> gmaxwell: Trying to make the network client as dumb and secure as possible. Sort of an internal bitcoin server, to which the wallet process connects. Going to use bloom filters to keep keys out of the network process. chroot (or selinux if present), ...
853 2012-11-27 19:33:11 <gmaxwell> yea, the non-wallet side of this should actually have no access to anything at all except the sockets and the block(headers) database and however you're doing IPC to the wallet. Ideally an attacker with full control of that process shouldn't be more powerful than someone who controls a server for a thinclient.
854 2012-11-27 19:33:12 <gmaxwell> yea, the non-wallet side of this should actually have no access to anything at all except the sockets and the block(headers) database and however you're doing IPC to the wallet. Ideally an attacker with full control of that process shouldn't be more powerful than someone who controls a server for a thinclient.
855 2012-11-27 19:42:01 <kjj> jgarzik: extensions don't solve the problem. the initial definition has to define a range, and people have to make sure they don't overlap in that range (which is admittedly large, larger than tcp/udp port numbers)
856 2012-11-27 19:42:48 <sipa> if there's really a potential for extensions, add some repeated key/value string pair to the protobuf
857 2012-11-27 19:42:49 <sipa> if there's really a potential for extensions, add some repeated key/value string pair to the protobuf
858 2012-11-27 19:43:35 <kjj> that's a hack though, a way to pretend to have a proper namespace, when you really don't
859 2012-11-27 19:43:36 <kjj> that's a hack though, a way to pretend to have a proper namespace, when you really don't
860 2012-11-27 19:44:18 <sipa> but the point is that you need some common structure for an "invoice" to be meaningful; PB allows you to define that structure and make it trivial to parse that part
861 2012-11-27 19:44:19 <sipa> but the point is that you need some common structure for an "invoice" to be meaningful; PB allows you to define that structure and make it trivial to parse that part
862 2012-11-27 19:45:09 <sipa> and yes, JSON in a way also allows that, if you require some structure, and use some JSON parsing library
863 2012-11-27 19:45:10 <sipa> and yes, JSON in a way also allows that, if you require some structure, and use some JSON parsing library
864 2012-11-27 19:45:17 <kjj> and difficult to safely extend, while ASCII string identifiers give all the same benefits, and is easy to extend, at the cost of a few extra bytes