1 2013-05-31 00:00:11 <phantomcircuit> sipa, yeah double check was fixed by adding an instance value fValid
2 2013-05-31 00:00:28 <sipa> NielDLR: http://bitcoin.stackexchange.com/a/9152/208
3 2013-05-31 00:00:30 <kadoban> NielDLR: because the transaction is not in your wallet would be my guess. you can make it work if you set txindex=1 and reindex once
4 2013-05-31 00:00:48 <sipa> kadoban: wallet or not is irrelevant; getrawtransaction is a blockchain RPC call
5 2013-05-31 00:01:17 <NielDLR> Yes, that's how I understand it as well. It's not a wallet lookup. I'll check out the bitcoin stackexchange link quickly.
6 2013-05-31 00:02:37 <kadoban> sipa: oh, huh. it's only unspent txes then? sorry, didn't know that
7 2013-05-31 00:02:38 <NielDLR> Ah I see: In that transaction both outputs have been spent. So I'll have set txindex=1. Lemme me see if I can do this quickly.
8 2013-05-31 00:02:54 <sipa> not quickly, no
9 2013-05-31 00:02:59 <sipa> you'll need to rebuild the index
10 2013-05-31 00:03:57 <NielDLR> Aah. I see.
11 2013-05-31 00:04:23 <NielDLR> Thanks for the help sipa.
12 2013-05-31 00:06:16 <phantomcircuit> sipa, GetTransaction with fAllowSlow=True will search CCoinsViewCache::GetCoins
13 2013-05-31 00:06:22 <phantomcircuit> im nto sure exactly what that does though
14 2013-05-31 00:07:41 <sipa> search the UTXO set
15 2013-05-31 00:07:50 <sipa> so it knows the block height in which the transaction was created
16 2013-05-31 00:07:58 <sipa> so it can read that whole block from disk
17 2013-05-31 00:08:07 <sipa> and then search it for the transaction you wanted
18 2013-05-31 00:08:33 <sipa> that's why getrawtransaction works for unspent transactions, even without txindex
19 2013-05-31 00:09:11 <phantomcircuit> sipa, mkes sense
20 2013-05-31 00:52:47 <phantomcircuit> sipa, 2013-05-31 02:09:48 LoadWallet 146 200075
21 2013-05-31 00:52:50 <phantomcircuit> that's HEAD
22 2013-05-31 00:53:39 <phantomcircuit> sipa, is wkey for watch keys?
23 2013-05-31 00:53:51 <phantomcircuit> nah nvm it's loading a priv key
24 2013-05-31 00:53:54 <sipa> 03:42:13 < sipa> i don't know of any bitcoin version that actually wrote wkey's
25 2013-05-31 00:54:09 <sipa> they're effective unused afaik
26 2013-05-31 01:07:14 <phantomcircuit> 1.4 ms per key
27 2013-05-31 01:07:29 <phantomcircuit> <1 usec for a "pool" entry
28 2013-05-31 01:07:39 <phantomcircuit> so clearly it isn't bdb's fault
29 2013-05-31 01:09:04 <phantomcircuit> if (key.GetPubKey() != vchPubKey) ~= 0.5 ms
30 2013-05-31 01:18:16 <phantomcircuit> sipa, lol the problem here is so obvious
31 2013-05-31 01:19:02 <phantomcircuit> CKey holds a serialized bignum vch, everytime anything is done with the key a new CECKey object is created/initialized/SetSecretBytes is called
32 2013-05-31 01:19:23 <phantomcircuit> which calls BN_bin2bn and then EC_KEY_regenerate_key
33 2013-05-31 01:19:36 <phantomcircuit> this is hilariously amazingly inefficient
34 2013-05-31 01:23:54 <MC1984_> what is the main benefit of bitcoind running through the last 288 blocks on startup
35 2013-05-31 01:24:06 <MC1984_> because right now ive got it disabled and its much faster
36 2013-05-31 01:24:17 <phantomcircuit> MC1984_, sanity checking mostly
37 2013-05-31 01:24:18 <MC1984_> scurity thing or integrity?
38 2013-05-31 01:24:35 <MC1984_> ok
39 2013-05-31 01:24:54 <MC1984_> !blocks
40 2013-05-31 01:25:28 <MC1984_> !BLOCKS
41 2013-05-31 01:26:07 <MC1984_> gribble what are you doing bots dont sleep
42 2013-05-31 01:26:10 <MC1984_> gribble pls
43 2013-05-31 01:26:34 <owowo> ;;seen gribble
44 2013-05-31 01:26:36 <altgribble> gribble was last seen in #bitcoin-dev 1 week, 0 days, 3 hours, 19 minutes, and 4 seconds ago: <gribble> Time since last block: 4 minutes and 57 seconds
45 2013-05-31 01:26:46 <MC1984_> wtf
46 2013-05-31 01:27:15 <phantomcircuit> sipa, seriously i dont think there is anything with with bdb here this is 100% a bitcoind cock up
47 2013-05-31 01:27:22 <phantomcircuit> the entire key.cpp file needs to be fixed
48 2013-05-31 01:27:24 <phantomcircuit> it's a mess
49 2013-05-31 01:27:45 <sipa> phantomcircuit: lol, you know i just rewrote it? :p
50 2013-05-31 01:27:54 <phantomcircuit> really?
51 2013-05-31 01:27:56 <MC1984_> where the hell is gribble
52 2013-05-31 01:27:59 <phantomcircuit> it's still wrong!
53 2013-05-31 01:28:14 <sipa> phantomcircuit: loading is sort of an exception that it does multiple operations on the same key consecutively
54 2013-05-31 01:28:40 <sipa> phantomcircuit: a CKey::Load(CPubKey, CPrivKey) that does all checks once is probably the solution
55 2013-05-31 01:29:11 <phantomcircuit> sipa, im guessing that of the ~1.5 ms it takes to load the key the CECKey stuff is probably 500 usec
56 2013-05-31 01:29:15 <phantomcircuit> if not more
57 2013-05-31 01:29:28 <sipa> i'm guessing it's 1.499
58 2013-05-31 01:29:39 <phantomcircuit> one of the things that's happening is the GetPubKey call ends up creating another CECKey
59 2013-05-31 01:29:56 <phantomcircuit> sipa, actually
60 2013-05-31 01:29:57 <phantomcircuit> <phantomcircuit> if (key.GetPubKey() != vchPubKey) ~= 0.5 ms
61 2013-05-31 01:30:15 <sipa> yes, we're doing the EC multiplication multiple times
62 2013-05-31 01:30:40 <sipa> i suppose
63 2013-05-31 01:33:57 <sipa> phantomcircuit: also, feel free to try the version using libsecp256k1 :p
64 2013-05-31 01:34:16 <sipa> https://github.com/sipa/bitcoin/commits/secp256k1
65 2013-05-31 01:34:28 <phantomcircuit> sipa, from my counting there's copy constructor, Set, SetPrivKey, MakeNeyKey which all set vch independently
66 2013-05-31 01:34:41 <phantomcircuit> :/
67 2013-05-31 01:34:53 <sipa> is that unexpected?
68 2013-05-31 01:35:05 <phantomcircuit> lol i guess it shouldn't be
69 2013-05-31 01:35:18 <sipa> no, do you think that's bad design?
70 2013-05-31 01:35:40 <phantomcircuit> yeah since it makes implementing RIAA hilariously hard
71 2013-05-31 01:35:59 <sipa> RAII, you mean :D
72 2013-05-31 01:36:02 <phantomcircuit> er
73 2013-05-31 01:36:04 <phantomcircuit> yeah that
74 2013-05-31 01:36:08 <sipa> ACTION will NOT implement RIAA
75 2013-05-31 01:36:24 <phantomcircuit> to make sure an EC_KEY instance is initialized when the priv key is set i have to make sure i get each place where the privkey can be changed
76 2013-05-31 01:36:25 <sipa> not sure why; there's just one destructor
77 2013-05-31 01:36:45 <sipa> ?
78 2013-05-31 01:37:45 <sipa> do you mean add a CECKey to CKey?
79 2013-05-31 01:37:47 <phantomcircuit> sipa, instead of using CECKey in each of the methods i'd rather allocate a single CECKey
80 2013-05-31 01:37:51 <sipa> please don't
81 2013-05-31 01:37:52 <phantomcircuit> CKey
82 2013-05-31 01:38:12 <phantomcircuit> sipa, bugs bugs everywhere should i try it?
83 2013-05-31 01:38:37 <sipa> the reason for this is exactly because almost everywhere you just do a single crypto operation every time
84 2013-05-31 01:38:54 <sipa> so everywhere you had CKey key; key.SetSecret(secret). key.DoSomething; ...
85 2013-05-31 01:39:12 <sipa> which was ugly, and hard to change the underlying operations
86 2013-05-31 01:39:32 <phantomcircuit> ah so CKey just having the bare minimum information means it doesn't have to do that
87 2013-05-31 01:39:33 <phantomcircuit> ok
88 2013-05-31 01:39:45 <sipa> the solution is just have a CKey::Load(CPrivKey, CPubkey), which does full checking of both
89 2013-05-31 01:39:49 <sipa> and check that they match
90 2013-05-31 01:39:55 <sipa> using a single CECKey
91 2013-05-31 01:40:05 <sipa> and call that from LoadWallet
92 2013-05-31 01:45:59 <MC1984_> do miners decide a block template once and hash on it for ten minutes, or does the template change as new tx come in
93 2013-05-31 01:46:19 <sipa> whenever they ask for a template, they get a current/new one
94 2013-05-31 01:46:31 <sipa> how often they ask for a template is another thing
95 2013-05-31 01:46:59 <MC1984_> oh
96 2013-05-31 01:47:06 <MC1984_> whats the default?
97 2013-05-31 01:47:24 <MC1984_> oh there is no default
98 2013-05-31 01:47:34 <sipa> depends on the software
99 2013-05-31 01:52:02 <phantomcircuit> sipa, appears to shave 0.5ms off the 1.5ms
100 2013-05-31 01:52:42 <phantomcircuit> 14385983 vs 9598030
101 2013-05-31 01:54:19 <sipa> can i see the code?
102 2013-05-31 01:54:42 <sipa> 1ms is till ridiculous
103 2013-05-31 01:54:45 <sipa> *still
104 2013-05-31 01:55:04 <phantomcircuit> sipa, that includes if (!pwallet->LoadKey(key, vchPubKey))
105 2013-05-31 01:55:15 <phantomcircuit> which i would hope is the rest of it
106 2013-05-31 01:55:26 <sipa> yes
107 2013-05-31 01:55:44 <sipa> but even if you get it to a single EC operation of 0.5ms or so, 0.5ms is perhaps mor ethan you can tolerate
108 2013-05-31 01:55:53 <sipa> a -nocheckwallet option is maybe useful :p
109 2013-05-31 01:56:10 <phantomcircuit> sipa, http://pastebin.com/raw.php?i=54x97yk1
110 2013-05-31 01:56:59 <phantomcircuit> walletdb.cpp is full of other nonsense but it's just replacing the SetPrivKey and GetPubKey lines
111 2013-05-31 01:57:50 <phantomcircuit> http://pastebin.com/raw.php?i=7YxsRCgm
112 2013-05-31 01:58:07 <sipa> i wonder whether GetPubKey recomputes the public key if it's already known to the EC_KEY object
113 2013-05-31 01:58:18 <phantomcircuit> ooh i think it does
114 2013-05-31 01:58:50 <phantomcircuit> hmm maybe not
115 2013-05-31 01:58:55 <sipa> i would hope not
116 2013-05-31 01:58:59 <sipa> what hardware is that?
117 2013-05-31 01:59:02 <sipa> 1ms is really long
118 2013-05-31 01:59:07 <phantomcircuit> i3-2100
119 2013-05-31 01:59:24 <phantomcircuit> entire datadir in a tmpfs
120 2013-05-31 01:59:26 <phantomcircuit> with plenty of ram
121 2013-05-31 01:59:35 <phantomcircuit> and of course this is all the openssl stuff which is 99.9999% undocumented
122 2013-05-31 01:59:48 <sipa> well, feel free to try libsecp256k1 :p
123 2013-05-31 02:00:06 <phantomcircuit> sipa, heh im sure it's faster im just trying to figure out why this is so slow
124 2013-05-31 02:00:08 <sipa> i haven't actually benchmarked its privkey stuff, but it's likely faster than openssl for that too
125 2013-05-31 02:00:11 <phantomcircuit> since it's been bugging me for like
126 2013-05-31 02:00:13 <phantomcircuit> a year
127 2013-05-31 02:00:34 <Krellan> Hi, hope this isn't a dumb question: bitcoin IPv6 - I haven't seen my bitcoin-qt find any IPv6 peers - is IPv6 enabled by default in 0.8.1?
128 2013-05-31 02:00:46 <sipa> Krellan: since 0.6 or so
129 2013-05-31 02:00:56 <phantomcircuit> Krellan, try -connect=momentovps.com
130 2013-05-31 02:00:58 <phantomcircuit> er
131 2013-05-31 02:01:03 <sipa> Krellan: try -onlynet=ipv6 if you want to only make outgoing connections to ipv6 nodes
132 2013-05-31 02:01:05 <phantomcircuit> actually that's not listening on ipv6
133 2013-05-31 02:01:06 <phantomcircuit> lol
134 2013-05-31 02:01:18 <phantomcircuit> i should fix that
135 2013-05-31 02:01:18 <sipa> well, the fastest i've seen OpenSSL do a verification is around 0.5ms
136 2013-05-31 02:01:33 <phantomcircuit> sipa, that is terrible
137 2013-05-31 02:01:54 <sipa> verifications ought to be a bit slower than generations
138 2013-05-31 02:02:03 <phantomcircuit> i was testing out various ecdsa implementations in python for a web authentication service and i had pure python ops that were that fast
139 2013-05-31 02:02:25 <sipa> for an EC multiplication? that i can't believe
140 2013-05-31 02:02:26 <phantomcircuit> wat are they doin
141 2013-05-31 02:03:02 <sipa> well, openssl is already an order of magnitude faster than cryptopp
142 2013-05-31 02:03:03 <phantomcircuit> sipa, ok i was cheating using the shortest curve i could find
143 2013-05-31 02:03:04 <phantomcircuit> lol
144 2013-05-31 02:03:18 <sipa> and it has pretty high-level assembly implementations
145 2013-05-31 02:03:31 <sipa> so for a generic-curve implementation it's very good actually
146 2013-05-31 02:03:38 <sipa> but secp256k1 has several curve-specific tricks
147 2013-05-31 02:05:33 <Diablo-D3> I swear I already saw this irc
148 2013-05-31 02:06:16 <phantomcircuit> sipa, well for now Load is definitely about 30% faster
149 2013-05-31 02:06:41 <phantomcircuit> and basically just makes an error message nobody ever sees slightly more opaque
150 2013-05-31 02:09:07 <phantomcircuit> sipa, also GetPubKey can be reduced to one call to i2o_ECPublicKey
151 2013-05-31 02:09:28 <phantomcircuit> actually it doesn't look like openssl handles a short buffer
152 2013-05-31 02:09:43 <Krellan> Thanks for IPv6 advice - got it to work - now it makes IPv6 connections
153 2013-05-31 02:10:09 <Krellan> played around with other command-line options, nice to know "-gen" still works - 4.1 Mhash/sec lol
154 2013-05-31 02:10:49 <phantomcircuit> wait maybe they do
155 2013-05-31 02:12:24 <phantomcircuit> no they dont since they cant know how long the pointer is
156 2013-05-31 02:14:12 <phantomcircuit> sipa, the actual conversation EC_POINT_point2oct ends up happening 3 times, once for bitcoind to figure out the size of the buffer, once for openssl to figure it out again, and finally to actually do it
157 2013-05-31 02:14:20 <phantomcircuit> :/
158 2013-05-31 02:19:04 <phantomcircuit> 9589847
159 2013-05-31 02:19:05 <phantomcircuit> meh
160 2013-05-31 02:19:49 <sipa> thankfully just serializing isn't very expensive
161 2013-05-31 02:20:03 <sipa> and i want to get rid of openssl...
162 2013-05-31 02:20:18 <phantomcircuit> sipa, check pm
163 2013-05-31 02:27:18 <phantomcircuit> hmm i was wrong
164 2013-05-31 02:28:00 <phantomcircuit> pwallet->LoadKey is taking ~60 usec per key, Load is taking ~900 usec per key
165 2013-05-31 02:31:50 <sipa> do you know how much of that time is i2o, loadprivkey, ... ?
166 2013-05-31 02:32:23 <sipa> if you run bitcoind -reindex -benchmark -par=1 -dbcache=1000 -nocheckpoints
167 2013-05-31 02:32:36 <sipa> you can see how long signature validations tKe
168 2013-05-31 02:32:47 <sipa> take
169 2013-05-31 02:33:04 <sipa> i wonder how that compares to that loading
170 2013-05-31 02:33:21 <sipa> if loading is not faster than verifying, there is something wrong
171 2013-05-31 02:34:17 <phantomcircuit> SetPrivKey 885.5229
172 2013-05-31 02:34:31 <sipa> fuuu
173 2013-05-31 02:34:51 <phantomcircuit> that's with some adhauc gettimeofday stuff but i've found gettimeofday takes all of about 10 ns
174 2013-05-31 02:34:55 <phantomcircuit> so it should be pretty accurate
175 2013-05-31 02:35:19 <Luke-Jr> it's nowhere near accurate on Windows, FWIW
176 2013-05-31 02:35:24 <Aurigae> hi guys, blockchain.info is a heavy security htread to unexperienced users, one can log IPs and eploit easily
177 2013-05-31 02:35:27 <phantomcircuit> ;)
178 2013-05-31 02:35:27 <phantomcircuit> Luke-Jr, of course not
179 2013-05-31 02:35:40 <sipa> Aurigae: what you gonna do about it?
180 2013-05-31 02:35:59 <Aurigae> no idea, i just figured
181 2013-05-31 02:36:12 <phantomcircuit> Aurigae, are you the guy who opened the issue on github?
182 2013-05-31 02:36:35 <Luke-Jr> doesn't sound like it
183 2013-05-31 02:36:37 <Aurigae> no
184 2013-05-31 02:36:44 <phantomcircuit> nvm then
185 2013-05-31 02:37:36 <phantomcircuit> sipa, ima try again with EC_KEY_check_key disabled
186 2013-05-31 02:37:59 <phantomcircuit> also printf in key.cpp appears to actually be printf instead of OutputDebug whatever
187 2013-05-31 02:38:07 <phantomcircuit> lol
188 2013-05-31 02:38:14 <phantomcircuit> 36.1861 usec per key
189 2013-05-31 02:38:15 <sipa> orly?
190 2013-05-31 02:38:40 <phantomcircuit> so the issue here is entirely EC_KEY_check_key
191 2013-05-31 02:38:49 <sipa> wait
192 2013-05-31 02:40:44 <phantomcircuit> sipa, lol yeah it is printf
193 2013-05-31 02:40:54 <phantomcircuit> ReadKeyValue SetPrivKey 10000 8693402
194 2013-05-31 02:40:58 <phantomcircuit> 2013-05-31 04:38:53 ReadKeyValue SetPrivKey 1000 875286
195 2013-05-31 02:40:58 <phantomcircuit> #include "util.h"
196 2013-05-31 02:45:16 <sipa> phantomcircuit: i think part of the explanation is that openssl does key checking in an inefficient way
197 2013-05-31 02:45:20 <phantomcircuit> sipa, does d2i_ECPrivateKey calculate the public key also?
198 2013-05-31 02:45:32 <sipa> unsure
199 2013-05-31 02:45:42 <sipa> does it?
200 2013-05-31 02:45:44 <phantomcircuit> one way to check
201 2013-05-31 02:45:52 <phantomcircuit> lets find out
202 2013-05-31 02:45:53 <sipa> use the source luke
203 2013-05-31 02:46:23 <phantomcircuit> lol d2i_ECPrivateKey calls d2i_EC_PRIVATEKEY
204 2013-05-31 02:46:26 <phantomcircuit> WHY?
205 2013-05-31 02:46:47 <sipa> but if you disable checkkey it's only 36us anymore?
206 2013-05-31 02:46:59 <sipa> i don't think that includes an ec mult anymore
207 2013-05-31 02:47:02 <phantomcircuit> right
208 2013-05-31 02:47:27 <sipa> now
209 2013-05-31 02:47:44 <sipa> tje sec standard gives an algorithm for verifying ppublic keys
210 2013-05-31 02:48:09 <sipa> which has one step that isn't technically required for curves with cofactor=1
211 2013-05-31 02:48:15 <phantomcircuit> d2i_EC_PRIVATEKEY is one of these magic asn1 functions in openssl that isn't defined anywhere
212 2013-05-31 02:48:17 <sipa> which is the case for secp256jq
213 2013-05-31 02:48:25 <sipa> secp256k1
214 2013-05-31 02:48:26 <phantomcircuit> it's magicked into existence by the preprocessor somewhere
215 2013-05-31 02:48:37 <sipa> but openssl implements it anyway, always
216 2013-05-31 02:48:48 <sipa> with a comment "required by the standard"
217 2013-05-31 02:49:46 <phantomcircuit> it vaguely looks like the public key can be included in the DER encoded value and if it is then it will be set
218 2013-05-31 02:49:50 <phantomcircuit> but if it isn't it wont
219 2013-05-31 02:50:00 <phantomcircuit> and afaikt bitcoind only has the private key
220 2013-05-31 02:50:08 <phantomcircuit> and the public key is set separately
221 2013-05-31 02:50:15 <sipa> nope, it has both
222 2013-05-31 02:50:32 <sipa> getprivkey also serializes the pubkey
223 2013-05-31 02:50:36 <sipa> by default
224 2013-05-31 02:50:41 <sipa> and we don't disable it
225 2013-05-31 02:50:59 <phantomcircuit> then i *think* it should be reading the publickey from that and it wont have to calculate the public key separately
226 2013-05-31 02:51:05 <sipa> ok
227 2013-05-31 02:51:19 <phantomcircuit> but again the function that actually does the decoding is some magic preprocessor hacks
228 2013-05-31 02:51:30 <phantomcircuit> d2i_EC_PRIVATEKEY doesn't exist but works
229 2013-05-31 02:51:32 <sipa> i know
230 2013-05-31 02:51:40 <sipa> but it just deserialozes
231 2013-05-31 02:52:45 <phantomcircuit> sipa, so what verification can to done to check that the private key matches the public key without taking 800 usec
232 2013-05-31 02:54:56 <sipa> a single ec multiplication with the generator, and a few bounds checks should be enough
233 2013-05-31 02:56:53 <phantomcircuit> hmm
234 2013-05-31 02:57:11 <phantomcircuit> sipa, well ima open an issue and just say something like "EC_KEY_check_key" is really really slow
235 2013-05-31 02:57:26 <phantomcircuit> cause i definitely shouldn't be trying to implement that type of check
236 2013-05-31 02:57:46 <sipa> i'll have a lool at ec key check key
237 2013-05-31 02:57:51 <sipa> look!
238 2013-05-31 02:58:00 <phantomcircuit> sipa, same thing
239 2013-05-31 02:58:01 <phantomcircuit> :)
240 2013-05-31 02:58:37 <phantomcircuit> something tells me copy/paste the openssl function into key.cpp will not work
241 2013-05-31 02:58:39 <phantomcircuit> heh
242 2013-05-31 02:59:27 <sipa> well what you could do is replace it with getsecret + setsecret
243 2013-05-31 02:59:32 <sipa> and then compare pubkeys
244 2013-05-31 03:00:48 <phantomcircuit> hmm
245 2013-05-31 03:01:16 <phantomcircuit> and actually the vchPubKey == GetPubKey check should already mean the key was at least valid when generated
246 2013-05-31 03:01:27 <phantomcircuit> so as long as nobody is adding invalid keys to their wallet
247 2013-05-31 03:01:29 <phantomcircuit> it's fine
248 2013-05-31 03:05:49 <phantomcircuit> sipa, that is soooo much better
249 2013-05-31 03:06:37 <phantomcircuit> 40x speed up :/
250 2013-05-31 03:07:29 <sipa> well that doesn't verify that pubkeys and privkeys match
251 2013-05-31 03:07:43 <CodeShark> what does verification have to do with the wallet?
252 2013-05-31 03:07:55 <CodeShark> and why do we need performance improvements for the wallet?
253 2013-05-31 03:08:07 <sipa> because phantomcircuit has a wallet with 100k keys
254 2013-05-31 03:08:20 <gmaxwell> So does midnightmagic.
255 2013-05-31 03:08:21 <CodeShark> oh, lol - the solution is to scrap that wallet and design one that scales :p
256 2013-05-31 03:08:21 <sipa> so if each takes 1ms to check, it takes 2 minutes
257 2013-05-31 03:08:49 <sipa> no, it has nothing to do with that
258 2013-05-31 03:09:02 <sipa> it's just that we verify that pubkeys and privkeys match as a sanity check
259 2013-05-31 03:09:09 <sipa> which means an EC multiplication for each key
260 2013-05-31 03:09:18 <CodeShark> oh
261 2013-05-31 03:09:34 <CodeShark> perhaps move it into an assert?
262 2013-05-31 03:09:41 <sipa> how will that help?
263 2013-05-31 03:09:48 <phantomcircuit> sipa, it seems like the pubkey is stored both as a separate serialized entry (vchPubKey) and as an entity in the DER for the privkey
264 2013-05-31 03:09:54 <sipa> phantomcircuit: that's correct
265 2013-05-31 03:10:08 <phantomcircuit> so im making the assumption that is vchPubKey matches privkey->publickey
266 2013-05-31 03:10:13 <phantomcircuit> that everything is good
267 2013-05-31 03:10:20 <phantomcircuit> which is of course an assumption
268 2013-05-31 03:10:30 <CodeShark> is this operation really the bottleneck for a 100k + key wallet?
269 2013-05-31 03:10:31 <sipa> CodeShark: we don't build with -DNDEBUG
270 2013-05-31 03:10:39 <sipa> for _loading_ it, sure
271 2013-05-31 03:11:07 <phantomcircuit> ima try this again with a million keys...
272 2013-05-31 03:15:42 <phantomcircuit> sipa, of course openssl provides only incomplete types for everything
273 2013-05-31 03:15:51 <sipa> of course
274 2013-05-31 03:19:21 <sipa> phantomcircuit: in any case, my library does an privkey->pubkey computation in 30us :p
275 2013-05-31 03:19:34 <phantomcircuit> lol yeah that's much better
276 2013-05-31 03:21:27 <phantomcircuit> sipa, i cant even tell if openssl calculates the public key when you dont provide it
277 2013-05-31 03:22:19 <sipa> (on a cpu which may be faster than yours)
278 2013-05-31 03:22:56 <phantomcircuit> sipa, this is 3.1 GHz and is basically under no real other load
279 2013-05-31 03:23:09 <phantomcircuit> since this is 100% single threaded GHZ probably wins the day
280 2013-05-31 03:23:58 <sipa> it's an i5-2520M here at 2.5 GHz, but it probably overclocks in single-thread mode
281 2013-05-31 03:25:09 <phantomcircuit> sipa, i've generally found the mobile cpus to be slower than a desktop even when it's faster on paper
282 2013-05-31 03:25:51 <phantomcircuit> the i3 in this never gets above about 45C
283 2013-05-31 03:26:06 <phantomcircuit> but my laptop with iirc has something equivalent is often at 70C
284 2013-05-31 03:34:12 <phantomcircuit> sipa, derp accessor macros EC_KEY_get0_group
285 2013-05-31 03:34:24 <sipa> ?
286 2013-05-31 03:35:03 <phantomcircuit> there's macros defined to access the incompetely typed EC_KEY struct
287 2013-05-31 03:40:47 <sipa> ok?
288 2013-05-31 03:41:29 <phantomcircuit> sipa, copy/paste EC_KEY_check_key, fix accessors figure out which op is slow
289 2013-05-31 03:41:32 <phantomcircuit> try to eliminate it
290 2013-05-31 03:42:01 <sipa> ah yes, let me have a look
291 2013-05-31 03:44:04 <sipa> phantomcircuit: it's doing two EC_mult
292 2013-05-31 03:44:15 <sipa> only the second is needed if the privkey is present
293 2013-05-31 03:45:18 <sipa> pretty much all of the pubkey checks are unnecessary if you're going to recompute the pubkey anyway from the privkey to see if it matches
294 2013-05-31 03:45:55 <phantomcircuit> sipa, which is what it's doing with the privkey at the bottom right
295 2013-05-31 03:46:01 <sipa> indeed
296 2013-05-31 03:46:21 <sipa> imho all problems are caused by having a single datastructure that holds both privkey and pubkey :p
297 2013-05-31 03:46:39 <sipa> sometimes you only have one, and don't care about the other
298 2013-05-31 03:49:12 <phantomcircuit> sipa, honestly i dont get the math so im not going to screw with it
299 2013-05-31 03:49:15 <phantomcircuit> http://pastebin.com/raw.php?i=3FMSRiJH
300 2013-05-31 03:49:30 <phantomcircuit> but this is a copy of the EC_KEY_check_key function fixing all the incomplete type issues
301 2013-05-31 03:49:36 <phantomcircuit> plz2reassemble
302 2013-05-31 03:49:43 <phantomcircuit> plz2re-arrange
303 2013-05-31 03:58:07 <phantomcircuit> sipa, lol EC_GROUP_get_order -> fails is order is zero and yet there is also a BN_is_zero check
304 2013-05-31 04:01:57 <sipa> ?
305 2013-05-31 04:03:16 <phantomcircuit> sipa, http://pastebin.com/EgsyM2z2
306 2013-05-31 04:03:35 <grau> sipa: Hi. Do you know who administers/decides on bitcoin.org content? I wonder what would be the right place for bits of proof there?
307 2013-05-31 04:03:52 <sipa> grau: github.com/bitcoin/bitcoin.org, send a pull request
308 2013-05-31 04:05:10 <grau> thanks, will do.
309 2013-05-31 04:05:42 <grau> Iwould also like to have an opinion. It is not primarely a wallet, so where should it go
310 2013-05-31 04:06:32 <sipa> i've withdrawn from all website discussion some time ago, as it drove me nuts
311 2013-05-31 04:06:59 <grau> ok, Is there som other forum takes it up?
312 2013-05-31 04:07:22 <sipa> i believe there's been discussion about the website on the foundation forum
313 2013-05-31 04:07:36 <grau> ok, I try there. Thanks a lot
314 2013-05-31 04:07:36 <sipa> but i'm not following those
315 2013-05-31 04:07:59 <wumpus> grau: imo, just add it to the list of clients
316 2013-05-31 04:08:53 <wumpus> oh it doesn't really show a list anymore
317 2013-05-31 04:08:56 <wumpus> nm
318 2013-05-31 04:09:31 <phantomcircuit> sipa, [3536534.798513] bitcoind[7399]: segfault at c ip 00007f5a5cd2d703 sp 00007fff936d2e30 error 4 in libcrypto.so.1.0.0[7f5a5cc8a000+1ad000]
319 2013-05-31 04:09:37 <phantomcircuit> lol well i guess that isn't quite right
320 2013-05-31 04:10:33 <wumpus> it's only aimed at newbies now :p I guess there needs to be a page for developers in which they can find ways to interface to the network such a bitcoind, bitcoinj, and bits of proofs
321 2013-05-31 04:11:26 <sipa> there is some business section on the site
322 2013-05-31 04:11:34 <sipa> so i guess BoP makes sense
323 2013-05-31 04:11:55 <sipa> but i choose not to join discussions about what ought to be on the site or not
324 2013-05-31 04:12:00 <wumpus> this site is confusing... I usually expect the top menu to be above the heirarchy to the bottom menu
325 2013-05-31 04:12:41 <wumpus> I'm not discussing whether it ought to be on the site, just where it should be, given that The Powers That Be accept that it should be on the site :p
326 2013-05-31 04:13:24 <grau> I think I will put it on business
327 2013-05-31 04:13:52 <grau> and developer
328 2013-05-31 04:14:02 <sipa> perhaps discuss with saivann first where he thinks it belongs
329 2013-05-31 04:14:19 <sipa> he's sort of the webmaster, but he usually asks for other's opinion
330 2013-05-31 04:14:28 <wumpus> but neither developer or business has a list of node implementation
331 2013-05-31 04:14:30 <phantomcircuit> sipa, ok so the simplified check cuts the time for EC_KEY_check_key in half
332 2013-05-31 04:14:43 <sipa> yes, expected
333 2013-05-31 04:15:03 <phantomcircuit> that combined with CKey::Load is having a significant impact on load time
334 2013-05-31 04:15:11 <wumpus> so that'd be the first thing to agree on... just adding bits of proof somewhere in the text wil likely be rejected :)
335 2013-05-31 04:15:22 <grau> best create a different page, not choose your wallet but choose your toolkit and put bitcoind, BitcoinJ, BOP on it
336 2013-05-31 04:15:25 <phantomcircuit> looks to be about 2k keys/second
337 2013-05-31 04:15:29 <sipa> that first mult check is actually completely unnecessary
338 2013-05-31 04:15:32 <sipa> for our curve
339 2013-05-31 04:17:07 <phantomcircuit> sipa, and this is really just being used as a very expensive checksum anyways
340 2013-05-31 04:17:19 <sipa> yeah
341 2013-05-31 04:17:38 <phantomcircuit> upto key 400k
342 2013-05-31 04:18:24 <phantomcircuit> man i should have done this before i started messing with postgresql
343 2013-05-31 04:18:30 <phantomcircuit> fixing this was a lot easier than that was
344 2013-05-31 04:19:26 <phantomcircuit> 600k
345 2013-05-31 04:24:38 <wumpus> grau: agreed
346 2013-05-31 04:24:40 <phantomcircuit> and loaded
347 2013-05-31 04:24:48 <phantomcircuit> sipa, loaded and apparently actually working
348 2013-05-31 04:25:09 <phantomcircuit> i do not want to see what happens when this receives a transaction though
349 2013-05-31 04:25:38 <sipa> phantomcircuit: try importing a correct horse staple battery :)
350 2013-05-31 04:25:44 <sipa> and rescan
351 2013-05-31 04:25:47 <sipa> if you want some pain
352 2013-05-31 04:26:19 <wumpus> grau: though mind that the page currently has a completely flat heirarchy (all links to leaf pages are always shown).. adding a new page may be easier said than done, as you'd need to find space to cram another link :)
353 2013-05-31 04:26:35 <Luke-Jr> I thought it was a correct horse battery staple?
354 2013-05-31 04:27:04 <phantomcircuit> Luke-Jr, it is
355 2013-05-31 04:27:41 <wumpus> coming next to a "secure" online wallet close to you, correct horse battery staple permutations
356 2013-05-31 04:27:51 <phantomcircuit> sipa, lol oh god
357 2013-05-31 04:27:54 <saivann_>
358 2013-05-31 04:27:54 <saivann_> <li><a href="/en/how-it-works">What is Bitcoin</a></li><li><a href="/en/how-it-works">How it works</a></li>
359 2013-05-31 04:27:54 <saivann_> <ul id=???"menu">???<ul id="menu">
360 2013-05-31 04:27:56 <saivann_>
361 2013-05-31 04:27:58 <saivann_> <li><a href="/en/resources">Resources</a></li>
362 2013-05-31 04:28:00 <saivann_>
363 2013-05-31 04:28:01 <phantomcircuit> lol
364 2013-05-31 04:28:02 <saivann_>
365 2013-05-31 04:28:03 <Luke-Jr> /kick saivann_
366 2013-05-31 04:28:04 <saivann_>
367 2013-05-31 04:28:06 <saivann_> <li><a href="/en/development">Development</a></li>
368 2013-05-31 04:28:08 <saivann_>
369 2013-05-31 04:28:10 <saivann_> <li><a href="/en/foundation">Foundation</a></li>
370 2013-05-31 04:28:12 <saivann_>
371 2013-05-31 04:28:16 <saivann_> <li><a href="/en/about">FAQ</a></li>
372 2013-05-31 04:28:18 <saivann_>
373 2013-05-31 04:28:20 <saivann_> </ul>
374 2013-05-31 04:28:22 <saivann_> Oh damn, sorry :P
375 2013-05-31 04:28:24 <saivann_> The joy of drag and drop
376 2013-05-31 04:28:27 <wumpus> hehe
377 2013-05-31 04:28:53 <sipa> wumpus: determined by a fair dice roll, guaranteed to be random
378 2013-05-31 04:29:46 <phantomcircuit> sipa, lol i forgot importprivkey defaults to doing a rescan
379 2013-05-31 04:30:14 <fimp> Bitcoin-QT quit just after I entered my password for doing a transaction. After restart, the tx shows in the tx list, but it doesn't appear on blockchain.info nor blockexplorer. This is more than 30 minutes ago, so it should have been rebroadcasted. I'm up to date with 238815 blocks. What to do?
380 2013-05-31 04:30:49 <wumpus> sipa: now using the calendar date and current mouse position for extra randomness!
381 2013-05-31 04:31:49 <phantomcircuit> sipa, now i just need to cache whether a CKey has run the EC_KEY_check_key stuff for the specific private key
382 2013-05-31 04:32:05 <phantomcircuit> again it would be useful if there was only one way to change the priv key :)
383 2013-05-31 04:32:23 <sipa> phantomcircuit: checking a private key is trivial
384 2013-05-31 04:32:47 <phantomcircuit> not when you have a million of them and trigger a rescan on accident
385 2013-05-31 04:33:13 <sipa> when does check_key every happen except for loading?
386 2013-05-31 04:33:23 <sipa> *ever
387 2013-05-31 04:33:57 <sipa> because CKey::Check is far faster and just as good, if you only need to check a private key
388 2013-05-31 04:34:01 <wumpus> oh gawd, goldbit89 is spamming his issue into all other issues that show even an extremely vague resemblance to his problem
389 2013-05-31 04:34:35 <phantomcircuit> sipa, EC_KEY_check_key is being called for every single sign/verify op
390 2013-05-31 04:34:48 <sipa> really?
391 2013-05-31 04:34:53 <phantomcircuit> lol yes
392 2013-05-31 04:35:44 <phantomcircuit> wait
393 2013-05-31 04:35:46 <phantomcircuit> maybe not
394 2013-05-31 04:35:48 <sipa> it's only in SetPrivKey
395 2013-05-31 04:35:49 <phantomcircuit> SetSecretBytes
396 2013-05-31 04:35:58 <sipa> and CPrivKey is only used when loading
397 2013-05-31 04:36:04 <phantomcircuit> oh it's just EC_KEY_regenerate_key
398 2013-05-31 04:36:06 <sipa> internally, CKey is always used
399 2013-05-31 04:36:53 <sipa> hmm that probably means 1 superfluous EC multiplication per signing
400 2013-05-31 04:37:05 <sipa> but for not for verifying
401 2013-05-31 04:37:10 <sipa> which is what is most important
402 2013-05-31 04:37:24 <phantomcircuit> brain is too tired
403 2013-05-31 04:37:36 <Scrat> hey sipa, is your verification code going to be in 0.9?
404 2013-05-31 04:38:04 <sipa> Scrat: unlikely
405 2013-05-31 04:39:20 <phantomcircuit> sipa, well either way this is really a huge massive improvement
406 2013-05-31 04:39:31 <phantomcircuit> at least for people with absurdly large wallets
407 2013-05-31 04:39:48 <Scrat> phantomcircuit: define large in MB
408 2013-05-31 04:40:00 <sipa> Scrat: >10k keys
409 2013-05-31 04:40:08 <phantomcircuit> -rw------- 1 patrick patrick 554M May 30 23:38 wallet.dat
410 2013-05-31 04:40:13 <phantomcircuit> thats 1 million keys
411 2013-05-31 04:40:27 <sipa> phantomcircuit: yes, i wasn't aware we were doing so many redundant operations
412 2013-05-31 04:40:32 <phantomcircuit> plus that silly xkcd pass phrase as a brain wallet
413 2013-05-31 04:41:04 <phantomcircuit> i'd like to import the private key for something like an sd address to see what happens
414 2013-05-31 04:41:09 <phantomcircuit> but something tells me they wont be sharing
415 2013-05-31 04:44:45 <Scrat> change addresses do contribute to wallet bloat. imagine if you could set one change address for the entire wallet. it would still have to track all the outputs so I don't know how much it would help
416 2013-05-31 04:45:33 <sipa> well if we want wallets to scale they'll likely need to use local bloom filters when scanning blocks for example
417 2013-05-31 04:45:55 <TheUni> sipa: ping
418 2013-05-31 04:46:18 <sipa> TheUni: yes?
419 2013-05-31 04:46:27 <TheUni> sipa: i'm a bit confused...
420 2013-05-31 04:46:36 <TheUni> is wine involved in the mingw win32 build?
421 2013-05-31 04:46:43 <sipa> not afaik
422 2013-05-31 04:47:22 <TheUni> ok
423 2013-05-31 04:47:48 <sipa> deps-win32 does refer to wine
424 2013-05-31 04:47:51 <sipa> hmmm
425 2013-05-31 04:47:57 <TheUni> killall wineserver services.exe explorer.exe winedevice.exe
426 2013-05-31 04:47:57 <TheUni> # Kill wine processes as gitian won't figure out we are done otherwise
427 2013-05-31 04:48:03 <TheUni> yea, that's the bit i stumbled upon
428 2013-05-31 04:48:31 <sipa> talk to BlueMatt
429 2013-05-31 04:48:43 <TheUni> ok
430 2013-05-31 04:48:46 <TheUni> BlueMatt: ping
431 2013-05-31 04:49:07 <TheUni> i finally got win32 binaries cranked out, then that got me confused. i'm hoping they're used to run the tests
432 2013-05-31 04:49:17 <TheUni> *it's
433 2013-05-31 04:49:28 <sipa> will gitian doesn't do tests afaik
434 2013-05-31 04:49:37 <phantomcircuit> sipa, wallets can scale just fine *if* every change to the blockchain is run through the wallet
435 2013-05-31 04:49:51 <phantomcircuit> actually guaranteeing that happens though is uh
436 2013-05-31 04:49:52 <phantomcircuit> lol right
437 2013-05-31 04:50:00 <sipa> TheUni: pulltester does, and i assume it does so using wine
438 2013-05-31 04:50:01 <Scrat> sipa: something weird though: does it fsync/lock in steps when sending? with a 10 MB wallet on a 5400 rpm drive I've seen up to ~5 sec delays while on an SSD it's always <0.1s
439 2013-05-31 04:50:19 <sipa> when _sending_ ?
440 2013-05-31 04:50:22 <TheUni> sipa: ok, thanks
441 2013-05-31 04:50:29 <Scrat> sendtoaddress
442 2013-05-31 04:51:14 <sipa> TheUni: i suppose that for deps-win32 one of the packages builds an intermediate binary to produce extra sources
443 2013-05-31 04:51:26 <sipa> TheUni: which is built for the target instead of the host arch
444 2013-05-31 04:51:28 <phantomcircuit> Scrat, iirc it's trying to lock outputs or something
445 2013-05-31 04:51:32 <sipa> but that's a pure guess
446 2013-05-31 04:51:46 <sipa> Scrat: i won't claim i understand bdb good
447 2013-05-31 04:52:08 <TheUni> ok. well the deps are still driving me crazy, but i'm just ignoring that part for now
448 2013-05-31 04:52:31 <sipa> Scrat: but i've seen for example that changing the wallet from bdb to an append-only file format, speeds up generating the initial keypool tremendously
449 2013-05-31 04:52:40 <sipa> Scrat: so i'm sure there's lots of weird syncing involved
450 2013-05-31 04:53:56 <Scrat> ACTION puts wallet.dat on a ramfs
451 2013-05-31 04:54:48 <Scrat> quite doable with a large keypool and frequent backups
452 2013-05-31 04:57:26 <wumpus> Scrat: does fit into the 'playing with knives' category :p
453 2013-05-31 04:59:11 <Scrat> bitcoind is racist towards 5400 rpm drives
454 2013-05-31 05:01:02 <CodeShark> dropping bdb is a good idea regardless of context :p
455 2013-05-31 05:01:42 <sipa> ^
456 2013-05-31 05:02:39 <CodeShark> there is not one real-world engineering situation I can imagine where someone could sanely ever say "you know, if we only were to use bdb this would work better."
457 2013-05-31 05:02:49 <Scrat> I'd love to play with grau's creation but java scares me
458 2013-05-31 05:03:28 <CodeShark> why does java scare you?
459 2013-05-31 05:04:08 <Scrat> I'd have to research how to hook it up to a non-java system
460 2013-05-31 05:04:17 <grau> Java is only insecure if you execute an applet code you download from a website and execute it. That however would apply to C++ too, but it can not be done, so the problem does not arise.
461 2013-05-31 05:04:24 <Scrat> and java scares me because I don't know it very well
462 2013-05-31 05:04:37 <Scrat> grau: not from a security standpoint
463 2013-05-31 05:05:23 <sipa> i agree with grau here
464 2013-05-31 05:05:26 <CodeShark> the difficulty in learning a language is seldom the actual language itself but the frameworks, libraries, and tools surrounding it
465 2013-05-31 05:05:48 <sipa> but being scared of what you don't know, is good practice in security
466 2013-05-31 05:08:07 <CodeShark> all runtime environments have security weaknesses - it's a good idea to go with what you already know for critical production stuff while also learning about and experimenting with the unfamiliar
467 2013-05-31 05:08:10 <Scrat> bitsofproof is probably safer than bitcoind wrt remote code execution since you have a sandbox to break out of
468 2013-05-31 05:08:22 <Scrat> sure you can make an argument the other way around but it's not as strong
469 2013-05-31 05:09:02 <Luke-Jr> Java doesn't scare me, just annoys me :P
470 2013-05-31 05:09:56 <CodeShark> dumbed down C++ with inefficient runtime environment - but with far more library standardization than C++
471 2013-05-31 05:11:03 <CodeShark> a more coherent coding style, generally speaking
472 2013-05-31 05:11:17 <CodeShark> whereas C++ is often a hodgepodge of a bunch of different conventions
473 2013-05-31 05:13:50 <wumpus> library standardization, yeah, but how... WidgetFactoryManagerInjectorFactory etc :p
474 2013-05-31 05:13:59 <CodeShark> quite verbose :)
475 2013-05-31 05:14:06 <CodeShark> that's certainly true
476 2013-05-31 05:15:10 <CodeShark> you really do need text editors with autocomplete if you don't want to spend your entire life typing :)
477 2013-05-31 05:16:22 <wumpus> then again I'm not really a fan of c++ either, it 's just the default choice for projects in which performance and predictable memory usage is important
478 2013-05-31 05:16:59 <CodeShark> nothing has come to replace it, for better or worse
479 2013-05-31 05:17:34 <Scrat> go Go!
480 2013-05-31 05:17:53 <wumpus> though I certainly hope a safer language with less foot-shooting possibility will replace it eventually
481 2013-05-31 05:18:36 <wumpus> which is not brain-deadly verbose like java
482 2013-05-31 05:19:02 <Scrat> wumpus: seems to be the natural evolution -> less foot-shooting capability, more productivity
483 2013-05-31 05:19:15 <CodeShark> right, wumpus - without sacrificing its greatest strengths...compile-time to-the-metal optimization, predictable memory use, and tight integration with OS libraries
484 2013-05-31 05:19:17 <wumpus> but not looking at the specifics of the language syntax and runtime and such, I like java
485 2013-05-31 05:20:38 <CodeShark> the trend right now is certainly towards dynamic languages
486 2013-05-31 05:21:20 <CodeShark> with runtime optimization rather than compiletime optimization
487 2013-05-31 05:21:49 <wumpus> dynamic languages are great for ui and user facing code, and scripts and tools
488 2013-05-31 05:23:26 <wumpus> but for backend code like bitcoind, not so much, I don't see how a python implementation (for example) would be more better in this case even if super-optimized
489 2013-05-31 05:23:28 <Scrat> GCs and JITs are becoming increasingly smarter and smarter
490 2013-05-31 05:24:00 <CodeShark> even a python implementation would surely require some routines implemented in something like C/C++ or asm to be practical
491 2013-05-31 05:24:29 <wumpus> it's not about smarter and smarter -- compile time type checking can be pretty useful -- I like my runtime dumb sometimes and the compiler smart, so that as much as possible is caught at compile time
492 2013-05-31 05:25:30 <wumpus> also for refactoring and code analysis tools is can help if the language is not too dynamic
493 2013-05-31 05:25:57 <CodeShark> dynamic languages can still be used to manipulate certain data structures and for control structures - with highly optimized implementations of certain algorithms written in C/C++
494 2013-05-31 05:26:20 <CodeShark> or asm
495 2013-05-31 05:26:35 <CodeShark> that's pretty much what it comes down to
496 2013-05-31 05:28:35 <CodeShark> I like compile-time optimization too :)
497 2013-05-31 05:29:24 <CodeShark> problem is that right now hardware is cheap and abundant while good programmers are still relatively scarce :)
498 2013-05-31 05:30:42 <Scrat> and the gap will only widen
499 2013-05-31 05:30:55 <Scrat> moore's law still has quite a few years left
500 2013-05-31 05:31:06 <wumpus> the good thing with good programmers is that you only need a few anyway :p
501 2013-05-31 05:33:42 <CodeShark> C++ gives you plenty of opportunities to really shoot yourself in the foot if you don't know what you're doing
502 2013-05-31 05:34:24 <CodeShark> it requires a considerable amount of discipline and good practices along with deeper theoretical knowledge and understanding to use it properly
503 2013-05-31 05:34:40 <CodeShark> you can't just rely on some framework to ensure your app works
504 2013-05-31 06:20:35 <BlueMatt> TheUni: deps-win32 requires wine to run some build tools for...dont remember, maybe miniupnpc?
505 2013-05-31 06:20:49 <BlueMatt> TheUni: ahh, sipa guessed it
506 2013-05-31 06:20:49 <TheUni> BlueMatt: for building or testing?
507 2013-05-31 06:21:01 <BlueMatt> building
508 2013-05-31 06:21:08 <BlueMatt> it builds something and then that builds more sources to compile
509 2013-05-31 06:21:57 <TheUni> ok
510 2013-05-31 06:24:06 <wumpus> yeah it builds a wingenminiupnpcstrings.exe and executes that to generate strings for the build
511 2013-05-31 06:24:59 <wumpus> would be nice if it used the native toolchain for that, then again, wine works :)
512 2013-05-31 06:27:47 <TheUni> agh, it actually seems to assume builder == target
513 2013-05-31 06:28:06 <Luke-Jr> >_<
514 2013-05-31 06:28:15 <TheUni> sigh
515 2013-05-31 06:34:31 <wumpus> I guess with some makefile patching in gitian you could avoid wine, if you really wanted
516 2013-05-31 06:35:03 <wumpus> or maybe even with two make calls, with different parameters
517 2013-05-31 06:35:05 <wumpus> and env
518 2013-05-31 06:36:20 <sipa> or include its generated output in the source?
519 2013-05-31 06:36:34 <sipa> or just drop upnp support for windows *ducks*
520 2013-05-31 06:36:46 <TheUni> wumpus: not sure how if it depends on runtime greps
521 2013-05-31 06:37:06 <Luke-Jr> is there a problem with wine? :p
522 2013-05-31 06:37:11 <TheUni> unless you just hard-code that there instead (which is equally portable.. zero)
523 2013-05-31 06:37:58 <wumpus> possible to include the generated output in the source, but that will require a manual step every time the upnp version is changed
524 2013-05-31 06:38:21 <wumpus> don't know Luke-Jr
525 2013-05-31 06:38:32 <sipa> it also means the official source tarballs can't be uaed
526 2013-05-31 06:38:48 <wumpus> sipa: not necessarily, it could be an extra input :P
527 2013-05-31 06:38:54 <wumpus> I have no problems with wine, but it does pull in the entire x library shebang into the vm
528 2013-05-31 06:39:18 <sipa> well the issue is here that TheUni wants it to work not only in a VM
529 2013-05-31 06:39:55 <Luke-Jr> build miniupnpc native first, then reuse the generated files?
530 2013-05-31 06:40:01 <BlueMatt> yay java, "if (msg == null || msg.getType() == null)" generates a NullPointerException.....
531 2013-05-31 06:40:11 <Luke-Jr> might even be a make target to get it to build just the generator..
532 2013-05-31 06:40:25 <wumpus> Luke-Jr: yes, that's what I meant with "two make calls"
533 2013-05-31 06:40:43 <sipa> BlueMatt: huh?
534 2013-05-31 06:40:53 <BlueMatt> sipa: yes
535 2013-05-31 06:40:56 <wumpus> as I doubt the generated file depends on the OS in any way
536 2013-05-31 06:41:03 <sipa> unless getType itself throws that, i don't see how
537 2013-05-31 06:41:16 <TheUni> all it's doing is grabbing some runtime data.. though that runtime data is coming from a phony emulated OS. so the runtime check is useless
538 2013-05-31 06:41:18 <wumpus> Luke-Jr: don't forget a clean in between :PP
539 2013-05-31 06:41:34 <BlueMatt> sipa: nope, that is the top of the stacktrace, so getType cant be it...
540 2013-05-31 06:41:53 <BlueMatt> TheUni: why is this a problem?
541 2013-05-31 06:42:16 <wumpus> TheUni: huh, I thought it generated a list of strings for something, but it actually does something os-dependent?
542 2013-05-31 06:42:17 <TheUni> BlueMatt: it's not. i'm just retooling the buildsystem, and i facepalmed when i saw the wine bits
543 2013-05-31 06:42:33 <sipa> i guess it'd be nice if wine could be avoided i guess
544 2013-05-31 06:42:35 <BlueMatt> why are you retooling the deps build?
545 2013-05-31 06:42:47 <Luke-Jr> [08:37:35] <sipa> well the issue is here that TheUni wants it to work not only in a VM
546 2013-05-31 06:42:51 <sipa> he's autotoolsifying
547 2013-05-31 06:42:59 <BlueMatt> autotools shouldnt touch deps, though
548 2013-05-31 06:43:12 <BlueMatt> Luke-Jr: it works fine outside of a vm
549 2013-05-31 06:43:13 <TheUni> BlueMatt: i'm not touching the deps yet
550 2013-05-31 06:43:14 <BlueMatt> you just need wine
551 2013-05-31 06:43:29 <TheUni> for now i'm just assuming the deps are handed to me in a usable form
552 2013-05-31 06:43:39 <BlueMatt> ok, as any reasonable build system should, no?
553 2013-05-31 06:44:02 <sipa> seems like a good assumption
554 2013-05-31 06:44:05 <Luke-Jr> ACTION notes the output of deps-win32 is not reasonable right now
555 2013-05-31 06:44:11 <sipa> that part can always be improved later
556 2013-05-31 06:44:16 <TheUni> what Luke-Jr said :)
557 2013-05-31 06:44:19 <BlueMatt> Luke-Jr: ehhh, it works, doesnt it?
558 2013-05-31 06:44:28 <Luke-Jr> BlueMatt: only because our build system is a disaster
559 2013-05-31 06:45:21 <BlueMatt> Im confused as to how deps-win32's output is unreasonable
560 2013-05-31 06:45:37 <TheUni> ok, let's just skip that for now
561 2013-05-31 06:45:44 <Luke-Jr> reasonable = .a files and include dir
562 2013-05-31 06:46:10 <Luke-Jr> (or even better, .so/.dll files and include???)
563 2013-05-31 06:46:23 <TheUni> well, reasonable would be a staging prefix
564 2013-05-31 06:46:54 <sipa> care to explain what you mean by that?
565 2013-05-31 06:46:57 <Luke-Jr> yes, that's what I mean
566 2013-05-31 06:47:06 <wumpus> Luke-Jr: feel free to "fix" it, we're not quite bored enough for such details, it works fine and deterministic now and that's the important bit
567 2013-05-31 06:47:09 <BlueMatt> ahh, yes, well hopefully that means it just needs to actually call make install at the end
568 2013-05-31 06:47:25 <TheUni> BlueMatt: yes, that's the major part that's missing
569 2013-05-31 06:47:38 <TheUni> doing that would improve things 10x
570 2013-05-31 06:47:43 <BlueMatt> improve?
571 2013-05-31 06:47:55 <BlueMatt> doing that would change things, not improve much...
572 2013-05-31 06:48:02 <sipa> for gitian's purpose it wouldn't change anything
573 2013-05-31 06:48:14 <TheUni> BlueMatt: it does when there's a single buildsystem that tries to build everything the same way
574 2013-05-31 06:48:30 <wumpus> how would it improve? it'd at most speed up the build a very little bit as there's less files to be packed/unpacked and bounced around...
575 2013-05-31 06:48:37 <TheUni> instead, the mingw build fights autotools most of the way through
576 2013-05-31 06:48:48 <wumpus> in any case you need to rebuild the deps only if something changed
577 2013-05-31 06:49:16 <Luke-Jr> TheUni: probably best off just doing it and demonstrating in a pullreq :p
578 2013-05-31 06:49:51 <TheUni> Luke-Jr: heh, i'm trying not to get ahead of myself. i want the foundation ready first
579 2013-05-31 06:49:57 <wumpus> in any case I'm confused, you don't need to even touch the deps for autotool-ization
580 2013-05-31 06:50:10 <wumpus> configure scripts *never* install dependencies
581 2013-05-31 06:50:14 <TheUni> i'd just prefer to do a pull req that's nice and clean rather than saying "it's ugly but i can't help it!" :)
582 2013-05-31 06:50:24 <TheUni> wumpus: it's not about installing anything
583 2013-05-31 06:50:28 <TheUni> it's about finding them
584 2013-05-31 06:50:41 <wumpus> just look in the default locations and allow custom paths to be passed
585 2013-05-31 06:50:55 <TheUni> wumpus: sure, that's how it works
586 2013-05-31 06:51:08 <TheUni> but, as a comparison, linux looks like this:
587 2013-05-31 06:51:10 <TheUni> ./configure
588 2013-05-31 06:51:15 <TheUni> windows is:
589 2013-05-31 06:51:33 <TheUni> LDFLAGS=-L/opt/bitcoin/lib CPPFLAGS="-I/opt/bitcoin/include -I/opt/bitcoin/include/boost -L/home/cory/dev/bitcoin/deps/boost/boost_1_50_0/stage/lib" ./configure --host=i586-mingw32msvc --with-boost=/home/cory/dev/bitcoin/deps/boost/boost_1_50_0/stage
590 2013-05-31 06:51:35 <wumpus> and yes, the gitian configure line will be longer because it uses non-standard locations, no problem
591 2013-05-31 06:52:18 <TheUni> wumpus: i'm not sure why everything is discussed in terms of gitian, rather than a casual dev who's hacking on the source
592 2013-05-31 06:52:24 <wumpus> cross-building always results in terrible configure lines
593 2013-05-31 06:52:26 <TheUni> maybe that is our disconnect
594 2013-05-31 06:52:39 <wumpus> if you're building on windows you don't need any of that shit :p
595 2013-05-31 06:53:44 <sipa> well i'd certainly like it if i could build windows binaries without gitian
596 2013-05-31 06:54:17 <sipa> i know it's possible
597 2013-05-31 06:54:26 <sipa> but it's just inconvenient
598 2013-05-31 06:54:27 <wumpus> when building on a modern debian/ubuntu, many of the deps are available for windows already
599 2013-05-31 06:54:46 <BlueMatt> sipa: you can: submit a pullreq...done
600 2013-05-31 06:54:57 <wumpus> no need to even build them from source, AFAIK, maybe only upnpc
601 2013-05-31 06:55:31 <BlueMatt> sipa: but, no, its not that bad if you already have the deps zip from gitian
602 2013-05-31 06:55:31 <sipa> BlueMatt: i said if *I* could build
603 2013-05-31 06:55:39 <sipa> yeah i know
604 2013-05-31 06:55:42 <sipa> i'm just lazy
605 2013-05-31 06:55:49 <TheUni> ok, well, i'll put all that aside and finish up this stage
606 2013-05-31 06:55:53 <TheUni> one question though
607 2013-05-31 06:56:11 <TheUni> can i assume that i have a reasonable set of binaries/includes for qt?
608 2013-05-31 06:56:24 <TheUni> and if so, anyone happen to have a tarball laying around? :)
609 2013-05-31 07:00:02 <wumpus> btw now we're talking about the deps, libpng isn't really needed as input either :-) libqrcode just uses it to build an executable that we throw away
610 2013-05-31 07:00:50 <wumpus> the qrcode library doesn't depend on libpng in any way, it just passes raw images
611 2013-05-31 07:01:23 <wumpus> TheUni: you mean the gitian win32-qt output?
612 2013-05-31 07:01:39 <TheUni> wumpus: i suppose
613 2013-05-31 07:02:18 <TheUni> wumpus: ./configure --without-tools
614 2013-05-31 07:02:39 <TheUni> should be all it takes to drop libpng, if that's all it's used for
615 2013-05-31 07:03:27 <wumpus> yes -- the only other that uses libpng is qt, which is not part of the deps and uses it's own
616 2013-05-31 07:07:10 <wumpus> hm I see that my cross build env for mingw64 does build all the dependencies just like gitian, I was a bit over-optimistic with saying they are available as packages :-) kind of forgot we want to produce a static executable, too
617 2013-05-31 07:14:57 <wumpus> building static executables is a black art, doubly so with windows/mingw, and can hardly ever be done with precompiled packages
618 2013-05-31 07:15:55 <TheUni> wumpus: right, which is why i'm suggesting a dependency builder as my followup to the autotools part
619 2013-05-31 07:16:38 <TheUni> do you have a min to check out an example of what i'm talking about? I've done this exact thing on a massive scale already, it'd be dead-simple to move over
620 2013-05-31 07:25:47 <wumpus> sure
621 2013-05-31 07:27:35 <TheUni> git clone git://github.com/xbmc/xbmc.git
622 2013-05-31 07:27:49 <TheUni> (it's a bit big, may take a few min)
623 2013-05-31 07:28:08 <TheUni> cd tools/depends; ./bootstrap && ./configure
624 2013-05-31 07:28:23 <TheUni> then make
625 2013-05-31 07:28:59 <TheUni> it fetches all sources, then builds build-side tools, then build about 60 target libs
626 2013-05-31 07:30:15 <TheUni> that is massive. but my point is that.. it can be accomplished with a single configure and make
627 2013-05-31 07:31:50 <wumpus> ok, yes indeed, it's like a mini gentoo :)
628 2013-05-31 07:33:05 <TheUni> sure, but it's very simple and makes no assumption about the native system
629 2013-05-31 07:33:18 <BlueMatt> ACTION thinks that is overkill...
630 2013-05-31 07:33:44 <wumpus> well in our case we really only need it for cross-compile to windows
631 2013-05-31 07:33:45 <TheUni> BlueMatt: well if i were to bring it over here, it'd be slimmed way down
632 2013-05-31 07:33:53 <TheUni> wumpus: and arm?
633 2013-05-31 07:34:09 <BlueMatt> ACTION isnt convinced we need it at all
634 2013-05-31 07:34:10 <wumpus> when compiling on a native platform, just install the native packages
635 2013-05-31 07:34:18 <BlueMatt> why are you cross compiling to arm?
636 2013-05-31 07:34:20 <TheUni> android? x64 to x86?
637 2013-05-31 07:34:30 <BlueMatt> you cant run bitcoind on android...
638 2013-05-31 07:34:36 <BlueMatt> to x86 -m32
639 2013-05-31 07:34:39 <wumpus> x64 to x86 is simple, just install the :i386 dev packages
640 2013-05-31 07:34:41 <BlueMatt> you just need cflags
641 2013-05-31 07:35:08 <TheUni> i just assumed portability was a goal
642 2013-05-31 07:35:27 <BlueMatt> it isnt portable at all right now...so, yes, some would be nice
643 2013-05-31 07:35:33 <wumpus> for arm, yeah, but when you're seriously cross-building for ARM you generally building a lot of stuff, and need the dependencies outside of the context of bitcoin itself
644 2013-05-31 07:35:49 <BlueMatt> but if you are xcompiling bitcoind to run on something like arm, your device essentially needs to have the umph to run a compiler anyway
645 2013-05-31 07:35:55 <wumpus> so building the dependencies *inside* bitcoin would be a bit strange
646 2013-05-31 07:35:56 <BlueMatt> (eg, running bitcoind on raspi is not ok)
647 2013-05-31 07:36:05 <TheUni> wumpus: i cross-built for arm using only that dependency system. had it up and running in less than an hour
648 2013-05-31 07:36:40 <TheUni> <BlueMatt> it isnt portable at all right now...so, yes, some would be nice
649 2013-05-31 07:36:43 <wumpus> also when building for an embedded target you usually want to use the shared libraries available on that rootfs
650 2013-05-31 07:36:56 <wumpus> not build a static executable that ignores everything
651 2013-05-31 07:37:09 <BlueMatt> bitcoind cant run on an embedded target...
652 2013-05-31 07:37:12 <TheUni> BlueMatt: just keep in mind that it would be nice, i'm raising my hand, and offering a plan
653 2013-05-31 07:37:19 <TheUni> huh?
654 2013-05-31 07:37:28 <BlueMatt> anything that qualifies as embedded doesnt have the processing power to be a full node
655 2013-05-31 07:37:28 <wumpus> oh it can
656 2013-05-31 07:37:32 <BlueMatt> or the iops or whatever
657 2013-05-31 07:37:41 <wumpus> there's some quite beefy arm boxes out there, these days
658 2013-05-31 07:37:43 <TheUni> what does embedded have anything to do with that?
659 2013-05-31 07:37:50 <BlueMatt> wumpus: yes, I have one
660 2013-05-31 07:37:56 <BlueMatt> I installed gcc there and compiled bitcoind...
661 2013-05-31 07:38:03 <wumpus> quad-core, lots of memory
662 2013-05-31 07:38:16 <BlueMatt> but it doesnt really qualify as embedded if its quad-core and takes a full debian install with gcc
663 2013-05-31 07:38:22 <BlueMatt> (and sata)
664 2013-05-31 07:38:24 <TheUni> ...
665 2013-05-31 07:39:26 <wumpus> hmyeah what 'really quailfies as embedded' is more like a philosophical point I'm afraid :)
666 2013-05-31 07:39:39 <TheUni> well "embedded" is of no significance here
667 2013-05-31 07:39:43 <BlueMatt> ok, well my point is: if it cant run gcc, there is no way it has enough cpu/iops to run bitcoind
668 2013-05-31 07:39:46 <wumpus> many really embedded devices just run some debian
669 2013-05-31 07:40:07 <TheUni> BlueMatt: what does running gcc have anything at all to do with running bitcoin?
670 2013-05-31 07:40:22 <TheUni> i don't compile on my nas either, but it's beefier than my desktop was 3 years ago
671 2013-05-31 07:40:24 <BlueMatt> my point is targeting arm xcompiling is not really at all important
672 2013-05-31 07:40:40 <TheUni> BlueMatt: i would argue that targeting portability is crucially important
673 2013-05-31 07:40:43 <BlueMatt> its a "would be nice, but really dont care"
674 2013-05-31 07:41:08 <wumpus> well the autoconf will have to support cross compiling to any environment by passing the right arguments and environment vars
675 2013-05-31 07:41:13 <wumpus> but aside from that...
676 2013-05-31 07:41:23 <BlueMatt> feel free to implement, but having a ./bootstrap-download-all-deps-and-build.sh is probably not something worth having...
677 2013-05-31 07:41:55 <wumpus> BlueMatt: it is, if it does deterministic builds for all platforms too :P
678 2013-05-31 07:42:13 <BlueMatt> who said anything about deterministic?
679 2013-05-31 07:42:19 <wumpus> but that'd be kind of a wrapper around gitian
680 2013-05-31 07:42:20 <BlueMatt> (it would be a pita to make that script deterministic)
681 2013-05-31 07:42:53 <BlueMatt> unless its a gitian wrapper, at which point its...pointless?
682 2013-05-31 07:42:53 <TheUni> why is that? it's self-hosting
683 2013-05-31 07:43:00 <wumpus> well it'd be nice if there is a new release and the builders can just do ./gitian-do-frigging-everything.sh :P
684 2013-05-31 07:43:17 <BlueMatt> wumpus: gitian is like 4 commands right now...get over it?
685 2013-05-31 07:44:01 <wumpus> sigh, I'm just saying it would be nice, nothing more
686 2013-05-31 07:44:12 <BlueMatt> ok...well ok
687 2013-05-31 07:44:30 <BlueMatt> ACTION copies 4 lines from .bash_history into a .sh, adds a header and a cd...done
688 2013-05-31 07:45:11 <wumpus> nah it'd also have to check the deps, see if they need to be rebuilt, and such things, it's not just those four lines, everything sounds trivial until you try to implement it robustly and notice all the edge cases :)
689 2013-05-31 07:46:56 <TheUni> ok, one last try for the night.. anyone can point me to gitian qt binaries?
690 2013-05-31 07:47:58 <wumpus> yes
691 2013-05-31 07:48:45 <wumpus> uploading
692 2013-05-31 07:49:04 <TheUni> thanks
693 2013-05-31 07:49:56 <webuser234> Hi, I just upgraded bitoind on both of my servers from 0.8.1 to 0.8.2. I've been helping the network this way for many month now. I've been hearing that 0.8.2 imposses some limits on how small transaction it relays. My question is how do I make my daemons more liberal (which settings) so I they still relay even the small transaction be it satoshidice or others. I've got unlimitted bandwidth. Thanks
694 2013-05-31 07:50:43 <BlueMatt> its not a question of your own bandwidth
695 2013-05-31 07:50:50 <BlueMatt> its a question of everyone's bandwidth
696 2013-05-31 07:50:59 <BlueMatt> please dont relay crap
697 2013-05-31 07:51:43 <wumpus> also satoshidice changed their minimum output, so there's no issue with them
698 2013-05-31 07:51:51 <BlueMatt> yes, sadly
699 2013-05-31 07:51:56 <foami> hi
700 2013-05-31 07:51:58 <wumpus> yeah :(
701 2013-05-31 07:52:33 <wumpus> then again, at least their loser payout has some kind of minimal value now, and can be spent, which is what was the point
702 2013-05-31 07:53:00 <BlueMatt> well, true, if only coin selection algorithms sweeped up change though...
703 2013-05-31 07:53:52 <webuser234> BlueMatt, I understand it with my head by my heart tells me I should not be trying to control what can or can not be relayed.
704 2013-05-31 07:54:01 <webuser234> By the way those are my stats if devs are interested
705 2013-05-31 07:54:09 <webuser234> eth0 / monthly month rx | tx | total | avg. rate ------------------------+-------------+-------------+--------------- Jun '12 11.28 GiB | 108.44 GiB | 119.72 GiB | 387.46 kbit/s Jul '12 8.90 GiB | 109.16 GiB | 118.06 GiB | 369.76 kbit/s Aug '12 7.10 GiB | 94.44 GiB | 101.54 GiB | 318.02 kbit/s Sep '12 8.31 GiB | 85.45 GiB | 93.7
706 2013-05-31 07:54:18 <webuser234> oh shit; sorry didn't come out righ
707 2013-05-31 07:54:49 <BlueMatt> webuser234: the limits were added for a reason
708 2013-05-31 07:54:54 <webuser234> http://pastebin.com/Rt3RuvxR
709 2013-05-31 07:55:03 <BlueMatt> (and its not arbitrary)
710 2013-05-31 07:55:26 <webuser234> I started bitcoind in Dec
711 2013-05-31 08:15:03 <TheUni> ahh, bitcoin-qt.exe built, so i can sleep :)
712 2013-05-31 08:15:13 <TheUni> thanks for the help wumpus, BlueMatt, Luke-Jr
713 2013-05-31 08:15:26 <BlueMatt> hah!, I was right, Assertion `"mempool transaction missing input" == 0' failed.
714 2013-05-31 08:15:28 <BlueMatt> it can fail
715 2013-05-31 08:15:38 <BlueMatt> now that said...no fucking clue what I did to make it
716 2013-05-31 08:31:42 <damo22> what is stopping someone from editing the code for the client and rewarding themselves more coins for each block mined?
717 2013-05-31 08:32:45 <jaakkos> damo22: nothing, if you get everyone else in the network to accept the new rules.
718 2013-05-31 08:33:01 <wumpus> no one, it's not always needed to stop people doing stupid things :)
719 2013-05-31 08:33:25 <damo22> huh, what stops them doing it silently just for their own benefit or their own mining rig
720 2013-05-31 08:33:38 <wumpus> nothing, but they'll have their own dream bitcoins
721 2013-05-31 08:33:42 <wumpus> that no one else will accept
722 2013-05-31 08:34:01 <damo22> oh
723 2013-05-31 08:34:13 <lianj> damo22: the network would not accept it because he changed the rules which everyone checks. so you can mine a block but no one will accept it
724 2013-05-31 08:34:26 <damo22> good
725 2013-05-31 08:34:34 <damo22> i thought i just found a hole in bitcoin
726 2013-05-31 08:34:39 <damo22> :P
727 2013-05-31 08:34:43 <wumpus> the network exists by virtue of agreeing on the rules with respect to block rewards and such
728 2013-05-31 08:35:02 <wumpus> if you use other rules, you'll have created an altcoin
729 2013-05-31 08:35:05 <jaakkos> well, dude, you are going to have to look a little harder than that.
730 2013-05-31 08:35:17 <lianj> hehe
731 2013-05-31 08:35:20 <damo22> :D
732 2013-05-31 08:35:41 <damo22> im not about finding holes to benefit, i just want to be certain its unbreakable before i invest in the cryptocurrency
733 2013-05-31 08:36:17 <lianj> look deeper, but dont be disappointed when you find out its sane
734 2013-05-31 08:36:59 <damo22> i hope its not too late
735 2013-05-31 08:37:07 <darsk1ez> you'll have to be _very_ creative to think of an exploit vector that someone hasnt considered already.
736 2013-05-31 08:37:23 <damo22> i know its too late to start mining
737 2013-05-31 08:37:26 <damo22> lol
738 2013-05-31 08:37:44 <darsk1ez> if you want bitcoins, offer some service or goods for them.
739 2013-05-31 08:37:47 <Aurigae> THE vulnerable in bitcoin are the USERS ....
740 2013-05-31 08:37:59 <Aurigae> lol
741 2013-05-31 08:38:04 <damo22> yeah
742 2013-05-31 08:38:13 <damo22> thats sad though
743 2013-05-31 08:38:31 <damo22> i bet millions of dollars have been lost by gambling bitcoin already
744 2013-05-31 08:39:03 <Aurigae> from a technical pure system exploi tperspective with the transpoarency background, to SE on stupid users by unethical individuals, .....
745 2013-05-31 08:39:37 <Aurigae> imo, gambling and financial assets as well as hard assets .....
746 2013-05-31 08:39:38 <Aurigae> lol
747 2013-05-31 08:43:27 <Aurigae> hehe, damo, the dollars arent lost, just somebody else owns them who was smarter in the first place
748 2013-05-31 08:43:48 <Aurigae> that easy, like 123
749 2013-05-31 08:44:10 <webuser234> damo22, gambling isn't stupid - it's life in its purest form
750 2013-05-31 08:45:01 <Aurigae> gambling is stupid because the casino in 99% of cases has a advantage, of course its a nice thrill if chances are equal like a home poker game i think
751 2013-05-31 08:45:38 <webuser234> Aurigae, not if it's provable
752 2013-05-31 08:45:56 <Aurigae> proveable?
753 2013-05-31 08:47:51 <webuser234> Aurigae, Take satoshidice as an example ... it's an investment in its purest form ... you chose how much you want to put in, you know exactly how much you gonna get. Bam, done. You eiether win or loose - pure binary outcome without building theories around it.
754 2013-05-31 08:48:23 <Aurigae> perfect example that the house has the edge
755 2013-05-31 08:48:43 <runeks> devs, do you think it would make sense to add a queue to the thread that runs -blocknotify commands? as it is currently, the command is run whenever a new block comes in, which can happen at a rate of 50 blocks per second. so the previous -blocknotify command might not have finished before the next is executed. I'm thinking this can be solved with a queue, but I'm not sure how to implement it, or even if you agree that it's needed.
756 2013-05-31 08:50:12 <webuser234> Aurigae, well yeah; edge is just fancy term for fee. You pay fee in everything you do - you gamble your life goingn to certain school for example; you pay the fee. Outcome will be 0 - not successful 1 - successful
757 2013-05-31 08:50:35 <wumpus> runeks: sounds like a very unscalable solution; I think it would be better to time-limit the requests somehow, and pass the block number so that the command knows how many blocks have passed
758 2013-05-31 08:50:58 <wumpus> runeks: alternatively you could look at low-overhead notifications such as the zeromq pull
759 2013-05-31 08:51:10 <wumpus> runeks: starting a command is not ideal for high-frequency events
760 2013-05-31 08:51:53 <runeks> wumpus: So you're saying it's unscalable because the queue might fill up with a large amount of blocks if the command is slow?
761 2013-05-31 08:52:34 <wumpus> runeks: yes, as launching a command has a fairly large overhead in itself it will needlessly burden the system
762 2013-05-31 08:52:38 <wumpus> also known as "fork bomb"
763 2013-05-31 08:53:02 <runeks> wumpus: Well that's what the current code does. It fires the command on every incoming block.
764 2013-05-31 08:53:12 <CodeShark> I'm debating between two architectures: 1) integrate a higher level streaming API into bitcoind, 2) run a separate process that connects to bitcoind via p2p and exposes a higher level streaming API to other apps (the architecture I'm currently using)
765 2013-05-31 08:53:21 <wumpus> yes, which is why I proposed two other solutions that would be better
766 2013-05-31 08:53:51 <wumpus> CodeShark: what are you trying to do?
767 2013-05-31 08:54:22 <CodeShark> develop a platform upon which app developers can build that affords full streaming capabilities but is higher level than the p2p protocol
768 2013-05-31 08:54:49 <CodeShark> REST is wholely inadequate for a system that requires handling of network events
769 2013-05-31 08:55:06 <CodeShark> and the p2p protocol is far too low level for most web developers
770 2013-05-31 08:55:26 <CodeShark> and solutions like -blocknotify are horrendous hacks
771 2013-05-31 08:55:46 <wumpus> CodeShark: I'd look at the zeromq pull, though not ideal it is very promising, at least it would allow for other processes to subscribe to events, and send commands/receive responses asynchronously
772 2013-05-31 08:56:14 <CodeShark> I've built a websockets server which has the advantage of being far more familiar to web app developers and can go through firewalls
773 2013-05-31 08:56:21 <wumpus> zeromq is also used in high frequency trading for example, it's a very good fit for these kinds of things
774 2013-05-31 08:56:54 <runeks> wumpus: Right. The thing with the timer is that we don't know how long the command takes to finish. If the command takes 20 seconds to finish, and our timer is executes the script with the last 10 seconds of incoming block hashes, it would still be too fast. A queue seems more flexible to me in that regard.
775 2013-05-31 08:56:56 <CodeShark> websockets only has minimal additional overhead to a regular TCP socket
776 2013-05-31 08:56:56 <wumpus> the idea is that you can build that o top
777 2013-05-31 08:57:03 <wumpus> in a seperate process
778 2013-05-31 08:57:30 <wumpus> zeromq has no overhead at all compared to normal sockets, it is very very optimized
779 2013-05-31 08:58:09 <CodeShark> the biggest downside to zeromq that I see is that most web developers aren't very familiar with it
780 2013-05-31 08:58:32 <wumpus> well yeah 'most web developers' is kind of a hard to quantify group
781 2013-05-31 08:59:24 <CodeShark> and it can't go through firewalls
782 2013-05-31 08:59:41 <wumpus> oh it can
783 2013-05-31 08:59:47 <wumpus> just configure your firewall properly
784 2013-05-31 08:59:53 <CodeShark> lol
785 2013-05-31 09:00:07 <Scrat> the difference is that if your backend is down, zeromq will queue up the callbacks while websocket will not
786 2013-05-31 09:00:43 <CodeShark> well, you can add queueing to the websocket server :)
787 2013-05-31 09:01:23 <wumpus> yes you can, but you don't want all that ballast in the bitcoind
788 2013-05-31 09:02:44 <wumpus> and once someone is writing real financial applications and not just some web frontend, investing in knowing zeromq is probably a good idea
789 2013-05-31 09:02:56 <runeks> blocks or not.
790 2013-05-31 09:02:56 <runeks> wumpus: Thinking about it some more, doesn't the -blocknotify feature lose its purpose if we rate-limit executed commands with a timer? I mean, if we only send out new blocks every n seconds, we might as well have a script that polls bitcoind with the getblockcount command every n seconds. That would be equivalent, except for the fact that bitcoind would need to respond to an RPC request every n seconds, regardless of whether there are new
791 2013-05-31 09:03:10 <CodeShark> zeromq is a very reasonable way to go
792 2013-05-31 09:04:10 <wumpus> runeks: sure, but it's always a compromise.. if the events you want to monitor occur faster than your communication channel allows you need to aggregate somehow...
793 2013-05-31 09:04:35 <wumpus> runeks: but yes that reasoning is why the UI polls the block number instead of getting notified on new blocks
794 2013-05-31 09:04:52 <wumpus> runeks: 'block notify' is only useful *after* the initial sync
795 2013-05-31 09:05:11 <wumpus> in the initial block download it clogs all the pipes :P
796 2013-05-31 09:05:54 <CodeShark> actually, the architecture I think I'd most like to see is super streamlined verification/relay nodes that just do p2p, SPV nodes that provide a higher level streaming API, and then apps built atop those nodes
797 2013-05-31 09:06:21 <wumpus> I'm sure we'd all rather see that
798 2013-05-31 09:06:33 <wumpus> but we have to work with what we've got, for now :)
799 2013-05-31 09:07:52 <runeks> smallest possible number of times, while retaining real-time notifications.
800 2013-05-31 09:07:52 <runeks> wumpus: So instead of building up a queue of commands to be executed, we could execute the command once upon the first incoming block, then if new blocks come while the command is still running we just add block hashes to the queue, and when the first command finishes we execute the -blocknotify command with the first and last block hash in the queue list. That would prevent the "fork bomb" while only executing the -blocknotify command the
801 2013-05-31 09:08:42 <wumpus> runeks: yes, executing a command provides natural rate limiting: just only re-execute the command once it comes back