1 2012-10-28 01:11:07 <jgarzik> w00t
2 2012-10-28 01:11:19 <jgarzik> picocoin exchanges P2P "version" messages
3 2012-10-28 01:16:33 <jgarzik> next step, blockchain database download and sycn
4 2012-10-28 01:16:35 <jgarzik> *sync
5 2012-10-28 02:42:51 <knotwork> Can the raw transactions RPC stuff enable people to create transactions such as user A icommitting to sending a specific sum to user B if oracle Y signs it?
6 2012-10-28 02:43:22 <knotwork> And also enable oracles Y and N to sign on demand any such transactions?
7 2012-10-28 02:44:12 <knotwork> So oracle Y and N could act as Yes and No oracles on a point of fact, and users can give each other transactions that will be able to be made effective by such oracles?
8 2012-10-28 02:45:55 <knotwork> Basically bet transactions, users will bet each other "I bet oracle A will not sign this!" or "I bet oracle N will not sign this!" type of effect?
9 2012-10-28 02:46:18 <knotwork> (Since if the oracle does sign it, oops you just parted with some coin)
10 2012-10-28 02:47:16 <knotwork> I am thinking the oracles will be publicly known addresses with whom users construct two of two addresses
11 2012-10-28 02:47:17 <gmaxwell> knotwork: you must two phase that sort of thing in bitcoin, otherwise you can defeat the oracle's signature by spending the input out from under it.
12 2012-10-28 02:47:59 <knotwork> isnt two of two supposed to be a kind of escrow?
13 2012-10-28 02:48:19 <gmaxwell> I'm missing the context of your question.
14 2012-10-28 02:48:22 <knotwork> you deposit your bet into an address where it will rot forever unless both you and the oracle sign it?
15 2012-10-28 02:48:37 <knotwork> oops yeah you also need a way to get it back out
16 2012-10-28 02:48:44 <gmaxwell> Was your question a response to something I said?
17 2012-10-28 02:49:08 <knotwork> No its related to ways of doing shorts and longs or calls and puts
18 2012-10-28 02:49:29 <gmaxwell> so first you'd relinquish control of coins to 2 of 3 (me, you, oracle), then after the matter is resolved me+you or one of us and the oracle pay the coins back to whereever the signing parties consent to.
19 2012-10-28 02:49:36 <knotwork> trying to figure a way to make bets using blockchain provided oracles exist that can settle questions of fact
20 2012-10-28 02:50:46 <gmaxwell> What you can't do is do this securely without first relinquishing sole control of the coins. (otherwise you'll just wait until the event happens not in your favor and rapidly spend the coins.
21 2012-10-28 02:50:52 <gmaxwell> )
22 2012-10-28 02:51:01 <knotwork> Hmm.
23 2012-10-28 02:51:46 <knotwork> One model that sounded promising amounted to two people agreeing on a strike price and volume either could exercise at any moment during some span of time
24 2012-10-28 02:51:59 <knotwork> like say we both might agree to $11 per bitcoin strke price
25 2012-10-28 02:52:23 <knotwork> then up until some expiry date, either of us can exercise it
26 2012-10-28 02:52:28 <knotwork> that model needed no oracle
27 2012-10-28 02:52:56 <knotwork> but not sure how that model would/could be enforced in blockchain either
28 2012-10-28 02:52:58 <gmaxwell> It doesn't? What is a $ and where can I find out about that in bitcoin? :P
29 2012-10-28 02:53:07 <gmaxwell> Right.
30 2012-10-28 02:53:28 <knotwork> oh yeah right that model needs two currencies
31 2012-10-28 02:53:46 <knotwork> the oracle method uses oracle instead of a second currency
32 2012-10-28 02:54:06 <knotwork> since given an oracle you can bet on whether it will or will not sign something
33 2012-10-28 02:55:10 <knotwork> so I was trying to figure out how I could set up an oracle, as in what it would actually sign and how, if there is for example an RPC call it would use nowadays to sign something
34 2012-10-28 02:55:30 <gmaxwell> if the oracle will sign nothing at all you'll get stuck if your counterparty is a dick.
35 2012-10-28 02:55:32 <knotwork> then hmm what would the users be building for it to sign and how would they build and sign it
36 2012-10-28 02:56:08 <knotwork> yeah so we need a y-junction, you put in coins and they either come back to you or proceed on to me
37 2012-10-28 02:56:13 <gmaxwell> knotwork: the rpc you want is signraw transaction. But really a proper oracle would run on tamper resistant hardware with remote attestation and not be based on bitcoin.
38 2012-10-28 02:56:18 <knotwork> with an oracle deciding which of us gets them
39 2012-10-28 02:56:26 <gmaxwell> knotwork: yea, thats easy to do.
40 2012-10-28 02:57:08 <gmaxwell> knotwork: here is an example of all the relevant commands: https://people.xiph.org/~greg/escrowexample.txt
41 2012-10-28 02:57:36 <knotwork> thanks
42 2012-10-28 02:58:29 <gmaxwell> you'd give the oracle a set of rules "If x pay a otherwise pay b" it gives you a public key. you include it in a 2-of-3.. then later you can ask it to sign according to the prearranged rule.
43 2012-10-28 02:59:02 <knotwork> great that sounds like just the thing thanks
44 2012-10-28 02:59:41 <knotwork> I hate having to use oracles really. I prefer the mutually exerciseable mutual option thing but that cannot be done on blockchain
45 2012-10-28 02:59:50 <knotwork> maybe could be done in Open Transactions though
46 2012-10-28 04:58:03 <jgarzik> sipa: will any DNS seed or seeds.txt ever pick up a !NODE_NETWORK node?
47 2012-10-28 05:15:01 <sipa> jgarzik: i think mine filters those out immediately
48 2012-10-28 11:26:09 <jeremias> https://fundhub.org/
49 2012-10-28 11:26:14 <jeremias> anyone know about this?
50 2012-10-28 11:26:19 <jeremias> seems really interesting idea
51 2012-10-28 11:49:50 <BCB> anybody up yet
52 2012-10-28 11:52:00 <sipa> no, sorry
53 2012-10-28 11:55:08 <BCB> ha
54 2012-10-28 11:58:16 <BCB> can bitcoind return the total received from a public key not in my wallet?
55 2012-10-28 11:58:23 <sipa> no
56 2012-10-28 11:59:02 <BCB> what info can it return on a public key not in my wallet
57 2012-10-28 11:59:16 <Luke-Jr> nothing
58 2012-10-28 11:59:27 <sipa> total received by an address would be possible, but that requires an index that is not necessary for general operation, so it's not done
59 2012-10-28 11:59:58 <sipa> total received by a public key would imply being able to find all addresses (including P2SH) that are related to the public key, which is impossible
60 2012-10-28 12:00:13 <BCB> can bitcoind generate that index
61 2012-10-28 12:00:16 <sipa> no
62 2012-10-28 12:00:27 <Luke-Jr> BCB: if you hack it a bit
63 2012-10-28 12:00:40 <BCB> so when I go to blockchain.info and put in an address I get total received
64 2012-10-28 12:00:53 <BCB> i would prefer to do that internally
65 2012-10-28 12:01:00 <sipa> what do you need it for?
66 2012-10-28 12:01:28 <BCB> researching deposit addresses
67 2012-10-28 12:01:28 <Luke-Jr> BCB: I think Armory supports this
68 2012-10-28 12:01:49 <BCB> so if its possible how is it done
69 2012-10-28 12:01:49 <sipa> yeah, we should support watch-only addresses/wallets
70 2012-10-28 12:03:10 <Luke-Jr> you tell bitcoind it's yours
71 2012-10-28 12:03:17 <BCB> hmmm
72 2012-10-28 12:03:28 <BCB> cmd?
73 2012-10-28 12:03:34 <Luke-Jr> there is no cmd
74 2012-10-28 12:03:46 <Luke-Jr> probably an hour of hacking if you know what you're doing
75 2012-10-28 12:04:06 <sipa> BCB: bitcoin's addresses are not intended to be reused, so requesting a balance for an address seems pointless
76 2012-10-28 12:04:09 <Luke-Jr> (which you probably don't, based on the questions you're asking)
77 2012-10-28 12:04:17 <sipa> well, there are some cases where it's useful of course
78 2012-10-28 12:04:18 <BCB> I don't
79 2012-10-28 12:04:25 <BCB> but i'm curious
80 2012-10-28 12:04:29 <BCB> best way to learn
81 2012-10-28 12:04:41 <sipa> but most often, you want to reason in function of wallets which collect several addresses, instead of just a single address
82 2012-10-28 12:05:01 <sipa> as those don't necessarily reveal connection to the outside world
83 2012-10-28 12:05:31 <BCB> but this external address is not in any of my wallets
84 2012-10-28 12:05:57 <sipa> yes, that's why we'd need to support watch-only wallets
85 2012-10-28 12:06:24 <BCB> and bitcoind current does not support that?
86 2012-10-28 12:06:24 <sipa> (and multiple wallets)
87 2012-10-28 12:06:27 <sipa> no
88 2012-10-28 12:06:41 <sipa> but that's planned
89 2012-10-28 12:06:47 <BCB> sipa: you are an armory developer
90 2012-10-28 12:06:51 <sipa> no
91 2012-10-28 12:06:56 <sipa> i'm a bitcoind developer
92 2012-10-28 12:07:42 <BCB> oh you meant "we" and in bitcoind not armory previously sorry
93 2012-10-28 12:07:53 <BCB> as in
94 2012-10-28 12:07:56 <sipa> yes
95 2012-10-28 12:08:20 <BCB> no a bit deal
96 2012-10-28 12:08:25 <sipa> ha
97 2012-10-28 12:08:27 <sipa> a bit deal
98 2012-10-28 12:08:34 <BCB> fat fingers
99 2012-10-28 12:08:36 <BCB> sorry
100 2012-10-28 12:08:41 <BCB> bit
101 2012-10-28 12:08:43 <BCB> big
102 2012-10-28 12:08:47 <BCB> the friggen g
103 2012-10-28 12:08:49 <sipa> no problem, just found it funny :)
104 2012-10-28 12:09:00 <sipa> ACTION afk
105 2012-10-28 12:09:10 <BCB> I'm just trying to keep all my api call on my own box
106 2012-10-28 12:09:42 <BCB> when possible
107 2012-10-28 12:10:09 <BCB> so if I send funds to a external address
108 2012-10-28 12:10:45 <BCB> and want to confirm funds, confirms etc
109 2012-10-28 12:11:10 <BCB> I get generate that from my bitcoind but not funds I did not send to that address
110 2012-10-28 12:11:26 <sipa> not following
111 2012-10-28 12:11:46 <BCB> if a user is accessesing my web service
112 2012-10-28 12:12:03 <BCB> I can call gettransaction and get confirmation info etc
113 2012-10-28 12:12:03 <sipa> the bitcoind wallet tracks transactions sent to any of your addresses, and spends from any transaction that was yours
114 2012-10-28 12:12:57 <BCB> but not transactions received by that external address if I did not initiate the transaction
115 2012-10-28 12:13:25 <sipa> indeed
116 2012-10-28 12:13:52 <BCB> but if i know what i was doing i could hack that out in an hour
117 2012-10-28 12:13:56 <BCB> any directions
118 2012-10-28 12:14:01 <BCB> maybe a clue
119 2012-10-28 12:14:13 <sipa> why do you need that?
120 2012-10-28 12:14:52 <sipa> ideally, addresses shouldn't be reused, so there shouldn't be any receives except yours to that address
121 2012-10-28 12:15:36 <sipa> and any clues: wallet.{h,cpp} have some IsMine/IsFromMe functions
122 2012-10-28 12:15:37 <BCB> buy user do reuse addresses often
123 2012-10-28 12:15:56 <sipa> yes, i know, and i hope that will decrease in the future
124 2012-10-28 12:16:08 <BCB> why
125 2012-10-28 12:16:16 <sipa> becaue it's bad for bitcoin
126 2012-10-28 12:16:24 <BCB> anonymity
127 2012-10-28 12:16:27 <BCB> ?
128 2012-10-28 12:16:30 <sipa> it decreases privacy for everyone in the system
129 2012-10-28 12:16:37 <BCB> is that the only reason
130 2012-10-28 12:16:46 <sipa> it's not enough?
131 2012-10-28 12:17:10 <BCB> I just thinking from a standard point of vie
132 2012-10-28 12:17:11 <BCB> w
133 2012-10-28 12:17:26 <sipa> well the standard point of view is going for the route that is easiest
134 2012-10-28 12:17:28 <BCB> maybe TBF needs to reccomend that to client develpers
135 2012-10-28 12:17:46 <sipa> so we should make it easy to not reuse addresses
136 2012-10-28 12:17:57 <BCB> yes
137 2012-10-28 12:18:08 <sipa> also, users shouldn't ever see a base58 address at all, in my opinion
138 2012-10-28 12:18:18 <BCB> what should you see
139 2012-10-28 12:18:28 <sipa> something like an email address or a url
140 2012-10-28 12:18:42 <BCB> sipa: i will be your evangulist
141 2012-10-28 12:18:44 <sipa> and have the bitcoin key negotiated automatically
142 2012-10-28 12:18:45 <BCB> I totally agree
143 2012-10-28 12:18:52 <sipa> lol
144 2012-10-28 12:18:56 <BCB> seriously
145 2012-10-28 12:19:01 <BCB> it fucking ugly and scary
146 2012-10-28 12:19:21 <BCB> you'll never see mass adobtion until it is abstracted
147 2012-10-28 12:19:26 <sipa> the fact that we call these base58 strings "addresses" is part of the problem, imho
148 2012-10-28 12:19:37 <sipa> they are not addresses if people don't expect payments to them
149 2012-10-28 12:19:49 <Diablo-D3> er, what?
150 2012-10-28 12:19:50 <sipa> they are public key identifiers, and nothing more
151 2012-10-28 12:20:00 <BCB> I took me 6 months to make the destinction between "address" and Public key
152 2012-10-28 12:20:04 <Diablo-D3> so basically you want pgp idents
153 2012-10-28 12:20:11 <BCB> total aggree
154 2012-10-28 12:20:13 <BCB> ly
155 2012-10-28 12:20:13 <Diablo-D3> "pgp sign to address@fuckyou.asshole
156 2012-10-28 12:20:19 <Diablo-D3> etc etc
157 2012-10-28 12:20:23 <sipa> not really
158 2012-10-28 12:21:10 <BCB> sipa: what do you mean have the bitoin key negotiated automatically
159 2012-10-28 12:21:38 <sipa> i want "i pay to https://www.mywebshop.com/order2134.btc", client fetches from that URI, result contains a signed message "pay amount X to address Y", client creates transaction that satisfies the requirements, and sends it back to the merchent (*not* to the p2p network)
160 2012-10-28 12:23:08 <BCB> signed message come from server? using private key?
161 2012-10-28 12:23:21 <sipa> there are several layers of security possible
162 2012-10-28 12:24:09 <BCB> so server generates bot x and y address?
163 2012-10-28 12:24:22 <BCB> both
164 2012-10-28 12:24:34 <sipa> X is an amount, that depends on what you bought or want to pay for
165 2012-10-28 12:24:52 <sipa> in some circumstances, it could be left to the sender (in case of a donation, for example)
166 2012-10-28 12:25:16 <BCB> and how does the server know who the client is
167 2012-10-28 12:25:21 <sipa> read this if you want an example in detail (it's more than a year old, and i'd do some things differently, but still): https://gist.github.com/1237788
168 2012-10-28 12:25:29 <sipa> BCB: why does the server need to know who the client is?
169 2012-10-28 12:25:36 <BCB> oh oh
170 2012-10-28 12:25:56 <sipa> if someone comes into your shop and says "hey i want to pay for order X", do you even care whether he is the one that ordered it?
171 2012-10-28 12:26:12 <BCB> i mistook x and y for addresses not amount address
172 2012-10-28 12:26:29 <BCB> Sipa: whay is that a YEAR old
173 2012-10-28 12:26:47 <BCB> why
174 2012-10-28 12:26:50 <sipa> nobody found it important enough, i guess
175 2012-10-28 12:27:37 <sipa> anyway, afk
176 2012-10-28 12:27:52 <BCB> but is not the dev team focus core operation and security of the client?
177 2012-10-28 12:28:30 <sipa> yes, and this has not much to do with the client anymore
178 2012-10-28 12:28:31 <BCB> in my humble opinion an impliment like you are discussing could be HUGE for adobtion
179 2012-10-28 12:28:46 <sipa> it's about an infrastructure on top of the current bitcoin network
180 2012-10-28 12:28:49 <BCB> ahh
181 2012-10-28 12:29:10 <sipa> obviously at some point you'd want clients to support it, but you need more than that
182 2012-10-28 12:29:11 <BCB> maybe there should be a UI team
183 2012-10-28 12:29:22 <BCB> like what
184 2012-10-28 12:29:33 <sipa> read the document first
185 2012-10-28 12:29:40 <BCB> ok
186 2012-10-28 12:29:47 <BCB> thx
187 2012-10-28 12:30:03 <sipa> you need servers that provide the payment information, and servers to handle incoming ones (they can be the same, though)
188 2012-10-28 12:30:16 <sipa> anyway, i believe gavin wants to focus on payment protocols in the future
189 2012-10-28 12:30:31 <BCB> I think you are on to something big there.
190 2012-10-28 12:30:43 <BCB> he should
191 2012-10-28 12:31:49 <BCB> sipa:servers on which the bitcoind resides or external
192 2012-10-28 12:33:22 <sipa> to generate public keys for payments does not need any connection to the network
193 2012-10-28 12:33:39 <sipa> handling incoming ones does, as you want to verify their validity
194 2012-10-28 12:37:05 <BCB> yes. I've hacked together a php api that generates key pairs on or off like so that each transaction is receiving a fresh off line address
195 2012-10-28 12:37:49 <BCB> so my desire is to be able to confirm the transaction with out importing the private key until it is time to pay out
196 2012-10-28 12:38:52 <BCB> so if the address is not in my walled I can't validate the transaction
197 2012-10-28 12:41:39 <sipa> bdbuse deterministic key generation, so public and private keys don't need to be one the same server
198 2012-10-28 12:41:51 <sipa> BCB
199 2012-10-28 12:43:16 <BCB> sipa: "deterministic key generation" - do you have a link
200 2012-10-28 12:44:43 <sipa> BCB: bip30 :)
201 2012-10-28 12:44:56 <BCB> thx
202 2012-10-28 12:45:49 <sipa> eh, 32!
203 2012-10-28 12:47:28 <BCB> THX: I was trying to figure out what duplicate transactions had to to with it !!
204 2012-10-28 12:49:50 <BCB> sipa: Dev teams need to be divided up
205 2012-10-28 12:50:06 <BCB> sipa:you need to lead the UI team
206 2012-10-28 12:50:17 <sipa> hell f*cking no
207 2012-10-28 12:50:26 <abrkn> are there any online wallet services that provide apis for sending funds, creating addresses and streaming/push apis for being notified when an address has received funds? (i want to integrate bitcoin balances into my service without storing the wallets directly)
208 2012-10-28 12:51:21 <BCB> sipa: why not
209 2012-10-28 12:52:10 <BCB> abrkn: blockchain.info/api
210 2012-10-28 12:52:59 <BCB> abrkn:there are other but they and their accessibily come and go so be careful
211 2012-10-28 12:54:02 <abrkn> ok, so blockinfo will notify me of funds. which bank is fairly safe? i wont be handling large amounts at this point
212 2012-10-28 12:54:28 <sipa> BCB: i don't know anytging about user interface, and don't care at all about how a program looks (i mostly use command lines myself). what i care about is interoperability and efficiency
213 2012-10-28 12:55:47 <BCB> sipa:then i'm using the wrong term.
214 2012-10-28 12:56:09 <BCB> sipa: you need to lead the interoperability/efficency team!
215 2012-10-28 12:56:12 <BCB> serious
216 2012-10-28 13:07:03 <BCB> arkin: pm me this isn't really a btcoin-dev issue
217 2012-10-28 13:07:29 <BCB> abrkn:
218 2012-10-28 13:09:09 <D34TH> abrkn: that would be a #bitcoin topic
219 2012-10-28 14:40:24 <sipa> gmaxwell: difference between importing time for 193k blocks between -O2 and -O3 (both on -flto): 30%
220 2012-10-28 14:41:42 <sipa> 22m -> 16m
221 2012-10-28 14:43:28 <sipa> hmm, objects were compiled with -O0 and linked with -O2; maybe the initial optimization level does matter
222 2012-10-28 15:05:30 <Diablo-D3> sipa: linker -O just controls link time optimization shit
223 2012-10-28 15:05:35 <Diablo-D3> like, cross object inlining
224 2012-10-28 15:05:38 <Diablo-D3> if your gcc supports it
225 2012-10-28 15:05:40 <Diablo-D3> otherwise it does nothing
226 2012-10-28 15:05:42 <sipa> yes, that's the theory
227 2012-10-28 15:05:48 <Diablo-D3> historically, it has never done anything
228 2012-10-28 15:05:58 <Diablo-D3> and iirc its only turned on at -O3
229 2012-10-28 15:06:21 <Diablo-D3> also, gcc doesnt support multiple -O levels across objects, shit can go bad
230 2012-10-28 15:41:17 <jgarzik> sipa: heh
231 2012-10-28 15:41:24 <OneEyed> Diablo-D3: FUD
232 2012-10-28 15:41:29 <jgarzik> sipa: compile optimization level matters far more than linker opt level ;p
233 2012-10-28 15:41:33 <OneEyed> Diablo-D3: GCC supports having multiple -O levels just fine
234 2012-10-28 15:41:53 <jgarzik> indeed
235 2012-10-28 15:42:01 <jgarzik> the ABI does not change based on opt level
236 2012-10-28 15:42:01 <OneEyed> Diablo-D3: and on some architectures, gold (the new ld from binutils) supports link time optimization
237 2012-10-28 15:42:06 <jgarzik> ABI is ABI
238 2012-10-28 15:42:45 <jgarzik> even if gcc inlines some code, ABI says it must emit a copy if it's a public function, etc.
239 2012-10-28 15:43:52 <OneEyed> I remember testing one file at -O0, then -O1, -O2, -O3 while keeping the rest unchanged when debugging some C or Ada compiler bugs in GCC
240 2012-10-28 15:44:10 <OneEyed> It would have been a nightmare if that didn't work :)
241 2012-10-28 15:46:46 <sipa> jgarzik: with -flto, the optimization is (at least in theory) only done at link time
242 2012-10-28 15:48:04 <sipa> and sure if you specify -O0, the .o file itself contain a non-optimized compiled object version of the code, but that shouldn't be used
243 2012-10-28 15:48:41 <OneEyed> sipa: if you have a linker that recognize -flto - as far as I remember, some platforms don't have it yet
244 2012-10-28 15:49:10 <OneEyed> and you'll end up with your pessimized code instead (since -O0 is not non-optimized code as another compiler would do, that would rather correspond to almost -O1)
245 2012-10-28 15:49:21 <sipa> OneEyed: given the fact that the linking stage takes far longer than the compilation stage, and forks several subprocesses, i figure it does :)
246 2012-10-28 15:49:49 <OneEyed> sipa: oh yeah, on x86/x86_64/ARM at least, the linker understands it
247 2012-10-28 16:21:25 <Diablo-D3> [12:41:24] <OneEyed> Diablo-D3: FUD
248 2012-10-28 16:21:27 <Diablo-D3> no FUD at all
249 2012-10-28 16:21:33 <Diablo-D3> FUD doesnt even make sense in this context
250 2012-10-28 16:21:39 <Diablo-D3> I've used gcc for 15 years
251 2012-10-28 16:21:43 <Diablo-D3> I see no problems with it
252 2012-10-28 16:21:50 <Diablo-D3> [12:42:45] <jgarzik> even if gcc inlines some code, ABI says it must emit a copy if it's a public function, etc.
253 2012-10-28 16:21:52 <Diablo-D3> jgarzik: yes
254 2012-10-28 16:21:58 <Diablo-D3> so in theory nothing should hoark
255 2012-10-28 16:22:36 <Diablo-D3> oh and as for bugs in -O0, yeaaaah, gcc is aware of this and doesnt care
256 2012-10-28 16:24:25 <gmaxwell> 09:06 < Diablo-D3> also, gcc doesnt support multiple -O levels across objects, shit can go bad
257 2012-10-28 16:24:28 <gmaxwell> ^ huh?
258 2012-10-28 16:24:54 <gmaxwell> I'm pretty sure thats not true, got a cite? There is even a pragma to change the optimization level/flags in the source.
259 2012-10-28 16:25:36 <sipa> gmaxwell: when estimating compressibility per-block, i forgot to take pubkey reconstruction from signatures into account; new estimate: 39% compression (from 100 to 61 bytes)
260 2012-10-28 16:28:10 <gmaxwell> sipa: that LTO speedup isn't inconsistent with some other things I've seen, but it would probably be good to figure out why??? since we might get even more out of fixing whatever it is directly
261 2012-10-28 16:28:53 <D34TH> what is happening
262 2012-10-28 16:29:04 <D34TH> im noticing MASSIVE speed improments
263 2012-10-28 16:29:15 <D34TH> loadblock is going atleast 10x faster
264 2012-10-28 16:30:11 <Diablo-D3> gmaxwell: Ive had that bite me in the ass in the past
265 2012-10-28 16:30:15 <sipa> D34TH: faster than what?
266 2012-10-28 16:30:19 <Diablo-D3> gmaxwell: its obviously a bug and shouldnt happen
267 2012-10-28 16:30:21 <D34TH> normal
268 2012-10-28 16:30:30 <sipa> well what changed?
269 2012-10-28 16:30:45 <Diablo-D3> D34TH: forgot to clear your file cache first?
270 2012-10-28 16:30:57 <D34TH> oct 25's commits are all ive pulled
271 2012-10-28 16:31:09 <sipa> D34TH: and what code where you on before that?
272 2012-10-28 16:31:22 <D34TH> 62e21fb5d0
273 2012-10-28 16:31:26 <D34TH> oct 24
274 2012-10-28 16:33:36 <D34TH> almost fully imported already
275 2012-10-28 16:33:55 <sipa> how many blocks on how much time?
276 2012-10-28 16:34:24 <D34TH> 146032 in roughly 5 minutes
277 2012-10-28 16:34:53 <sipa> looks normal to me
278 2012-10-28 16:34:59 <D34TH> quite faster for me
279 2012-10-28 16:35:04 <sipa> the slow part is yet to come
280 2012-10-28 16:35:13 <D34TH> and my loadavg isnt being raped
281 2012-10-28 16:41:23 <gmaxwell> CPU usage will go up when you go over the highest checkpoint.
282 2012-10-28 16:41:46 <D34TH> for verifying right?
283 2012-10-28 16:41:55 <gmaxwell> For ECDSA.
284 2012-10-28 16:41:59 <D34TH> whats the highest checkpoint?
285 2012-10-28 16:42:09 <D34TH> ill let you know the results
286 2012-10-28 16:42:19 <sipa> 193k
287 2012-10-28 16:42:32 <D34TH> k, shouldnt take long im @ 180k
288 2012-10-28 16:42:42 <sipa> you'll likely spend more time on 193k-now than on everything before 193k
289 2012-10-28 16:52:04 <sipa> gmaxwell: not sure what you mean; i wasn't comparing LTO to non-LTO
290 2012-10-28 16:52:16 <sipa> only optization levels within lto
291 2012-10-28 16:52:39 <D34TH> gmaxwell, ive actually seen a small decrease ~2%
292 2012-10-28 16:52:54 <D34TH> but what is rev*.dat
293 2012-10-28 16:53:17 <sipa> D34TH: undo data for blocks
294 2012-10-28 16:53:25 <D34TH> ahh
295 2012-10-28 16:53:37 <sipa> it's only needed to reorganize away from a block
296 2012-10-28 17:16:23 <jgarzik> <gmaxwell> 09:06 < Diablo-D3> also, gcc doesnt support multiple -O levels across objects, shit can go bad
297 2012-10-28 17:16:25 <jgarzik> 100% not true
298 2012-10-28 17:17:04 <sipa> well i have seen cases were trying to mix object files compiled with and without -flto causes a link failures
299 2012-10-28 17:17:05 <jgarzik> sipa: there is definitely optimization that occurs at compile time, with -flto
300 2012-10-28 17:17:17 <jgarzik> sipa: oh definitely, but that is separate from -O
301 2012-10-28 17:18:04 <sipa> jgarzik: sure, for the object code in the .o files, but i didn't know the compilation-time -O flag influenced the intermediate format code in the .o files
302 2012-10-28 17:18:45 <jgarzik> sipa: yes, it influences that too
303 2012-10-28 17:18:52 <sipa> well, apparently :)
304 2012-10-28 17:18:55 <jgarzik> sipa: simple constant folding, loop opts and other stuff
305 2012-10-28 17:19:01 <sipa> right
306 2012-10-28 17:19:06 <sipa> early optimizations
307 2012-10-28 17:19:16 <jgarzik> yep
308 2012-10-28 17:20:26 <jgarzik> TD: (resend earlier message) JFYI, smartcoin (pybond) is idling, while I poke at picocoin
309 2012-10-28 17:20:32 <TD> what is picocoin?
310 2012-10-28 17:20:42 <sipa> yet another C implementation of Bitcoin
311 2012-10-28 17:20:46 <jgarzik> TD: SPV client, written in C
312 2012-10-28 17:20:58 <TD> ok
313 2012-10-28 17:21:07 <TD> if you want you can compile bitcoinj to native using GCJ
314 2012-10-28 17:21:22 <sipa> ACTION thinks jgarzik prefers writing C over reading Java
315 2012-10-28 17:21:38 <jgarzik> hehehe a fair assumption
316 2012-10-28 17:22:05 <gmaxwell> TD: though that still won't result in something with very minimal memory usage and such.
317 2012-10-28 17:22:44 <TD> i'm sure you can make it use a bit less memory than any java codebase, but i doubt it'll make so much difference that it matters in any real use case
318 2012-10-28 17:22:56 <TD> but go ahead. just remember that it's a lot of work :)
319 2012-10-28 17:24:15 <sipa> \\o/ a 15th .onion node
320 2012-10-28 17:25:08 <sipa> 11 reachable ones
321 2012-10-28 17:27:09 <jgarzik> ACTION would like to figure out how to do an AES decryption stream, rather than (1) read whole file, (2) decrypt entire memory buffer to another memory buffer, (3) read plaintext memory buffer
322 2012-10-28 17:27:35 <sipa> if you use stream encryption, that is slightly easier
323 2012-10-28 17:27:53 <sipa> but even per-block it's not very hard to implement CBC or so
324 2012-10-28 17:30:09 <gmaxwell> or OFB mode instead of a stream cipher.
325 2012-10-28 17:35:44 <sipa> brainstorm: how should database upgrade/copy/move work in bitcoind?
326 2012-10-28 17:36:31 <jgarzik> sipa: ideal world? start, note "upgrading database", reindex. progress bar for UI.
327 2012-10-28 17:37:10 <jgarzik> RE stream, I'm using AES-CBC right now
328 2012-10-28 17:37:12 <gmaxwell> Refuse to start until someone deletes the old data or gives it -upgrade option.
329 2012-10-28 17:37:28 <jgarzik> just the API is whole-region, and I need to drill down and figure out how to iterate
330 2012-10-28 17:37:36 <sipa> in bitcoin-qt, i'd say check the size of blk000?.dat vs blocks/blk0000?.dat, and if larger, show a dialog that asks "Do you want to (a) import the earlier database, but leave it intact, or (b) move the old one to the new", with (b) greyed out if some blocks/* files already exist
331 2012-10-28 17:38:52 <sipa> oh, and (c) skip import and download from scratch
332 2012-10-28 17:39:31 <gmaxwell> why bother offering C? I also thought the question was really "leave the old data behind, allowing downgrades, or recover space?"
333 2012-10-28 17:40:04 <sipa> well, if there already are existing new files, you can't really offer (b)
334 2012-10-28 17:40:18 <sipa> so the only option becomes "hey, we're going to import your old data!"
335 2012-10-28 17:41:16 <sipa> i suppose there will be cases where the old database is still larger, but already imported
336 2012-10-28 17:41:24 <sipa> so you want to prevent asking over and over again
337 2012-10-28 17:41:54 <gmaxwell> yea, quite a few people have dropped in blk files downloaded without an index or just deleted their index at some point.
338 2012-10-28 17:42:04 <jgarzik> strangely enough.... I think UI experience should delete old data. bitcoind should leave old data intact.
339 2012-10-28 17:42:09 <sipa> maybe don't ask anything if the new datafiles already exists
340 2012-10-28 17:42:30 <jgarzik> high level UI users should just rely on "database details" being wholly internal, and out of their knowledge. "upgrading... [progress bar]" is sufficient. just delete old data, for UI users.
341 2012-10-28 17:42:33 <sipa> a tools -> import data menu entry could exist
342 2012-10-28 17:42:39 <jgarzik> bitcoind users are more knowledgeable (in theory)
343 2012-10-28 17:42:47 <jgarzik> so do minimal upgrade, leaving files intact, for bitcoind
344 2012-10-28 17:43:42 <sipa> sounds reasonable
345 2012-10-28 17:43:43 <jrmithdobbs> jgarzik: I agree.
346 2012-10-28 17:44:15 <sipa> though the UI should still show a "we're upgrading your database, may take a while!" warning
347 2012-10-28 17:44:30 <jgarzik> yes
348 2012-10-28 17:44:34 <jgarzik> though ideally, a progress bar
349 2012-10-28 17:44:42 <jrmithdobbs> sipa: maybe even a "HOLD UP BUDDY IF YOU PROCEDE YOU CAN'T RUN OLDER VERSIONS" warning
350 2012-10-28 17:44:52 <gmaxwell> well, because it will take _an hour_ there ought to be a [uhh. quit, I'll do this later].
351 2012-10-28 17:44:59 <sipa> agree
352 2012-10-28 17:45:00 <gmaxwell> otherwise you'll get people killing it.
353 2012-10-28 17:45:03 <jgarzik> true
354 2012-10-28 17:45:12 <sipa> jgarzik: there is a progress bar, the normal IBD bar
355 2012-10-28 17:45:17 <jgarzik> ok
356 2012-10-28 17:45:24 <sipa> (i've even shown it at the conference!)
357 2012-10-28 17:45:27 <gmaxwell> jrmithdobbs: well you can but it'll be a full chain download.
358 2012-10-28 17:45:43 <jrmithdobbs> i'm amazed at how many people are using -onlynet=tor or at least have tor connected nodes btw
359 2012-10-28 17:45:55 <jgarzik> so a dialog: "About to begin required database upgrade. This may take > 1 hour. Proceed? OK / Cancel"
360 2012-10-28 17:45:56 <jgarzik> ?
361 2012-10-28 17:46:02 <jgarzik> then the normal IBD progress bar
362 2012-10-28 17:46:23 <sipa> jrmithdobbs: we need a ton more reachable onion nodes though
363 2012-10-28 17:46:43 <sipa> my vps gets more incoming onion connections than there are publicly reachable onion nodes
364 2012-10-28 17:46:54 <jrmithdobbs> sipa: i can tell. one of mine has >128 and my laptop i just turned on to sync last night has 32 right now
365 2012-10-28 17:47:26 <jrmithdobbs> (both have onions, so doing what i can ;p)
366 2012-10-28 17:47:51 <gmaxwell> hm. we could make it much faster if we turned off ecdsa closer to old best block. bleh. I feel uneasy about that but it would be a big speedup.
367 2012-10-28 17:48:00 <jrmithdobbs> sipa: the laptop isn't using upnp or v6, all inbounds are to the onions
368 2012-10-28 17:48:03 <gmaxwell> (e.g. for the import from your own data)
369 2012-10-28 17:48:19 <jrmithdobbs> s/onions/onion/
370 2012-10-28 17:48:25 <sipa> gmaxwell: though about that yes, but that requires BDB code to see how far the old node has things verified already
371 2012-10-28 17:48:52 <sipa> as there is a chance he got stuck, and got fed an invalid chain while being stuck
372 2012-10-28 17:48:53 <gmaxwell> sipa: here is a hack: check the wallet.
373 2012-10-28 17:49:02 <sipa> gmaxwell: ha!
374 2012-10-28 17:49:22 <sipa> also, ewww!
375 2012-10-28 17:49:27 <jgarzik> hehehe
376 2012-10-28 17:49:29 <gmaxwell> hey, I said a hack.
377 2012-10-28 17:50:57 <jgarzik> ACTION nyah nyah's that his SPV client does a better job of seeking peers than that other, java-based client ;p
378 2012-10-28 17:52:34 <gmaxwell> sipa: I only suggest such awfulness because it's probably 3/4 of the loadblock time.
379 2012-10-28 17:59:37 <gmaxwell> sipa: another option would be to sync all the way with ecdsa disabled, then rewind N blocks (e.g. N=1000) and run again with the validation running.
380 2012-10-28 18:00:44 <gmaxwell> probably as much code as reading the old index to get the best block... but it would mean a big reorg would be widely tested.
381 2012-10-28 18:01:34 <sipa> meh for coding effort
382 2012-10-28 18:19:59 <sipa> gmaxwell: can you re-test #1943 ?
383 2012-10-28 18:24:33 <Guest17788> are hidden nodes announced just by adding them to the list of .onions on the wiki?
384 2012-10-28 18:24:43 <sipa> they announce themself
385 2012-10-28 18:24:56 <sipa> at least if you use -externalip
386 2012-10-28 18:26:09 <Guest17788> sipa: thanks, so i can just set externalip to xxxxx.onion and be done with it? :)
387 2012-10-28 18:26:12 <Guest17788> excellent
388 2012-10-28 18:26:31 <sipa> Guest17788: read doc/Tor.txt
389 2012-10-28 18:26:49 <Guest17788> thanks
390 2012-10-28 18:30:24 <Zarutian> ACTION was going to add that you also need to configure a hidden service to forward xxxx.onion connections to the listening port of the bitcoin node but sees that Guest17788 already left.
391 2012-10-28 18:32:55 <sipa> gmaxwell: first 193k blocks reindexed in 14m17s; in the following 14m17s, 2686 blocks
392 2012-10-28 18:33:19 <sipa> so some hack to prevent re-validating of scripts may well be worth it...
393 2012-10-28 18:38:53 <gmaxwell> Zarutian: the doc at least gives everything he needs to know.
394 2012-10-28 18:44:07 <djoot> does bitcoind only announce one ip at a time? I'm guessing I only announce my .onion now with externalip...
395 2012-10-28 18:44:15 <D34TH> abe is all funky now
396 2012-10-28 18:44:16 <D34TH> D:
397 2012-10-28 18:45:08 <sipa> djoot: you can specify multiple -externalip's (or use it in combination with -discover)... it will use smart guessing to decide which ip to give to who
398 2012-10-28 18:45:20 <djoot> tyty
399 2012-10-28 18:49:17 <gmaxwell> Who runs weusecoins? it's linking directly to 0.7.0 (and we're still getting significant downloads of it)
400 2012-10-28 18:49:27 <sipa> justmoon, iirc
401 2012-10-28 18:58:29 <Guest95155> deeplinking asshole
402 2012-10-28 19:00:15 <jrmithdobbs> gmaxwell: what's fixed in .7.1 anways?
403 2012-10-28 19:00:40 <sipa> wallet recovery was added, and some bugfixes
404 2012-10-28 19:01:02 <gmaxwell> It's not super duper urgent. Some anti DOS fixes too IIRC...
405 2012-10-28 19:45:07 <Diapolo> sipa: is there any maximum size for undofiles?
406 2012-10-28 19:45:17 <sipa> not really
407 2012-10-28 19:45:30 <Diapolo> normal block files is 128 MiB, but there is at least no separate limit in the code for undofiles it seems
408 2012-10-28 19:45:57 <sipa> no, undo data goes to rev0000N.dat for blocks whose data is in blk0000N.dat
409 2012-10-28 19:46:13 <sipa> so you can delete both at once, when pruning is implemented
410 2012-10-28 19:46:58 <Diapolo> I just need to know for an efficient pre-alloc alog on Windows, what the max size of a rev0000x.dat file can be.
411 2012-10-28 19:47:04 <Diapolo> algo not alog
412 2012-10-28 19:47:11 <sipa> why do you need to know?
413 2012-10-28 19:47:33 <Diapolo> because I can't alloc in chunks as that leads to a non-contingous file
414 2012-10-28 19:47:46 <sipa> *sigh*
415 2012-10-28 19:48:02 <Diapolo> the ideal block-file should be 1 fragment for optimum performance IMO
416 2012-10-28 19:48:05 <sipa> non-continuous files isn't a problem for random access
417 2012-10-28 19:48:17 <sipa> as long as there is no excessive fragmentations
418 2012-10-28 19:48:28 <Diapolo> right, but for linear reads at least it matters
419 2012-10-28 19:48:34 <jgarzik> undo is generated all at once, at connect-block time, yes?
420 2012-10-28 19:48:39 <jgarzik> seems unlikely to be fragmented
421 2012-10-28 19:48:43 <Diapolo> and I know how bad it was pre Ultraprune ^^
422 2012-10-28 19:48:43 <sipa> we never do linear reads
423 2012-10-28 19:48:52 <sipa> undo files are allocated in batches of 1 MiB
424 2012-10-28 19:48:57 <sipa> that should be plenty
425 2012-10-28 19:48:59 <jgarzik> indeed
426 2012-10-28 19:49:01 <Diapolo> in terms of number of fragments of the block files
427 2012-10-28 19:49:14 <sipa> and block files in batches of 16 MiB
428 2012-10-28 19:49:26 <sipa> at some point we may want to make that configurable if it seems sub-optimal
429 2012-10-28 19:49:36 <Diapolo> you seem to miss that Windows can't pre-alloc the way you do it ... at least it seems non-beneficial
430 2012-10-28 19:49:42 <sipa> but please, just implement AllocateFileRange
431 2012-10-28 19:50:01 <sipa> i don't care what dirty windows-specific tricks you use for that
432 2012-10-28 19:50:20 <sipa> but if there's a problem with the allocation policy, that should be fixed in general, but i don't expect problems
433 2012-10-28 19:51:07 <jgarzik> sipa: current AllocateFileRange doesn't pre-allocate actual blocks
434 2012-10-28 19:51:07 <sipa> and i'm very sure that writing zeroes has exactly the same effect as calling a specific prealloc function, except for the fact that nothing is actually written
435 2012-10-28 19:51:14 <sipa> jgarzik: huh?
436 2012-10-28 19:51:21 <jgarzik> sipa: it creates a sparse file
437 2012-10-28 19:51:31 <sipa> writing zeroes doesn't cause a sparse file?
438 2012-10-28 19:51:42 <Diapolo> your code has no errors and works, but I'm only telling it leads to fragmented files
439 2012-10-28 19:51:56 <jgarzik> sipa: oh, nevermind, it's fine. I misread "1" as writing a byte, not a block.
440 2012-10-28 19:51:57 <sipa> Diapolo: how many fragments do you see?
441 2012-10-28 19:52:44 <jgarzik> not sure why AllocateFileRange() would cause problems on Windows
442 2012-10-28 19:52:49 <jgarzik> need numbers not "seems"
443 2012-10-28 19:52:54 <sipa> indeed
444 2012-10-28 19:53:43 <sipa> Diapolo: if you can show me the current code leads to excessive fragmentation (and not just more than 1 fragment) on a not-close-to-full filesystem om windows, we'll discuss improvements
445 2012-10-28 19:54:00 <gmaxwell> Diapolo: you really shouldn't be adding any preallocation complexity for undo files. I will NAK changes that do unless they come with concrete performance measurements that show must-be-visible speedups.
446 2012-10-28 19:54:27 <Diapolo> I have no original Ultaprune file anymore, as I was playing around with own code, but the files had > 1 fragment. Perhaps our goal is different here, I tried to create a patch that leads to a 1 fragment file, which should be optimal, what is your goal?
447 2012-10-28 19:55:00 <jgarzik> Diapolo: as long as the fragment size is sufficiently large, who cares?
448 2012-10-28 19:55:05 <Diapolo> gmaxwell: I said I'm playing around and I'm fine with NAKing changs to undo files :-P.
449 2012-10-28 19:55:15 <gmaxwell> Diapolo: I don't care how many fragments the file has, unless its some insane number. We randomly access the files so small numbers will not result in performance differences.
450 2012-10-28 19:55:34 <sipa> Diapolo: also, in ultraprune, block files are almost never accessed at all
451 2012-10-28 19:55:45 <jgarzik> undo files even more so
452 2012-10-28 19:55:49 <sipa> indeed
453 2012-10-28 19:56:16 <Diapolo> so, what is the purpose of the chunks then anyway?
454 2012-10-28 19:56:26 <sipa> to prevent *excessive* fragmentation