1 2012-12-19 00:08:29 <stealth222> donnchac, I have built a class which connects to a bitcoind node and provides onTx and onBlock handlers
2 2012-12-19 00:08:33 <stealth222> but I haven't published it
3 2012-12-19 00:08:38 <stealth222> thinking about publishing it
4 2012-12-19 00:09:43 <sipa> i think such code already exists
5 2012-12-19 00:09:44 <gmaxwell> again. an _ontx_ callback does not do what you need to accomplish to safely keep an application txn database in sync.
6 2012-12-19 00:09:55 <stealth222> I'm sure I'm not the only one who has written something like that, sipa
7 2012-12-19 00:10:04 <stealth222> there's really a desperate need for that functionality :)
8 2012-12-19 00:10:28 <donnchac> That would be very helpfully! I think it would really help developers implement Bitcoin more directly in their apps and not have to rely on 3rd parties like BitPay etc. to do the heavy lifting
9 2012-12-19 00:10:30 <gmaxwell> stealth222: we have onblock built into the reference client.
10 2012-12-19 00:11:10 <gmaxwell> And onblock is sufficient, callback wise, to keep an application database insync.
11 2012-12-19 00:11:15 <sipa> woah... we don't have any unit tests for CreateTransaction??
12 2012-12-19 00:11:24 <donnchac> The whole JSON-RPC doesn't seem to cut it for this kind of functionality if the bitcoind "accounts" functionality is non scalable.
13 2012-12-19 00:11:51 <stealth222> a couple problems I'm seeing with the reference client: 1) the callback executes a cmd which is both resource-inefficient as well as a potential security vulnerability. 2) the reference implementation does not have an indexed database that can be quickly queried to obtain all relevant information for arbitrary transactions quickly
14 2012-12-19 00:11:56 <gmaxwell> donnchac: your problem isn't that accounts isn't scalable, its that it actually doesn't do what you want.
15 2012-12-19 00:12:54 <gmaxwell> stealth222: "resource-inefficient" for something that happens once per ten minutes on average?
16 2012-12-19 00:12:54 <stealth222> what exactly are you trying to do, donnchac?
17 2012-12-19 00:13:10 <stealth222> for onblock, it's not a problem, gmaxwell - for ontx, it would be
18 2012-12-19 00:13:33 <gmaxwell> stealth222: yes, I agree, but for what donnchac is wanting to do ontx is not what he wants.
19 2012-12-19 00:13:47 <sipa> stealth222: you can quickly query any transaction you want; just not by address
20 2012-12-19 00:14:06 <stealth222> precisely
21 2012-12-19 00:14:26 <sipa> and even by txid is not a very nice solution, imho - as it requires an ever-growing database
22 2012-12-19 00:14:31 <gmaxwell> stealth222: ... how is that going to help someone running a wallet service?
23 2012-12-19 00:14:39 <donnchac> So I am looking to develop a freelancing site where people can pay with bitcoin. Users would send bitcoins to the web app and they would be stored linked to their account. When they agree on a deal those bitcoins would be locked in an "escrow" and released to the the freelancer once deliverables have been provided
24 2012-12-19 00:15:02 <gmaxwell> stealth222: you very rapidly have many more addresses than there are txn in a block, so the sane thing to do is to grab the block transactions.
25 2012-12-19 00:15:06 <stealth222> the wallet can be kept completely offline, gmaxwell - only the public keys are kept in a database against which incoming transactions are searched
26 2012-12-19 00:15:15 <sipa> i think the solution is a nice an easy wallet library, which works as an SPV client, and you can connect to a trusted validation node
27 2012-12-19 00:15:17 <donnchac> Just looking for what is the best practice to update accounts and keep everything in sync with the actual data from bitcoind
28 2012-12-19 00:15:28 <stealth222> you can then alert applications of when payments are received (at 0-n confirmations)
29 2012-12-19 00:16:03 <gmaxwell> stealth222: because you will have many more addresses than there are txn in a block it's more useful to dump whats in the block instead of query by address...
30 2012-12-19 00:16:19 <gmaxwell> esp since you're only interested in scanning the blocks that have changed since you last scanned.
31 2012-12-19 00:16:36 <gmaxwell> Unless you expect something to have a K-d tree index of addresses and height???
32 2012-12-19 00:17:05 <stealth222> I guess onblock might be sufficient if you're not interested in tracking the network in realtime
33 2012-12-19 00:17:12 <stealth222> and only care about confirmed transactions
34 2012-12-19 00:17:23 <sipa> we do need support for watch-only addresses though
35 2012-12-19 00:17:52 <gmaxwell> stealth222: unconfirmed transactions are quite dangerous... lots of subtle risks.
36 2012-12-19 00:18:02 <donnchac> It might be, I would just need to be careful to manage reorgs in the block chain.
37 2012-12-19 00:18:23 <stealth222> there are many online services where accepting unconfirmed transactions is perfectly fine - for instance, paying for a VPS
38 2012-12-19 00:18:37 <stealth222> if it gets doublespent, kill the guy's account
39 2012-12-19 00:18:38 <stealth222> simple
40 2012-12-19 00:18:59 <gmaxwell> donnchac: there isn't any technical need in bitcoin to place funds with a third party to escrow them?????doing so has big trust issues, and potentially legal and regulatory issues for the holder.
41 2012-12-19 00:19:24 <gmaxwell> stealth222: yea, good job there... we just spent an hour talking about how anyone can 'doublespend' someone elses transactions. :P
42 2012-12-19 00:19:38 <gmaxwell> Thats what I mean about subtle risks.
43 2012-12-19 00:19:47 <donnchac> How would you prepose exchanging the coins in a secure manner?
44 2012-12-19 00:20:06 <donnchac> without a trusted 3rd party holding them in escrow?
45 2012-12-19 00:20:10 <stealth222> if someone changes the signature but the transaction is still valid and the outputs are the same, the merchant still gets paid
46 2012-12-19 00:20:14 <gmaxwell> donnchac: Users can author escrow transactions.
47 2012-12-19 00:20:27 <gmaxwell> stealth222: You're not following me.
48 2012-12-19 00:20:46 <stealth222> I guess I'm not. could you elaborate?
49 2012-12-19 00:20:56 <donnchac> So would the best solution for me be to log pending/unconfirmed transactions and only accept and confirm them once I have recieved say, 6 subsequent blocks on that chain?
50 2012-12-19 00:21:02 <gmaxwell> stealth222: if you'd had this fantastic idea three hours ago you would have implimented the detection that was triggered by a different txid most likely because you weren't aware of the malleability.
51 2012-12-19 00:21:28 <stealth222> I wasn't saying kill the guy's account literally
52 2012-12-19 00:21:32 <stealth222> it's more like a credit relationship
53 2012-12-19 00:21:57 <stealth222> moreover, the maleability has a simple workaround in this case
54 2012-12-19 00:22:22 <gmaxwell> but yes, in some cases it can be fine??? but people need to be very cautious. Personally I would use mempool polling for unconfirmed transactions instead of trying to pick them up from the network.
55 2012-12-19 00:22:53 <stealth222> when I send the bitcoins to a service provider, I would like to see my invoice updated to "payment pending" at the very least
56 2012-12-19 00:22:59 <gmaxwell> Less likely to get tripped up by transactons that are very unlikely to confirm, and you can also see when they've left the mempool.
57 2012-12-19 00:23:27 <stealth222> moreover, the merchant might want to send an automated email telling the customer that the payment has been received and is pending confirmation
58 2012-12-19 00:23:46 <gmaxwell> ::nods::
59 2012-12-19 00:24:27 <gmaxwell> Never said there wasn't a use for it??? but it doesn't do much of anything useful for keeping a transaction database in sync.
60 2012-12-19 00:24:55 <donnchac> Okay, I was hoping there would be a safe way to implement near realtime feedback and tracking of transfers.
61 2012-12-19 00:24:57 <jgarzik> sipa: yes, data-driven unit tests for create-transaction would be nice, especially
62 2012-12-19 00:25:16 <donnchac> I assumed this is the kind of functionality that would already have been developed by someone.
63 2012-12-19 00:26:26 <gmaxwell> donnchac: it's been implemented by many parties, in their own money making services.
64 2012-12-19 00:26:42 <stealth222> it's such a useful feature it should almost be standard :)
65 2012-12-19 00:26:48 <gmaxwell> There isn't just some call that could be done to bitcoin, though??? it really has to be part of the application.
66 2012-12-19 00:28:01 <donnchac> I understand that, but I thought that bitcoind would expose more than just a block-notify callback and that there would be at least some sample implementation of this kind of polling and transaction tracking publicily/open source
67 2012-12-19 00:28:39 <stealth222> donnchac: if you're interested in using my library, PM me
68 2012-12-19 00:28:52 <stealth222> that goes for anyone else
69 2012-12-19 00:29:54 <gmaxwell> What do you expect it to expose? Ignoring unconfirmed transactions, blocknotify/getblock is all that bitcoind should need to have for a tracking app. I agree its unfortunate that there are no open users of it.
70 2012-12-19 00:33:01 <donnchac> I suppose it would be very convienient if bitcoind could be configured to make a callback after a particular number of confirmations of a transaction in its wallet.
71 2012-12-19 00:33:07 <gmaxwell> sipa: hm. kind of a bummer that in current git you can't <getblock {penultimate block}> and getrawtransaction each result anymore.
72 2012-12-19 00:33:27 <gmaxwell> donnchac: blocknotify, listsinceblock
73 2012-12-19 00:33:33 <sipa> gmaxwell: feel free to implement a txid index :)
74 2012-12-19 00:33:43 <gmaxwell> sipa: doesn't need one for that case!
75 2012-12-19 00:33:57 <gmaxwell> sipa: e.g. getblocktransaction <blockhash> <txid>
76 2012-12-19 00:34:26 <donnchac> for example have it configured for 1,6 confirmations. Call back /tx_recv?id=[TX_ID]&confirm=6
77 2012-12-19 00:34:29 <sipa> gmaxwell: yup, with the risk of applications go depend on that, with slow fetching and complains :)
78 2012-12-19 00:34:48 <gmaxwell> donnchac: thats almost that exactly that??? you'd just call it every block and it's empty if there is nothing new.
79 2012-12-19 00:34:54 <donnchac> But I agree a lot of that is probably better implemented in individual applications
80 2012-12-19 00:35:41 <donnchac> Okay :) Thanks for all the advice. I'll try read threw the docs some more with these starting points
81 2012-12-19 00:35:47 <gmaxwell> donnchac: Good luck!
82 2012-12-19 00:36:03 <donnchac> Much appreciate the help guys.
83 2012-12-19 00:36:07 <gmaxwell> donnchac: please do report back what you learn, I spent a bunch of time saying "no don't do that" but your feedback is really helpful.
84 2012-12-19 00:37:31 <donnchac> Will do!
85 2012-12-19 00:44:41 <wizkid057> ok, finally getting around to seeing how long a full node setup takes from scratch using git head
86 2012-12-19 00:45:34 <sipa> wizkid057: use -connect or -loadblock
87 2012-12-19 00:45:48 <wizkid057> cheating
88 2012-12-19 00:45:58 <sipa> well, yes
89 2012-12-19 00:46:00 <gmaxwell> no, not unless you want to make 50 measurments.
90 2012-12-19 00:46:11 <sipa> but we know the block fetching mechanism is a hack that barely works anymore
91 2012-12-19 00:46:34 <wizkid057> ACTION shrugs
92 2012-12-19 00:46:37 <gmaxwell> ??? without connect to a good node (or loadblock) you're luck of the draw. so the time could span many hours extra.
93 2012-12-19 00:46:40 <sipa> and in many situation, on git head it's the bottleneck
94 2012-12-19 00:46:52 <wizkid057> well, I added my local node and Eligius as addnodes
95 2012-12-19 00:46:57 <sipa> won't help
96 2012-12-19 00:47:08 <sipa> or at least, won't necessarily help
97 2012-12-19 00:47:12 <gmaxwell> e.g. if you spend most of your time pulling from some satcom connected node in south africa.
98 2012-12-19 00:47:21 <wizkid057> well, thats as much as I'm going to cheat
99 2012-12-19 00:47:45 <gmaxwell> wizkid057: it's fine, but your result may not represent anything in particular.
100 2012-12-19 00:48:08 <wizkid057> i'm mainly curious as to the block processing times post-satoshidice
101 2012-12-19 00:48:17 <gmaxwell> sure, but you won't be measuring that.
102 2012-12-19 00:48:31 <gmaxwell> You'll be measuring whichever random node you're pulling from most likely.
103 2012-12-19 00:48:34 <wizkid057> well, i can compare CPU usage
104 2012-12-19 00:48:51 <sipa> if you only receive one block every 5 seconds, CPU usage will be low
105 2012-12-19 00:48:53 <wizkid057> and everyone has been saying forever now that the download isnt the bottleneck
106 2012-12-19 00:48:59 <wizkid057> sipa: cpu time them
107 2012-12-19 00:49:01 <wizkid057> *then
108 2012-12-19 00:49:06 <gmaxwell> wizkid057: yes, it is now in githead.
109 2012-12-19 00:49:39 <sipa> and certainly with parallel sigcheck enabled
110 2012-12-19 00:49:46 <sipa> (which isn't in git head yet)
111 2012-12-19 00:49:57 <gmaxwell> It was disk IO, and we fixed that, and now it's often your peer's responsiveness (latency as much as throughput) and if not that cpu.
112 2012-12-19 00:50:06 <jgarzik> sig cache impacts block processing times
113 2012-12-19 00:50:12 <sipa> jgarzik: hardly
114 2012-12-19 00:50:14 <gmaxwell> By "we" I mean mostly sipa.
115 2012-12-19 00:50:24 <jgarzik> sipa: greatly
116 2012-12-19 00:50:36 <gmaxwell> jgarzik: for new blocks not IBD.
117 2012-12-19 00:50:49 <sipa> oh, for new blocks it's day vs night
118 2012-12-19 00:50:52 <jgarzik> sipa: I often see greatly reduced checks thanks to caching
119 2012-12-19 00:50:56 <jgarzik> yeah
120 2012-12-19 00:50:56 <sipa> but for IBD it's hardly hurts
121 2012-12-19 00:51:02 <jgarzik> agreed
122 2012-12-19 00:51:02 <wizkid057> hmm
123 2012-12-19 00:51:17 <wizkid057> rpc not work until the chain is loaded now or something?
124 2012-12-19 00:51:27 <gmaxwell> if you want to see it scream grab the parallel sigchecking and halcode. pulls too.
125 2012-12-19 00:51:34 <sipa> wizkid057: always been that way
126 2012-12-19 00:51:44 <wizkid057> i used to be able to getinfo
127 2012-12-19 00:51:52 <sipa> not until the DB was loaded
128 2012-12-19 00:52:00 <gmaxwell> wizkid057: nah, it has to connect the db first.
129 2012-12-19 00:52:02 <wizkid057> i mean, while its downloading
130 2012-12-19 00:52:13 <sipa> oh, that should work fine
131 2012-12-19 00:52:16 <wizkid057> i'm watching it download blocks (on 52k already in 6 minutes)
132 2012-12-19 00:52:24 <sipa> slow!
133 2012-12-19 00:52:24 <wizkid057> in debug log
134 2012-12-19 00:52:26 <gmaxwell> wizkid057: works fine for me.
135 2012-12-19 00:52:30 <gmaxwell> yea, wow. thats slow.
136 2012-12-19 00:52:34 <wizkid057> 65k rather
137 2012-12-19 00:52:45 <sipa> i typically see 140k blocks in 3 minutes
138 2012-12-19 00:52:49 <sipa> from local disk, that is
139 2012-12-19 00:52:50 <wizkid057> lol
140 2012-12-19 00:52:56 <wizkid057> well ofc from local disk
141 2012-12-19 00:53:01 <wizkid057> thats a fake test :P
142 2012-12-19 00:53:27 <sipa> it's a very real test for measuring how long it will take for people to upgrade their nodes :)
143 2012-12-19 00:53:50 <gmaxwell> 12/17/12 03:28:27 SetBestChain: new best=00000000839a8e6886ab5951d76f411475428afc90947ee320161bbf18eb6048 height=1 work=8590065666 tx=2 date=01/09/09 02:54:25
144 2012-12-19 00:53:53 <gmaxwell> 12/17/12 03:29:27 SetBestChain: new best=0000000006502bbeba998efd13a8dfa7fb6f213da62143f5cddc4b709daa2cc2 height=65000 work=1266146729122629 tx=73412 date=07/09/10 03:
145 2012-12-19 00:53:53 <wizkid057> not a real test for when random user discovers bitcoin and installs the client
146 2012-12-19 00:54:16 <sipa> wizkid057: agree, but there'll be several changes before release of 0.8 anyway
147 2012-12-19 00:54:17 <gmaxwell> (that a over the network sync, but it was -connect to a known not-slow node)
148 2012-12-19 00:54:25 <wizkid057> hmm... yeah, and RPC is giving me only 403's...
149 2012-12-19 00:54:30 <wizkid057> with correct user/pass
150 2012-12-19 00:54:44 <sipa> sure it's correct?
151 2012-12-19 00:55:43 <gmaxwell> wizkid057: is 34ea321ccdb234fe695f6279e3d53cce31c28f4a killing you?
152 2012-12-19 00:56:06 <wizkid057> nope
153 2012-12-19 00:56:12 <wizkid057> rpcpassword is a keyboard mash
154 2012-12-19 00:57:12 <gmaxwell> right ports?
155 2012-12-19 00:57:35 <wizkid057> http://pastebin.com/6SPLF2ZD
156 2012-12-19 00:58:40 <wizkid057> i assume it should still use values from bitcoin.conf
157 2012-12-19 00:59:08 <gmaxwell> why are you giving it a datadir parameter in any case?
158 2012-12-19 00:59:12 <wizkid057> well
159 2012-12-19 00:59:17 <wizkid057> just to be sure
160 2012-12-19 00:59:24 <wizkid057> same result without it
161 2012-12-19 00:59:58 <sipa> should be the same, indeed
162 2012-12-19 01:00:02 <sipa> unless -testnet
163 2012-12-19 01:00:08 <wizkid057> nope
164 2012-12-19 01:00:20 <wizkid057> wizkid057@wizroute:~$ ps ux | grep bitcoind
165 2012-12-19 01:00:23 <wizkid057> 1000 24942 59.1 55.4 1207268 998576 ? SLsl 20:45 9:40 bitcoin/src/bitcoind -daemon
166 2012-12-19 01:00:57 <gmaxwell> is this a fresh keyboard mash?
167 2012-12-19 01:01:07 <wizkid057> passwd is 28 alphanumeric
168 2012-12-19 01:01:08 <gmaxwell> it has issues with some html specialchars in the rpcpassword.
169 2012-12-19 01:01:48 <wizkid057> well, not like I cant change it
170 2012-12-19 01:02:00 <wizkid057> rpcuser=wizrouteuser1
171 2012-12-19 01:03:23 <gmaxwell> looks great to me.
172 2012-12-19 01:04:04 <wizkid057> yeah, dunno why its 403'ing me
173 2012-12-19 01:04:07 <sipa> no other bitcoind listening?
174 2012-12-19 01:04:17 <wizkid057> nope, no bitcoind on this machine at all except this one
175 2012-12-19 01:04:27 <sipa> very strange
176 2012-12-19 01:04:47 <wizkid057> tcp 0 0 127.0.0.1:2833 0.0.0.0:* LISTEN 24942/bitcoind
177 2012-12-19 01:04:59 <sipa> looks perfect
178 2012-12-19 01:05:03 <wizkid057> correct process, correct port
179 2012-12-19 01:05:19 <gmaxwell> crazy iptables dnat rule? :P
180 2012-12-19 01:05:30 <wizkid057> not on that port
181 2012-12-19 01:06:19 <sipa> http://xkcd.com/341/ ?
182 2012-12-19 01:07:06 <wizkid057> haha
183 2012-12-19 01:08:18 <sipa> bitcoind.conf didn't change after you started bitcoind, right?
184 2012-12-19 01:08:27 <wizkid057> actually
185 2012-12-19 01:08:39 <wizkid057> gmaxwell is partially correct it seems
186 2012-12-19 01:11:15 <gmaxwell> ACTION never believes the (l)users for good reason! :P
187 2012-12-19 01:11:49 <wizkid057> gmaxwell: my iptables is insane complicated
188 2012-12-19 01:12:36 <wizkid057> somewhere in there, there is a module interpretting it as http traffic, sending it to my proxy, and nat'ing from my WAN interface IP to localhost on interface lo
189 2012-12-19 01:12:58 <wizkid057> which... is broken
190 2012-12-19 01:14:52 <wizkid057> root@wizroute:~# iptables-save | wc -l
191 2012-12-19 01:14:53 <wizkid057> 519
192 2012-12-19 01:15:58 <wizkid057> there we go
193 2012-12-19 01:16:21 <jgarzik> BTW, the current iptables gadgetry that compiles rules directly into machine instructions is pretty cool
194 2012-12-19 01:16:33 <jgarzik> </offtopic>
195 2012-12-19 01:16:38 <wizkid057> :D
196 2012-12-19 01:17:06 <wizkid057> admittedly, I had a buddy of mine setup my http catching stuff and such
197 2012-12-19 01:17:14 <wizkid057> so I don't fully understand it
198 2012-12-19 01:17:17 <wizkid057> but it works
199 2012-12-19 01:17:22 <wizkid057> ... usually
200 2012-12-19 01:17:23 <sipa> jgarzik: ooh, didn't know that
201 2012-12-19 01:17:36 <sipa> like varnish does :)
202 2012-12-19 01:18:31 <wizkid057> hmm
203 2012-12-19 01:18:41 <wizkid057> seems I have a lot of packets routing through the wrong interfaces
204 2012-12-19 01:18:44 <wizkid057> ACTION sighs
205 2012-12-19 01:24:03 <wizkid057> iptables fail :P
206 2012-12-19 01:25:09 <phantomcircuit> jgarzik, there's security issues with it related to using it
207 2012-12-19 01:25:42 <phantomcircuit> that was a fairly redundantly redundant sentence
208 2012-12-19 01:27:11 <gmaxwell> phantomcircuit: are you thinking of the BFP compiler being used to get executible code in kernel memory to bypass the additional kernel mode execute disable stuff in the lastest cpus?
209 2012-12-19 01:27:21 <phantomcircuit> yes
210 2012-12-19 01:27:29 <jgarzik> that's something different
211 2012-12-19 01:27:33 <gmaxwell> ^
212 2012-12-19 01:28:33 <phantomcircuit> ah
213 2012-12-19 01:28:38 <phantomcircuit> live and learn
214 2012-12-19 01:29:21 <sipa> no need for all that difficulty stuff; just use /dev/exynos-mem
215 2012-12-19 01:32:22 <wizkid057> 2012-12-19T01:45:40 SetBestChain: new best=00000000839a8e6886ab5951d76f411475428afc90947ee320161bbf18eb6048 height=1 work=8590065666 tx=2 date=2009-01-09T02:54:25
216 2012-12-19 01:32:24 <wizkid057> 2012-12-19T02:33:55 SetBestChain: new best=00000000000009e203e255c4140a95a0500325b40c17dac0982837d4cbfef5ed height=137153 work=51437118718226147970 tx=1083972 date=2011-07-20T07:29:53
217 2012-12-19 01:33:42 <gmaxwell> so.. about 45 minutes so far?
218 2012-12-19 01:33:51 <wizkid057> yeah, give or take
219 2012-12-19 01:34:01 <sipa> my guess is that 40 of those 45 where spent waiting for blocks
220 2012-12-19 01:34:05 <sipa> *were
221 2012-12-19 01:34:05 <stealth222> I just extended gettransaction <txid> to work for transactions that are not in your wallet. rpcwallet.cpp starting line 1193, http://pastebin.com/hZAw7VYa
222 2012-12-19 01:34:23 <stealth222> would it be possible to get this feature added to MAIN?
223 2012-12-19 01:34:26 <sipa> stealth222: you reimplemented getrawtransaction, you mean?
224 2012-12-19 01:34:32 <stealth222> oh...lol
225 2012-12-19 01:34:34 <wizkid057> well, i'll stop it and -connect my machine and see what happens (gigabit lan)
226 2012-12-19 01:34:41 <gmaxwell> behold, the suckyness of your peers:
227 2012-12-19 01:34:45 <gmaxwell> 12/17/12 03:32:32 SetBestChain: new best=00000000000009e203e255c4140a95a0500325b40c17dac0982837d4cbfef5ed height=137153 work=51437118718226147970 tx=1083972 date=07/20
228 2012-12-19 01:35:06 <stealth222> wait, does getrawtransaction give JSON?
229 2012-12-19 01:35:09 <sipa> yes
230 2012-12-19 01:35:11 <gmaxwell> yes.
231 2012-12-19 01:35:19 <gmaxwell> (with the verbose argument)
232 2012-12-19 01:35:19 <wizkid057> it does?
233 2012-12-19 01:35:30 <wizkid057> oh, wow
234 2012-12-19 01:35:40 <wizkid057> i've been doing getrawtransactions then decoderawtransaction :P
235 2012-12-19 01:35:46 <gmaxwell> we explicitly don't do that on getrawtransaction to avoid people getting screwed because they think if getrawtransaction returns the txn is theirs.
236 2012-12-19 01:36:01 <gmaxwell> er don't do that on gettransaction*
237 2012-12-19 01:36:36 <stealth222> I see. nevermind :)
238 2012-12-19 01:37:30 <stealth222> I'm so used to having my wallet be completely separate from my relay/block chain nodes
239 2012-12-19 01:37:37 <stealth222> that I sometimes forget how other people use bitcoin
240 2012-12-19 01:37:38 <stealth222> lol
241 2012-12-19 01:40:27 <wizkid057> going more quickly with -connect to my local machine
242 2012-12-19 01:40:38 <wizkid057> not gmaxwell quick, but, quicker
243 2012-12-19 01:41:23 <gmaxwell> wizkid057: if it's not as quick as my numbers above then thats interesting.
244 2012-12-19 01:41:34 <gmaxwell> especially considering thats far below the top checkpoint.
245 2012-12-19 01:43:08 <wizkid057> 2012-12-19T02:38:49 SetBestChain: new best=0000000000000044c7b6a5511c0b2ae64ec545abccac8053f31cf7bba23bb886 height=138000 work=57588416401850432161 tx=1130286 date=2011-07-25T18:17:58
246 2012-12-19 01:43:11 <wizkid057> 2012-12-19T02:44:30 SetBestChain: new best=0000000000000a3290f20e75860d505ce0e948a1d1d846bec7e39015d242884b height=150000 work=147961445995450882498 tx=1718407 date=2011-10-20T13:44:51
247 2012-12-19 01:43:36 <gmaxwell> 12/17/12 03:34:10 SetBestChain: new best=0000000000000a3290f20e75860d505ce0e948a1d1d846bec7e39015d242884b height=150000 work=147961445995450882498 tx=1718407 date=10/2
248 2012-12-19 01:43:56 <gmaxwell> hm. much slower. interesting.
249 2012-12-19 01:44:02 <gmaxwell> Tell me about this system?
250 2012-12-19 01:44:38 <sipa> Was it abused as a child?
251 2012-12-19 01:44:59 <wizkid057> its not a monster... Core2 Duo 2.6GHz w/2GB/ram
252 2012-12-19 01:45:05 <wizkid057> just my router machine
253 2012-12-19 01:45:29 <wizkid057> bitcoind is not maxing the CPU, though
254 2012-12-19 01:45:43 <gmaxwell> No, it won't until it gets to the signature validation above 210000.
255 2012-12-19 01:46:02 <gmaxwell> (well, and will only use one core unless you have the parallel validation pull request)
256 2012-12-19 01:46:10 <gmaxwell> wizkid057: whats the OS?
257 2012-12-19 01:46:58 <sipa> at least something with iptables :)
258 2012-12-19 01:46:58 <wizkid057> debian, kernel 2.6.32-5-amd64
259 2012-12-19 01:47:56 <sipa> disk?
260 2012-12-19 01:48:08 <wizkid057> some random drive, not sure
261 2012-12-19 01:48:09 <sipa> though that shouldn't matter much at this point
262 2012-12-19 01:48:17 <sipa> ok, no network or encrypted stuff
263 2012-12-19 01:48:50 <wizkid057> Model=WDC WD2500BEVT-60A23T0, FwRev=02.01A02, SerialNo=WD-WX61A80P0068
264 2012-12-19 01:49:09 <wizkid057> so, 250GB western digital SATA drive
265 2012-12-19 01:49:12 <sipa> won't be the problem :)
266 2012-12-19 01:49:22 <sipa> how much is the CPU usage?
267 2012-12-19 01:49:29 <gmaxwell> my numbers are on a fast E3-1230 with lots of ram and a fast ssd, so I expect faster but the difference is more than I'd expect.
268 2012-12-19 01:49:36 <wizkid057> sipa: ~20%
269 2012-12-19 01:49:54 <wizkid057> 20-30% really
270 2012-12-19 01:49:58 <sipa> 20% of one core, or of all cores?
271 2012-12-19 01:50:08 <gmaxwell> wizkid057: is this node listening?
272 2012-12-19 01:50:17 <sipa> with -connect it shouldn't be
273 2012-12-19 01:50:26 <sipa> unless enabled explicitly
274 2012-12-19 01:50:32 <wizkid057> gmaxwell: its probably trying, but, iptables wont let it
275 2012-12-19 01:50:42 <gmaxwell> wizkid057: so getinfo shows one connection?
276 2012-12-19 01:51:11 <wizkid057> yeah
277 2012-12-19 01:52:07 <Kiba> hello
278 2012-12-19 01:52:16 <wizkid057> sipa: looks like its using avg of between 20-30% of one core, spikes to 100% for short moments, then back
279 2012-12-19 01:52:20 <Kiba> bitcoin-qt is like never ever syncing
280 2012-12-19 01:52:39 <Kiba> I got 8 peers connected
281 2012-12-19 01:52:42 <wizkid057> ~400MB of ram
282 2012-12-19 01:52:58 <gmaxwell> Kiba: what is your current block height?
283 2012-12-19 01:53:07 <gmaxwell> ;;bc,blocks
284 2012-12-19 01:53:07 <gribble> 212402
285 2012-12-19 01:53:23 <Kiba> and Warning: Displayed Transaction may not be correct! You may need to upgrade, or other nodes may need to upgrade.
286 2012-12-19 01:53:41 <sipa> yet another corrupted block chain index :'(
287 2012-12-19 01:53:42 <wizkid057> blockexplorer.com has been incorrect for days, btw
288 2012-12-19 01:53:47 <sipa> wizkid057: we know
289 2012-12-19 01:53:51 <Kiba> 209859
290 2012-12-19 01:54:03 <wizkid057> 212729
291 2012-12-19 01:54:09 <gmaxwell> Kiba: search your debug.log for Invalid
292 2012-12-19 01:54:17 <Kiba> yes, sir
293 2012-12-19 01:54:38 <Kiba> InvalidChainFound
294 2012-12-19 01:54:48 <wizkid057> hmm... what was the satoshidice starting height?
295 2012-12-19 01:54:59 <wizkid057> was in the upper 180k wasnt it?
296 2012-12-19 01:55:14 <gmaxwell> Kiba: go find the earliest instance of that, and pastebin a dozen lines before and after or so.
297 2012-12-19 01:55:35 <sipa> wizkid057: i think so
298 2012-12-19 01:55:36 <wizkid057> hmm.. maybe before then
299 2012-12-19 01:55:42 <gmaxwell> sipa: I wish I understood why BDB seems to have gone from being pretty reliable to not being reliable at all for many people.
300 2012-12-19 01:56:18 <sipa> gmaxwell: indeed; i myself never encountered any problems, ever (except when the locks ran out, or some easily identifable problems like deleting database/)
301 2012-12-19 01:56:31 <wizkid057> sipa: looks like 176781 has the first SD txn
302 2012-12-19 01:56:49 <sipa> wizkid057: at how many transactions are you now?
303 2012-12-19 01:57:03 <wizkid057> 2012-12-19T02:58:36 SetBestChain: new best=000000000000043f3db8ff44c587e4ebabb298a9ea978c11133416e464fd193d height=177391 work=305347815834450590288 tx=2926205 date=2012-04-27T04:33:53
304 2012-12-19 01:57:28 <Kiba> gmaxwell: mkay
305 2012-12-19 01:57:43 <stealth222> theymos's node died?
306 2012-12-19 01:57:57 <jgarzik> I always wondered if BDB's problems were largely on Windows
307 2012-12-19 01:58:15 <stealth222> BDB looks ugly on any platform :p
308 2012-12-19 01:58:46 <wizkid057> you know, i've always been curious
309 2012-12-19 01:58:51 <Kiba> http://pastie.org/5550356
310 2012-12-19 01:58:54 <stealth222> is BDB going to be completely replaced by LevelDB?
311 2012-12-19 01:59:02 <Kiba> tachikomas!
312 2012-12-19 01:59:02 <sipa> wallet is still BDB... for now
313 2012-12-19 01:59:03 <wizkid057> what is the "work" value in the SetBestChain debug.log line?
314 2012-12-19 01:59:11 <sipa> wizkid057: expected number of hashes
315 2012-12-19 01:59:20 <sipa> in the chain until that block
316 2012-12-19 01:59:26 <wizkid057> ah
317 2012-12-19 01:59:35 <wizkid057> so, sum of block difficulties * 2**32 ?
318 2012-12-19 01:59:36 <sipa> it's 668976227326636550981 now
319 2012-12-19 01:59:41 <sipa> wizkid057: approximately, yes
320 2012-12-19 01:59:46 <stealth222> is the wallet the only thing that's still BDB?
321 2012-12-19 01:59:51 <sipa> yes
322 2012-12-19 01:59:54 <stealth222> I understand the binary compatibility requirements here
323 2012-12-19 02:00:02 <stealth222> any other reasons to keep it?
324 2012-12-19 02:00:16 <sipa> lazyness
325 2012-12-19 02:00:24 <sipa> (read: no replacement was written yet)
326 2012-12-19 02:00:56 <sipa> but we want to git rid of there as well
327 2012-12-19 02:01:01 <Ukto> is there any chance of a listtransactions <bitcoinaddress> ?
328 2012-12-19 02:01:21 <wizkid057> ah here we go... CPI usage rising as we pass 180k...
329 2012-12-19 02:01:25 <wizkid057> *CPU
330 2012-12-19 02:01:28 <gmaxwell> Kiba: well I wanted lines before it too, but that looks like enough... looks like your blockchain file itself is corrupted.
331 2012-12-19 02:01:35 <Kiba> boohooe
332 2012-12-19 02:01:43 <Kiba> ACTION goes delete the file
333 2012-12-19 02:01:58 <Kiba> what is it? blk*.dat?
334 2012-12-19 02:02:05 <sipa> yes
335 2012-12-19 02:02:11 <sipa> blk000* and blkindex
336 2012-12-19 02:02:14 <stealth222> listtransactions <bitcoinaddress> and listunspenttxouts <bitcoinaddress> would be sweet
337 2012-12-19 02:02:17 <wizkid057> Ukto: cant you getaccount bitcoinaddress, listtransactions accounts ?
338 2012-12-19 02:02:20 <gmaxwell> wizkid057: 180?? what version of bitcoin are you running?
339 2012-12-19 02:02:56 <Ukto> wizkid057: no, not my wallet
340 2012-12-19 02:03:13 <sipa> stealth222: that last one is reasonable, it only requires an address-based index for the UTXO set
341 2012-12-19 02:03:17 <wizkid057> gmaxwell: git clone https://github.com/bitcoin/bitcoin.git ; cd bitcoin/src ; make -f makefile.unix USE_UPNP=
342 2012-12-19 02:03:23 <sipa> wizkid057: from when?
343 2012-12-19 02:03:30 <wizkid057> hour ago?
344 2012-12-19 02:03:35 <sipa> oh, weird
345 2012-12-19 02:04:04 <stealth222> sipa: would it be possible to query arbitrary bitcoin addresses? or only ones belonging to the wallet?
346 2012-12-19 02:04:10 <stealth222> only the former is useful to me right now
347 2012-12-19 02:04:31 <sipa> stealth222: both require an index that is not necessary for normal operation
348 2012-12-19 02:04:46 <sipa> stealth222: so it would be optional to have such an index anyway, and it needs to be implemented of course
349 2012-12-19 02:04:48 <Kiba> will bitcoin take over the world?
350 2012-12-19 02:05:39 <sipa> stealth222: but i think for several (but not all) use cases, watch-only addresses in a wallet could do a lot
351 2012-12-19 02:05:39 <wizkid057> 2012-12-19T03:02:39 SetBestChain: new best=00000000000004ff83b6c10460b239ef4a6aa320e5fffd6c7bcedefa8c78593c height=180000 work=322814099992850108886 tx=3142020 date=2012-05-13T18:21:11
352 2012-12-19 02:05:40 <wizkid057> 2012-12-19T03:06:53 SetBestChain: new best=000000000000026f9f99ccde88c96bdedf5870b5c36a3657f6eb735a3be5423c height=180800 work=328769446535555022486 tx=3286753 date=2012-05-19T20:00:49
353 2012-12-19 02:05:56 <stealth222> what do you mean by watch-only?
354 2012-12-19 02:05:56 <wizkid057> 4 minutes to do 800 blocks
355 2012-12-19 02:06:21 <sipa> stealth222: pretend they are yours, and have all RPC calls work as if they were yours
356 2012-12-19 02:06:27 <sipa> stealth222: but only not able to spend coins from them
357 2012-12-19 02:06:30 <stealth222> oh, I see
358 2012-12-19 02:06:34 <stealth222> yeah, that makes sense
359 2012-12-19 02:06:39 <stealth222> that could be useful for sure
360 2012-12-19 02:07:06 <stealth222> anyone working on that?
361 2012-12-19 02:07:12 <sipa> no
362 2012-12-19 02:07:20 <sipa> it's not hard, but it's not a priority
363 2012-12-19 02:07:20 <stealth222> can I volunteer? :)
364 2012-12-19 02:07:29 <sipa> sure
365 2012-12-19 02:08:10 <Ukto> wizkid057: when polling bitcoind every minute looking for new deposits, trying to keep from having check ALL transactions aginst the db fore recognition
366 2012-12-19 02:08:15 <Ukto> will get heavy on the sql server pretty quick
367 2012-12-19 02:08:57 <wizkid057> Ukto: i figured the bitcoind accounts stuff would handle that
368 2012-12-19 02:08:58 <Ukto> I have a project that will take 10,000's of deposits over time from ppl
369 2012-12-19 02:09:04 <Ukto> 1 address
370 2012-12-19 02:09:25 <Ukto> how do I mark a transaction and set it to not show it next list?
371 2012-12-19 02:09:27 <Ukto> if it does?
372 2012-12-19 02:09:37 <Ukto> otherwise it doesnt handle it
373 2012-12-19 02:09:46 <wizkid057> i think thats unsafe
374 2012-12-19 02:10:07 <Ukto> still, I dont want to have to run 10,000+ select statements every minute
375 2012-12-19 02:10:14 <wizkid057> i think listtransactions purposely doesnt have any "dont show already shown" stuff because status of transactions can change
376 2012-12-19 02:10:16 <Ukto> to check all transactions
377 2012-12-19 02:10:54 <wizkid057> sounds like you need a custom database tailored to this purpose
378 2012-12-19 02:10:58 <wizkid057> and not SQL
379 2012-12-19 02:11:14 <Ukto> listrtransactions doesnt even have a min confirms field
380 2012-12-19 02:11:41 <wizkid057> gmaxwell: my local node hasnt been updated in ages... wonder if that could be a slowdown?
381 2012-12-19 02:11:48 <Ukto> or a max confirms for that matter which would be better
382 2012-12-19 02:11:57 <wizkid057> min/max confirms?
383 2012-12-19 02:12:02 <wizkid057> what good would that do?
384 2012-12-19 02:12:15 <Ukto> if i could set max confirms 250, thats plenty of leway
385 2012-12-19 02:12:20 <Ukto> and not have to check ALL deposits ever made
386 2012-12-19 02:12:24 <sipa> wizkid057: what version is it?
387 2012-12-19 02:12:24 <wizkid057> oh oh
388 2012-12-19 02:12:28 <Ukto> and cut back the sql queries
389 2012-12-19 02:12:31 <wizkid057> sipa: not telling... lol
390 2012-12-19 02:12:39 <wizkid057> its embarassingly outdated
391 2012-12-19 02:12:40 <wizkid057> lol
392 2012-12-19 02:12:52 <gmaxwell> Ukto: you mean you want listsinceblock ?
393 2012-12-19 02:13:13 <wizkid057> sipa: actually, not that bad i guess: "version" : 60103,
394 2012-12-19 02:13:15 <Ukto> gmaxwell: for a specific address
395 2012-12-19 02:13:20 <sipa> wizkid057: good enough
396 2012-12-19 02:13:44 <gmaxwell> wizkid057: I can't think of anything specific since then... my tests were against a 0.7.1 node.
397 2012-12-19 02:14:03 <wizkid057> dunno
398 2012-12-19 02:14:09 <wizkid057> its definitely faster than before, though
399 2012-12-19 02:14:21 <wizkid057> at 184300
400 2012-12-19 02:14:32 <sipa> 800 blocks in 4 minutes... that's only 2 blocks per second
401 2012-12-19 02:14:43 <sipa> i do more than that after the checkpoint on a single core
402 2012-12-19 02:14:44 <wizkid057> sipa: its sped up since then for some reason
403 2012-12-19 02:15:22 <wizkid057> 2012-12-19T03:13:26 SetBestChain: new best=00000000000007be8b15bd307b7bc04aa22369e4079c12914a415f6c96ab2844 height=183000 work=344193715259143008966 tx=3721560 date=2012-06-04T16:13:37
404 2012-12-19 02:15:23 <wizkid057> 2012-12-19T03:15:15 SetBestChain: new best=000000000000060405a235c6b968ccb18fd6b3800ae9742c2524e28863367359 height=184000 work=351008948950763208291 tx=3934378 date=2012-06-11T04:02:23
405 2012-12-19 02:16:44 <Ukto> err
406 2012-12-19 02:16:52 <gmaxwell> okay, ten blocks per second sounds somewhat saner.
407 2012-12-19 02:17:04 <wizkid057> wow, we split blkNNNN at 134217728 bytes now?
408 2012-12-19 02:17:12 <Ukto> if i do listsinceblock 210000 it should only show transactions that appeared on/after 210000 right ? :/
409 2012-12-19 02:17:31 <sipa> wizkid057: yes
410 2012-12-19 02:18:45 <wizkid057> Ukto: need a hash not a height
411 2012-12-19 02:19:08 <Ukto> the help said blockid :P
412 2012-12-19 02:19:16 <Ukto> took that as height
413 2012-12-19 02:19:27 <Ukto> still, seems should be optional between them
414 2012-12-19 02:20:18 <Ukto> incrimenting the height is alot easier then wasting an rpc call to first getblockhash
415 2012-12-19 02:20:29 <wizkid057> Ukto: well, hash for a height can change
416 2012-12-19 02:20:42 <Ukto> .. what?
417 2012-12-19 02:20:50 <wizkid057> Ukto: orphaned blocks
418 2012-12-19 02:21:10 <Ukto> but so far never after 6 blocks later, right?
419 2012-12-19 02:21:26 <sipa> so far never after more than 2 blocks
420 2012-12-19 02:21:33 <Ukto> that should be impossible w/o a 51% right?
421 2012-12-19 02:21:40 <sipa> it's always possible
422 2012-12-19 02:21:45 <wizkid057> sipa: havent there been some huge reorgs in the past?
423 2012-12-19 02:21:48 <Ukto> hehe
424 2012-12-19 02:21:54 <Ukto> i guess i wrote an oxymoron
425 2012-12-19 02:22:13 <sipa> wizkid057: some because of bugs, yes
426 2012-12-19 02:22:17 <sipa> but in normal cases, not afaik
427 2012-12-19 02:22:56 <Ukto> i check the blockhash, and use it for public verification of data. adter about 2~3 months, if the blockhash changed, it wont matter anymore
428 2012-12-19 02:23:13 <wizkid057> probably not
429 2012-12-19 02:23:20 <wizkid057> but can never be too careful
430 2012-12-19 02:23:42 <Ukto> if it changes, all it does is cause someone to say "hey... thats not wht I see in the blockchain"
431 2012-12-19 02:23:48 <Ukto> payouts are already completed and done
432 2012-12-19 02:23:48 <wizkid057> probably safest to account for this stuff now, while its in front of you and easy, then it will be after the hack :P
433 2012-12-19 02:23:52 <sipa> the problem is that there's an mismatch between how people do transactions in real life, and in bitcoin
434 2012-12-19 02:24:04 <sipa> at some point you're going to consider the transaction as completed
435 2012-12-19 02:24:12 <Ukto> I am just using the blockhash as a random data source
436 2012-12-19 02:24:16 <sipa> and you probably don't even care about it being reversed after that anyway
437 2012-12-19 02:24:30 <Ukto> that shows I didnt "make it up"
438 2012-12-19 02:24:42 <Ukto> and am not controling the choices
439 2012-12-19 02:24:43 <sipa> Ukto: you know miners have very strong control over blocks, right?
440 2012-12-19 02:24:51 <Ukto> yes
441 2012-12-19 02:25:05 <phantomcircuit> sipa, is there anyway reasonable way to detect a reversal?
442 2012-12-19 02:25:29 <sipa> phantomcircuit: number of confirmations drops to 0?
443 2012-12-19 02:25:34 <phantomcircuit> i actually have one of the (very few) services where accepting 0 confirmations and then cancelling if it doesn't either confirm or gets double spent
444 2012-12-19 02:25:40 <phantomcircuit> makes sense
445 2012-12-19 02:25:44 <Ukto> phantomcircuit: which should only happen within 10 blocks
446 2012-12-19 02:25:51 <sipa> "should"
447 2012-12-19 02:25:51 <Ukto> _should_
448 2012-12-19 02:26:05 <phantomcircuit> i dont actually give a crap about how many blocks
449 2012-12-19 02:26:16 <Ukto> if it happens past that point.. then something major happened :P
450 2012-12-19 02:27:43 <phantomcircuit> it's self correcting
451 2012-12-19 02:28:30 <Ukto> wizkid057: think of those bitcoin lotto sites that use blockhash for their 'winning numbers' to show proof that they are not picking the winners. They only need the data to stay correct for at least a few weeks. the longer the better obviously.
452 2012-12-19 02:28:44 <Ukto> thats an equivlent to what I am using it for
453 2012-12-19 02:29:23 <Ukto> get the blockhash at 12 confirms, do X with it, leave it stored as reference
454 2012-12-19 02:29:32 <Ukto> if it changes matter it is of no consequence
455 2012-12-19 02:31:05 <wizkid057> gotcha
456 2012-12-19 02:31:26 <wizkid057> on block 194k btw
457 2012-12-19 02:32:54 <gmaxwell> Ukto: using a blockhash for that is not a good idea, as it can be rigged. There are plenty of ways of doing nothing up my sleeve randomness. that don't require a blockhash.
458 2012-12-19 02:33:47 <wizkid057> gmaxwell: rigging the blockhash at these difficulties would likely be more costly than whatever ukto is using it for
459 2012-12-19 02:34:38 <gmaxwell> I dunno what ukto is using it for, but if you only need 1 bit of rigging and can pick which blocks you'll use, then its not that costly. (12.5 btc per instance, I guess)
460 2012-12-19 03:09:02 <wizkid057> sipa: wow, we are CRAWLING > 210000
461 2012-12-19 03:09:55 <wizkid057> 2012-12-19T04:08:53 SetBestChain: new best=000000000000048b95347e83192f69cf0366076336c639f9b7228e9ba171342e height=210000 work=628963747775700992096 tx=9344662 date=2012-11-28T15:24:38
462 2012-12-19 03:10:15 <wizkid057> has a core maxed
463 2012-12-19 03:12:02 <phantomcircuit> wizkid057, satoshi dice in action
464 2012-12-19 03:12:53 <sipa> 130s for 50 blocks?
465 2012-12-19 03:12:58 <sipa> that's ridiculous
466 2012-12-19 03:13:24 <sipa> well... maybe not, on my CPU i do 2-3 blocks per second on a single core
467 2012-12-19 03:14:08 <sipa> still 6x slower
468 2012-12-19 03:14:20 <wizkid057> yeah, crazy slow
469 2012-12-19 03:14:38 <wizkid057> that the database? or some type of verification?
470 2012-12-19 03:14:48 <sipa> purely signature checks
471 2012-12-19 03:14:50 <sipa> ECDSA
472 2012-12-19 03:14:56 <wizkid057> way to disable?
473 2012-12-19 03:15:11 <sipa> you shouldn't :)
474 2012-12-19 03:15:31 <wizkid057> pretty sure my local bitcoind did the work already
475 2012-12-19 03:15:37 <wizkid057> no sense doing it again
476 2012-12-19 03:16:08 <sipa> search for CS_AFTER_CHECKPOINT in a call to CheckInputs in main.cpp, and replace it by CS_NEVER
477 2012-12-19 03:16:49 <wizkid057> main.cpp:1628: if (!tx.CheckInputs(view, CS_AFTER_CHECKPOINT, fStrictPayToScriptHash ? SCRIPT_VERIFY_P2SH : SCRIPT_VERIFY_NONE))
478 2012-12-19 03:16:55 <sipa> yup
479 2012-12-19 03:17:57 <wizkid057> so, the initial speedup was fake because it wasnt actually verifying stuff?
480 2012-12-19 03:18:06 <gmaxwell> ...
481 2012-12-19 03:18:16 <gmaxwell> wizkid057: this has long been the behavior.
482 2012-12-19 03:18:30 <gmaxwell> The speedup is all in the transaction database handling.
483 2012-12-19 03:18:50 <wizkid057> hm
484 2012-12-19 03:19:00 <wizkid057> why not a distributed hash table?
485 2012-12-19 03:19:02 <wizkid057> :D
486 2012-12-19 03:19:35 <gmaxwell> ACTION stabs wizkid057 in the face
487 2012-12-19 03:19:40 <sipa> wizkid057: it was verifying tons of stuff, just not the scripts
488 2012-12-19 03:20:06 <sipa> but indeed ECDSA checks is the most expensive part, and it's only done after the last checkpoint
489 2012-12-19 03:20:08 <wizkid057> well, i'll compile this out, let it catch up, then compile it in
490 2012-12-19 03:20:16 <gmaxwell> wizkid057: the verfication it does is sufficient to constrain inflation and to build a valid txout set, it just doesn't validate whos spending what before the checkpoint.
491 2012-12-19 03:20:16 <sipa> parallel script checking helps, though :)
492 2012-12-19 03:20:37 <gmaxwell> though I still want to uncover why its slow for you. may have an interesting cause.
493 2012-12-19 03:20:50 <gmaxwell> but I'm not sure of where to start??? benchmark patch would be helpful there I guess.
494 2012-12-19 03:21:10 <sipa> benchmark is merged :)
495 2012-12-19 03:21:21 <sipa> wizkid057: can you pass -benchmark when running?
496 2012-12-19 03:21:37 <wizkid057> sure
497 2012-12-19 03:21:38 <sipa> it should dump some timing statistics in debug.log
498 2012-12-19 03:22:23 <sipa> you didn't accidentally remove/overwrite -O2 when compiling, or something like that?
499 2012-12-19 03:22:33 <sipa> hmm... 32 bit or 64 bit system?
500 2012-12-19 03:22:49 <wizkid057> 64-bit
501 2012-12-19 03:22:54 <wizkid057> and I didnt mess with the makefile
502 2012-12-19 03:22:56 <sipa> ok
503 2012-12-19 03:26:04 <wizkid057> i guess -daemon doesnt work with -benchmark?
504 2012-12-19 03:26:10 <sipa> it should
505 2012-12-19 03:26:18 <wizkid057> its not ;)
506 2012-12-19 03:26:25 <sipa> why do you think so?
507 2012-12-19 03:26:41 <wizkid057> actually
508 2012-12-19 03:26:51 <wizkid057> i'm just going to stop saying stuff isnt working
509 2012-12-19 03:27:00 <wizkid057> bitcoin/src/bitcoind -deamon -benchmark -connect=192.168.5.54
510 2012-12-19 03:27:03 <wizkid057> lol
511 2012-12-19 03:27:28 <sipa> when a block is processed, you should see some timings in debug.log
512 2012-12-19 03:28:05 <wizkid057> http://pastebin.com/YF9ybGYW
513 2012-12-19 03:29:22 <gmaxwell> wow, those connect numbers seem really slow.
514 2012-12-19 03:30:12 <sipa> yeah, too bad there's no way to see script vs the rest in his version of the code
515 2012-12-19 03:30:24 <sipa> (added by parallel)
516 2012-12-19 03:30:32 <wizkid057> i can merge whatever
517 2012-12-19 03:30:46 <wizkid057> there a branch that does what you want?
518 2012-12-19 03:30:58 <sipa> merge my 'hal' and 'parallel' branches
519 2012-12-19 03:32:03 <sipa> ACTION -> zZzZ
520 2012-12-19 03:33:29 <gmaxwell> 2160
521 2012-12-19 03:33:40 <gribble> use the 'gentime' command instead
522 2012-12-19 03:33:40 <sipa> ;;bc,calc 2000000
523 2012-12-19 03:33:44 <gmaxwell> er. 2060
524 2012-12-19 03:33:46 <sipa> ;;gentime 2000000
525 2012-12-19 03:33:47 <gribble> The average time to generate a block at 2000000.0 Mhps, given difficulty of 3370181.79928, is 2 hours and 37 seconds
526 2012-12-19 03:33:54 <gribble> The average time to generate a block at 2000.0 Mhps, given difficulty of 3370181.79928, is 11 weeks, 6 days, 18 hours, 23 minutes, and 30 seconds
527 2012-12-19 03:33:54 <sipa> ;;gentime 2000
528 2012-12-19 03:34:13 <gmaxwell> wizkid057: https://github.com/bitcoin/bitcoin/pull/2060.patch and https://github.com/bitcoin/bitcoin/pull/2061.patch
529 2012-12-19 03:34:28 <gmaxwell> wizkid057: git am those bad boys.
530 2012-12-19 03:34:59 <wizkid057> gmaxwell: i already merged sipa's hal + parallel... add those patches on top?
531 2012-12-19 03:35:11 <sipa> no, those are those two patches
532 2012-12-19 03:35:33 <gribble> The average time to generate a block at 2.0 Mhps, given difficulty of 3370181.79928, is 229 years, 25 weeks, 6 days, 7 hours, 45 minutes, and 4 seconds
533 2012-12-19 03:35:33 <sipa> ;;gentime 2
534 2012-12-19 03:35:45 <wizkid057> so, no need to patch those
535 2012-12-19 03:37:22 <wizkid057> commmpillllliiiinnnnng
536 2012-12-19 03:45:19 <wizkid057> sipa: this is faster
537 2012-12-19 03:45:42 <wizkid057> but, presumably because its using both cores now
538 2012-12-19 03:46:33 <wizkid057> http://pastebin.com/HP375B6r
539 2012-12-19 04:32:07 <gmaxwell> Hey, testnet is going to start having node discovery problems.
540 2012-12-19 04:32:18 <gmaxwell> We probably need to get seednodes / dnsseed working for it. :(
541 2012-12-19 04:33:26 <gmaxwell> irc.lfnet.org is now suppressing /who is all the bitcoin idling channels, including #bitcoinTEST
542 2012-12-19 04:33:30 <gmaxwell> er #bitcoinTEST3
543 2012-12-19 04:41:33 <MC1984> Composed of about 300 atoms suspended in space, the "ion-crystal" device "has the potential to be the most powerful computer ever developed, beating the computational capacity of any existing machine by 10^80 times
544 2012-12-19 04:41:41 <MC1984> that sounds bad for encryption
545 2012-12-19 04:41:49 <MC1984> its pretty bad isnt it
546 2012-12-19 04:42:08 <etotheipi_> does it run linux?
547 2012-12-19 04:43:41 <MC1984> its probably sensationalised bullshit
548 2012-12-19 04:43:44 <phantomcircuit> MC1984, so far the "best" quamtum computer publically disclosed could only factor the number 15
549 2012-12-19 04:43:47 <MC1984> i hope
550 2012-12-19 04:44:17 <etotheipi_> "best" "quantum" "computer"
551 2012-12-19 04:44:19 <phantomcircuit> which sort of makes me question whether it was actually a quantum computer or not because the difference in # of operations between a quantum computer and a conventional computer here would be a rounding error
552 2012-12-19 04:45:18 <MC1984> quantum computers could be total bullshit and we would never know because they only thing that can verify a result is another QC
553 2012-12-19 04:45:35 <MC1984> quantum computers are great, ACCORDING TO QUANTUM COMPUTERS
554 2012-12-19 04:45:48 <phantomcircuit> gmaxwell, why was lfnet used
555 2012-12-19 04:45:58 <phantomcircuit> the last time i checked the *only* thing on it was bitcoin clients
556 2012-12-19 04:46:28 <phantomcircuit> MC1984, wrong there are entire classes of problems which can be solved on a quantum computer in a reasonable amount of time which you could pretty much never solve on a conventional computer
557 2012-12-19 04:46:38 <phantomcircuit> solving one of those problems proves you've built a qc
558 2012-12-19 04:47:00 <MC1984> what if the answer is wrong
559 2012-12-19 04:47:33 <phantomcircuit> hard to answer easy to verify
560 2012-12-19 04:48:03 <MC1984> how to verify if its never been answered before
561 2012-12-19 04:49:23 <MC1984> am i being stupid again
562 2012-12-19 04:49:32 <phantomcircuit> yes
563 2012-12-19 04:49:38 <MC1984> ok
564 2012-12-19 04:51:24 <gmaxwell> phantomcircuit: freenode kicked us off.
565 2012-12-19 04:52:55 <phantomcircuit> https://bitcointalk.org/index.php?topic=1735.msg21433#msg21433
566 2012-12-19 04:52:56 <phantomcircuit> ahaha
567 2012-12-19 04:52:59 <phantomcircuit> that's great
568 2012-12-19 04:54:57 <MC1984> cia false flaggin erryday
569 2012-12-19 04:55:07 <MC1984> WL does take bitcoin now
570 2012-12-19 04:55:41 <gmaxwell> yea, ever since gavin presented about it at the cia.
571 2012-12-19 04:55:44 <gmaxwell> :P
572 2012-12-19 04:57:23 <phantomcircuit> https://bitcointalk.org/index.php?topic=215.0
573 2012-12-19 04:57:24 <MC1984> as if we dont know gavin is a plant
574 2012-12-19 04:57:24 <phantomcircuit> ah yes
575 2012-12-19 04:59:17 <gmaxwell> oops. /query and /quit are not the same.
576 2012-12-19 05:01:12 <MC1984> thats a strange mask
577 2012-12-19 05:01:34 <phantomcircuit> oh wow i just found the tom williams thread there crazy people tried to find him
578 2012-12-19 05:01:47 <phantomcircuit> seems like they accused pretty much everybody of being tom
579 2012-12-19 05:01:49 <phantomcircuit> lol
580 2012-12-19 05:02:01 <phantomcircuit> sometimes i forget how ridiculous this all is
581 2012-12-19 05:02:29 <MC1984> as if the real world is any less ridiculous
582 2012-12-19 05:02:53 <MC1984> as if USD is
583 2012-12-19 05:03:07 <phantomcircuit> possibly not the thing itself
584 2012-12-19 05:03:13 <phantomcircuit> but certainly the community around it :P
585 2012-12-19 05:03:35 <MC1984> welcome to the internet enjoy your stay
586 2012-12-19 05:22:18 <jgarzik> gmaxwell: I thought we knew the lfnet IRC admin?
587 2012-12-19 05:22:26 <jgarzik> gmaxwell: isn't that laszlo?
588 2012-12-19 05:22:41 <jgarzik> if not laszlo, I'm pretty sure it was _somebody_ we could ping
589 2012-12-19 05:24:13 <jgarzik> gmaxwell: Regardless, I can set up a static DNS seed @ dnspark very quickly
590 2012-12-19 05:26:20 <gmaxwell> I've asked on #lfnet on lfnet, but haven't had response yet.
591 2012-12-19 05:31:37 <jgarzik> gmaxwell: email sent, cc'd you
592 2012-12-19 07:27:38 <Ukto> hwo could you rig the blockhash ?
593 2012-12-19 07:33:57 <forrestv> sipa, can you look at https://github.com/bitcoin/bitcoin/pull/2115/files ?
594 2012-12-19 09:52:53 <stealth222> anyone here?
595 2012-12-19 09:53:22 <sturles> No
596 2012-12-19 09:55:07 <stealth222> lol
597 2012-12-19 12:53:09 <Supa> why is the dev team filled with a bunch of fuckin losers ?
598 2012-12-19 13:02:00 <Diapolo> I first read sipa ^^.
599 2012-12-19 13:15:26 <t7> hehe me too
600 2012-12-19 13:17:50 <sipa> those damn devs!
601 2012-12-19 13:18:22 <kinlo> why do you think you are a loser sipa ? :p
602 2012-12-19 14:12:38 <Matt_von_Mises> To read from blk0001.dat, it seems it just contains blocks one after the other including the message headers?
603 2012-12-19 14:17:02 <m0mchil> used znort's blockparser to check sum of all UTXO - 10567357.18910617
604 2012-12-19 14:17:07 <jgarzik> Matt_von_Mises: correct. blk000?.dat is a stream of {pchMessageStart}{nSize}{CBlock}
605 2012-12-19 14:17:38 <jgarzik> Matt_von_Mises: v0.8 may append zeroes after the very last block in the file
606 2012-12-19 14:17:40 <m0mchil> whoever took the remainder to 10 570 000 to return it immediately to the authorities
607 2012-12-19 14:17:58 <Matt_von_Mises> jgarzik: OK thanks.
608 2012-12-19 14:27:48 <stealth222> when I try to do a git pull, I'm getting:
609 2012-12-19 14:27:52 <stealth222> fatal: You have not concluded your cherry-pick (CHERRY_PICK_HEAD exists).
610 2012-12-19 14:28:02 <stealth222> but then I git commit and get:
611 2012-12-19 14:28:12 <stealth222> # On branch master
612 2012-12-19 14:28:13 <stealth222> #
613 2012-12-19 14:28:15 <stealth222> nothing to commit (working directory clean)
614 2012-12-19 14:28:25 <stealth222> this is extremely frustrating - anyone have any suggestions?
615 2012-12-19 14:29:13 <denisx> sounds like your cherry pick had a conflict
616 2012-12-19 14:29:39 <stealth222> so how do I undo?
617 2012-12-19 14:29:52 <stealth222> I want to go back to the state that is at my remote
618 2012-12-19 14:30:14 <sipa> git reset --hard yourremote/yourbranch
619 2012-12-19 14:31:04 <stealth222> thank you! :)
620 2012-12-19 14:31:36 <gmaxwell> m0mchil: blame gavinandresen and midnightmagic. :)
621 2012-12-19 14:32:51 <m0mchil> gmaxwell: what did they do?
622 2012-12-19 14:34:13 <Ukto> gmaxwell: hwo could you rig the blockhash ?
623 2012-12-19 14:34:30 <gmaxwell> m0mchil: things that resulted in some fees gettng burned
624 2012-12-19 14:35:15 <gmaxwell> Ukto: by mining blocks which meet whatever criteria you like.
625 2012-12-19 14:35:33 <gmaxwell> (by throwing out ones that don't)
626 2012-12-19 14:36:07 <m0mchil> A check for outputs+fees == inputs was rejected? Instead of <= ?
627 2012-12-19 14:36:32 <Ukto> but dont you have to find the block?
628 2012-12-19 14:36:36 <gmaxwell> m0mchil: the network rule is <=.
629 2012-12-19 14:36:38 <Ukto> first ?
630 2012-12-19 14:37:48 <m0mchil> gmaxwell: but what is the advantage over == ?
631 2012-12-19 14:37:50 <Ukto> wait, so the "blockhash", is that the hash of that won the "block" ?
632 2012-12-19 14:38:37 <gmaxwell> Ukto: sure. and that means that someone with 30% of the hashpower (e.g. by controlling a pool or via a hashpower buying servce), can make 1 bit of selection 15% of the time at a cost of half a block subsidy (12.5 btc now) on average when successful. Probably somewhat less because they could still race for it.
633 2012-12-19 14:39:15 <sipa> m0mchil: not splitting the network :)
634 2012-12-19 14:39:41 <sipa> m0mchil: i haven't seen an actual good reason though, except compatibility
635 2012-12-19 14:40:00 <gmaxwell> m0mchil: <= is sufficient, and it's just how it is and can't easily be changed. There aren't any major advantages, though it may make rule enforcement a little easier.
636 2012-12-19 14:40:11 <Ukto> was the 'sure' in answer to my question that the 'hash' that is found that gets teh 'block' is what becomes the 'blockhash' ?
637 2012-12-19 14:40:22 <m0mchil> sipa: no, seriously... I see, it was this way and can't be changed because of existing transactions
638 2012-12-19 14:40:26 <gmaxwell> E.g. > is somewhat easier to exclude.
639 2012-12-19 14:41:08 <gmaxwell> Ukto: it was to 'but dont you have to find the block', but the answer to so the "blockhash", is that the hash of that won the "block" is yes too.
640 2012-12-19 14:41:27 <Ukto> ah k
641 2012-12-19 14:41:34 <Ukto> hrm
642 2012-12-19 14:42:14 <Ukto> i think you would be pretty hardpressed to get the eaxct block you want though, but using a service like hashpower could def. increase the chances
643 2012-12-19 14:43:02 <Ukto> althgough it would be hard enough to normally find the block, but to mine for a winner AND exact placements of numbers/letters?
644 2012-12-19 14:43:19 <gmaxwell> Ukto: you can't get 'the exact' but you can control 1 bit worth of output for moderate cost as I described.
645 2012-12-19 14:43:47 <Ukto> in this case, I am taking the blockhash, droppping the leters, and only using the last 3 numbers remaining
646 2012-12-19 14:43:57 <gmaxwell> it doesn't matter how you mutate it.
647 2012-12-19 14:43:59 <Ukto> in specific order
648 2012-12-19 14:44:31 <Ukto> that wouldnt fall under 1 bit worth now would it?
649 2012-12-19 14:44:36 <m0mchil> pity... it would've been nice to have this enforced, being sure that at some height sum of spendable outputs equals generation
650 2012-12-19 14:44:38 <gmaxwell> ..
651 2012-12-19 14:44:52 <Ukto> the more exact it has to search for, the more impossible it would become?
652 2012-12-19 14:45:02 <Ukto> sorry if I am not understanding fully, thats why I am here, and sking
653 2012-12-19 14:45:05 <Ukto> asking*
654 2012-12-19 14:45:16 <sipa> Ukto: what decision will you be basing on it?
655 2012-12-19 14:45:29 <gmaxwell> Ukto: You're asking a question devoid of context, so it's hard to give a useful answer.
656 2012-12-19 14:46:26 <gmaxwell> For example, if you use the value to select between N orderings of winnings, I can make one half of those orderings more likely than another half through mining... at moderate cost.
657 2012-12-19 14:46:50 <Ukto> the question outright would now be, how difficult would it be for someone to mine a winning block were the last 3 numbers in it, in order, what they wanted?
658 2012-12-19 14:46:55 <gmaxwell> There are ways to do secure coin tosses that don't require bitcoin and aren't vulnerable to this.
659 2012-12-19 14:47:13 <Ukto> i agree
660 2012-12-19 14:47:19 <Ukto> but this is something specifically for bitcoin :)
661 2012-12-19 14:47:57 <gmaxwell> Ukto: thats hard but usually irrelevant??? they can't pick an exact value, but they can bias the selection. e.g. I can't pick who wins the lottery, but I can make my friends more likely than everyone else.
662 2012-12-19 14:48:14 <Ukto> so if I told you 5 blocks from now, the last 3 numbers of the hash (dropping hte letters) has to be 321, how hard woudl it be to make that happen?
663 2012-12-19 14:48:47 <gmaxwell> Okay, you're not speaking concretely enough so I'm not going to play any more guessing games.
664 2012-12-19 14:48:58 <Ukto> thats all there was
665 2012-12-19 14:49:00 <Ukto> no guessing
666 2012-12-19 14:49:35 <Ukto> i am just asking how hard/easy it would be to forcefully manipulate/mine a block where the last 3 numerics of the blockhas was an extact 3 numbers in order
667 2012-12-19 14:49:50 <sipa> Ukto: the cost is 25000 BTC * chance_for_match
668 2012-12-19 14:50:32 <Ukto> sipa: I feel like I am getting trolled. :P
669 2012-12-19 14:50:35 <gmaxwell> yes, there is??? you're asking overly specific questions without describing the payoff matrix. So while I can answer the direct question I'm reasonably confident that doing so will make you less knowledgeable than if I didn't.
670 2012-12-19 14:50:39 <Ukto> to some extent
671 2012-12-19 14:50:42 <sipa> so for 250 BTC you can buy a 1% chance
672 2012-12-19 14:51:24 <Ukto> gmaxwell: can I pm you?
673 2012-12-19 14:51:40 <Ukto> I will be happy to give a full break down
674 2012-12-19 14:51:56 <Ukto> very quick and simple
675 2012-12-19 14:52:02 <gmaxwell> E.g. say you and I were to bet that the hash 5 blocks from now wouldn't end n 321. It would be fairly costly for me to make more likely to end in that, but it would be quite cheap for you to make it more likely that it wouldn't end in that.
676 2012-12-19 14:52:06 <Ukto> in fact i already said it but
677 2012-12-19 14:52:17 <Ukto> alright
678 2012-12-19 14:52:19 <Ukto> let me put it this way
679 2012-12-19 14:52:21 <gmaxwell> And so the system can be cheaply cheated.
680 2012-12-19 14:52:27 <gmaxwell> (in that example)
681 2012-12-19 14:52:44 <Ukto> lets say you have the number 123. and if you can get the last three numerics of the block has to be 123 in order, ingoring alphas, you win 5,000 btc
682 2012-12-19 14:52:49 <Ukto> how doable is forcing it
683 2012-12-19 14:52:53 <Ukto> end of story
684 2012-12-19 14:53:06 <Ukto> thats all there is to it, theres nothing else
685 2012-12-19 14:53:36 <Ukto> after that block hits or does not it, the number changes to (random 3) for X blocks from then
686 2012-12-19 14:53:41 <gmaxwell> Why do I only have number 123? Why don't I bet multiple times and have 128-1152?
687 2012-12-19 14:54:07 <Ukto> you could bet multiple times, and get 533, 612, 164, 327
688 2012-12-19 14:54:11 <Ukto> each bet costs more
689 2012-12-19 14:54:27 <Ukto> the odds are currently 1/1000 of winning
690 2012-12-19 14:54:37 <Ukto> which is fairly good for what I am doing
691 2012-12-19 14:54:46 <Ukto> but, regardless of the odds based on number of entries
692 2012-12-19 14:55:04 <Ukto> your saying they get 10~20 sets of 3 numbers
693 2012-12-19 14:55:07 <gmaxwell> Ukto: well what you're describing is slightly more likely to pay better 000 to begin with, so I'd want that number for sure.
694 2012-12-19 14:55:07 <Ukto> look for ANY of those
695 2012-12-19 14:55:57 <Ukto> yeah, looking at it that way, I can see where it can be easily abused, and forced now
696 2012-12-19 14:58:20 <gmaxwell> Ukto: I'd have to sit down and work out the payoffs, but it's pretty straight forward: by betting multiple times I can control my baseline odds of winning. By buying/controlling hashpower I can throw out blocks that don't make one of my accounts win in the hopes that I or someone else will find a block that does. The loss of tossing a block is 25 BTC times how likely you are to do it. If I'm half the betters I only need 1 bit of selection on
697 2012-12-19 14:58:38 <gmaxwell> Moreover, if I'm a pool operator and a little dishonest, I don't even have to pay the 25 BTC.
698 2012-12-19 14:58:49 <Ukto> corect
699 2012-12-19 14:58:56 <Ukto> blockhash would not fit my needs
700 2012-12-19 14:59:12 <gmaxwell> I instead lose half the pool fees I collect (+ whatever loss of miners I get from having 'bad luck' due to this activity)
701 2012-12-19 14:59:33 <denisx> we need a new channel, like bitcoin-gambling
702 2012-12-19 15:01:07 <gmaxwell> well this also applies to the whole exosystem... someone creating a big (e.g. 5000 btc) lottery thing like this would be bad because it would create a bigger market for dishonest mining. (and god knows??? give some regulator an excuse to claim that bitcoin == gambling)
703 2012-12-19 15:01:34 <Ukto> liek satoshi dice doesnt already?
704 2012-12-19 15:01:39 <Ukto> have you seen their numbers this month?
705 2012-12-19 15:02:15 <Ukto> i am not making something thats ment to get rich off of
706 2012-12-19 15:02:23 <Ukto> i am making something thats ment for honest fun
707 2012-12-19 15:02:35 <Ukto> and have some returns worth playing, but not mega millions winner
708 2012-12-19 15:02:46 <gmaxwell> Ukto: if you want to have a lottery with defined participants just use a regular cryptographic coin flip. Everyone transmits a hash of a random value. Once everyone has seen these hashes and agrees, then everyone discloses the numbers that hashed to the random values, you compose and hash those, and you get your group random value.
709 2012-12-19 15:03:07 <Ukto> yup
710 2012-12-19 15:03:19 <Ukto> but does not apply to my situation unfortunately
711 2012-12-19 15:03:48 <gmaxwell> Ukto: welp, I suggest you go find some cryptographer to consult for you before you invent another insecure system. :)
712 2012-12-19 15:04:24 <Ukto> :P
713 2012-12-19 15:04:40 <Ukto> You sure have a way with people, you know that? hehe
714 2012-12-19 15:04:53 <Ukto> but
715 2012-12-19 15:04:57 <Ukto> Thank you for answering my qusetions
716 2012-12-19 15:39:51 <helo> allowing people to win more than the block reward based on some characteristic of the block hash encourages dishonest mining
717 2012-12-19 15:40:47 <helo> if the prizes are less than the reward, maybe it wouldn't be bad
718 2012-12-19 15:45:08 <helo> using only info in the blockchain to determine prizes at least allows the paying node to be well hidden
719 2012-12-19 16:29:36 <sipa> Qt 5!
720 2012-12-19 16:43:44 <gmaxwell> sipa: I thought of another way we can encourage people to generate IsCanonicalScript() passing transactions... when someone recieves a non-canonical txn, flag it in the gui with a <weird txn> that you can click on and it asks you to tell the party who sent you the txn to go to <forum post about IsCanonicalScript()>. Might be a way to mop up the unidentified straggling implementations.
721 2012-12-19 16:44:17 <sipa> gmaxwell: well, first thing we should do is test how many non-standard ones are still being created
722 2012-12-19 16:44:33 <sipa> since last time we identified two big offenders, which should be fixed now
723 2012-12-19 16:47:30 <gmaxwell> ::nods:: Certantly, I meant after the big offenders were gone, of course??? though we're going to have this excercise again for the even/oddness... and that will be harder to clean up.
724 2012-12-19 16:52:31 <sipa> gmaxwell: first 22 transactions accepted to my mempool... none violate
725 2012-12-19 16:52:32 <sipa> weird
726 2012-12-19 16:52:47 <sipa> ACTION wonders whether the verifier works at all
727 2012-12-19 17:00:08 <sipa> several hundreds, and no fail canonical checks :s
728 2012-12-19 17:20:53 <Pucilowski> Is there a java API out there for determnistic address creation? One that also allows a public seed that only allows public keys to be generated.
729 2012-12-19 17:28:41 <sipa> gmaxwell: two non-canonical ones out of over 1200
730 2012-12-19 17:29:28 <sipa> in 45 minutes
731 2012-12-19 17:29:33 <sipa> ACTION likes
732 2012-12-19 17:32:02 <sipa> let's see how many i accumulate in a day or so
733 2012-12-19 17:32:30 <sipa> if it's really that low, maybe we can even propose not relaying them anymore in 0.8 already...
734 2012-12-19 17:32:52 <gmaxwell> haha. now add the even/odd check. :P But yes, at that rate it does sound like something 0.8 could stop relaying.
735 2012-12-19 17:33:13 <sipa> gmaxwell: that means it'll be 601 failures in 1200 :)
736 2012-12-19 17:33:58 <etotheipi_> sipa: what other things are you looking for?
737 2012-12-19 17:34:06 <sipa> etotheipi_: ?
738 2012-12-19 17:34:07 <etotheipi_> I fixed the signature (r,s) bytes
739 2012-12-19 17:34:12 <sipa> i know
740 2012-12-19 17:34:18 <etotheipi_> anything else?
741 2012-12-19 17:34:31 <sipa> the other proposed change is requiring s to be always even
742 2012-12-19 17:34:51 <etotheipi_> how/why?
743 2012-12-19 17:34:54 <sipa> (you can take the complement modulo the curve order if it's not, without invalidating the signature)
744 2012-12-19 17:34:57 <sipa> removing malleability
745 2012-12-19 17:35:18 <etotheipi_> that seems to be taking it kind of far, don't you think?
746 2012-12-19 17:35:29 <gmaxwell> etotheipi_: suggestions on the simplest possible code for people to conform would be helpful.
747 2012-12-19 17:35:30 <etotheipi_> I could also just use a different random number and produce a completely different s
748 2012-12-19 17:35:34 <sipa> i don't want *any* malleability, ideally
749 2012-12-19 17:35:51 <gmaxwell> etotheipi_: uh. it's not taking it kind of far, it's preventing _third party_ malleability, which is a vulnerability.
750 2012-12-19 17:36:04 <etotheipi_> oh
751 2012-12-19 17:36:36 <gmaxwell> self-malleability is fine, there are lots of things in a transaction you can change to make a different one, even if our signatures were determinstic.
752 2012-12-19 17:37:56 <sipa> sure, the problem is that anyone on the network can do that transformation without invalidating the signature
753 2012-12-19 17:38:02 <etotheipi_> okay
754 2012-12-19 17:38:04 <etotheipi_> I get it
755 2012-12-19 17:38:19 <etotheipi_> it just seems like an awful lot of implementation details
756 2012-12-19 17:38:29 <etotheipi_> and no longer the behavior of any default libraries
757 2012-12-19 17:38:37 <sipa> indeed :(
758 2012-12-19 17:38:38 <etotheipi_> err.. standard
759 2012-12-19 17:39:16 <etotheipi_> your forcing people who already have ECDSA implementations to dig into the math and add extra operations... it doesn't sit right with me
760 2012-12-19 17:39:23 <etotheipi_> (I know it's simple, but tha'ts not the point)
761 2012-12-19 17:45:59 <gmaxwell> etotheipi_: thats why I said "suggestions on the simplest possible code for people to conform would be helpful."
762 2012-12-19 18:15:00 <helo> maybe repackage openssl's quirky behavior into a standalone lib, and reimplement it in various languages so they can share the suck?
763 2012-12-19 18:16:20 <phantomcircuit> helo, im nto sure that's possible honestly
764 2012-12-19 18:16:38 <phantomcircuit> testing and verifying correctness is hugely expensive
765 2012-12-19 18:17:59 <gmaxwell> helo: that doesn't solve the problem at all.
766 2012-12-19 18:18:27 <gmaxwell> The concern about the weirdness with non-canonical signatures is not just that it makes it harder to write a safe node??? though it does??? its that it makes it so that third parties can modify your transactions.
767 2012-12-19 18:19:55 <gmaxwell> fortunately there isn't currently any obvious way to profit from doing that... but it can be a nuisance at a minimum.
768 2012-12-19 18:20:50 <helo> so when a node receives a transaction, they could modify it and distribute the modified one instead?
769 2012-12-19 18:20:57 <gmaxwell> Correct.
770 2012-12-19 18:21:01 <gmaxwell> Or a miner.
771 2012-12-19 18:21:05 <helo> what can they modify?
772 2012-12-19 18:21:32 <gmaxwell> they can change between any of the signature encodings openssl will accept. Which changes the txnid.
773 2012-12-19 18:22:09 <gmaxwell> and results in something that looks like a double spend, but isn't really??? which may or may not be accepted by the recipents wallet (depending on how their signature validation code is implemented).
774 2012-12-19 18:23:09 <gmaxwell> It isn't currently an urgent problem, but it's why we'd take the route of forcing everything into a canonical form instead of trying to get everyone to copy openssl's specific permissiveness.
775 2012-12-19 18:25:23 <helo> so at some point there will be blockheight-dictated acceptable sig encodings?
776 2012-12-19 18:37:08 <gmaxwell> helo: I think thats likely. But thats a ways off.
777 2012-12-19 18:38:24 <gmaxwell> First we would make them non-standard??? non-relayed and non-mined. And once that is good and settled then thinking can begin about making it enforced. But I think the latter part is probably only interesting once all the malleability is out, and unfortunately there is one kind that all current signers produce randomly.
778 2012-12-19 19:45:39 <sipa> gmaxwell: i think therr is one very widely possible malleability left: extra data elements in the input scripts
779 2012-12-19 19:55:08 <sipa> with a relatively easy but perhaps controversial fix: require that the final stack has only one element left, which must be nonzero
780 2012-12-19 19:55:33 <sipa> hmm... may be incompatible with some multisig stuff
781 2012-12-19 20:43:52 <midnightmagic> what are these microtransactions that mike hearn is talking about in his talk at bitcoin 2012?
782 2012-12-19 20:44:11 <midnightmagic> is this related to what gavin is doing?
783 2012-12-19 20:44:20 <midnightmagic> https://www.youtube.com/watch?v=mD4L7xDNCmA
784 2012-12-19 20:46:50 <gmaxwell> sipa: we need to think carefully about the use of NOPs in the future and how that interacts. (in particular, how it interacts with using them to add another cryptographic signature type)
785 2012-12-19 21:00:08 <gavinandresen> gmaxwell: if we re-define a NOP in the future we should bump the transaction.version number; if you get a transaction.version you don't understand, consider it non-standard but accept it if it gets into a block.
786 2012-12-19 21:00:20 <gavinandresen> (and validates, of course)
787 2012-12-19 21:00:26 <gavinandresen> (under the old rules)
788 2012-12-19 21:16:37 <gmaxwell> oh beautful, that should allow us to close down the malleability for everything without burdening the future.
789 2012-12-19 21:38:08 <forrestv> sipa, any comments on the updated pullreq? https://github.com/bitcoin/bitcoin/pull/2115
790 2012-12-19 21:45:54 <kuzetsa> what's the idea behind the "keypool" line of the bitcoin.conf file anyway? the comments say something about: "...so wallet backups will be valid for both prior transactions and several dozen future transactions..."
791 2012-12-19 21:46:13 <kuzetsa> I suspect... is it related to the "change" addreses or something?
792 2012-12-19 21:48:55 <gmaxwell> kuzetsa: not just change, though that too.
793 2012-12-19 21:49:14 <gmaxwell> e.g. you make a backup. then 10 seconds later you get a new address from bitcoin so someone can pay you.
794 2012-12-19 21:49:23 <gmaxwell> It would be nice if that didn't invalidate your backup.
795 2012-12-19 21:49:29 <gmaxwell> And it doesn't??? because of the keypool.
796 2012-12-19 21:51:01 <kuzetsa> right
797 2012-12-19 21:52:20 <gmaxwell> It just precomputes the next N addresses that it will use, so they end up in the backup and if you repeat backups at least every N addresses that you consume your backups will be good.
798 2012-12-19 21:52:48 <kuzetsa> yeah
799 2012-12-19 21:55:35 <kuzetsa> mind if I quote you on that?
800 2012-12-19 21:56:01 <kuzetsa> I mean, it sounds factual / common sense, but I like your explaination better than I would've worded it.
801 2012-12-19 21:56:25 <gmaxwell> fine with me
802 2012-12-19 22:04:00 <midnightmagic> oh, that's interesting. CWallet::ReserveKeyFromKeyPool() and the rpc calls getnewaddress() keypoolrefill() fillthe keypool immediately with new keys.
803 2012-12-19 22:04:17 <midnightmagic> I thought they didn't top up until it was heading towards empty
804 2012-12-19 22:05:05 <kuzetsa> Thanks gmaxwell ---> https://bitcointalk.org/index.php?topic=86854.msg1410337#msg1410337 (( going AFK for a bit though ))
805 2012-12-19 22:10:50 <gmaxwell> midnightmagic: oh no, that wouldn't be good. It fills when it can??? though if you're encrypted it can't fill if you're locked and getnewaddress though it will fill when you unlock.
806 2012-12-19 22:14:04 <midnightmagic> gmaxwell: Well.. for some definition of "empty": anyway I didn't realise at all that every key consumed appears to generate a fresh key in the keypool. I do see that there is a keygen thread started in walletpassword() which implies eventually it will be possible to retrieve a key from the keypool before the keypool is fully topped up; but currently it looks like the process is synchronous. That suggests to me that it was only
807 2012-12-19 22:14:10 <midnightmagic> done to facilitate longer-term periods for backups.
808 2012-12-19 22:16:10 <midnightmagic> holy crap you talk a lot. it takes forever to grep back for old things you've said. :)
809 2012-12-19 22:16:47 <Jouke> I have once heard that mtgox used the satoshi-client, but doesn't anymore because the walletformate didn't scale to such proportions. Is this still the case?
810 2012-12-19 22:17:12 <midnightmagic> Jouke: it seems to scale horizontally..
811 2012-12-19 22:17:43 <Jouke> What does that mean?
812 2012-12-19 22:18:44 <midnightmagic> Even if MagicalTux didn't have customizations, I think he could just put 20-50 individual bitcoind instances behind a load balancer
813 2012-12-19 22:18:46 <gmaxwell> you can run multiple bitcoinds to scale??? but it would be kind of crazy to do something like mtgox with bitcoind's wallet code as it stands today. Though you should think twice before doing something like that yourself, mtgox lost thousands of btc through buggy software plus their R&D costs.