1 2012-12-02 00:41:09 <riush> hm, what does this flag mean? https://github.com/bitcoin/bitcoin/blob/master/src/main.h#L1392
2 2012-12-02 00:44:23 <sipa> riush: that that block is invalid
3 2012-12-02 00:45:14 <sipa> riush: e.g. BLOCK_VALID_TREE | BLOCK_FAILED_VALID means that the checks to satisfy BLOCK_VALID_TRANSACTIONS failed
4 2012-12-02 00:45:25 <sipa> note that far from all combinations are used right now
5 2012-12-02 00:47:46 <riush> ah, i see.
6 2012-12-02 00:48:51 <riush> so a BLOCK_VALID_TRANSACTIONS | BLOCK_FAILED_VALID would be reconsidered in case of a reorg
7 2012-12-02 00:49:10 <sipa> why would it?
8 2012-12-02 00:49:22 <sipa> reorganisations don't change the validness of a block at all
9 2012-12-02 00:50:05 <sipa> validness is a const boolean property of a block, but there are several steps in verifying it
10 2012-12-02 00:50:40 <riush> but a tx that now looks like a doublespend can become valid in a reorg, no?
11 2012-12-02 00:51:26 <sipa> there are no double spends
12 2012-12-02 00:51:45 <sipa> they simply don't exist from the point of view of a single chain
13 2012-12-02 00:52:32 <sipa> if you mean a transaction that tries to spend a spent input... that's invalid, and nothing will change that for a given block
14 2012-12-02 00:52:42 <sipa> for the transaction, yes, it may change
15 2012-12-02 00:53:08 <riush> ah now i see what you mean
16 2012-12-02 00:53:23 <sipa> but there is no way that a block (which references its own history) could go from invalid to valid, ever
17 2012-12-02 00:53:49 <riush> of course, the tx will be in a new block after the reorg :)
18 2012-12-02 00:54:04 <sipa> indeed, and for that new block everything will need to be revalidated
19 2012-12-02 00:54:40 <riush> ok.. then what's the use in knowing which validation failed?
20 2012-12-02 00:54:50 <sipa> 0
21 2012-12-02 00:54:58 <riush> ah :)
22 2012-12-02 00:55:38 <sipa> why would there be a use for that?
23 2012-12-02 00:58:04 <riush> well because there are separate flags for them i assumed they might have some kind of deeper meaning
24 2012-12-02 00:58:21 <gmaxwell> It's useful for diagnostics to know what failed (e.g. in logs).
25 2012-12-02 00:59:34 <sipa> riush: those flags don't tell you which transaction failed; they tell you a) which checks still have to be performed b) whether it's considered valid so far or not
26 2012-12-02 00:59:42 <sipa> for a block
27 2012-12-02 01:01:02 <riush> aah! so you don't have to do all checks at once and can store the intermediary result
28 2012-12-02 01:01:14 <riush> (that can be picked up by another thread/process)
29 2012-12-02 01:03:00 <sipa> exactly
30 2012-12-02 01:04:35 <sipa> the checks are already done is separate steps, by the way: 1) at receive time (e.g. for orphan blocks) 2) at store time (e.g. for stale blocks) 3) at connect time
31 2012-12-02 01:05:32 <sipa> 1) corresponds to BLOCK_VALID_HEADER (+ a bit more), 2) to BLOCK_VALID_TRANSACTIONS, 3) to BLOCK_VALID_SCRITPS
32 2012-12-02 01:08:58 <riush> cool. it seems very well thought out btw. nice work!
33 2012-12-02 01:09:21 <riush> can read all headers/blocks through blktree now in ruby :)
34 2012-12-02 01:21:26 <andrew12-> ;;seen noagendamarket
35 2012-12-02 01:21:26 <gribble> noagendamarket was last seen in #bitcoin-dev 8 weeks, 0 days, 20 hours, 37 minutes, and 13 seconds ago: <noagendamarket> thats where you end up with brokers who get squeezed by the feds :P
36 2012-12-02 01:21:46 <noagendamarket> what ?
37 2012-12-02 01:22:06 <noagendamarket> :P
38 2012-12-02 01:23:26 <sipa> riush: you're implementing a parser for bitcoin's new data files in ruby?
39 2012-12-02 01:23:44 <andrew12-> noagendamarket: ah you do still exist
40 2012-12-02 01:24:05 <noagendamarket> im a figment of your imagination :P
41 2012-12-02 01:24:24 <andrew12-> i remember you from like two years ago.. weren't you the guy that made that reddit clone that let you upvote things for btc?
42 2012-12-02 01:24:27 <andrew12-> something like that...
43 2012-12-02 01:24:50 <noagendamarket> sort of
44 2012-12-02 01:25:01 <noagendamarket> till the developer stole the site
45 2012-12-02 01:25:15 <andrew12-> fun
46 2012-12-02 01:25:21 <noagendamarket> I seem to have bad luck with that :D
47 2012-12-02 01:25:37 <andrew12-> hard to trust people sometimes.
48 2012-12-02 01:25:39 <andrew12-> do you remember me?
49 2012-12-02 01:27:20 <gmaxwell> sipa: recovered on reindex. No evidence of a reorg anywhere near that block the logs.
50 2012-12-02 01:28:58 <sipa> gmaxwell: meh :S
51 2012-12-02 01:30:48 <gmaxwell> hm. the reindex took it as far as "blocks" : 210524,
52 2012-12-02 01:30:54 <gmaxwell> And it seems to just be sitting there now.
53 2012-12-02 01:31:03 <noagendamarket> yes andrew12 I do
54 2012-12-02 01:31:26 <gmaxwell> been about 7 minutes.
55 2012-12-02 01:31:45 <sipa> gmaxwell: yeah, there is no auto-start of IBD after -reindex or -loadblock
56 2012-12-02 01:31:52 <sipa> so you'll need to a wait for a new block
57 2012-12-02 01:32:04 <sipa> or restart
58 2012-12-02 01:32:15 <gmaxwell> ah. okay.
59 2012-12-02 01:33:11 <gmaxwell> ;;bc,blocks
60 2012-12-02 01:33:11 <gribble> 210532
61 2012-12-02 01:33:17 <gmaxwell> k, synced up after a restart.
62 2012-12-02 01:38:16 <Pucilowski> I want to work with secp256k1 in java - what could I use?
63 2012-12-02 01:38:49 <sipa> bouncycastle
64 2012-12-02 01:38:52 <sipa> see BitcoinJ
65 2012-12-02 01:39:05 <Pucilowski> cheers
66 2012-12-02 01:39:16 <gmaxwell> sipa: 2060 is kinda leaky.
67 2012-12-02 01:39:36 <gmaxwell> (I mean it includes commits from your other pulls)
68 2012-12-02 01:39:37 <gmaxwell> (I mean it includes commits from your other pulls)
69 2012-12-02 01:39:55 <sipa> gmaxwell: and that is even mentioned :)
70 2012-12-02 01:40:25 <etotheipi_> anyone here use "Coin Control"?
71 2012-12-02 01:40:55 <etotheipi_> I'm trying to figure out what are the common use cases for it
72 2012-12-02 01:41:08 <etotheipi_> (specifically, how people use it)
73 2012-12-02 01:42:33 <sipa> i think it's useful for one purpose only: transparency (letting people see how things work underneath, educationally)
74 2012-12-02 01:43:22 <sipa> for every other use case, you either want multi-wallet support or a coin selection algorithm that automatically tries to prevent unnecessary coin linkage
75 2012-12-02 01:44:07 <riush> sipa: yes. the current postgres database we have in bitcoin-ruby ist just too big and slow for regular clients and generally having a separate utxo index seems to be the way to go
76 2012-12-02 01:44:27 <riush> so i figured might as well look at what you've already done, instead of reinventing that wheel
77 2012-12-02 01:44:42 <sipa> you know you can't access those files while bitcoin is running?
78 2012-12-02 01:44:50 <riush> yes sure
79 2012-12-02 01:45:04 <sipa> ok, just making sure
80 2012-12-02 01:45:43 <riush> but it would still be awesome to be able to switch easily, without waiting for ruby to do IBD
81 2012-12-02 01:46:58 <riush> but i guess reading is the easy part so have to see how it goes ;)
82 2012-12-02 01:51:45 <gmaxwell> sipa: your par pull appears to have added an unexpected shutdown delay?
83 2012-12-02 01:51:46 <gmaxwell> 12/02/12 02:49:57 ThreadMessageHandler exited
84 2012-12-02 01:52:27 <sipa> that's strange
85 2012-12-02 01:52:30 <gmaxwell> hm maybe just a coincidence.
86 2012-12-02 01:52:52 <gmaxwell> (It isn't like I normally watch shutdown closely)
87 2012-12-02 02:05:41 <sipa> ;;bc,blocks
88 2012-12-02 02:05:42 <gribble> 210535
89 2012-12-02 03:23:56 <etotheipi_> well, I am implementing coin control, because of extremely high demand for it... but I am not entirely clear how to tailor it
90 2012-12-02 03:24:32 <etotheipi_> I guess I just show them a list of non-empty addresses and let them select a subset from which they want their transaction to be created
91 2012-12-02 03:25:44 <etotheipi_> while experimenting with it, I did find one use: I was able to select every address that had more than 4 decimal places, then create a single transaction that splits it in an integer part and non-integer (actually, I donated the non-int)... it really cleans up the wallet
92 2012-12-02 03:26:39 <etotheipi_> collapsing all your BTC crumbles into a single output is probably good for the network
93 2012-12-02 03:27:23 <helo> filtering might be nice... tag various addresses to create sets for filtering?
94 2012-12-02 03:28:04 <etotheipi_> helo: filtering for what?
95 2012-12-02 03:28:31 <helo> "create this transaction using only funds sent to addresses in the "donations" category
96 2012-12-02 03:28:45 <helo> or "tagged with 'donations'"
97 2012-12-02 03:28:53 <freewil> that's what accounts are for
98 2012-12-02 03:29:06 <etotheipi_> that's what multiple wallets are for, in Armory
99 2012-12-02 03:29:13 <freewil> in bitcoind accounts are just groups of addresses
100 2012-12-02 03:29:19 <etotheipi_> but it doesn't quite have the same flexibility as "tagging"
101 2012-12-02 03:29:45 <helo> expand accounts to tagging, allow tag filtering for coin control?
102 2012-12-02 03:30:02 <helo> would kind of connect receiving and sending
103 2012-12-02 03:30:10 <etotheipi_> helo: it's an interesting idea... I've always been a fan of tag-sorting sets
104 2012-12-02 03:30:25 <Luke-Jr> etotheipi_: I found Coin Control annoying since it didn't give me actual per-coin control.
105 2012-12-02 03:30:45 <etotheipi_> Luke-Jr: selecting source addresses wasn't enough?
106 2012-12-02 03:30:54 <Luke-Jr> etotheipi_: no
107 2012-12-02 03:31:12 <etotheipi_> gah, I was afraid someone would complain about that
108 2012-12-02 03:31:24 <etotheipi_> I don't really want to expand the coin-control further to accommodate per-coin
109 2012-12-02 03:32:17 <etotheipi_> Luke-Jr: but what's the use of that? I thought that most use cases for coin control was really so users could select how their addresses were linked
110 2012-12-02 03:32:45 <Luke-Jr> etotheipi_: sending a specific dirty coin back to its rightful owner
111 2012-12-02 03:32:54 <helo> the best analogy i've heard describing how bitcoin transactions work is "take a bunch of buckets of water, and divide it up into several new buckets"
112 2012-12-02 03:33:13 <helo> so make it easy to select which are the input buckets, and the output
113 2012-12-02 03:33:47 <helo> tagging on both sides?
114 2012-12-02 03:34:35 <etotheipi_> Luke-Jr: I'm not a fan of treating BTC as non-fungible
115 2012-12-02 03:34:51 <helo> it would be nice if it would work with colored coins, btw
116 2012-12-02 03:35:02 <etotheipi_> man, a lot of talk of colored coins these days...
117 2012-12-02 03:35:32 <helo> if you're making something, you might as well factor that in just to cover your bases
118 2012-12-02 03:35:49 <helo> coin control is basically the only requirement for colored bitcoin
119 2012-12-02 03:36:14 <etotheipi_> does it require per-coin control? or per-address?
120 2012-12-02 03:37:18 <helo> it requires some kind of agreement on arbitrary rules regarding how color is transferred in a transaction with colored and non-colored coin :/
121 2012-12-02 03:37:33 <helo> (quickly making coin control a pita)
122 2012-12-02 03:37:59 <etotheipi_> helo: sounds like my interface would need to display individual TxOuts, not just non-empty addresses
123 2012-12-02 03:38:07 <helo> i think sipa was at least talking about atomic transactions...
124 2012-12-02 03:38:37 <helo> which imo are colored coin's killer app feature
125 2012-12-02 03:40:09 <etotheipi_> I think I'll leave what I've got and move onto the new wallet format with multi-sig
126 2012-12-02 03:40:37 <etotheipi_> I think it makes sense to make multi-sig a priority
127 2012-12-02 03:40:50 <helo> when coin control and accounts are rehauled, ideally it would be long term enough to facilitate colored coin
128 2012-12-02 03:40:59 <helo> yeah, and multi-sig... coin control goes pretty deep
129 2012-12-02 03:43:03 <helo> i think that's mainly why no work has been done heh
130 2012-12-02 04:18:41 <etotheipi_> so what are some good places to work that you can work completely remotely? I know Red Hat had some such opportunities...
131 2012-12-02 04:22:17 <weex> etotheipi_: i think most places want you to come in 1x per week at least
132 2012-12-02 04:22:42 <weex> but if you go for international coding jobs for btc, it's unlikely you'll need to leave to do anything
133 2012-12-02 04:24:25 <etotheipi_> I'm not sure that I'd be willing to make the leap to getting paid in BTC
134 2012-12-02 04:25:11 <gmaxwell> weex: lots of tech companies support teleworking, though it's generally more available to senior positions and people who've previously had a deskjob at the same place.
135 2012-12-02 04:25:12 <gmaxwell> weex: lots of tech companies support teleworking, though it's generally more available to senior positions and people who've previously had a deskjob at the same place.
136 2012-12-02 04:26:05 <etotheipi_> ...or perhaps people who have a history of producing good work at home...? (like, say, 2000 hours on an open-source project in my spare time?)
137 2012-12-02 04:26:33 <gmaxwell> I'm not a big fan of it, I was work-from-home for the first 5 years of my current job and requested the company move me across the company so I could work out of an office.
138 2012-12-02 04:27:02 <weex> there are definite benefits to being in offices with people from time to time
139 2012-12-02 04:27:04 <etotheipi_> I've been working in an office for 6 years, and I'm just so much more comfortable and productive at home
140 2012-12-02 04:27:52 <gmaxwell> I would rather work at home than at an office I lived more than biking distance from though... being in the office is not enough of a benefit to offset a long commute.
141 2012-12-02 04:28:03 <etotheipi_> the last 2 months, I've gotten very little done, because... I'm on four different projects now, constantly going to meetings, and dragged into every problem-solving/troubleshooting exercise everyone else has...
142 2012-12-02 04:29:40 <gmaxwell> Thats something that happen even when you're remote too. Worse, when you get pulled onto a dozen projects and you're falling behind on all of them while remote people assume you're slacking because they only see you when you advance the needle on the project they care about.
143 2012-12-02 04:30:18 <etotheipi_> but at least I can segregate...
144 2012-12-02 04:30:19 <randy-waterhouse> people who are productive working from home are goldmines for the corp since they are cheap to accomodate and by necessity are highly-motivated, independent and focussed
145 2012-12-02 04:30:50 <etotheipi_> the problem at work is that 5+ times per day, someone walks in my office/lab, and drags me out of whatever state of mind I was in
146 2012-12-02 04:30:53 <randy-waterhouse> self-disciplined
147 2012-12-02 04:31:03 <etotheipi_> the context switching is killing me
148 2012-12-02 04:31:36 <etotheipi_> and because of the sensitive nature of my projects, my lab can't have internet... so I have to walk two buildings over to look up stupid stuff
149 2012-12-02 04:31:51 <gmaxwell> etotheipi_: just saying??? that can be fixed in the context of an office. Working from home might also fix that, but it also leaves you less connected unless the org is highly distributed.
150 2012-12-02 04:31:55 <etotheipi_> as you can tell, I'm reaching a threshold...
151 2012-12-02 04:32:32 <gmaxwell> (and a org that doesn't let people concentrate on jobs that need concentration is dysfunctional)
152 2012-12-02 04:32:48 <etotheipi_> my job is great.... but maybe I just don't say "no" enough
153 2012-12-02 04:32:57 <etotheipi_> I have LOVED working there so far...
154 2012-12-02 04:33:05 <etotheipi_> and if I could just do this work from home, I'd be super happy
155 2012-12-02 04:33:32 <gmaxwell> I've certantly run into time management problems from not saying no enough.
156 2012-12-02 04:33:42 <gmaxwell> Perhaps you need a tent: http://www.jwz.org/tent-of-doom/net1.jpg to isolate you from the office. :P
157 2012-12-02 04:34:01 <etotheipi_> haha
158 2012-12-02 04:35:08 <randy-waterhouse> a pair of large headphones and a bad attitude has been known to work also
159 2012-12-02 04:35:49 <etotheipi_> it doesn't work
160 2012-12-02 04:36:01 <etotheipi_> if I'm not talking to someone, I'm prime for question asking
161 2012-12-02 04:36:09 <gmaxwell> A cute "do not disturb, next context switch in N minutes" sign/display might help?
162 2012-12-02 04:36:42 <etotheipi_> I had an idea... I get one stopwatch for each of the four people that take up most of my time
163 2012-12-02 04:36:42 <randy-waterhouse> IRQ flag system on a flipboard?
164 2012-12-02 04:36:50 <gmaxwell> perhaps a ticket dispencer and a NOW SERVING PROJECT [88] display? :P
165 2012-12-02 04:37:09 <etotheipi_> tell them they have 20 min per day of my time, and with a 10 min latency after ascking
166 2012-12-02 04:37:16 <randy-waterhouse> something to handle the context threads would be useful
167 2012-12-02 04:37:59 <etotheipi_> because a lot of times they would just figure out what they're doing without me, if they spent more time... but since I'm *right there* they just ask me
168 2012-12-02 04:38:12 <gmaxwell> etotheipi_: I assume that since you're handling security sensitive things that you couldn't work some number of days a week from home with the current employer?
169 2012-12-02 04:38:25 <etotheipi_> gmaxwell: that's correct
170 2012-12-02 04:38:59 <etotheipi_> my employer does make such arrangements, but I'm not lucky enough to be on a suitable project for htat
171 2012-12-02 04:39:17 <gmaxwell> is there an unused portion of your building? or just another division thats still security cleared but where no one knows you? and perhaps you could ask to have your seat relocated?
172 2012-12-02 04:39:35 <etotheipi_> haha, interesting idea
173 2012-12-02 04:40:32 <etotheipi_> I think my supervisor really likes me where I am, since I am helping everyone else get their work done, all the time
174 2012-12-02 04:41:11 <etotheipi_> I had tried to get a secure computer in my office, so I could escape the open lab that has 10 people working together
175 2012-12-02 04:41:22 <etotheipi_> but it's been 8 months, and they still havne't approved it...
176 2012-12-02 04:42:41 <etotheipi_> maybe I just need to pester the security people about that, and that would be 50% better....not to mention I could be near an internet connection, too
177 2012-12-02 04:43:03 <etotheipi_> okay, i'm going to stop rambling about my job... I just wanted to know where I could "look around" for remote employment
178 2012-12-02 04:43:40 <doublec> etotheipi_: mozilla has many remote employees
179 2012-12-02 04:43:57 <etotheipi_> doublec: that's a good point
180 2012-12-02 04:44:13 <doublec> etotheipi_: disclaimer being I work for them
181 2012-12-02 04:48:09 <doublec> etotheipi_: not much listed in the security area but who knows http://careers.mozilla.org/en-US/
182 2012-12-02 04:48:25 <doublec> I'm sure browsers could always do with more security work
183 2012-12-02 04:48:29 <etotheipi_> meh... I'm more interested in algorithm development/optimization
184 2012-12-02 04:49:27 <etotheipi_> things like image/signal processing... I don't think Mozilla's got that for me...
185 2012-12-02 04:49:40 <doublec> hmm, how about video/audio codec development
186 2012-12-02 04:49:42 <etotheipi_> AI, datamining
187 2012-12-02 04:50:18 <etotheipi_> doublec: probably not
188 2012-12-02 04:50:32 <etotheipi_> but I haven't done it before, so who knows
189 2012-12-02 04:53:11 <etotheipi_> gah, maybe I'll just focus on getting that system approved in my office, that would improve a lot of things...
190 2012-12-02 04:53:14 <etotheipi_> for now, bedtime...
191 2012-12-02 04:53:26 <etotheipi_> thanks for listening to my rant
192 2012-12-02 09:35:56 <remiforall> hello
193 2012-12-02 12:33:38 <BitDev> hi all
194 2012-12-02 12:33:47 <BitDev> can some one help me again :)
195 2012-12-02 12:34:18 <BitDev> i dont get it, how to calculate merkle root hash
196 2012-12-02 12:35:16 <sipa> http://bitcoin.stackexchange.com/questions/1110/how-do-i-implement-a-merkle-tree
197 2012-12-02 12:37:57 <BitDev> yes, and if i have 4 hashes, then First round is: R1 = sha256(H1 + H2); R2 = sha256(H3 + H4); then next round is: result = sha256(R1 + R2) ?
198 2012-12-02 12:38:42 <BitDev> where in sources i can find how its work?
199 2012-12-02 12:38:48 <BitDev> maybe you know sipa?
200 2012-12-02 12:39:47 <phantomcircuit> BitDev, it isn't a straight sha256
201 2012-12-02 12:39:48 <phantomcircuit> BitDev, it isn't a straight sha256
202 2012-12-02 12:39:54 <phantomcircuit> iirc it's sha256(sha256())
203 2012-12-02 12:40:12 <BitDev> ow, sorry, i forget that - thnx
204 2012-12-02 12:40:14 <jeremias> hmm is there a specific reason for double sha256
205 2012-12-02 12:40:20 <sipa> indeed, bitcoin always uses sha256^2 or ripemd160.sha256
206 2012-12-02 12:40:27 <phantomcircuit> ;;bc,blocks
207 2012-12-02 12:40:28 <gribble> 210595
208 2012-12-02 12:40:34 <sipa> BitDev: see the obscurely-named function BuildMerkleTree in main.h
209 2012-12-02 12:40:56 <BitDev> sipa thnx!!! you help me so many times, God bless you!
210 2012-12-02 12:41:51 <phantomcircuit> sipa, lol when genjix was first starting work on libbitcoin he got to the merkletree and was just like
211 2012-12-02 12:41:52 <phantomcircuit> "wat"
212 2012-12-02 12:42:00 <phantomcircuit> that code is so weird
213 2012-12-02 12:43:36 <sipa> yes, concatenating the different levels in one vector makes it kinda hard to read
214 2012-12-02 12:43:52 <sipa> also, why does that code use std::vector::reserve()?
215 2012-12-02 12:46:20 <sipa> *doesn't
216 2012-12-02 12:51:20 <BitDev> two hashes of txn must be added or some other operation?
217 2012-12-02 12:51:48 <sipa> concatenated
218 2012-12-02 12:52:13 <sipa> so the input to double-sha256 in the merkle tree algorithm is always a 64-byte structure
219 2012-12-02 12:52:15 <BitDev> how?
220 2012-12-02 12:52:38 <sipa> the 32 first bytes are the first input hash, the 32 second bytes are the second input hash
221 2012-12-02 12:53:52 <sipa> gmaxwell, jgarzik: reindexing with script check enabled everywhere until block 210000: 1h14m
222 2012-12-02 12:55:27 <BitDev> so first of all i must find out txn hash (T-hash1) then i must double this hash (cuz just 1 txn hash) (T-hash2) after that i must find hash of [T-hash1 32 bytes as high of 64 bytes + T-hash2 32 bytes if low part of 64 bytes]?
223 2012-12-02 12:56:12 <sipa> the output of sha256 is only 32 bytes
224 2012-12-02 12:56:17 <sipa> *sigh*
225 2012-12-02 12:56:31 <sipa> just concatenate the two hashes and feed it to the next hash function
226 2012-12-02 12:56:48 <sipa> gmaxwell, jgarzik: that's with 4 script verification threads; without parallellism, it was 3h22m
227 2012-12-02 12:57:21 <BitDev> so i must hash 64 bytes?
228 2012-12-02 12:57:26 <sipa> yes
229 2012-12-02 12:57:40 <sipa> 14:52:15 < sipa> so the input to double-sha256 in the merkle tree algorithm is always a 64-byte structure
230 2012-12-02 12:57:43 <BitDev> ow, its simple )
231 2012-12-02 12:57:44 <BitDev> thnx
232 2012-12-02 13:23:36 <Pucilowski> What's the correct practice in converting a user-given BTC input to satoshis in java?
233 2012-12-02 14:14:28 <abrkn> how do i make an address show in "receive coins" after i used importprivkey?
234 2012-12-02 14:14:44 <abrkn> trying to get my mywallet-address into bitcoin-qt for signing on otc
235 2012-12-02 14:14:57 <sipa> it should do that automatically
236 2012-12-02 14:15:11 <abrkn> definitely not seeing it
237 2012-12-02 14:15:12 <sipa> but maybe the Qt interface doesn't show it immediately
238 2012-12-02 14:15:18 <sipa> you may need to restart
239 2012-12-02 14:15:26 <abrkn> ok, trying that
240 2012-12-02 14:16:04 <abrkn> :D
241 2012-12-02 14:16:08 <abrkn> thanks again sipa
242 2012-12-02 14:16:20 <sipa> feel free to file a bug for that
243 2012-12-02 14:16:47 <phantomcircuit> mywallet-address ?
244 2012-12-02 14:17:11 <abrkn> phantomcircuit: yeah, a guy on otc wanted to use bitcoin address verificiation while still using blockchain.info wallet to manage his btc
245 2012-12-02 14:17:28 <abrkn> sipa: https://github.com/bitcoin/bitcoin/issues/835
246 2012-12-02 14:17:36 <sipa> i was wondering how you got to the private key of a mywallet account
247 2012-12-02 14:17:45 <sipa> but you're talking about blockchain.info?
248 2012-12-02 14:17:48 <abrkn> yes
249 2012-12-02 14:18:05 <abrkn> just used the unencrypted export on blockchain.info/wallet
250 2012-12-02 14:18:11 <phantomcircuit> abrkn, that is mildly strange but ok
251 2012-12-02 14:18:27 <abrkn> phantomcircuit: well, it's just because mywallet doesnt support signing messages that i can find
252 2012-12-02 14:18:27 <sipa> what is mildly strange?
253 2012-12-02 14:19:06 <phantomcircuit> using bitcoin private keys for message signing
254 2012-12-02 14:19:13 <phantomcircuit> i just dont see the point
255 2012-12-02 14:19:19 <phantomcircuit> but i guess other people do
256 2012-12-02 14:19:22 <sipa> proving you own an address
257 2012-12-02 14:19:28 <abrkn> i dont find it strange at all
258 2012-12-02 14:19:53 <abrkn> what better way to prove who you are than proving ownership of your money :)
259 2012-12-02 14:21:00 <abrkn> sipa: i see the issue was closed 7 months ago, but which version is that? im using 70100
260 2012-12-02 14:23:11 <sipa> abrkn: i commented
261 2012-12-02 14:25:17 <phantomcircuit> lol
262 2012-12-02 14:25:26 <phantomcircuit> it's ridiculous but i need a faster workstation
263 2012-12-02 14:25:34 <abrkn> workstation? in 2012? :-D
264 2012-12-02 14:25:42 <abrkn> i need a slower one, some kind of tablet with monitor support
265 2012-12-02 14:25:54 <phantomcircuit> yes workstation
266 2012-12-02 14:26:08 <phantomcircuit> laptops are significantly more likely to break than a desktop
267 2012-12-02 14:26:22 <phantomcircuit> and it's almost impossible to get a laptop with >8GB of ram
268 2012-12-02 14:27:04 <sipa> i'm benchmarking my parallel signature check patch for bitcoind on my laptop... i'm afraid i'm killing it, judging by the temperature of the air coming out of the fan...
269 2012-12-02 14:27:17 <phantomcircuit> yeah that's a major issue with laptops
270 2012-12-02 14:27:34 <phantomcircuit> you can only run them full bore for about a minute before they start to overheat usually
271 2012-12-02 14:28:11 <phantomcircuit> i can run this workstation totally maxed out all day and the cpu wont break 60C
272 2012-12-02 14:29:34 <sipa> temp1: +99.0??C (crit = +103.0??C)
273 2012-12-02 14:29:36 <sipa> :S
274 2012-12-02 14:29:54 <phantomcircuit> ahaha
275 2012-12-02 14:30:19 <phantomcircuit> that's not good
276 2012-12-02 14:30:20 <phantomcircuit> mkk?
277 2012-12-02 14:30:24 <phantomcircuit> mkkay?
278 2012-12-02 14:30:35 <sipa> if that is true, indeed it isn't
279 2012-12-02 14:31:01 <phantomcircuit> intel machine?
280 2012-12-02 14:31:13 <sipa> yes, i7-2670QM
281 2012-12-02 14:31:28 <phantomcircuit> it's right
282 2012-12-02 14:31:49 <sipa> running at 2.2 GHz now
283 2012-12-02 14:32:19 <sipa> 12k blocks left to go...
284 2012-12-02 14:32:34 <phantomcircuit> the main problem you'll have from that kind of temp isn't the temp itself but the damage from thermo-cycling a lot
285 2012-12-02 14:32:57 <sipa> yup
286 2012-12-02 14:33:08 <phantomcircuit> i have a T61 which is all screwed up because the heat from the gpu caused parts of the mobo to disconnect
287 2012-12-02 14:33:08 <sipa> it smells slightly burnt-pasticy
288 2012-12-02 14:33:10 <phantomcircuit> i uh
289 2012-12-02 14:33:13 <phantomcircuit> "repaired" it
290 2012-12-02 14:33:17 <phantomcircuit> but it's not reliable
291 2012-12-02 14:33:20 <sipa> *plasticy
292 2012-12-02 14:33:28 <phantomcircuit> yeah that smell is bad news
293 2012-12-02 14:33:33 <_dr> where is the crit 103 value from?
294 2012-12-02 14:33:37 <_dr> that doesn't sound right :)
295 2012-12-02 14:36:24 <phantomcircuit> _dr, for a laptop it's right
296 2012-12-02 14:38:29 <_dr> running a laptop permanently (>5m) sounds right to you?
297 2012-12-02 14:38:38 <_dr> *at 100C
298 2012-12-02 14:39:20 <_dr> maybe i'm just pampered by my macbooks, they hardly get any warmer than 30C on the outside
299 2012-12-02 14:41:15 <phantomcircuit> _dr, 103C is the critical point for the sensor in the cpu
300 2012-12-02 14:41:26 <phantomcircuit> the case isn't going to get anywhere near that hot
301 2012-12-02 14:41:32 <sipa> i don't care about outside temperature (as long as touching the keyboard doesn't burn my fingers)
302 2012-12-02 14:41:53 <phantomcircuit> that's the point at which your cpu will start to turn off cores and/or lower the clock
303 2012-12-02 14:42:07 <abrkn> i heard that the new macbooks cant play back a dvd without getting so hot between GH keys that it's unusable =)
304 2012-12-02 14:42:41 <phantomcircuit> sigh
305 2012-12-02 14:42:42 <_dr> phantomcircuit: they can actually take much more than that
306 2012-12-02 14:42:42 <phantomcircuit> sigh
307 2012-12-02 14:42:51 <phantomcircuit> stable version of libvirt doesn't support VIR_STORAGE_VOL_RESIZE_SHRINK
308 2012-12-02 14:42:55 <phantomcircuit> oh well
309 2012-12-02 14:43:11 <_dr> intel says they can take up to 125C, but i wouldn't want it ever to actually get that hot :)
310 2012-12-02 14:44:10 <_dr> maybe it's just a psychological barrier, 100C just sounds really hot
311 2012-12-02 14:45:11 <sipa> 94C... what happened?
312 2012-12-02 14:47:14 <phantomcircuit> sipa, you probably broke 103.0 which triggered clock to scale back
313 2012-12-02 14:49:07 <sipa> gmaxwell, jgarzik: with 4 threads + Hal's optimization: 1h1m to fully verify 210k blocks
314 2012-12-02 14:49:27 <sipa> 66C !
315 2012-12-02 14:51:56 <D34TH> ACTION doesn't know what that means, but it seems fast
316 2012-12-02 14:52:52 <phantomcircuit> ludicrous speed
317 2012-12-02 14:53:30 <sipa> D34TH: right now, the code only verifies scripts after block 193k
318 2012-12-02 14:53:52 <sipa> D34TH: i just did a benchmark that has it enabled everywhere, importing 210k blocks in 1h1m
319 2012-12-02 14:54:21 <D34TH> sipa, phantomcircuit: does it go plaid
320 2012-12-02 14:54:45 <sipa> before parallel verification and Hal's optimized secp256k1 verifier, that took 3h22m
321 2012-12-02 14:54:55 <sipa> plaid?
322 2012-12-02 14:55:03 <phantomcircuit> sipa, spaceballs
323 2012-12-02 14:55:07 <D34TH> sipa: spaceballs
324 2012-12-02 14:55:11 <phantomcircuit> oh god my brain is full of useless shit
325 2012-12-02 14:55:27 <sipa> i have no clue what you're talking about
326 2012-12-02 14:55:29 <D34TH> phantomcircuit, i came to that realization 6 years ago
327 2012-12-02 14:55:31 <phantomcircuit> D34TH, i was going to make that joke but i could remember how to spell plaid
328 2012-12-02 14:55:36 <D34TH> sipa: funny movie
329 2012-12-02 14:55:48 <D34TH> phantomcircuit, i had to ask someone D:
330 2012-12-02 14:55:53 <sipa> ok, small explanation of the context?
331 2012-12-02 14:56:05 <D34TH> ill get you a youtube link
332 2012-12-02 14:56:11 <phantomcircuit> sipa, millennium falcon at lightspeed
333 2012-12-02 14:56:16 <phantomcircuit> instead of the star pattern
334 2012-12-02 14:56:18 <phantomcircuit> it's plaid
335 2012-12-02 14:56:26 <sipa> eh ok
336 2012-12-02 14:56:34 <phantomcircuit> it's funnier in the movie
337 2012-12-02 14:56:42 <phantomcircuit> also it's funnier when you're 16
338 2012-12-02 14:57:40 <D34TH> sipa: https://www.youtube.com/watch?feature=player_detailpage&v=ygE01sOhzz0#t=80s
339 2012-12-02 15:00:41 <sipa> eh ok
340 2012-12-02 15:07:10 <_dr> that speedup doesn't seem too high
341 2012-12-02 15:07:19 <_dr> or does your machine only have two physical cores?
342 2012-12-02 15:07:31 <_dr> in which case it's a very nice speedup :)
343 2012-12-02 15:07:53 <D34TH> _dr isnt that like 300%?
344 2012-12-02 15:08:12 <sipa> _dr: no, but when running single-threaded, it runs at 3.1GHz, and when stressing all cores, it switches to 2.2GHz
345 2012-12-02 15:08:38 <_dr> D34TH: yeah, using 4x the hardware and some software optimization
346 2012-12-02 15:09:10 <_dr> sipa: you could try with 8 threads, crypto-foo tends to have lots of dependencies during calculations
347 2012-12-02 15:09:34 <sipa> _dr: already tried that using more than 4 doesn't seem to give any benefit
348 2012-12-02 15:09:35 <_dr> so SMT could avoid some empty blobs in the pipeline
349 2012-12-02 15:09:40 <_dr> oh, okay
350 2012-12-02 15:10:07 <_dr> did you pin the threads?
351 2012-12-02 15:10:16 <sipa> no
352 2012-12-02 15:11:08 <_dr> if only i had more time :)
353 2012-12-02 15:11:37 <_dr> i'm pretty sure you could even use simd, with avx and 32bit integers that should give another speedup of 8x
354 2012-12-02 15:12:00 <sipa> i have no idea what openssl is already using underneath, but i have no doubt further optimization is possible
355 2012-12-02 15:13:32 <phantomcircuit> sipa, you're pipelining the entire transaction verification or only the script verification?
356 2012-12-02 15:14:19 <sipa> just script verification
357 2012-12-02 15:14:53 <sipa> so there's one thread that verifies blocks, and it pushes script-verification actions to a queue, and then waits for that queue to go empty
358 2012-12-02 15:15:52 <sipa> so far from the most efficient implementation (where block verification doesn't wait for scripts to be finished processing), but certainly a less drastic change for now
359 2012-12-02 15:15:53 <sipa> so far from the most efficient implementation (where block verification doesn't wait for scripts to be finished processing), but certainly a less drastic change for now
360 2012-12-02 15:19:14 <_dr> ah i see, the block thread not helping with the script-queue explains the 3x speedup
361 2012-12-02 15:19:44 <sipa> yeah, i could improve things a bit by instead of waiting, also working on the queue
362 2012-12-02 15:19:52 <sipa> and only have 3 extra threads instead of 4
363 2012-12-02 15:20:18 <sipa> so there's a bit less switching overhead
364 2012-12-02 15:35:38 <_dr> mh, using additional threads for the queue means you should get a speedup of 4x nevertheless
365 2012-12-02 15:35:39 <_dr> mh, using additional threads for the queue means you should get a speedup of 4x nevertheless
366 2012-12-02 15:36:20 <_dr> maybe it's queue contention overhead, since the actual work is pretty small
367 2012-12-02 15:37:17 <sipa> there's certainly some queue contention, but the time for a single entry in the queue is around 0.6ms, which should be enough
368 2012-12-02 15:37:41 <_dr> i'd ditch the queue and use some sort of static scheduling. that way you don't need synchronization
369 2012-12-02 15:38:03 <sipa> the problem is that you have to wait for all threads to finish every time, so they can't remain 100% busy all the time
370 2012-12-02 15:38:28 <sipa> also, cpu speed going from 3.1GHz to 2.2GHz limits potential speedup
371 2012-12-02 15:38:57 <sipa> i think maybe 10% can be gained still
372 2012-12-02 15:39:35 <_dr> the 0.6ms is for the script verification? i'm not entirely familiar with ecdsa and how openssl handles that stuff
373 2012-12-02 15:39:50 <sipa> one script verification averages around 0.6ms here, yes
374 2012-12-02 15:40:10 <sipa> with hal's code it's around 0.45ms
375 2012-12-02 15:40:33 <_dr> does that code use openssl as well?
376 2012-12-02 15:40:38 <sipa> yes
377 2012-12-02 15:41:04 <sipa> it just precalculates some things, and preprocesses it so the actual numbers to work with are much smaller
378 2012-12-02 15:41:10 <_dr> where can i find it? maybe there's a way to ditch openssl entirely. i bet the implementation isn't that fast
379 2012-12-02 15:41:39 <sipa> i haven't seen any code anywhere faster than openssl's for secp256k1
380 2012-12-02 15:41:55 <riush> strange.. trying to sync a fresh node from a local leveldb bitcoind and it seems the database is corrupted.. but the scary thing is, it's sending me the broken stuff!
381 2012-12-02 15:41:56 <sipa> but a very specialized implementation will certainly be faster
382 2012-12-02 15:41:59 <riush> http://paste.mhanne.net/p/a217ab2bafb0a2ea4df5f94bea4b6a2512bc25bd
383 2012-12-02 15:42:10 <_dr> sipa: sure, but i bet it doesn't do simd
384 2012-12-02 15:43:06 <sipa> riush: eww :S
385 2012-12-02 15:43:25 <riush> (the corruption might likely be caused by me, but it shouldn't be sending it i guess)
386 2012-12-02 15:44:13 <sipa> riush: not sure it is sending bad data
387 2012-12-02 15:44:28 <sipa> riush: the first error is GetHash() doesn't match index... which is about data on disk
388 2012-12-02 15:51:06 <riush> sipa: but my node is receiving broken data.. the invs are ok but the blocks are somehow messed up
389 2012-12-02 15:51:10 <riush> http://paste.mhanne.net/p/238b24af637cd957e661416639ef1c724a6dd25a
390 2012-12-02 15:52:41 <sipa> i see no evidence the received data is bad/
391 2012-12-02 15:54:01 <riush> at the bottom, the ">> block" lines..
392 2012-12-02 15:54:30 <sipa> yes?
393 2012-12-02 15:54:36 <sipa> what do those mean?
394 2012-12-02 15:54:52 <sipa> it's receiving the same block over and over again?
395 2012-12-02 15:54:53 <sipa> it's receiving the same block over and over again?
396 2012-12-02 15:54:54 <riush> that should be the block hashes
397 2012-12-02 15:55:29 <sipa> are you sure your algorithm for calculating block hashes is right?
398 2012-12-02 15:55:33 <riush> yes, at first it seems to be random garbage (not starting with 0's) and then it's always the same
399 2012-12-02 15:56:11 <riush> well it *could* be a bug there of course, but since that always worked, and bitcoind is logging freaky stuff, i don't think that's it
400 2012-12-02 15:56:38 <riush> i can sync fine from random network nodes
401 2012-12-02 15:56:43 <sipa> oh, ok
402 2012-12-02 15:56:45 <sipa> now i get it
403 2012-12-02 15:57:15 <sipa> so in case of a corrupted index, bitcoind may be sending invalid block data (even when it's aware it couldn't read the data)
404 2012-12-02 15:57:28 <riush> yes, that's what i mean :)
405 2012-12-02 15:59:14 <BitDev> hi all, in bitcoin - how to generate my own address?
406 2012-12-02 15:59:42 <abracadabra> use bitcoin client
407 2012-12-02 15:59:48 <abracadabra> click "generate new address"
408 2012-12-02 16:00:00 <BitDev> now, i mean by my software )
409 2012-12-02 16:00:04 <sipa> BitDev: i don't mind answering questions, but if you need to ask about such basic things, perhaps you should try to do some research before trying to implement your own client
410 2012-12-02 16:00:34 <abracadabra> heh
411 2012-12-02 16:00:36 <BitDev> i was reading about it
412 2012-12-02 16:00:38 <abracadabra> ACTION crawls back under rock
413 2012-12-02 16:00:48 <BitDev> i need private and public keys
414 2012-12-02 16:01:11 <BitDev> and two keys is needed for each address?
415 2012-12-02 16:01:55 <sipa> so sorry, i'll stop answering questions until you at least have looked at existing implementations and read about the technologies used (EC crypto in this case); if you don't understand something at that point, please ask
416 2012-12-02 16:02:03 <abracadabra> ACTION 's irc troll spidy sense is tingling
417 2012-12-02 16:02:29 <BitDev> ok thnx
418 2012-12-02 16:03:23 <abracadabra> ACTION makes note not to install any software from bitdev
419 2012-12-02 16:10:00 <sipa> _dr: seems I was wrong; the 3h22m figure is also when running at 2.2GHz
420 2012-12-02 16:24:44 <gmaxwell> sipa: I wonder how much speed you're losing to queue management? perhaps each thread thould pull out total/(par*2) items at once or something?
421 2012-12-02 16:37:35 <sipa> gmaxwell: well i prefer not to use my laptop as benchmark system anymore, now that i've seen the temperatures it reaches (and dmesg contains reports of overheating and lowering clocks, so it's probably not reliable either)
422 2012-12-02 16:38:14 <sipa> but indeed, processing multiple elements at once can certainly help
423 2012-12-02 16:40:32 <gmaxwell> sipa: how are you benchmarking? reindex with checkpoints disabled?
424 2012-12-02 16:41:07 <denisx> sipa: what model is your laptop?
425 2012-12-02 16:45:21 <_dr> i'd ditch the queue entirely and use a static scheduling
426 2012-12-02 16:45:55 <_dr> since each kind of script has a deterministic runtime it should be possible
427 2012-12-02 16:46:24 <_dr> queue management and thread synchronization using gcc is evilish slow
428 2012-12-02 16:47:02 <_dr> because GCC uses libpthread for that stuff, i.e. kernel stuff to synchronize, that's in the order of 30.000-100.000 cycles per simple synchrnization
429 2012-12-02 16:47:29 <gmaxwell> GCC???!
430 2012-12-02 16:47:44 <_dr> -openmp
431 2012-12-02 16:47:50 <gmaxwell> We're not using openmp.
432 2012-12-02 16:47:53 <_dr> or where are you getting your parallelism from?
433 2012-12-02 16:48:54 <gmaxwell> But you're probably right there... just run PAR queues and have the input side fill all of them, then have the readers run to completion.
434 2012-12-02 16:49:01 <_dr> well, the same argument hold if you're using pthreads directly, of course. synchronization is slow, thus i'd avoid a queue where possible
435 2012-12-02 16:49:13 <gmaxwell> Thats even better than my suggestion to read bigger blocks out.
436 2012-12-02 16:52:46 <_dr> could you point me to the routine that does the block chain verification? i'd like to have a look at the openssl functions involved for ecdsa
437 2012-12-02 16:53:01 <_dr> you can switch it for intel's ipp to get an idea how fast it can go with optimization
438 2012-12-02 16:53:24 <_dr> i just found out they have that particular curve in the crypto ipp
439 2012-12-02 16:58:35 <sipa> synchronization is very fast if the lock is uncontended
440 2012-12-02 17:00:53 <sipa> having a queue per worker indeed reduces contention, but does mean that when many elements need to be pushed, you need locks on more queues
441 2012-12-02 17:05:23 <sipa> _dr: also, scripts don't have deterministic runtime (most are between 500 and 700 microseconds, but some can be a lot more)
442 2012-12-02 17:07:30 <_dr> okay, that's a problem
443 2012-12-02 17:08:04 <_dr> i assume that has to do with multiplications in EC? they're somehow translated into $lower_factor additions or something iirc
444 2012-12-02 17:09:21 <sipa> yes, that, but also simply the fact that there are several types of scripts
445 2012-12-02 17:09:52 <gmaxwell> _dr: e.g. we can have a single script that does many EC signature checks.
446 2012-12-02 17:09:55 <sipa> a pubkey script is slightly easier than a pubkeyhash script (an additional sha256 + ripemd16), but that is negligable compared to EC verification
447 2012-12-02 17:10:17 <sipa> but indeed, multisig scripts can do up to 20 EC verifications
448 2012-12-02 17:11:35 <gmaxwell> This still wouldn't prevent you from doing input side roundrobbin over all the writable queues (without spinning if it's not available). A bit of imbalanace is almost certantly better than more time spent handling the locks in the readers.
449 2012-12-02 17:12:17 <gmaxwell> (doesn't boost have lock free queues available?)
450 2012-12-02 17:13:11 <sipa> right, but now the writer needs 1 lock to push many elements, and the readers need to lock once for every N/k elements (contending with eachother)
451 2012-12-02 17:13:38 <sipa> using a per-worker queue means readers only contend with the writer, but the writer needs more locks
452 2012-12-02 17:13:55 <gmaxwell> oh they're already pulling multiple elements? K.
453 2012-12-02 17:14:05 <_dr> why push many elements? you could just push a struct containing a pointer to the first script and the workers will process 100 scripts form that pointer
454 2012-12-02 17:15:16 <sipa> gmaxwell: no, they're not; i'm just comparing the two proposed changes
455 2012-12-02 17:15:43 <_dr> and implement the 'queue' as a blocking ringbuffer. since the workers will take much longer to process 100 elements than the master to push such a struct onto the other end there'll be no contention if the buffer is large enough
456 2012-12-02 17:16:11 <_dr> and if it takes one worker twice the time to process 100 scripts than another it won't matter in the long run
457 2012-12-02 17:16:41 <gmaxwell> 12/02/12 17:49:53 -> 12/02/12 18:14:05 from 0->210000 with checkpoints on, par=4, and hal on E31230.
458 2012-12-02 17:17:00 <sipa> 25 minutes? :o
459 2012-12-02 17:17:07 <gmaxwell> _checkpoints on_
460 2012-12-02 17:17:11 <sipa> oh
461 2012-12-02 17:17:56 <gmaxwell> tested that first because thats what people are actually going to see (unless we turn off the 'optimization' wrt checkpoints)
462 2012-12-02 17:18:14 <gmaxwell> thats a big improvement in any case!
463 2012-12-02 17:18:29 <gmaxwell> It was over an hour, to a lower height, a few weeks ago.
464 2012-12-02 17:19:05 <gmaxwell> ;;bc,blocks
465 2012-12-02 17:19:06 <gribble> 210625
466 2012-12-02 17:20:20 <gmaxwell> par probably needs to get hooked up to cpu count detection, and a gui option to adjust the number of CPUs that bitcoin uses for verification (for desktop users irritated that bitcoin is using all their cores)
467 2012-12-02 17:20:43 <gmaxwell> the former should be another commit, though the current code isn't really setup for runtime par changes.
468 2012-12-02 17:21:02 <sipa> shouldn't be too hard to do
469 2012-12-02 17:21:30 <sipa> keep a counter of actually active workers inside the queue object, and have some function to stop/start threads as necessary
470 2012-12-02 17:29:10 <sipa> gmaxwell: trying to the pick-several-objects strategy
471 2012-12-02 17:29:32 <sipa> small optimization: keep a counter of idle threads, and pick up to N/(idle+1)/2
472 2012-12-02 17:30:56 <gmaxwell> or N/(M+idle) (all idle it picks N/M/2 ?
473 2012-12-02 17:31:32 <sipa> with M=?
474 2012-12-02 17:31:38 <sipa> seems better indeed
475 2012-12-02 17:31:44 <gmaxwell> executors.
476 2012-12-02 17:32:53 <gmaxwell> I'd guess that most of the benefit happens in the quantum=2-16 range or so. Above that the improvements are small and it's mostly about not making the imbalance too bad.
477 2012-12-02 17:33:56 <sipa> 16 already means 10ms
478 2012-12-02 17:42:47 <_dr> you can't work on more than one block in parallel?
479 2012-12-02 17:44:06 <sipa> i could, but that's trickier
480 2012-12-02 17:44:51 <gmaxwell> _dr: ultimately we will, but sipa's prior attempt at that had some issues and since this is now on the performance critical path having something sooner is better.
481 2012-12-02 17:45:14 <_dr> gmaxwell: i understand
482 2012-12-02 17:45:54 <gmaxwell> (making it work on multiple blocks means that you need to accept a block that you haven't checked yet... then if it fails you blacklist it and trigger a reorg off of it)
483 2012-12-02 17:50:59 <_dr> is there a function like VerifyChain(starting_from)? i'm starting to feel stupid about asking this stuff when it's all there in the code
484 2012-12-02 17:51:20 <sipa> ehh, it not as easy as that
485 2012-12-02 17:51:59 <sipa> because to verify a piece of chain, you need to have thr UTXO set corresponding to its predecessor block
486 2012-12-02 17:52:41 <sipa> SetBestChain moves the "correctly active chain tip" to a given block (wherever it was)
487 2012-12-02 17:53:57 <_dr> then i'll try to start there, thanks :)
488 2012-12-02 18:08:29 <gmaxwell> 12/02/12 18:23:46 -> 12/02/12 19:07:45 without checkpoints
489 2012-12-02 18:09:00 <gmaxwell> 44 minutes.
490 2012-12-02 18:10:45 <sipa> Nice.
491 2012-12-02 18:11:26 <gmaxwell> I'll see what it is when you post a batching patch.
492 2012-12-02 18:13:52 <sipa> pushed
493 2012-12-02 18:14:37 <sipa> changes: have a CScriptCheck::swap, so the vector in it doesn't need to be copied all the time when passing from one queue to another
494 2012-12-02 18:15:01 <sipa> use a std::vector as a stack instead of a std::deque for the actual queue
495 2012-12-02 18:15:07 <sipa> and batching up to 16 elements
496 2012-12-02 18:15:42 <sipa> ... but i have no idea whether it's actually faster
497 2012-12-02 18:21:03 <sipa> maybe it isn't actually faster, but has lower CPU usage
498 2012-12-02 18:21:18 <sipa> that would also be nice (though being faster at the same CPU usage is better)
499 2012-12-02 18:26:07 <sipa> gmaxwell: can you retry?
500 2012-12-02 18:27:22 <gmaxwell> started it seconds after you said 'pushed'
501 2012-12-02 18:27:26 <sipa> haha :D
502 2012-12-02 18:33:54 <sipa> gmaxwell: that 44 minutes... with or without Halcode?
503 2012-12-02 18:34:09 <gmaxwell> thats w/ halcode.
504 2012-12-02 18:34:28 <gmaxwell> oh crud. forgot to reapply that patch.
505 2012-12-02 18:34:59 <gmaxwell> hm its a height=178654 ... apply hal and restart or just let it finish?
506 2012-12-02 18:35:24 <gmaxwell> I guess I'll let it finish so that it doesn't do anything weird with the aborted reindex.
507 2012-12-02 18:35:43 <sipa> oh you can aboirt a reindex just fine
508 2012-12-02 18:36:30 <sipa> -reindex on commandline just means "wipe databases at startup", so it's not like any corruption from aborting could have any influence
509 2012-12-02 18:38:33 <gmaxwell> okay, restarted with halcode. (my prior 44 number was halcode too, I'd just pulled off those commits when I switched your parallel patches)
510 2012-12-02 18:39:07 <gmaxwell> I will eventually do a w/ and w/o hal number??? though we know its worthwhile already, just so we have some figures to tell hal.
511 2012-12-02 18:40:17 <sipa> my test commandline is getting long though... ./bitcoind -connect=vps.sipa.be -dns -dbcache=900 -reindex -debug -logtimestamps -nodaemon -par=4 -benchmark
512 2012-12-02 18:44:54 <gmaxwell> I'm not botherhering with dbcache, though my node is in tmpfs.
513 2012-12-02 18:45:56 <gmaxwell> sipa: I wonder if disabling the sig cache when 'its unlikely to be helpful' (I'd say IBD but I'm kinda tired of making things conditional on that!) would produce a measurable improvement.
514 2012-12-02 18:47:01 <sipa> iirc i tested that once, and it didn't help (which would mean that the overhead of storing/querying is very low (expected), and the lock on the sigcache is uncontended (maybe not so expected))
515 2012-12-02 18:47:18 <gmaxwell> Yea, I don't expect it to be uncontended.
516 2012-12-02 18:47:50 <gmaxwell> (one way to do that, perhaps, would be to make it so that blocks don't add to it, and so it doesn't get tested during block validation when there is <x items it)
517 2012-12-02 18:48:00 <gmaxwell> I can try doing a run with if off entirely later.
518 2012-12-02 18:48:31 <sipa> i suppose numbers with all 8 combinations would be nice: parallelism on/off, halcode on/off, sigcache on/off
519 2012-12-02 18:48:56 <sipa> thought without parallellism, sigcache on/off won't make much difference
520 2012-12-02 18:49:11 <gmaxwell> Right.
521 2012-12-02 18:50:11 <sipa> kind of a pity that reindex now works in parallel with a running node, makes the numbers somewhat less reliable as cs_main also gets used for incoming transactions e.g.
522 2012-12-02 18:50:20 <gmaxwell> Blocks not adding to it might have a general improvement, as they add enteries which are very likely to never be checked again. So elimiating them would lower overhead (unimportant) and increase hitrate (probably more important).
523 2012-12-02 18:50:43 <gmaxwell> well I can do numbers on an isolated node too, though I'm not now.
524 2012-12-02 18:50:44 <sipa> good point
525 2012-12-02 18:50:45 <sipa> good point
526 2012-12-02 18:51:36 <sipa> actually... there is an even better benchmarking scenario clear coins/ at startup - it will reconnect the best block in the blkindex
527 2012-12-02 18:51:51 <sipa> which also means no overhead from writing undo data or blktree data
528 2012-12-02 18:52:23 <sipa> ... except 1) it corresponds even less to a realistic situation 2) it's broken
529 2012-12-02 18:53:43 <gmaxwell> I need to head out for a bit, so I'll be a little late getting you the performance numbers on this run. (probably be gone for an hour)
530 2012-12-02 18:56:56 <sipa> no problem
531 2012-12-02 19:22:18 <korozion> I've made 8 GPUs work, but they seem to be only loading at 60%. When I was working with 6 GPUs, they were loading around 90%. Any ideas?
532 2012-12-02 19:26:35 <xIsalty> load balancing
533 2012-12-02 19:38:31 <korozion> how can I make them all work harder?
534 2012-12-02 20:02:37 <_dr> korozion: what are they working on? mining? how do you determine the load?
535 2012-12-02 20:03:33 <gmaxwell> 19:38:06 -> 20:22:15
536 2012-12-02 20:03:55 <_dr> korozion: usually there's an intensity parameter, try increasing it
537 2012-12-02 20:05:02 <korozion> _dr: was using GPU caps to determine the load
538 2012-12-02 20:05:11 <korozion> it also matches up with the hash/s value
539 2012-12-02 20:05:23 <korozion> I'll try intensity
540 2012-12-02 20:05:32 <sipa> gmaxwell: 44 minutes...
541 2012-12-02 20:06:11 <gmaxwell> I had the impression that the htop numbers looked more consistent, but I wouldn't really bet on that for anything.
542 2012-12-02 20:11:53 <korozion> man it's hard to make 8 GPUs stable
543 2012-12-02 20:11:58 <korozion> hrm
544 2012-12-02 20:12:10 <korozion> I wonder if it has _anything_ to do with the fact that cgminer for windows is 32bit
545 2012-12-02 20:12:26 <korozion> I know the OS has to be 64 bit for 8 GPUs, but I wonder about the problem
546 2012-12-02 20:12:31 <korozion> need to flip back to Linux and test ...
547 2012-12-02 20:12:53 <korozion> *program
548 2012-12-02 20:46:58 <gmaxwell> sipa: would probably be good if a corrupted coins had a fighting chance of triggering that.
549 2012-12-02 20:47:39 <sipa> triggering what?
550 2012-12-02 20:48:00 <sipa> rebuild the coin db?
551 2012-12-02 20:48:37 <gmaxwell> yes, sorry. .. what? you can't see my screen right now? :P
552 2012-12-02 20:48:54 <sipa> sure i can; i just pretend i don't
553 2012-12-02 20:49:10 <gmaxwell> hah
554 2012-12-02 20:55:22 <sipa> gmaxwell: that's certainly an option... there is some work left on the automatic error detection/correction front
555 2012-12-02 20:56:05 <gmaxwell> alternatively in that case perhaps it should just yell at you to run reindex and shut down... so that people actually report the problems.
556 2012-12-02 20:56:36 <sipa> something that was always missing from the old -checkblocks code was distinguishing between errors in the block tree (which basically means the blocks aren't what you hope they are, and you should just drop them from the index) and errors in the transaction state (coin state now), where you want reorganize to have a chance for recovery
557 2012-12-02 20:57:46 <sipa> i suppose in the reorganize case, if the code detects it needs to reorganize too far back, it probably is indeed better to start over
558 2012-12-02 20:57:52 <sipa> assuming all blocks are still present
559 2012-12-02 21:02:15 <gmaxwell> sipa: running with par=8 21:11:45 -> 21:52:25 (machine has SMT, so a cpu detection would likely come up with 8)
560 2012-12-02 21:02:41 <gmaxwell> interesting that it was faster.
561 2012-12-02 21:05:22 <sipa> not sure if 3 minutes is really significant
562 2012-12-02 21:05:32 <sipa> though i'd rather expect it to slow down
563 2012-12-02 21:10:13 <gmaxwell> well, 3/44 is like 6% should probably do a run with 4 again but isolated and just a coins rebuild ... may be that it was faster with the changes but noise masked it.
564 2012-12-02 21:11:34 <sipa> gmaxwell: you can try #2062 to benchmark more precisely
565 2012-12-02 21:12:03 <sipa> start with an indexed chain up to or above 210000, shut down, delete coins/ entirely, start up with whathever you want to trst
566 2012-12-02 21:12:07 <gmaxwell> yes, already doing.
567 2012-12-02 21:12:20 <sipa> LD
568 2012-12-02 21:12:21 <sipa> :D
569 2012-12-02 21:56:12 <gmaxwell> 12/02/12 22:13:56 SetBestChain: new best=00000000839a8e6886ab5951d76f411475428afc90947ee320161bbf18eb6048 height=1 work=8590065666 tx=2 date=01/09/09 02:54:25
570 2012-12-02 21:56:16 <gmaxwell> 12/02/12 22:54:27 SetBestChain: new best=000000000000048b95347e83192f69cf0366076336c639f9b7228e9ba171342e height=210000 work=628963747775700992096 tx=9344662 date=11/28/12 15:24:38
571 2012-12-02 21:56:20 <gmaxwell> thats with par=4
572 2012-12-02 21:57:12 <gmaxwell> running again in the same config with par=8 then will run with the older version of the parallel check patch.
573 2012-12-02 21:57:51 <gmaxwell> or maybe before doing that I'll first try with the sigcache disabled.
574 2012-12-02 21:58:56 <sipa> gmaxwell: maybe it even makes sense to _remove_ entries from the sigcache when they were used in a block
575 2012-12-02 22:03:46 <etotheipi_> are the Bitcoin Foundation forums being used? is it a good, lower-noise environment than the bitcointalk forums?
576 2012-12-02 22:04:13 <gmaxwell> sipa: ah! point, if we've already taken a lock to do the lookup and found the entry... why not remove it while we're there?
577 2012-12-02 22:04:15 <sipa> it's certainly a lot lower noise, but there's also less signal :)
578 2012-12-02 22:04:27 <sipa> gmaxwell: exactly
579 2012-12-02 22:04:33 <etotheipi_> is that perhaps a better place to start important conversations?
580 2012-12-02 22:04:43 <sipa> maybe we should try that, yes
581 2012-12-02 22:05:33 <gmaxwell> etotheipi_: I'd like it if more of the serious conversations moved there for sure. The forums feel a bit unsalvagable at times. Esp outside of the tech subforum.
582 2012-12-02 22:05:34 <sipa> but i'm worried about not reaching people... now it seems that the dev mailing lists have some number of people who have interest in contributing to discussions, but some are only on bitcointalk
583 2012-12-02 22:05:58 <gmaxwell> The dev mailing lists aren't working well. I mean??? we've got client authors who aren't on them.
584 2012-12-02 22:06:07 <sipa> yes :(
585 2012-12-02 22:06:23 <gmaxwell> (and they're authors of software people use)
586 2012-12-02 22:06:24 <gmaxwell> (and they're authors of software people use)
587 2012-12-02 22:07:23 <gmaxwell> sipa: it'll be fun to benchmark this on one of the 32-way machines here.
588 2012-12-02 22:07:32 <sipa> ha
589 2012-12-02 22:07:38 <sipa> please try :)
590 2012-12-02 22:07:55 <sipa> though i don't think the parallellism scales particularly well in this implementation
591 2012-12-02 22:08:01 <gmaxwell> I will, right now I have another job running but when it finishes (probably tomorrow) I'll run it.
592 2012-12-02 22:08:02 <gmaxwell> I will, right now I have another job running but when it finishes (probably tomorrow) I'll run it.
593 2012-12-02 22:08:38 <sipa> "not particularly well" might mean "no use for more than 6 thread" or "no use for more than 200 threads", though
594 2012-12-02 22:08:39 <sipa> "not particularly well" might mean "no use for more than 6 thread" or "no use for more than 200 threads", though
595 2012-12-02 22:09:07 <anzy_> hi ... Anyone knows how I can delete all the files that Bitchain-qt puts on a Mac? ... It's taking way too much space (Google says ~/Library/Application Support/Bitcoin/ , but the folder is not there and my diskspace is still gone)
596 2012-12-02 22:11:03 <sipa> gmaxwell: it may make sense to change the minimum number of elements taken from the queue to something higher than 1, if switching overhead is significant enough
597 2012-12-02 22:11:37 <sipa> anzy_: hmm, afaik that is where it should store them
598 2012-12-02 22:11:44 <sipa> anzy_: but i don't have mac experience
599 2012-12-02 22:13:56 <anzy_> ok, anyone else has any idea? =(
600 2012-12-02 22:14:20 <sipa> isn't there some nifty search file function in OSX?
601 2012-12-02 22:14:24 <sipa> look for a file called wallet.dar
602 2012-12-02 22:14:25 <sipa> look for a file called wallet.dar
603 2012-12-02 22:14:29 <sipa> wallet.dat
604 2012-12-02 22:15:37 <anzy_> oh geez found it ... It's really stupidly hidden in the new OS, not even searc finds it
605 2012-12-02 22:15:44 <anzy_> 5.6 GB :o
606 2012-12-02 22:16:21 <sipa> anzy_: if you find that excessive, you're problably better off running a light client
607 2012-12-02 22:16:22 <sipa> anzy_: if you find that excessive, you're problably better off running a light client
608 2012-12-02 22:17:36 <anzy_> yeah, I'll look into it ... I just have some big programs installed that I really need, so I only have like 6 gigs (including Bitcoin data)
609 2012-12-02 22:19:02 <anzy_> but thanks for the will to help ;)
610 2012-12-02 22:21:21 <anzy_> but yeah it'bb
611 2012-12-02 22:37:11 <gmaxwell> 12/02/12 22:56:50 SetBestChain: new best=00000000839a8e6886ab5951d76f411475428afc90947ee320161bbf18eb6048 height=1 work=8590065666 tx=2 date=01/09/09 02:54:25
612 2012-12-02 22:37:14 <gmaxwell> 12/02/12 23:34:10 SetBestChain: new best=000000000000048b95347e83192f69cf0366076336c639f9b7228e9ba171342e height=210000 work=628963747775700992096 tx=9344662 date=11/28/12 15:24:38
613 2012-12-02 22:37:33 <sipa> settings?
614 2012-12-02 22:37:34 <sipa> settings?
615 2012-12-02 22:37:59 <gmaxwell> Seems that it really is faster with par=8 on this system. This isn't _that_ much of a shock, the newer ht stuff actually works, and more threads may just make the scheduler happier
616 2012-12-02 22:38:00 <gmaxwell> Seems that it really is faster with par=8 on this system. This isn't _that_ much of a shock, the newer ht stuff actually works, and more threads may just make the scheduler happier
617 2012-12-02 22:38:17 <gmaxwell> This is hal+new-parallel+par-8+coins rebuild.
618 2012-12-02 22:38:47 <sipa> may also be the case that now smaller chunks are used (because of a larger divider), and the switching overhead is actually really low
619 2012-12-02 22:38:48 <sipa> may also be the case that now smaller chunks are used (because of a larger divider), and the switching overhead is actually really low
620 2012-12-02 22:39:02 <sipa> so the variance caused by threads not finishing at the same moment is lower
621 2012-12-02 22:39:16 <gmaxwell> (checkpoints=0)
622 2012-12-02 22:39:31 <gmaxwell> that may be the case, esp with there being lots of unfilled blocks in this test.
623 2012-12-02 22:39:32 <gmaxwell> that may be the case, esp with there being lots of unfilled blocks in this test.
624 2012-12-02 22:39:43 <gmaxwell> well lemme try the old parallel code.
625 2012-12-02 22:39:44 <gmaxwell> well lemme try the old parallel code.
626 2012-12-02 22:39:50 <gmaxwell> hm wait, that changes more stuff.
627 2012-12-02 22:39:59 <gmaxwell> let me try just having it always pull 1.
628 2012-12-02 22:40:29 <sipa> i think all other changes (apart from the batch processing) are pretty much certain not to harm performance
629 2012-12-02 23:06:03 <sipa> boost seems to have a function to request the number of threads the hardware can execture simultaneously, but i doubt it distinguishes between real cores and hyperthreading
630 2012-12-02 23:06:31 <maaku> c++ question: if I use something like printf("???", FormatMoney(value).c_str()); is the std::string object returned by FormatMoney properly destroyed, or is that a memory leak?
631 2012-12-02 23:07:02 <sipa> it will be destroyed
632 2012-12-02 23:07:07 <sipa> c++ even specifies when
633 2012-12-02 23:07:14 <sipa> (after the printf call)
634 2012-12-02 23:07:19 <maaku> thank you
635 2012-12-02 23:07:36 <maaku> i've been using dynamic languages for too long; i used to know this stuff :\\
636 2012-12-02 23:16:44 <phantomcircuit> sipa, im sure it doesn't
637 2012-12-02 23:19:18 <sipa> phantomcircuit: i'm sure it does
638 2012-12-02 23:19:19 <sipa> phantomcircuit: i'm sure it does
639 2012-12-02 23:19:36 <phantomcircuit> talking about boost
640 2012-12-02 23:19:40 <sipa> oh
641 2012-12-02 23:32:08 <gmaxwell> 12/02/12 23:43:29 SetBestChain: new best=00000000839a8e6886ab5951d76f411475428afc90947ee320161bbf18eb6048 height=1 work=8590065666 tx=2 date=01/09/09 02:54:25
642 2012-12-02 23:32:11 <gmaxwell> 12/03/12 00:20:39 SetBestChain: new best=000000000000048b95347e83192f69cf0366076336c639f9b7228e9ba171342e height=210000 work=628963747775700992096 tx=9344662 date=11/28/12 15:24:38
643 2012-12-02 23:32:47 <gmaxwell> 37 minutes, thats hal+checkpoint=0+par=8+coins_rebuild with it set to only pull one from the queue.
644 2012-12-02 23:35:37 <sipa> same as with laeger batches?
645 2012-12-02 23:35:38 <sipa> same as with laeger batches?
646 2012-12-02 23:37:01 <sipa> 37:20
647 2012-12-02 23:37:39 <sipa> now 37:10
648 2012-12-02 23:47:58 <jgarzik> ACTION returns
649 2012-12-02 23:48:12 <jgarzik> catching up on the jgarzik thread... nothing new apparently ;p
650 2012-12-02 23:58:37 <sipa> gmaxwell: where do you get those numbers? the fraction of signature-verification load before the checkpoint vs. all is certainly above 50%
651 2012-12-02 23:59:20 <sipa> 58% even