1 2015-02-27 00:40:18 <maaku> justanotheruser: in my experience "without strong typing" doesn't go with "easier to read" or "less buggy"
2 2015-02-27 00:43:13 <justanotheruser> maaku: It is time to move to bitcoin-Js eval
3 2015-02-27 00:43:41 <maaku> justanotheruser: i don't get it
4 2015-02-27 00:44:19 <justanotheruser> maaku: sarcasm doesn't translate well on the internet
5 2015-02-27 00:44:33 <maaku> justanotheruser: ah :(
6 2015-02-27 00:50:12 <Luke-Jr> justanotheruser: In my case, it was because I perceived you to be contrasting C++ with C ;)
7 2015-02-27 00:56:30 <instagibbs> do people actually want my gitian sig since I went through with it? :P
8 2015-02-27 03:59:22 <sipa> gavinandresen, gmaxwell: still, if there is a dos worry we can always keep both caching mechanism
9 2015-02-27 04:00:02 <sipa> i don't really like using the mempool for this, as it makes a big piece of code with many responsabilities consensus-critical
10 2015-02-27 04:00:26 <sipa> gavin's numbers are pretty surprisingly high
11 2015-02-27 04:03:19 <gmaxwell> yea, that was my thought, but hard to argue with those numbers.
12 2015-02-27 04:03:52 <gmaxwell> (perhaps it's somewhat suspicious, which is why I was asking if there was any profiling to find out where the time was going)
13 2015-02-27 04:04:25 <sipa> maybe it's due to contention
14 2015-02-27 04:05:58 <sipa> or maybe script execution (without sigchecking) is really slow compared to a cache lookup
15 2015-02-27 05:10:43 <Jerk> betcoin
16 2015-02-27 05:12:26 <Jerk> selling bitcoins half dollar each
17 2015-02-27 05:14:03 <sipa> Jerk: not here
18 2015-02-27 05:15:22 <Jerk> where then?
19 2015-02-27 05:24:14 <phroziac> LOL
20 2015-02-27 07:29:34 <wallet42> any block version 3 mined ?
21 2015-02-27 07:30:15 <sipa> yes
22 2015-02-27 07:30:44 <sipa> http://bitcoin.sipa.be/ver-10k.png
23 2015-02-27 07:31:52 <phantomcircuit> sipa, what's the rough time per validation with libsecp256k ?
24 2015-02-27 07:32:06 <sipa> 100us
25 2015-02-27 07:32:22 <sipa> on 2.6GHz i7
26 2015-02-27 09:09:14 <gmaxwell> sipa: https://github.com/bitcoin/bitcoin/pull/5835#issuecomment-76359741
27 2015-02-27 09:10:31 <sipa> heh?
28 2015-02-27 09:10:49 <sipa> the inputs shoukd be fetched anyway later
29 2015-02-27 09:10:57 <sipa> i'll have a look soon
30 2015-02-27 09:12:39 <gmaxwell> I'm confused by it for sure.
31 2015-02-27 09:14:00 <gmaxwell> that change also seems to make undo very fast.
32 2015-02-27 09:15:32 <sipa> this is all without signature checking?
33 2015-02-27 09:15:54 <gmaxwell> no, its' with signature checking (well signature cache is still there).
34 2015-02-27 09:16:10 <gmaxwell> (undo is 3.5x faster in a quick test here)
35 2015-02-27 09:16:46 <phantomcircuit> gmaxwell, i think the one in ConnectBlock is redundant not the one in CheckInputs
36 2015-02-27 09:17:24 <phantomcircuit> ie if (!view.HaveInputs(tx))
37 2015-02-27 09:17:30 <gmaxwell> well with gavin's changes the one in checkinputs is no longer fully functional; since the result is cached indifferently to the viewcache state.
38 2015-02-27 09:17:56 <gmaxwell> So that change is only safe if the test in checkinputs is redundant.
39 2015-02-27 09:18:00 <phantomcircuit> right
40 2015-02-27 09:18:03 <gmaxwell> (at least within the block verification)
41 2015-02-27 09:18:30 <gmaxwell> I looked briefly at acceptblock and accepttomempool and both appear to already check. I think the only other place we call checkinputs is in mining code.
42 2015-02-27 09:18:41 <gmaxwell> which I didn't check, but I think the call there is just precaution.
43 2015-02-27 09:20:05 <gmaxwell> wrt undo, it went from 13 seconds to disconnect 20 blocks, to <4 seconds. (which was enough faster that I noticed)
44 2015-02-27 09:21:39 <phantomcircuit> gmaxwell, yeah i believe there's actually a bunch of redundant checks
45 2015-02-27 09:21:49 <phantomcircuit> like i think basic block header checks are run like 4 times
46 2015-02-27 09:22:14 <gmaxwell> at least the ones in mining/mempool had been somewhat intentionally redundant; paranoia about corruption.
47 2015-02-27 09:22:42 <gmaxwell> though looks like some of this redundancy is really costly for some reason. (which itself is probably a bug. :) )
48 2015-02-27 09:24:17 <phantomcircuit> i'd guess some of it ends up being quasi exponential
49 2015-02-27 09:24:40 <phantomcircuit> add redundancy in 2 places and you get x4 runtime type of thing
50 2015-02-27 09:24:42 <sipa> i'll have a look in an hour or so
51 2015-02-27 09:25:34 <sipa> phantomcircuit: well if those are rcursive calls...
52 2015-02-27 09:25:44 <sipa> which *very* hopefully nkt the case here
53 2015-02-27 09:26:03 <sipa> not
54 2015-02-27 09:26:54 <gmaxwell> I've been busy with other things so I've not really given it more than a glance the performance numbers just seemed highly confusing to me, so I just did the first exploritory test that came to mind.
55 2015-02-27 09:27:52 <sipa> 0.1ms/txin sounds excessively high in any case
56 2015-02-27 09:28:38 <sipa> libsecp256k1 validates a signature in that time?
57 2015-02-27 09:28:51 <gmaxwell> well this is on my 2.1ghz laptop, so probably a bit slower than you were expecting.
58 2015-02-27 09:29:29 <sipa> ok, libsecp256k1 validates a signature in that time, with GLV enabled, in that case :p
59 2015-02-27 09:34:14 <gmaxwell> 90.9us without GLV, fwiw.
60 2015-02-27 09:36:34 <sipa> with turbo boost, i assume?
61 2015-02-27 09:36:45 <sipa> sounds very fast for 2.1GHz
62 2015-02-27 09:39:09 <gmaxwell> oh indeed, thats probably it. :)
63 2015-02-27 09:42:08 <vayvanne> hi folks. is it appropriate to discuss here some connectivity issues of bitcoind I have?
64 2015-02-27 09:42:42 <vayvanne> I certainly need some fresh look from aside to it
65 2015-02-27 09:42:47 <Luke-Jr> probably #bitcoin first
66 2015-02-27 09:43:07 <vayvanne> okay thank you
67 2015-02-27 10:01:40 <sipa> gmaxwell: i really don't get it
68 2015-02-27 10:01:53 <sipa> all HaveInputs does is iterate over the inputs, and call AccessCoins on them
69 2015-02-27 10:02:12 <sipa> which is exactly what happens in the loop further in CheckInputs too
70 2015-02-27 10:09:54 <sipa> Luke-Jr: are you again enabling policy patches by default in gentoo?
71 2015-02-27 10:10:49 <Luke-Jr> sipa: yep, as planned
72 2015-02-27 10:10:55 <sipa> please stop doing that
73 2015-02-27 10:11:07 <Luke-Jr> why?
74 2015-02-27 10:11:34 <sipa> forcing a political choice onto users feels incredibly wrong
75 2015-02-27 10:11:40 <Luke-Jr> it's not forced in any sense.
76 2015-02-27 10:11:47 <Luke-Jr> and I agree with that.
77 2015-02-27 10:12:15 <sipa> defaults matter
78 2015-02-27 10:12:28 <sipa> people feel betrayed, thinking they got something else
79 2015-02-27 10:12:38 <Luke-Jr> too much. and reference defaults are no better than non-reference in that regard.
80 2015-02-27 10:12:43 <Luke-Jr> um, no
81 2015-02-27 10:12:52 <Luke-Jr> it is impossible to get this without knowing it
82 2015-02-27 10:12:53 <sipa> which patches are these?
83 2015-02-27 10:13:41 <Luke-Jr> of http://luke.dashjr.org/programs/bitcoin/files/bitcoind/luke-jr/0.10.x/0.10.0.ljr20150220/bitcoin-0.10.0.ljr20150220.desc.txt , syslibs is required, and ljrF+cpfp+spamfilter+dcmp are default
84 2015-02-27 10:14:01 <phantomcircuit> Luke-Jr, what's the use flag?
85 2015-02-27 10:14:12 <gmaxwell> sipa: I rebenchmarked again, with the 'redundant' test in place and it's now 80ms for that block. Wondering if the inconsistency is related to the size of the viewcache.
86 2015-02-27 10:14:34 <Luke-Jr> phantomcircuit: http://luke.dashjr.org/tmp/code/bcljr-20150227.html
87 2015-02-27 10:14:36 <sipa> I don't understand how you can maintain that a spamfilter is a reasonable default to have.
88 2015-02-27 10:14:52 <Luke-Jr> sipa: why would users want their node relaying spam?
89 2015-02-27 10:14:59 <sipa> They may disagree with it being spam.
90 2015-02-27 10:15:04 <Luke-Jr> then they can disable it
91 2015-02-27 10:15:10 <sipa> Please.
92 2015-02-27 10:15:23 <sipa> Please keep things neutral.
93 2015-02-27 10:15:31 <phantomcircuit> this actually seems pretty reasonable for a gentoo package
94 2015-02-27 10:15:41 <Luke-Jr> relaying spam is no more neutral than not-relaying it.
95 2015-02-27 10:16:02 <sipa> It specifically targets some use cases, which benefits certain users.
96 2015-02-27 10:16:09 <phantomcircuit> sipa, im not sure about other gentoo users but i never install anything without checking the use flags
97 2015-02-27 10:16:09 <sipa> It's not just a constant that's being changed.
98 2015-02-27 10:16:31 <sipa> phantomcircuit: even then, this feels completely out of touch with realtity to me.
99 2015-02-27 10:16:57 <sipa> the only thing you can (and should) expect from this is outcry
100 2015-02-27 10:17:23 <gmaxwell> It would be certantly better if it were actually behavioral rather than the weird kludge that it is.
101 2015-02-27 10:17:29 <sipa> yes
102 2015-02-27 10:17:48 <phantomcircuit> oh hey
103 2015-02-27 10:18:03 <phantomcircuit> https://github.com/bitcoin/bitcoin/pull/2340/files
104 2015-02-27 10:18:07 <sipa> i believe software should have reasonable default policy, and that will inevitably hurt some users
105 2015-02-27 10:18:10 <phantomcircuit> i guess that's where the non final txs are coming from
106 2015-02-27 10:18:28 <sipa> but having specific patches that actively target specific hardcoded users is just... weird
107 2015-02-27 10:18:39 <gmaxwell> phantomcircuit: those are always quite final.
108 2015-02-27 10:18:43 <sipa> and not what anyone would expect as a default
109 2015-02-27 10:18:56 <sipa> gmaxwell: not before you're fully synced yourself
110 2015-02-27 10:18:58 <moa> 'spam' seems disengenious phraseology given the subjective nature also
111 2015-02-27 10:19:05 <Luke-Jr> sipa: spam != users
112 2015-02-27 10:19:05 <phantomcircuit> gmaxwell, not if im 1 block behind
113 2015-02-27 10:19:24 <sipa> Luke-Jr: it's subjective
114 2015-02-27 10:19:26 <phantomcircuit> ie it's a loverly race
115 2015-02-27 10:19:27 <gmaxwell> being behind just makes it even more final.
116 2015-02-27 10:19:31 <sipa> Luke-Jr: and i agree with you
117 2015-02-27 10:19:32 <Luke-Jr> sipa: any of the spammers are able to use Bitcoin just fine; it only affects their spam, not their person
118 2015-02-27 10:19:37 <sipa> Luke-Jr: but you can't make this choice for others
119 2015-02-27 10:19:41 <phantomcircuit> gmaxwell, they're nlocktime tip + 1
120 2015-02-27 10:19:59 <phantomcircuit> oh no it's -1
121 2015-02-27 10:20:19 <sipa> Luke-Jr: and if you feel that anyone should be checking use flags anyway,you can equally not enable this by default
122 2015-02-27 10:20:37 <Luke-Jr> sipa: the defaults are supposed to be what is best for most people
123 2015-02-27 10:20:38 <phantomcircuit> hmm
124 2015-02-27 10:20:39 <phantomcircuit> nvm
125 2015-02-27 10:21:11 <sipa> Luke-Jr: do you understand why we'd never merge a patch like yours upstream?
126 2015-02-27 10:21:24 <sipa> you probably do, otherwise you'd already have pull requested it
127 2015-02-27 10:21:28 <gmaxwell> Luke-Jr: existing structure is just too easy to construe as something you don't intend it to be, regardless; so you're going to waste your time time and other people's time again and again this way.
128 2015-02-27 10:22:26 <sipa> To me (and to many others, I'm sure) this feels as you pushing your personal views onto people who do not care about changing the defaults.
129 2015-02-27 10:22:36 <gmaxwell> You really should take my advice and come up with a generic spamfilter that targets the capacity wasting behavior, rather than the kludgy hardcoded filter.
130 2015-02-27 10:23:26 <sipa> And whether they agree or disagree with the policy in general doesn't matter - it is wrong to make the choice for them.
131 2015-02-27 10:24:01 <gmaxwell> sipa: hm. you're actually treading on a line where I might disagree with you. Not blocking things is also making a choice for other people.
132 2015-02-27 10:24:11 <gmaxwell> It's incorrect to privledge inaction.
133 2015-02-27 10:24:12 <sipa> fair enough
134 2015-02-27 10:24:13 <adlai> ACTION thinks the only 'magic numbers' in the upstream source should be the genesis hash (and the wire protocol constants)
135 2015-02-27 10:24:32 <CodeShark> we already block things like insufficient fees
136 2015-02-27 10:24:34 <gmaxwell> But yea, this is lame for reasons far simpler.
137 2015-02-27 10:24:35 <sipa> it's probably inevitable to make some sort of choice, if you need to set a default for them
138 2015-02-27 10:24:43 <Luke-Jr> sipa: I'd never merge a hacky patch like that upstream.
139 2015-02-27 10:25:12 <moa> so why force it on gentoo users?
140 2015-02-27 10:25:14 <CodeShark> the real issue is one of misplaced incentives...but that requires wizards to fix
141 2015-02-27 10:25:39 <moa> ok force is a little strong
142 2015-02-27 10:25:46 <Luke-Jr> gmaxwell: address reuse is the only way I knew to target *some* of the spam - what do you have in mind for "capacity wasting behaviour" detection?
143 2015-02-27 10:25:47 <sipa> I guess the thing that bothers people is that this doesn't feel like a generic best policy, but as a personal view.
144 2015-02-27 10:26:02 <gmaxwell> moa: it's not forced on gentoo users, it's a build flag, and it yells at you about it. It is a default and thats probably a bad idea. But as phantomcircuit says, pretty much every gentoo user checks and adjusts useflags.
145 2015-02-27 10:26:15 <moa> yeah, i qualified that ...
146 2015-02-27 10:26:25 <gmaxwell> moa: I typed slowly. :)
147 2015-02-27 10:26:35 <Luke-Jr> sipa: it's no more wrong for me to set defaults like this, than it is for the reference code to start with defaults.
148 2015-02-27 10:26:58 <sipa> Luke-Jr: agree there; it's inevitable to set defaults if we want the code to behave reasonably
149 2015-02-27 10:26:59 <gmaxwell> Luke-Jr: well the reuse generalizes to cover anything that this fool list thing could possibly work on. After all, something has to produce gazillions of txouts before you notice and bother to update the list.
150 2015-02-27 10:27:03 <Luke-Jr> sipa: for an end user, this seems clearly the best policy. you disagree? O.o
151 2015-02-27 10:27:37 <sipa> Luke-Jr: if i were in a position to make this choice for everyone, i would not make it
152 2015-02-27 10:28:00 <gmaxwell> Luke-Jr: if you keep the kludgy method, (which I don't really think you should) you could at least make the list not hardcoded, but read from a file periodically; so that even people who are clueless about gentoo and useflags will see that the user is in full control of it.
153 2015-02-27 10:28:06 <CodeShark> you know what's funny, sipa and Luke-Jr, is that just a short while ago I was testing the DNS seeds and only you two guys seem to be working
154 2015-02-27 10:28:19 <Luke-Jr> sipa: a choice has to be made, short of refusing to install without a + or - set, which Gentoo does not actually support
155 2015-02-27 10:28:42 <sipa> Luke-Jr: i agree it's the best policy, but not that it is *clearly* the best policy
156 2015-02-27 10:28:44 <moa> like a tor node blacklist
157 2015-02-27 10:28:46 <gmaxwell> and people could distribute alternative lists, I suppose.
158 2015-02-27 10:28:50 <Luke-Jr> gmaxwell: I'd rather spend that time removing the need for kludgy
159 2015-02-27 10:29:05 <CodeShark> so right now the ability for a new node to discover peers depends on two people
160 2015-02-27 10:29:11 <gmaxwell> I agree, I think the static lists are basically dumb from a number of perspectives and have potential for misuse.
161 2015-02-27 10:29:12 <CodeShark> that's a little scary :p
162 2015-02-27 10:29:15 <sipa> Luke-Jr: and it's by far the most controversial one you can make
163 2015-02-27 10:29:24 <gmaxwell> CodeShark: not so, it'll work okay with no dnsseeds at all. Try it. :)
164 2015-02-27 10:29:44 <gmaxwell> just takes a couple minutes in that case, normally.
165 2015-02-27 10:29:45 <Luke-Jr> sipa: oh, I think I can come up with a number of more controversial ideas.. :P
166 2015-02-27 10:29:52 <sipa> Luke-Jr: please don't try :)
167 2015-02-27 10:30:00 <phantomcircuit> lol
168 2015-02-27 10:30:14 <sipa> I really hope you understand why people find this controversial.
169 2015-02-27 10:30:20 <Luke-Jr> (like actually doing what I've been accused of)
170 2015-02-27 10:30:32 <Luke-Jr> sipa: because trolls mislead them about what it does? ;)
171 2015-02-27 10:31:24 <sipa> i'm not interested in continuing this discussion, but really *really* hope you stop doing this.
172 2015-02-27 10:31:28 <moa> messing with fungibilty is like dancing across the flames
173 2015-02-27 10:31:30 <sipa> it looks bad in so many ways
174 2015-02-27 10:31:55 <gmaxwell> it's more than we're discussing here; some of these people complaining aren't even bitcoin users; they're people luke picked up with foolish arguments elsewhere on the net on things unrelated to bitcoin, who are creating drama because this is such an easy target (they claim that its some religious related thing, which I know to be completely untrue). But regardless, by making it so easy for them you
175 2015-02-27 10:32:02 <gmaxwell> 're playing right into their hands.
176 2015-02-27 10:32:31 <gmaxwell> moa: arguably its the foolish applications which require address reuse that broke fungibility, the fool patch is just taking the next logical step. :(
177 2015-02-27 10:33:22 <moa> right
178 2015-02-27 10:33:44 <phantomcircuit> gmaxwell, yes but then who would Luke-Jr have to argue with?
179 2015-02-27 10:33:58 <sipa> Luke-Jr: i also believe we should just make the default op_return disabled, but if i can't convince enough other people that this is the way to go, i shouldn't use my write access to bitcoin core to make it so
180 2015-02-27 10:33:59 <gmaxwell> phantomcircuit: I somehow doubt that would be a problem.
181 2015-02-27 10:34:13 <phantomcircuit> gmaxwell, http://i.imgur.com/8YL0FAE.jpg
182 2015-02-27 10:34:23 <moa> phantomcircuit: any volunteers?
183 2015-02-27 10:34:46 <Luke-Jr> sipa: generally, I get more positive/supportive responses from Gentoo users than negative on this particular issue.
184 2015-02-27 10:35:38 <gmaxwell> Luke-Jr: if nothing else, think of the rest of us, who get to suffer from drama created on this. It's a huge stupid waste of time, and it can be avoided by changing your approach to one which serves the users even better by your own judgement but has better optics.
185 2015-02-27 10:36:09 <gmaxwell> it's not like the level of usage of that stuff is currently high enough to have a real impact, they'll still get stuck processing those txn as they show up in blocks.
186 2015-02-27 10:36:24 <gmaxwell> but a better design may be more widely used and actually have an impact.
187 2015-02-27 10:36:46 <Luke-Jr> gmaxwell: I suppose the time savings would outweight the spam costs
188 2015-02-27 10:36:53 <gmaxwell> but you'll not get a chance to deploy a better design if everyone is fed up with the drama this kludgy one has created.
189 2015-02-27 10:36:58 <phantomcircuit> yes but... real effort
190 2015-02-27 10:37:08 <Luke-Jr> although I question whether the trolls will *really* stop, or just find something else to troll on
191 2015-02-27 10:37:29 <moa> Luke-Jr: reddit bait makes easy fodder though
192 2015-02-27 10:37:33 <gmaxwell> well they may go back to trolling you in IRC, which none of the rest of us see. So for at least our sanity it would resolve it. :)
193 2015-02-27 10:37:34 <Luke-Jr> the default is actually OFF in the main Gentoo tree for the past 48+ hours, by accident
194 2015-02-27 10:38:30 <Luke-Jr> meh, I'll flip it off (intentionally) for a bit and see if the trolls actually stfu
195 2015-02-27 10:39:02 <gmaxwell> Please don't turn it back without improving it in some way. There are a half dozen things you could do that would be trivial that would make it more paletable to people.
196 2015-02-27 10:39:07 <sipa> i'm also seeing "AcceptToMemoryPool: non-final" messages while syncing
197 2015-02-27 10:39:18 <sipa> makes sense
198 2015-02-27 10:40:01 <Luke-Jr> gmaxwell: even if the trolls just move on to some other random thing?
199 2015-02-27 10:40:18 <gmaxwell> Luke-Jr: e.g. something as simple as the two lines of code to make it a commandline switch instead of only a useflag; for people who are not gentoo users the fact that its a useflag (which is meaningless to them) supports the narative that its forced on people.
200 2015-02-27 10:40:52 <gmaxwell> Well I don't suggest you do things you think harm users; but I think you'd agree a couple small improvements are improvements no matter what antagonistic voices think.
201 2015-02-27 10:40:57 <gmaxwell> Better is better.
202 2015-02-27 10:41:19 <Luke-Jr> sure
203 2015-02-27 10:42:33 <gmaxwell> reading from a list would be 'best' but the whole feature concerns me; next week after that some genius will be distributing a 'redlist' to feed it.
204 2015-02-27 10:42:50 <Luke-Jr> yeah, that's a concern
205 2015-02-27 10:43:16 <gmaxwell> well even your current code has that risk; see also my last post on bitcoin-development. :)
206 2015-02-27 10:43:27 <gmaxwell> just an ugly path in general.
207 2015-02-27 10:46:52 <Luke-Jr> gmaxwell: not quite the same risk - it won't "just work" with binaries :p
208 2015-02-27 11:19:17 <gmaxwell> anyone have an ability to benchmark on windows. I am testing with a user's large wallet that they complained was taking 20+ seconds to process a block. Testing for me shows it taking <800ms. My two leads on possible differences are bitcoind vs bitcoin-qt, and windows vs linux as being the source for the differences.
209 2015-02-27 11:20:10 <fronti> ls neuer job von Marcel D'avis
210 2015-02-27 11:20:35 <fronti> ww sorry
211 2015-02-27 11:21:58 <sipa> ;;blocks
212 2015-02-27 11:21:59 <gribble> 345379
213 2015-02-27 11:30:25 <wumpus> gmaxwell: if he's using bitcoin-qt, that's very likely to be the cause
214 2015-02-27 11:31:13 <gmaxwell> wumpus: I suppose it's pinging the GUI about each of the 400k transactions in the wallet or something?
215 2015-02-27 11:31:31 <gmaxwell> (I've currently got -qt compiling to check that; though I'd pipeline asking about windows benchmarking)
216 2015-02-27 11:32:31 <wumpus> I think so - it's very realistic if it cannot cope well with 400k transactions
217 2015-02-27 11:33:23 <wumpus> it's not surprising at all it becomes slow
218 2015-02-27 11:34:17 <gmaxwell> daemon works just fine at least.
219 2015-02-27 11:34:36 <gmaxwell> I admit I've never tried the gui with a huge wallet before.
220 2015-02-27 11:34:45 <wumpus> me neither.
221 2015-02-27 11:35:01 <wumpus> (well - I have some quite large wallets, but not 400k-large)
222 2015-02-27 11:37:09 <wumpus> it's the *reference* wallet we've never claimed that it would scale that far either
223 2015-02-27 11:37:59 <Luke-Jr> unfortunately there's no better wallet out there yet
224 2015-02-27 11:39:11 <wumpus> I'd say a company processing 400k transactions ought to have some resources to invest in optimizing their wallet
225 2015-02-27 11:39:22 <Luke-Jr> heh
226 2015-02-27 11:39:43 <gmaxwell> You might be surprised.
227 2015-02-27 11:39:50 <wumpus> and that's exactly the kind of tragedy-of-the-commons problem we're stuck with
228 2015-02-27 11:44:48 <Luke-Jr> ACTION ponders if B-Qt updates every transactionlist item for every block
229 2015-02-27 11:47:07 <wumpus> I vaguely remember that something went into 0.10 to avoid that
230 2015-02-27 11:47:26 <sipa> i do too
231 2015-02-27 11:48:31 <CodeShark> radix trees?
232 2015-02-27 11:49:01 <sipa> no, just not updating within X ms after the previous update
233 2015-02-27 11:49:04 <sipa> or something like that
234 2015-02-27 11:49:40 <Luke-Jr> eh, blocks aren't within X ms of each other
235 2015-02-27 11:49:58 <sipa> during ibd they are
236 2015-02-27 11:50:09 <sipa> but i'm probably misremembering
237 2015-02-27 11:50:14 <sipa> i tend to ignore gui things
238 2015-02-27 11:52:10 <CodeShark> unless the block contains transactions for the wallet all that should get updated is the confirmation count, no?
239 2015-02-27 11:52:10 <Luke-Jr> sipa: but the problem wasn't during IBD
240 2015-02-27 11:52:19 <Luke-Jr> CodeShark: right
241 2015-02-27 11:52:49 <sipa> ok
242 2015-02-27 11:53:16 <CodeShark> I have an issue with that currently in mSIGNA, too - could be solved by pagination or by just not updating confirmations while synching
243 2015-02-27 11:53:54 <CodeShark> actually, pagination AND not updating confirmations while synching
244 2015-02-27 11:53:59 <Luke-Jr> the ideal solution is to probably not fetch depth until it's about to be displayed
245 2015-02-27 11:54:27 <Luke-Jr> or any tx info for that matter (I guess that's what you mean by pagination)
246 2015-02-27 11:54:34 <CodeShark> yes :)
247 2015-02-27 11:55:22 <CodeShark> buffer a bit before and after so that scrolling is smooth
248 2015-02-27 11:55:28 <wumpus> there's ton of good ideas to speed it up, probablyl even low-hanging fruit, it's just that no one is implementing them
249 2015-02-27 11:56:20 <Luke-Jr> it's all because CodeShark left BCCore :<
250 2015-02-27 11:56:22 <Luke-Jr> :p
251 2015-02-27 11:56:53 <wumpus> well if he's working on a better, commercially funded wallet that's all good.
252 2015-02-27 11:58:00 <Luke-Jr> yes, it was a joke ;)
253 2015-02-27 12:01:36 <CodeShark> well, I didn'
254 2015-02-27 12:01:42 <CodeShark> I didn't entirely leave ;)
255 2015-02-27 12:02:59 <wumpus> pagination instead of one-big-list sounds good
256 2015-02-27 12:03:01 <CodeShark> but obviously bitcoin-qt wasn't designed to scale for enterprise usage, so trying to optimize the GUI seems like a dead-end investment
257 2015-02-27 12:03:07 <CodeShark> at least overdoing it
258 2015-02-27 12:03:24 <CodeShark> although as wumpus says, there's a lot of low-hanging fruit
259 2015-02-27 12:04:27 <wumpus> CodeShark: I agree.
260 2015-02-27 12:04:41 <gmaxwell> We do miss you.
261 2015-02-27 12:04:49 <CodeShark> awww
262 2015-02-27 12:04:52 <CodeShark> :P)
263 2015-02-27 12:06:05 <Luke-Jr> ACTION peers at freenode
264 2015-02-27 12:06:11 <wumpus> CodeShark: the traditional open source paradigm would be that companies interested in scaling would perform profiling and optimization of themselves and contribute back, but somehow it's not happening
265 2015-02-27 12:06:59 <gmaxwell> CodeShark: we "single system gui app" + "enterprise usage" ... thats some military intelligence there; I mean, I am aware that in practice "enterprise usage" often means very low quality java software written by incompetent IDE clicking monkeys... but I doubt thats what you meant. :)
266 2015-02-27 12:07:01 <Luke-Jr> wumpus: they're too busy writing full nodes in Ruby
267 2015-02-27 12:07:15 <wumpus> in Linux that happens a lot, for example. Is it a license issue?
268 2015-02-27 12:07:47 <Luke-Jr> I don't see any evidence anyone is improving it in-house and not publishing the changes
269 2015-02-27 12:07:54 <gmaxwell> Bitcoin Core necessarily handles the whole network, and does so at very high speed (impressively so compared to many of the more commercial attempts at network nodes).. yea, the wallet sure doesn't, but the much harder consensus parts handling it are an existance proof; people are just not putting time into the wallet.
270 2015-02-27 12:07:57 <Luke-Jr> it may be a difficulty-of-getting-things-merged issue, maybe
271 2015-02-27 12:08:00 <wumpus> Luke-Jr: me neither, but you wouldn't see that
272 2015-02-27 12:08:34 <wumpus> gmaxwell: yes, bitcoind's node implementation is very good, we should focus on that
273 2015-02-27 12:08:35 <gmaxwell> I've seen some evidence of that, but it's for the same reason most in house changes aren't made public: They're awful hacks, and "there isn't enough time" to clean them up.
274 2015-02-27 12:08:53 <Luke-Jr> wumpus: well, I mean we *do* see them using other code. maybe.
275 2015-02-27 12:09:01 <CodeShark> I would venture that most of the in-house implementations indeed are single-use hacks
276 2015-02-27 12:09:07 <Luke-Jr> BitPay has CoPay. Coinbase has Ruby stuff.
277 2015-02-27 12:09:46 <Luke-Jr> I suppose it's possible the "wallet pruning" stuff might have been otherwise-in-house
278 2015-02-27 12:10:13 <wumpus> Luke-Jr: yes, wallet changes are also hard to get merged, there is both lack of user and developer interest in them
279 2015-02-27 12:10:24 <Luke-Jr> perhaps an issue is that business needs differ significantly from user needs
280 2015-02-27 12:11:31 <gmaxwell> I haven't seen evidence of that.
281 2015-02-27 12:11:48 <gmaxwell> unless you count willingness to accept risky hacks as a business need. :)
282 2015-02-27 12:12:44 <CodeShark> the single biggest issue with writing a good wallet app, methinks, is the impossibility of using an address to retrieve the txout set in an efficient manner...and even worse, the fact that it's impossible to construct a short proof of the validity of the result
283 2015-02-27 12:12:57 <Eliel> wumpus: it's not happening because bitcoind does not do what hey want it to do.
284 2015-02-27 12:13:15 <CodeShark> so instead you get these implementations with humongous DB indices to support these queries
285 2015-02-27 12:13:23 <CodeShark> so it's a complete fail for scalability and decentralization
286 2015-02-27 12:13:40 <sipa> CodeShark: s/implementations/services/
287 2015-02-27 12:13:41 <sipa> even worse
288 2015-02-27 12:14:09 <wumpus> Eliel: hm?
289 2015-02-27 12:14:46 <gmaxwell> CodeShark: that shouldn't be a problem at all, because one can straightforwardly have a design which does not depend on that.
290 2015-02-27 12:15:55 <CodeShark> the thing is that by calling addresses addresses it's sort of implicit that you can perform these queries
291 2015-02-27 12:16:18 <CodeShark> that's what makes them addresses and not, say, a hash of a cryptographic challenge
292 2015-02-27 12:16:22 <sipa> i think part of bitcoind's low popularity (as wallet interaction service) is because we're not willing to merge things that encourage bad design :)
293 2015-02-27 12:16:25 <CodeShark> or not merely
294 2015-02-27 12:16:30 <Eliel> wumpus: Many users want to search based on addresses. Core dev team seems to be actively refusing to implement anything like that.
295 2015-02-27 12:16:49 <CodeShark> the UI use model promotes that, Eliel
296 2015-02-27 12:16:58 <CodeShark> but the protocol makes it impossible to scale that
297 2015-02-27 12:17:09 <sipa> Eliel: yes, because we want people to encourage building services that don't need that; and i've yet to see a use case that can't work without it, it's just harder to build
298 2015-02-27 12:17:19 <gmaxwell> Eliel: Yes, we're not going to add functionality that requires 30+gb of disk space on every users system.
299 2015-02-27 12:17:50 <Eliel> gmaxwell: even as an option, like txindex?
300 2015-02-27 12:18:00 <wumpus> it wouldn't work with pruning either
301 2015-02-27 12:18:03 <Eliel> anyway, the result is that people build their own solutions.
302 2015-02-27 12:18:15 <sipa> i think it's fine that people build their own solutions; bitcoind can't do everything
303 2015-02-27 12:18:23 <wumpus> yes, that's fine
304 2015-02-27 12:18:28 <sipa> having a more pluggable infrastructure for building your own indexes would be nice
305 2015-02-27 12:18:37 <gmaxwell> (as an optional thing: well we've already written that; but the applications have not materalized for it)
306 2015-02-27 12:18:38 <sipa> but there is risk associated with that too
307 2015-02-27 12:18:39 <wumpus> it's good to have things outside of bitcoind
308 2015-02-27 12:18:46 <Eliel> and that leads to bitcoind receiving much less developer resources.
309 2015-02-27 12:18:53 <CodeShark> but the other part of the problem is that there's no way to construct a short proof of the validity of the result of such queries
310 2015-02-27 12:18:55 <sipa> yeah, sunk costs
311 2015-02-27 12:19:02 <CodeShark> so any node implementing such services must be trusted
312 2015-02-27 12:19:24 <sipa> Eliel: well, i'm not sure; i think all we'd get from that is people helping build indexing on top, not core improvements
313 2015-02-27 12:19:41 <sipa> and in fact, would potentially make core improvements harder due to it breaking external stuff
314 2015-02-27 12:19:56 <sipa> but yes, it does shift attention away
315 2015-02-27 12:20:03 <wumpus> I'd be fine with bitcoind not having a wallet in the first place, but that's what we inherited, and people insist on using it but don't improve it
316 2015-02-27 12:20:21 <gmaxwell> Eliel: I'm completely confident that you're incorrect there. Sipa wrote a addrindex patch, and none of that other interest showed up at all for it. At the end of the day the cost of it is just enormous, and it can be completely avoided, but not if you build your service around a centeralized database like bc.i.
317 2015-02-27 12:20:47 <sipa> i'm sure people would use addrindex if it were merged
318 2015-02-27 12:20:51 <CodeShark> if we want to support tx indexing we probably need to rethink the core structures a bit...not just add indices
319 2015-02-27 12:21:00 <CodeShark> err, address indexing
320 2015-02-27 12:21:03 <wumpus> tx indexing of the utxo set would be nice
321 2015-02-27 12:21:10 <wumpus> not of the entire block chain
322 2015-02-27 12:21:16 <wumpus> eh address indexing
323 2015-02-27 12:21:20 <CodeShark> heh
324 2015-02-27 12:21:27 <Eliel> if an addrindex were merged, I'd personally start using it immediately. txindex is useless for me.
325 2015-02-27 12:21:36 <sipa> Eliel: that;s my problem
326 2015-02-27 12:21:38 <wumpus> there's no point in indexing historical, no-longer-relevant stuff
327 2015-02-27 12:21:54 <sipa> i think every use of addrindex except blockexplorer/debug is a step in the wrong direction
328 2015-02-27 12:22:12 <sipa> that will lock the ecosystem in a less scalable position
329 2015-02-27 12:22:13 <gmaxwell> it certantly pushing the ecosystem very strongly towards complete centeralization.
330 2015-02-27 12:22:48 <Eliel> the reason bc.i is getting as much use as it is is because people want to search by address.
331 2015-02-27 12:23:14 <CodeShark> there's certainly a desire for it - and the fact that they are called "addresses" sort of implies that this should be possible to do
332 2015-02-27 12:23:57 <CodeShark> if at least it were possible to construct short proofs of query results
333 2015-02-27 12:24:05 <sipa> Eliel: yes, and the problem is that they want to :)
334 2015-02-27 12:24:27 <CodeShark> then we could still use big untrusted servers - at least if care is taken as far as privacy protection
335 2015-02-27 12:24:41 <sipa> If I could build a feature in bitcoin core that would stop people from thinking a lookup by address is the way to go, I would.
336 2015-02-27 12:24:59 <CodeShark> change the name to "invoice id" or something else :p
337 2015-02-27 12:25:04 <sipa> CodeShark: way too late
338 2015-02-27 12:25:06 <CodeShark> hehe
339 2015-02-27 12:25:43 <Eliel> even if it were named invoice id, searching by that would be the simplest way to implement a check if a certain invoice is paid.
340 2015-02-27 12:26:06 <gmaxwell> Eliel: sure but we already search by index id cheaply and easily, within the wallet the index belongs to.
341 2015-02-27 12:26:21 <gmaxwell> s/index/invoice/ :)
342 2015-02-27 12:26:36 <CodeShark> to avoid having to do that you need to consider the wallet account structure when synching
343 2015-02-27 12:26:54 <Eliel> granted, I believe this support for watch-only addresses will help a lot.
344 2015-02-27 12:27:21 <Luke-Jr> wallets*
345 2015-02-27 12:27:23 <Eliel> I'd be beyond happy if it supported BIP32 public master key import too :P
346 2015-02-27 12:27:25 <CodeShark> if you want to be able to sync without concern for account structures and then overlay accounts atop the full database being able to query by address is necessary
347 2015-02-27 12:28:45 <CodeShark> this type of thing works well for a wallet deployment that's only intended to support a specific account structure for specific accounts - it doesn't work well for a general service that is account-agnostic
348 2015-02-27 12:29:08 <CodeShark> hence bc.i
349 2015-02-27 12:39:39 <Luke-Jr> would anyone object to having the GUI first-run present policy settings (unrelated to the spamfilter patch)?
350 2015-02-27 12:40:25 <wumpus> yes, I would
351 2015-02-27 12:40:42 <Luke-Jr> :<
352 2015-02-27 12:41:05 <wumpus> I understand where you're coming from, but it's just not realistic to present GUI users with those choices
353 2015-02-27 12:42:00 <Eliel> sipa: I think the main reason the addrindex patch doesn't get used is that people prefer using binaries to compiling from source. Also, there's the thinking "there must be something wrong with it since it's not part of the main binary" that keeps people from patching and compiling their own.
354 2015-02-27 12:42:34 <sipa> Eliel: well, working as intended
355 2015-02-27 12:42:36 <CodeShark> I built a bc.i clone just using txindex but addrindex wasn't supported at the time
356 2015-02-27 12:42:37 <Eliel> and since bitcoins are quite valuable, people tend to be _very_ conservative with what they do.
357 2015-02-27 12:44:09 <CodeShark> I think it's a problem, though - that querying by address is so inefficient yet the way things are presented to users strongly suggests that this should be a standard operation
358 2015-02-27 12:44:42 <CodeShark> the way I see it either the use model is broken or the protocol is broken or both :p
359 2015-02-27 12:44:44 <wumpus> but in a bank you can't just query for someone else's account number either.
360 2015-02-27 12:44:59 <CodeShark> wumpus: but that's only because of access controls
361 2015-02-27 12:45:03 <wumpus> it's kind of the lousy privacy of bitcoin that people expect to be able to do this at all
362 2015-02-27 12:45:07 <CodeShark> it's not about inherent db inefficiencies
363 2015-02-27 12:45:27 <CodeShark> bank tellers can query other people's account numbers
364 2015-02-27 12:46:53 <wumpus> personally I'd rather get rid of the stored public history than improve query-ability of it
365 2015-02-27 12:47:27 <gmaxwell> I don't think anything in Bitcoin Core presents it that way.
366 2015-02-27 12:47:54 <gmaxwell> Unfortunately there is a lot of commercial benefit to be had in getting people relying on that model, since it forces you into using service providers. :(
367 2015-02-27 12:47:58 <sipa> it's only sinxe blockexplorer that people think about addresses that way :)
368 2015-02-27 12:48:00 <wumpus> I don't think so either. It presents as a fairly straightforward wallet in which you manage your own transactions, not list those of others...
369 2015-02-27 12:48:02 <sipa> *since
370 2015-02-27 12:48:47 <CodeShark> it's useful to be able to share bc.i links, though
371 2015-02-27 12:49:06 <sipa> CodeShark: but broken from a trudt modrl
372 2015-02-27 12:49:11 <sipa> trust
373 2015-02-27 12:49:21 <wumpus> useful for what?
374 2015-02-27 12:49:48 <sipa> it's convenient
375 2015-02-27 12:49:50 <CodeShark> well, besides analysis and debug?
376 2015-02-27 12:49:55 <sipa> "look, i paid you!"
377 2015-02-27 12:50:03 <sipa> doesn't make it a good idea
378 2015-02-27 12:50:04 <CodeShark> yeah, it's a simple kind of "receipt"
379 2015-02-27 12:50:09 <wumpus> maybe from a kind of reality show nosyness
380 2015-02-27 12:50:42 <CodeShark> the broken trust model is the biggest problem, IMHO
381 2015-02-27 12:51:09 <CodeShark> even the inefficiencies could be tolerated if it were possible to prove the validity of query results and to perform queries with some measure of privacy
382 2015-02-27 12:51:19 <sipa> yes, bitcoin users expect a payment system
383 2015-02-27 12:51:36 <sipa> rather than a trustless currency
384 2015-02-27 12:51:42 <CodeShark> without a payment system it's next to useless for most people, though
385 2015-02-27 12:51:56 <sipa> sure
386 2015-02-27 12:51:56 <wumpus> how does a payment system require to query other people's balances?
387 2015-02-27 12:52:09 <sipa> but paymemt systems can be built on top
388 2015-02-27 12:53:32 <wumpus> or not even only the balances (which is possible with the utxo set) but the whole history throughout gigabytes of transaction data too
389 2015-02-27 12:53:42 <CodeShark> the way things are it strongly encourages centralization of tx verification
390 2015-02-27 12:54:07 <CodeShark> if for no other reason than that if you make this assumption it drastically simplifies writing a wallet app
391 2015-02-27 12:54:18 <sipa> i think we're preaching to the choir here
392 2015-02-27 12:54:27 <gmaxwell> yes, indeed making something totally centeralized vastly simplifies it.
393 2015-02-27 12:55:04 <wumpus> it simplifies making a broken wallet app
394 2015-02-27 12:56:36 <CodeShark> anyhow, this is a serious issue for adoption...and is especially an issue for mobile PoS and stuff like that
395 2015-02-27 12:56:51 <CodeShark> and I don't see any simple solution
396 2015-02-27 12:57:53 <CodeShark> so resistance is futile, I'm afraid, sipa :p
397 2015-02-27 13:01:11 <CodeShark> the only thing that might somewhat make it tenable is storing the merkle root of the txout set in blocks
398 2015-02-27 13:01:23 <CodeShark> because then at least it would be possible to construct O(log n) proofs
399 2015-02-27 13:01:56 <gmaxwell> very expensive though.
400 2015-02-27 13:02:18 <CodeShark> for whom?
401 2015-02-27 13:02:30 <sipa> full nodes
402 2015-02-27 13:02:36 <sipa> to validate all updates to it
403 2015-02-27 13:02:59 <CodeShark> yeah, I guess there's the tradeoff
404 2015-02-27 13:03:12 <sipa> it's pretty significant
405 2015-02-27 13:03:24 <CodeShark> and without an incentive structure to offset this cost it is a problem
406 2015-02-27 13:05:12 <wumpus> yes, commit-to-utxo-root in blocks would be incredibly useful
407 2015-02-27 13:06:27 <gmaxwell> in any case, just building it without the commitment in blocks is interesting and also useful (not as useful), and doesn't have the suicidepactness of being a network rule.
408 2015-02-27 13:06:59 <gmaxwell> Part of the challenge is that there is a major competing design (STXO) and it's not clear which is actually better. (though both have roughly similar costs, and it's not clear of those costs are worth it)
409 2015-02-27 13:07:35 <Eliel> I don't think people generally want to use the address query thing for querying other people's balances much. I'd think most are only interested in their own wallets as well as the status of whatever payments they've made.
410 2015-02-27 13:07:52 <gmaxwell> keep in mind, commited TXOs just make it harder for centeralized servers to dos attack you silently. Users can still be dos attacked detectably, and they lose basically all their privacy.
411 2015-02-27 13:08:17 <sipa> Eliel: yes, that's the problem
412 2015-02-27 13:08:32 <sipa> if they use mechanism that allow easy querying of their own addresses, others can too
413 2015-02-27 13:08:45 <gmaxwell> Eliel: bitcoin core already has robust support for querying your _own_ wallet, which is efficient and doesn't have huge externalized costs or local costs.
414 2015-02-27 13:08:50 <sipa> we shouldn't use mechanisms where querying a balance is useful information at all
415 2015-02-27 13:08:56 <wumpus> but you can already query your own.. .right @ gmaxwell
416 2015-02-27 13:09:17 <Eliel> gmaxwell: until recently, that was only possible by risking your private keys online as well.
417 2015-02-27 13:09:24 <CodeShark> the only reason it's efficient is because the wallet structure is considered during sync, gmaxwell - I think many app developers would like to separate the sync logic from the account management logic
418 2015-02-27 13:09:41 <gmaxwell> Eliel: thats not true. Unless recently you mean 2011. :(
419 2015-02-27 13:09:56 <gmaxwell> (you can use wallet encryption to achieve a watching wallet)
420 2015-02-27 13:10:26 <wumpus> CodeShark: but it's only due to the privacy loophole in bitcoin that it's possible to make address indexes at all
421 2015-02-27 13:10:40 <Eliel> gmaxwell: yes, if you're willing to constantly update the wallet after adding addresses to it on another system.
422 2015-02-27 13:10:51 <CodeShark> wumpus: are you suggesting disallowing reuse of addresses? :)
423 2015-02-27 13:11:12 <gmaxwell> indeed, zero cash utterly breaks that, and you _must_ build it for that like bitcoin core currently does. As does BRS, and stealth addresses at least if you are to keep any privacy at all.
424 2015-02-27 13:11:28 <Eliel> it's only practical with 0.10.0 now with the watch-only address function.
425 2015-02-27 13:11:31 <gmaxwell> CodeShark: disallowing address reuse requires a forever growing unprunable index, otherwise it might have always been disallowed.
426 2015-02-27 13:11:36 <adlai> CodeShark: regardless of whether it's a good practice or not, it's impractical to disallow
427 2015-02-27 13:11:51 <Eliel> and even there you still need to jump through extra hoops to get BIP32 support.
428 2015-02-27 13:11:51 <gmaxwell> Eliel: hm? you set the keypool size to 100k or what have you.
429 2015-02-27 13:11:54 <wumpus> Eliel: it's still the same though, doing that properly would require BIP32 support
430 2015-02-27 13:12:27 <wumpus> Eliel: bitcoin core really can't watch a remote wallet, it can only watch it's own
431 2015-02-27 13:12:59 <gmaxwell> I think it's really concering that so many wallets were in such a rush to deploy novel cryptographic systems adead of peer review and their own authors.
432 2015-02-27 13:13:02 <wumpus> with watchonly and importing individual addresses you can get close, but you'd have to import every new address
433 2015-02-27 13:13:26 <Eliel> wumpus: yes, and that rules bitcoind out as the primary software for anyone wanting to receive bitcoins securely online.
434 2015-02-27 13:13:27 <wumpus> watchonly is foremost useful for multisig
435 2015-02-27 13:13:52 <wumpus> Eliel: patches welcome?
436 2015-02-27 13:13:57 <gmaxwell> Eliel: lol.
437 2015-02-27 13:14:05 <CodeShark> I thought watchonly is foremost useful for inbound payment processing without having to keep the private keys on servers
438 2015-02-27 13:14:06 <CodeShark> :)
439 2015-02-27 13:14:20 <CodeShark> at least that was the first reason I encountered for it and what originally motivated it
440 2015-02-27 13:14:36 <wumpus> CodeShark: that would require deterministic address generation of some kind right?
441 2015-02-27 13:15:02 <CodeShark> not necessarily, wumpus - you can deploy lists of randomly generated addresses - deterministic address generation makes it simpler to deploy accounts
442 2015-02-27 13:15:21 <CodeShark> and to keep them in sync
443 2015-02-27 13:15:35 <wumpus> CodeShark: sure, you could do manual key management...
444 2015-02-27 13:16:01 <gmaxwell> Eliel: look, as the person who introduced the public derrivation, I think you're crazy. There are known limitations regarding security, and it's novel cryptography. The benefit they provide for that application is relatively modest considering their newness. A million pubkeys takes up 33 megabytes-- against a many gigabyte bitcoin core install, and would cover almost anyone's conceviable needs for
445 2015-02-27 13:16:07 <gmaxwell> a decade or more.
446 2015-02-27 13:16:31 <gmaxwell> Eliel: so I just have to laugh when you throw out hyperbole that bitcoin core has been "ruled out" because it doesn't implement public derrivation.
447 2015-02-27 13:18:47 <wumpus> heh
448 2015-02-27 13:19:10 <gmaxwell> The public derrivation approach also has the key fragility problem, leak one private key, you've leaked them all. I think people often underestimate the risks from that.
449 2015-02-27 13:21:08 <CodeShark> you have to consider the different ways the keys can be compromised, though - weak RNG, someone breaking into a server, etc...
450 2015-02-27 13:21:35 <CodeShark> if you generate private keys on different systems with different RNGs then I would concur with you, gmaxwell
451 2015-02-27 13:21:43 <gmaxwell> public derrive gets you the "I exported a private key and gave it to someone; hey where did all my funds go?" too
452 2015-02-27 13:21:57 <wumpus> right, so the idea would be to: generate N privkey/address pairs offline, send the pubkeys to the watching server, importaddress them there and store them in a database. Then give customers a new address from that list every time.
453 2015-02-27 13:22:00 <CodeShark> but if someone breaks into your bitcoind server chances are if they can steal one private key they can steal them all
454 2015-02-27 13:22:33 <gmaxwell> CodeShark: there have been a lot of leaked private keys actually in practice. I used to argue things like that I think I wouldn't anymore. Turns out crazy things happen.
455 2015-02-27 13:22:53 <CodeShark> I'm guessing the problem in those instances wasn't so much the software but the policies
456 2015-02-27 13:24:47 <wumpus> then if you want to spend any of the income you can create the transaction on a watching server, just need the private keys to sign which can happen in isolation offline
457 2015-02-27 13:26:51 <wumpus> the latter could be done e.g. with a non-internet-connected bitcoind using signtransaction, or with other software I suppose
458 2015-02-27 13:27:53 <wumpus> right now the wallet can't generate a spending transaction from watch-only balance, and return it unsigned
459 2015-02-27 13:27:55 <gmaxwell> CodeShark: nah, lots of crazy software out there. Things setting the nonce to constants, or H(message) or ...
460 2015-02-27 13:28:11 <CodeShark> lol
461 2015-02-27 13:28:25 <sipa> this is not a joke.
462 2015-02-27 13:28:57 <CodeShark> I know, after the SWARM snafu nothing surprizes me
463 2015-02-27 13:28:59 <gmaxwell> Or like one altcoin recently H(message) ^ privatekey. which is totally busted but just strong enough that proving its busted take a lot of work.
464 2015-02-27 13:29:33 <CodeShark> anyhow, I'm off - goodnight everyone
465 2015-02-27 13:29:39 <gmaxwell> there is some out there who is producing large numbers of txn where every signature uses the same nonce, but seemingly unrelated private keys.
466 2015-02-27 13:31:02 <wumpus> ah yes, there is a PR for that, which adds watch-only support to fundrawtransaction, that's nice https://github.com/bitcoin/bitcoin/pull/5524
467 2015-02-27 13:31:43 <gmaxwell> BlueMatt: ^ you gonna comment on that?
468 2015-02-27 13:32:37 <Eliel> gmaxwell: I see. So, in your opinion, is pregenerating a million pubkeys a better idea than using BIP32 wallet?
469 2015-02-27 13:33:42 <gmaxwell> maybe? The gap between them is not so large. The pregen keys have some setup costs (moving 33mb of key data). They have more robustness against some kinds of errors.
470 2015-02-27 13:37:13 <Eliel> ok, now I just need to find out how much more RAM bitcoind needs for 1 million watch-only addresses in wallet and how it affects the startup speed.
471 2015-02-27 13:38:06 <Eliel> 0.9.2.1 took around 15 minutes on the current system to read a wallet with 25k addresses
472 2015-02-27 13:38:21 <hearn> Eliel: if you do that, make sure you either generate a ridiculous number of keys up front, or modify Core so it quits rather than extending the key pool. otherwise you can set up a successful business and then 18 months from now your off-site key backup silently become stale.
473 2015-02-27 13:38:28 <wumpus> that's due to the number of transactions, not keys
474 2015-02-27 13:38:44 <hearn> Eliel: i have encountered at least one business where they didn't reconfigure the key pool size and were making nightly backups of their wallet. but they were running out of keys in the early afternoon.
475 2015-02-27 13:38:48 <hearn> data loss == doom in that case
476 2015-02-27 13:39:00 <Eliel> I'm already running with a pool of 10k
477 2015-02-27 13:39:33 <wumpus> hearn: yes, changing core to error when the keypool is exhausted instead of automatically regenerating seems prudent
478 2015-02-27 13:39:56 <wumpus> hearn: especially if you have backups or other actions to perform
479 2015-02-27 13:40:24 <hearn> wumpus: yes, adjusting the default size upwards bty 100x or so would also be a good idea. it's easy to exhaust the keypool during the day with the current size, with reasonable sorts of use cases like a busy hot wallet
480 2015-02-27 13:40:47 <wumpus> hearn: sure, probably both
481 2015-02-27 13:41:02 <gmaxwell> Eliel: 15 minutes? uh. thats VPS or spinning HD latency on the utxo consistency check.
482 2015-02-27 13:41:06 <wumpus> hearn: e.g. increase the default, then require manual action when it runs out
483 2015-02-27 13:41:53 <Eliel> gmaxwell: the system most definitely doesn't have an SSD.
484 2015-02-27 13:42:25 <Eliel> there's a check before wallet load that also takes 15 minutes.
485 2015-02-27 13:42:28 <Eliel> so total of 30
486 2015-02-27 13:43:34 <gmaxwell> Eliel: node here with >400k transactions in the wallet, and 13k addresses on my 2.1ghz laptop took 19 seconds to start cold cache.
487 2015-02-27 13:44:12 <gmaxwell> (this is a real production wallet on the mainnet that someone sent me)
488 2015-02-27 13:47:02 <Eliel> the performance difference from SSD is sometimes quite unbelievable...
489 2015-02-27 13:48:51 <gmaxwell> one would think that it shouldn't be so large an effect. :( but indeed, it's pretty big.
490 2015-02-27 13:49:43 <hearn> it might be improved by tweaking the leveldb settings
491 2015-02-27 13:49:53 <hearn> leveldb was designed for spinning disks, but i don't know if we ever optimised it for that use case
492 2015-02-27 13:50:00 <hearn> when i developed the original code i did so on a mac laptop that had an SSD
493 2015-02-27 13:50:13 <hearn> not sure if sipa tweaked the settings when he merged it with ultraprune
494 2015-02-27 13:52:55 <gmaxwell> I imagine whats happening is just a pessimal read pattern, we must do the consistency check backwards, so if the updates ended up on disk fowards, its probably looks like semi-random backwards reading.
495 2015-02-27 14:00:34 <Eliel> I think I'll be upgrading to 0.10 tonight. That performance is with 0.9.2.1, so it's not quite current from your point of view.
496 2015-02-27 14:01:38 <gmaxwell> 0.10 may be somewhat better, likely even, but sadly I think the hd is the bigger issue there.
497 2015-02-27 14:03:59 <sipa> hearn: i don't think we rely on leveldb much anymore, certainly with larger in-application utxo caching
498 2015-02-27 14:04:04 <sipa> *performance
499 2015-02-27 14:04:45 <hearn> sipa: well the cache still has to be filled somehow
500 2015-02-27 14:04:58 <gmaxwell> sipa: context here is the initial startup consistency check taking 15 minutes for Eliel
501 2015-02-27 14:05:04 <sipa> ah
502 2015-02-27 14:09:42 <Eliel> the wallet load also takes 15 minutes after the consistency check. there are 15k active and 10k pool addresses.
503 2015-02-27 14:11:24 <gmaxwell> how big is the wallet file?
504 2015-02-27 14:11:35 <ahf> a/94
505 2015-02-27 14:12:58 <Eliel> gmaxwell: 62468096 bytes
506 2015-02-27 14:13:53 <gmaxwell> the one here is 89MB.
507 2015-02-27 14:15:18 <Eliel> I guess it's checking the transactions as it loads the wallet?
508 2015-02-27 14:19:54 <wumpus> it loads them into memory, it does not perform extensive checking on them
509 2015-02-27 14:21:43 <Eliel> in that case I'm a little puzzled that it takes this long. Even without an SSD.
510 2015-02-27 14:22:34 <wumpus> run a profiler I'd say, then you know where it spends the time
511 2015-02-27 14:25:47 <wumpus> pre 0.10 versions did store the predecessor transactions with a transaction in the wallet, which was a waste of space and i/o
512 2015-02-27 14:27:08 <wumpus> keys are checked when read, but this is done using a simple hash, not by doing a full validation
513 2015-02-27 16:29:54 <dgenr8> re address reuse... the idea to publish scriptpubkey bloom filters in blocks might have the effect of allowing nodes to build a "bloofi" index and prevent reuse, with false positives though
514 2015-02-27 16:48:02 <dgenr8> re spamfilter ... the "dice" spamfilter is a convenient way to get a bunch of mining power replace a tx they otherwise wouldn't. just include a tiny payment to the "spam" address. identity of the real victim is unknown
515 2015-02-27 16:48:19 <dgenr8> viz. this 6 BTC double-spend a couple hours ago 7b67ab717008c3e366eee82e947923b10638bbe58424ce09ee2a1b2ec39ddfd2
516 2015-02-27 16:56:01 <gmaxwell> dgenr8: thats true for any mempool / proiority logic difference though. E.g. dust outputs, or changes in IsStandard.
517 2015-02-27 16:56:20 <gmaxwell> e.g. right now you can probably get a nice 40/60 split via p2sh with a not in it.
518 2015-02-27 16:58:13 <gmaxwell> in a decenteralized network there will always be inconsistency; mining exists to resolve that. Trying to ask mining to be instantly consistent would be basically saying you need another blockchain in your blockchain (yo dawg). :)
519 2015-02-27 17:01:22 <dgenr8> er, yes indeed. quite impossible. not sure why you mention it though
520 2015-02-27 17:02:03 <dgenr8> anyway somebody likes using the spamfilter for this purpose. just a data point
521 2015-02-27 17:44:44 <hearn> dgenr8: what's the url of your dashboard again?
522 2015-02-27 17:44:53 <hearn> dgenr8: i keep meaning to add dice addresses to the bitcoinj tx risk analysis code
523 2015-02-27 17:45:07 <hearn> dgenr8: or rather, duplicate luke's spam filter in the risk analysis. seriously painful and annoying to have to do this.
524 2015-02-27 17:46:59 <dgenr8> you have it in your release notes... http://respends.thinlink.com the tx I mentioned above is interesting ... for some reason he chopped up tx1 into .5BTC increments in a dependent chain
525 2015-02-27 17:47:37 <hearn> ah yes
526 2015-02-27 17:47:41 <hearn> i should bookmark it
527 2015-02-27 17:47:51 <hearn> for some reason chrome thinks the page is in greek
528 2015-02-27 17:48:57 <dgenr8> too many mu's prolly
529 2015-02-27 17:54:22 <hearn> dgenr8: any idea who was being attacked?
530 2015-02-27 17:54:56 <morcos> it seems like the first transaction had no fees and insufficient priority to be relayed by 0.10 clients
531 2015-02-27 17:56:04 <dgenr8> hearn: no. I hear OP_MUL knows who is behind every address tho ;)
532 2015-02-27 17:56:34 <gmaxwell> only about 20% of them.
533 2015-02-27 17:58:51 <dgenr8> are any of those UTXO's with new addresses? Or does he only make conjectures after a spend?
534 2015-02-27 18:00:05 <gmaxwell> he knows things about ones that haven't spent too.
535 2015-02-27 18:00:13 <dgenr8> morcos: belt-and-suspenders i guess
536 2015-02-27 18:01:12 <morcos> dgenr8: the fees reported on your webpage, are those accumulated among all children as well?
537 2015-02-27 18:02:36 <dgenr8> yes
538 2015-02-27 18:10:40 <kanzure> under what circumstances might bitcoind be slow to respond to rpc requests?
539 2015-02-27 18:17:03 <Raccoon> Does anyone here want the channel #bitcoin-coders before I /drop/ it, preventing it from being easily re-registered?
540 2015-02-27 18:17:28 <Raccoon> it would become property of "freenode-staff"
541 2015-02-27 18:18:37 <goatpig> hi
542 2015-02-27 18:20:19 <goatpig> is there a condition under which bitcoind would rearrange data in blkxxxxx.dat files starting 0.10? Particularly if it's catching up a few days worth of history
543 2015-02-27 18:20:36 <sipa> rearrange? no
544 2015-02-27 18:20:43 <sipa> store out of order? yes
545 2015-02-27 18:20:51 <sipa> it's in the release notes
546 2015-02-27 18:21:01 <goatpig> i got the out of order part months ago
547 2015-02-27 18:21:36 <goatpig> im seeing strange behavior where tons of blocks do not appear where they were last seen after a few days of blocks were udpated into the blk files
548 2015-02-27 18:21:54 <skittylx> does that mean the blockchain from .10 can't be imported to lesser versions?
549 2015-02-27 18:22:00 <goatpig> is there documentation on the out of order change?
550 2015-02-27 18:22:29 <sipa> gmaxwell: the release notes
551 2015-02-27 18:22:34 <sipa> eh, goatpig
552 2015-02-27 18:22:40 <goatpig> aight thanks
553 2015-02-27 18:22:42 <sipa> skittylx: see the release notes
554 2015-02-27 18:24:01 <sipa> goatpig: i'm not sure what you mean with rearrange
555 2015-02-27 18:24:13 <sipa> data inside the blk* files is append only
556 2015-02-27 18:24:22 <sipa> so blocks in it shouldn't ever change
557 2015-02-27 18:25:04 <sipa> goatpig: unless your node crashed, in which case it may overwrite things
558 2015-02-27 18:25:15 <sipa> which weren't properly synced
559 2015-02-27 18:25:16 <goatpig> Not a node issue, it's an issue with Armory
560 2015-02-27 18:25:30 <sipa> ask them, in that case
561 2015-02-27 18:25:37 <goatpig> I am them
562 2015-02-27 18:25:37 <sipa> no clue about armory here
563 2015-02-27 18:25:58 <goatpig> im trying to figure out if there was more to the out of order blocks change
564 2015-02-27 18:26:19 <sipa> feel free to ask
565 2015-02-27 18:26:34 <sipa> i just don't really understand what you're saying
566 2015-02-27 18:27:04 <goatpig> as long as you confirm that block data is never reordered in blkxxxxx.dat files i can blame my own code and mode on
567 2015-02-27 18:27:16 <goatpig> move*
568 2015-02-27 18:28:05 <sipa> unless the nodes crashes after writing blocks to disk but before flushing the index database, no
569 2015-02-27 18:28:19 <sipa> if yes, then the last blocks on disk may be overwritten
570 2015-02-27 18:28:38 <goatpig> so a forceful shotdown of bitcoind could cause that condition?
571 2015-02-27 18:28:48 <goatpig> shutdown*
572 2015-02-27 18:29:00 <sipa> yes
573 2015-02-27 18:29:12 <goatpig> ok then it's probably what is causing me so much pain
574 2015-02-27 18:29:12 <sipa> if it shuts down cleanly it won't happen
575 2015-02-27 18:29:20 <goatpig> thanks for the help, really appreciated
576 2015-02-27 18:39:21 <hearn> dgenr8: he does? is the db public?
577 2015-02-27 18:42:55 <hearn> hmm, has anyone heard of bitnet before?
578 2015-02-27 18:43:10 <hearn> they seem to have an impressively large team but i never encountered them before today
579 2015-02-27 18:53:35 <Eliel> gmaxwell: you could improve double spend situation detectability if miners were somehow incentiviced to publish some of their better attempts at finding a block. That would allow the network to have a reasonably good idea of what transactions are being mined and by what percentage of miners.
580 2015-02-27 18:57:55 <Eliel> If those were published, you could also implement a kind of an automated pooling with them. Miners could then reduce their variance without pools by including each other in the coinbase outputs with a tit for tat algorithm.
581 2015-02-27 19:42:26 <cfields> sipa: grr, yet more PRs to fix the un-supported libsecp256k1 verification. If it's still what you would prefer, I'll PR the removal of that code right now so people quit trying to "fix" it.
582 2015-02-27 20:05:06 <jtimon> phantomcircuit there's definitely redundant checks and maybe also missing checks that should be in more places (like enforcing bip30?), anyway this should become clearer once we separate consensus and policy code...
583 2015-02-27 20:05:25 <jtimon> ACTION nags people about #5696
584 2015-02-27 20:12:35 <cfields> jtimon: poking around yesterday, i finally managed to figure out an approach to the keystore that works with the lib concept
585 2015-02-27 20:12:40 <cfields> jtimon: got a min to discuss?
586 2015-02-27 20:13:05 <cfields> (it conflicts pretty hard with #5251)
587 2015-02-27 20:14:46 <jtimon> sure, I can discuss
588 2015-02-27 20:15:06 <jtimon> I'm happy closing #5251 if something better is found
589 2015-02-27 20:16:00 <jtimon> feel free to reuse my commits and/or change them if you find it useful too
590 2015-02-27 20:17:03 <jtimon> cfields ?
591 2015-02-27 20:17:44 <cfields> jtimon: sorry, food just got here. will you be around in 15min or so?
592 2015-02-27 20:18:16 <jtimon> I was planning on going to sleep soon, but sure, I can wait 15 mins
593 2015-02-27 20:18:55 <cfields> ok nm, food can wait..
594 2015-02-27 20:19:06 <cfields> my change is similar to yours in establishing a base class for the key store...
595 2015-02-27 20:19:33 <jtimon> and the difference is?
596 2015-02-27 20:19:49 <jtimon> removing what only stuff from it?
597 2015-02-27 20:19:57 <jtimon> watch-only
598 2015-02-27 20:20:15 <cfields> but hugely simplifying it. My goal is to make the keystore indifferent to how the keys are actually stored (which sounds stupid, but bear with me)
599 2015-02-27 20:20:43 <jtimon> doesn't sound stupid to me...
600 2015-02-27 20:21:07 <cfields> so it retains the getters (GetPubKey() etc), but it leaves the actual setting of priv keys to the derived classes
601 2015-02-27 20:21:36 <jtimon> ok, do you have a PR for this?
602 2015-02-27 20:22:07 <cfields> that way the derived classes can store the keys however they want, as long as they implement the getters properly. That would mean that an app author could create their own store with their own whacky scheme and not have to use CKey
603 2015-02-27 20:22:26 <cfields> not yet. I just have a hacked up version for now, wanted to run the idea by you first
604 2015-02-27 20:23:08 <cfields> jtimon: the second change is to add Sign() to the keystore, which i'm really unsure about
605 2015-02-27 20:23:55 <jtimon> ok, sounds good to me, when it's ready please post the PR # on #5251 so that I don't forget to review it (and probably close mine)
606 2015-02-27 20:24:00 <cfields> but that would mean that in script/sign.o, the keystore/keyid/hash would be all that's necessary to create a sig. So again, no reliance on CKey
607 2015-02-27 20:24:32 <jtimon> hmm, I'm not so sure about the last thing, would need to see the code
608 2015-02-27 20:24:45 <cfields> at that point, CKey becomes an app detail, and an external signing api could be created
609 2015-02-27 20:24:45 <jtimon> can you keep that in a different commit just in case?
610 2015-02-27 20:24:58 <cfields> yep, fair enough. I'll clean up what i've got and push it out
611 2015-02-27 20:25:06 <jtimon> great
612 2015-02-27 20:25:33 <cfields> the base interface sucks though, it's a collection of functions that really make no sense together. I could use some help coming up with a more reasonable separation
613 2015-02-27 20:26:33 <cfields> thanks for the quick chat. Get some sleep :)