1 2014-08-14 00:19:06 <Luke-Jr> dgenr8: the problem would be getting the first transaction relayed and in memory pools
2 2014-08-14 00:19:18 <Luke-Jr> dgenr8: once that's done, #1647 works
3 2014-08-14 01:03:56 <njaard> greetings. does anyone have a list of active/interesting wallet addresses?
4 2014-08-14 01:04:03 <njaard> like 1k or 10k
5 2014-08-14 01:04:26 <njaard> um
6 2014-08-14 01:04:27 <njaard> on testnet
7 2014-08-14 01:06:23 <Lloydy> what do you mean by active/interesting?
8 2014-08-14 01:07:34 <njaard> ones that appear in the blockchain would be a good start
9 2014-08-14 01:08:02 <njaard> I mean, I could write a program to do so
10 2014-08-14 01:08:08 <njaard> but I'm lazy!
11 2014-08-14 01:09:02 <Lloydy> why not check out the blockchain website?
12 2014-08-14 01:09:19 <Lloydy> https://blockchain.info/largest-recent-transactions
13 2014-08-14 01:09:28 <njaard> I need testnet
14 2014-08-14 01:09:36 <Lloydy> oh! sorry
15 2014-08-14 01:12:10 <mr_burdell> njaard: http://tbtc.blockr.io/trivia/address
16 2014-08-14 01:12:34 <njaard> that's perfect, other than that it's mainnet
17 2014-08-14 01:12:42 <njaard> oh it's not
18 2014-08-14 01:12:44 <njaard> derp
19 2014-08-14 01:13:12 <njaard> shame it's not machine readable :(
20 2014-08-14 01:48:14 <wumpus> cfields: how's the travis 'pull tester' coming along?
21 2014-08-14 01:49:58 <jrick> njaard: if by active you mean recently paid to, you can just iterate the last however many blocks, iterate through each tx, and extract the payment addresses (if any)
22 2014-08-14 01:53:02 <jrick> I have a program that can do that if you want addresses recently found in the testnet chain
23 2014-08-14 01:53:35 <jrick> (written in go, if you don't have go then I could just give you the output from the last 100 blocks)
24 2014-08-14 01:59:53 <njaard> hey jrick, sorry for the long delay
25 2014-08-14 01:59:59 <njaard> I got a solution similar to that
26 2014-08-14 02:00:04 <jrick> cool
27 2014-08-14 02:00:08 <njaard> thanks for the offer though
28 2014-08-14 02:00:27 <njaard> I gotta disconnect due to computer being haunted
29 2014-08-14 02:53:48 <Luke-Jr> BlueMatt: reckon I should split base58 out of libblkmaker?
30 2014-08-14 02:54:19 <BlueMatt> huh?
31 2014-08-14 02:54:28 <BlueMatt> why?
32 2014-08-14 02:54:44 <BlueMatt> I must admit I'm rather unfarmiliar with the capabilities of libblkmaker
33 2014-08-14 02:55:01 <Luke-Jr> BlueMatt: so it can be used by things without having to pull in all of libblkmaker
34 2014-08-14 02:55:09 <Luke-Jr> libblkmaker mainly just interprets GBT and makes block headers
35 2014-08-14 02:55:12 <BlueMatt> more libraries is always good, I guess?
36 2014-08-14 02:55:18 <BlueMatt> though I'm wondering why you're asking me?
37 2014-08-14 02:55:30 <Luke-Jr> BlueMatt: related to library-ification of things :p
38 2014-08-14 02:55:40 <BlueMatt> heh, well, yea, I'm always in favor of more libraries
39 2014-08-14 02:55:56 <Luke-Jr> otoh, libblkmaker only can decode base58, not encode
40 2014-08-14 02:56:04 <BlueMatt> but, specifically, I'm doing the libbitcoinscript stuff so that we dont end up with people mining on btcd with its own script engine...
41 2014-08-14 02:56:05 <Luke-Jr> ACTION ponders the complexity of encoding
42 2014-08-14 02:56:21 <Luke-Jr> BlueMatt: well, Eligius plans to do just that ;)
43 2014-08-14 02:56:32 <BlueMatt> waattttt?
44 2014-08-14 02:56:49 <BlueMatt> you're gonna mine on a reimplementation with its own script+block processing code???
45 2014-08-14 02:56:56 <Luke-Jr> that's the plan
46 2014-08-14 02:57:02 <BlueMatt> why???
47 2014-08-14 02:57:07 <Luke-Jr> verifying bitcoind accepts the block proposal of course
48 2014-08-14 02:57:10 <Luke-Jr> BlueMatt: to test
49 2014-08-14 02:57:17 <BlueMatt> ok, yea,
50 2014-08-14 02:57:26 <BlueMatt> I suppose this makes sense, as long as you're piping the verification through bitcoind
51 2014-08-14 02:57:50 <Luke-Jr> same as we did for the 0.8 update basically
52 2014-08-14 02:58:09 <Luke-Jr> (which caught the bdb issue)
53 2014-08-14 02:58:16 <BlueMatt> yea
54 2014-08-14 02:58:49 <jrick> Luke-Jr: thanks :)
55 2014-08-14 02:59:01 <BlueMatt> Luke-Jr: whats the motivation for btcd, here?
56 2014-08-14 02:59:08 <BlueMatt> is it somehow better/different than core?
57 2014-08-14 02:59:08 <Luke-Jr> BlueMatt: ?
58 2014-08-14 02:59:12 <BlueMatt> or other options
59 2014-08-14 02:59:30 <Luke-Jr> BlueMatt: to encourage more full node implementations, properly tested etc
60 2014-08-14 03:00:02 <BlueMatt> that is not properly tested
61 2014-08-14 03:00:09 <Luke-Jr> it's part of it
62 2014-08-14 03:00:11 <BlueMatt> properly tested is spending a few man-months writing test cases
63 2014-08-14 03:00:15 <BlueMatt> its a very, very time part of it
64 2014-08-14 03:00:20 <Luke-Jr> yes
65 2014-08-14 03:00:29 <Luke-Jr> but real world testing is important too
66 2014-08-14 03:00:39 <BlueMatt> and, afaik, no one, not a single implementation is even trying to do that
67 2014-08-14 03:01:15 <davec> I guess 100% test coverage between internal unit tests and the block tester tool isn't doing that?
68 2014-08-14 03:01:42 <Luke-Jr> BlueMatt: btw, does libbitcoinscript.so get *used* by the resulting bitcoind/-qt?
69 2014-08-14 03:01:54 <BlueMatt> Luke-Jr: no
70 2014-08-14 03:02:02 <BlueMatt> but its the same code
71 2014-08-14 03:02:03 <Luke-Jr> davec: I think the keyword there is "afaik"
72 2014-08-14 03:02:15 <Luke-Jr> BlueMatt: I guess that can be a later PR
73 2014-08-14 03:03:05 <BlueMatt> Luke-Jr: sure, if even needed
74 2014-08-14 03:03:19 <dhill> BlueMatt: you should try out btcd :)
75 2014-08-14 03:03:44 <davec> fair point. Just a sore spot at this point. At the risk of sounding abrasive, and that's not my goal here, multiple full nodes are going to happen. It's better for the bitcoin ecosystem as a whole that the network has facilities in place to deal with it.
76 2014-08-14 03:04:34 <BlueMatt> I have no problem with more full nodes, but if people want to use them for consensus-critical applications they need to be putting in the proper time to test them
77 2014-08-14 03:04:52 <BlueMatt> and no one is yet, so I'm going to be very vocal and explain to people that using them for consensus-critical applications is a very, very bad idea
78 2014-08-14 03:04:58 <davec> for sure - totally agree with you about testing
79 2014-08-14 03:04:59 <BlueMatt> (ie no taking large sums of money, no mining, etc)
80 2014-08-14 03:05:23 <Luke-Jr> BlueMatt: davec's point is that btcd has lots of unit tests providing 100% coverage
81 2014-08-14 03:05:27 <BlueMatt> but testing, here, essentially means like using libbitcoinscript and running static analysis tools to prove 100% equivalence, not just tests
82 2014-08-14 03:05:32 <BlueMatt> Luke-Jr: so?
83 2014-08-14 03:05:49 <BlueMatt> Luke-Jr: thats like the first, baby step in testing consensus critical code
84 2014-08-14 03:05:55 <davec> we also have the bitcoind script tests
85 2014-08-14 03:06:05 <BlueMatt> which are also very minimal and not nearly sufficient
86 2014-08-14 03:06:25 <BlueMatt> davec: btcd should pretty much immediately switch to using libbitcoinscript.so
87 2014-08-14 03:06:29 <dhill> lol
88 2014-08-14 03:06:31 <davec> and the block tester tool you wrote
89 2014-08-14 03:06:38 <jrick> that won't happen
90 2014-08-14 03:06:46 <Luke-Jr> jrick: not even as an option?
91 2014-08-14 03:06:48 <BlueMatt> sure, and all of these things are useful, but not nearly sufficient
92 2014-08-14 03:06:53 <BlueMatt> jrick: dhill ummmmmm...what?
93 2014-08-14 03:07:11 <jrick> there's little value in an alt implemetnation that is no different from the reference
94 2014-08-14 03:07:26 <BlueMatt> in that case I'm gonna be vocal about people not using btcd even if it invests a few man-years in testing, because its clear there is a misunderstanding of the problems at play here, then
95 2014-08-14 03:07:41 <jrick> and it greatly overcomplicates our build process
96 2014-08-14 03:07:56 <BlueMatt> jrick: sure, if you're looking at like reusing the whole thing, but libbitcoinscript will (eventually) be fairly minimal script-only stuff
97 2014-08-14 03:08:01 <BlueMatt> which very clearly should be standardized
98 2014-08-14 03:08:11 <BlueMatt> jrick: huh? it would just be another dependency
99 2014-08-14 03:08:18 <BlueMatt> hell, go isnt even easy to install
100 2014-08-14 03:08:35 <jrick> I can install it in 3 commands
101 2014-08-14 03:08:39 <jrick> but anyways..
102 2014-08-14 03:08:50 <BlueMatt> and I can install libbitcoinscript (after it ends up in a release build) in one
103 2014-08-14 03:08:52 <davec> there is no misunderstanding - just a disagreement. I'm fully aware, but you're completely glossing over the point I made above. For the bitcoin ecosystem to truly mature, it needs to deal with multiple consensus critical implementations
104 2014-08-14 03:08:53 <dhill> oh the hate
105 2014-08-14 03:09:06 <BlueMatt> davec: no
106 2014-08-14 03:09:09 <davec> this mean things like using BIP0023 gbt proprosals across multiple impls
107 2014-08-14 03:09:10 <BlueMatt> no it doesnt
108 2014-08-14 03:09:34 <BlueMatt> it needs to have a single consensus, which either means one implementation, or multiple implementations which are proven to be identical
109 2014-08-14 03:09:53 <BlueMatt> and by proven I dont just mean test cases, I mean compiling them and using static analysis to mathematically prove equivalence
110 2014-08-14 03:10:04 <BlueMatt> (which is doable)
111 2014-08-14 03:10:10 <jrick> BlueMatt: it wouldn't be just another dependancy, it would require cgo (complicates the build procsses and makes cross compiling nearly impossible), stack swapping (BIG performance hit), etc.
112 2014-08-14 03:10:31 <Luke-Jr> BlueMatt: how do you prove they are identical across different CPUs?
113 2014-08-14 03:10:32 <BlueMatt> go cant call C libraries without effort? that sounds verrryyyy broken
114 2014-08-14 03:10:45 <BlueMatt> Luke-Jr: indeed
115 2014-08-14 03:10:49 <dhill> i am glad there is a tcp.so and udp.so that everyone uses.
116 2014-08-14 03:10:56 <Luke-Jr> â¦
117 2014-08-14 03:11:05 <Luke-Jr> dhill: networking is not consensus-critical
118 2014-08-14 03:11:07 <BlueMatt> Luke-Jr: you're just making my point
119 2014-08-14 03:11:07 <gwillen> dhill: your lack of understanding is disturbing to me
120 2014-08-14 03:11:10 <Luke-Jr> and there kinda are
121 2014-08-14 03:11:40 <dhill> haha
122 2014-08-14 03:11:42 <jrick> BlueMatt: I would be all for doing static analysis between both
123 2014-08-14 03:11:44 <gwillen> dhill: perhaps what is meant here by "consensus-critical" is not clear
124 2014-08-14 03:11:48 <jrick> but not switching to the .so
125 2014-08-14 03:11:52 <Luke-Jr> BlueMatt: my point is that we cannot do such comparisons, even with one implemnetation
126 2014-08-14 03:12:12 <BlueMatt> Luke-Jr: yes, and thus we need to do the best we possibly can
127 2014-08-14 03:12:30 <gwillen> dhill: it's specifically referring to places where any variation whatsoever, whether more liberal or more conservative, in what is accepted, will result in a permanent fork of the blockchain between implementations
128 2014-08-14 03:12:40 <dhill> yes
129 2014-08-14 03:12:42 <BlueMatt> jrick: please do, I'll be very impressed if someone does that (it means compiling down to llvm and then analysing, not sure if go can compile to llvm yet, but it should)
130 2014-08-14 03:12:49 <gwillen> that's a bit harsher of a result than a disagreement between implementations of TCP, isn't it?
131 2014-08-14 03:13:10 <jrick> BlueMatt: llgo is in the works, there actually may be enough of it finished to do just that, but it isn't ready to produce binaries yet
132 2014-08-14 03:14:03 <BlueMatt> jrick: meh, as long as you can compile everything from one entry point that is like HereIsABlockDoSomethingUseful(byte array) and returns a serialized chain/utxo set or something, you should be able to test it
133 2014-08-14 03:15:00 <jrick> that's a lot more than just comparing the script engines
134 2014-08-14 03:15:10 <jrick> yes I know both are consensus critical
135 2014-08-14 03:15:46 <BlueMatt> yea, and reimplementations have had consensus bugs in both in very subtle, incredibly hard-to-test ways
136 2014-08-14 03:15:57 <BlueMatt> script is only a first step
137 2014-08-14 03:16:44 <BlueMatt> but, really, I'd settle for a verification of script verification and sciprt sigopcount and such as being pretty decent evidence
138 2014-08-14 03:16:52 <BlueMatt> though I'd never risk my mining reward on it
139 2014-08-14 03:17:55 <Luke-Jr> ACTION wishes the mining reward were secondary to consensus of the network :x
140 2014-08-14 03:18:31 <davec> I personally don't think that just because something is an exceedingly hard task (and it is super difficult to get right), that means we shouldn't bother
141 2014-08-14 03:19:14 <davec> BlueMatt: and I didn't intend to come off poorly - I'm aware and appreciate all of the work you've done in the area of alt implementations
142 2014-08-14 03:19:28 <Luke-Jr> davec: I think it would be a good idea to have btcd (perhaps optionally) use libbitcoinscript to double-check its results
143 2014-08-14 03:20:11 <davec> I wouldn't have any issue with that as an optional build tag
144 2014-08-14 03:20:12 <BlueMatt> davec: oh, I agree
145 2014-08-14 03:20:21 <BlueMatt> I dont think we disagree here, my only issue is the marketing involved
146 2014-08-14 03:20:48 <Luke-Jr> BlueMatt: well, nothing Bitcoin should really be marketted as safe today â¦
147 2014-08-14 03:20:54 <BlueMatt> not necessarily your own, but the fact is, today, some people are looking at using btcd to accept large sums of money or mine (without luke's belt-and-suspenders-I-cant-trust-this-implementation approach)
148 2014-08-14 03:20:57 <BlueMatt> and that is simply not ok
149 2014-08-14 03:21:56 <Luke-Jr> BlueMatt: btw, did you miss the guy who was just watching other pools' stratum for new prevblock hashes, and mining empty blocks against those? :P
150 2014-08-14 03:22:02 <gwillen> well, the main risk of mining with a buggy implementation is getting yourself forked off the network by generating a bad block, yes?
151 2014-08-14 03:22:11 <BlueMatt> Luke-Jr: heh, shit
152 2014-08-14 03:22:19 <gwillen> accepting large sums of money, on the other hand... (I guess mining rewards are somewhat large sums.)
153 2014-08-14 03:23:07 <Luke-Jr> BlueMatt: he got mad when bitcoind rejected his block because it was on the other side of a fork ;)
154 2014-08-14 03:23:31 <phantomcircuit> BlueMatt, sure it is... as long as it's not other peoples money
155 2014-08-14 03:23:45 <BlueMatt> phantomcircuit: ok, I stand corrected :)
156 2014-08-14 03:24:27 <gmaxwell> gwillen: the risk with accepting is someone mines something which forks off your implementation, and then rents hashpower to mine on that fork enough to make you consider the faited coins confirmed. It's not a cheap attack.
157 2014-08-14 03:24:31 <BlueMatt> davec: specifically, the reason I did the libbitcoinscript.so was for things like btcd and bitcoinj...if they cant/dont have the time to invest in a full-on mathematical proof, then they really need to be using bitcoin core for this
158 2014-08-14 03:24:37 <gwillen> well, the other risk is that a buggy implementation gets significant marketshare before the bugs are discovered
159 2014-08-14 03:24:45 <gwillen> in which case you get a large network fork
160 2014-08-14 03:25:11 <BlueMatt> ^ this
161 2014-08-14 03:25:12 <Luke-Jr> gwillen: in that case, it's no longer clear which implementation "wins"
162 2014-08-14 03:25:25 <gwillen> gmaxwell: if you get forked off it's not a lot of hashpower to screw you, if they're willing to wait for a retarget
163 2014-08-14 03:25:35 <gwillen> although that gives you a week at least to notice you've been forked
164 2014-08-14 03:25:55 <BlueMatt> Luke-Jr: yes, and this is a big problem
165 2014-08-14 03:25:56 <gwillen> Luke-Jr: well, it's clear but it depends on which implementation has the majority, and which one is stricter in what it accepts
166 2014-08-14 03:26:00 <BlueMatt> how do we recover in such a case?
167 2014-08-14 03:26:01 <gmaxwell> gwillen: there is no 'wait for a retarget'
168 2014-08-14 03:26:05 <gwillen> there are two outcomes where the network gets flaky but stays together
169 2014-08-14 03:26:13 <gwillen> and two where the minority is permanently forked off
170 2014-08-14 03:26:16 <gwillen> gmaxwell: oh, point
171 2014-08-14 03:26:23 <gwillen> gmaxwell: that does make it tougher
172 2014-08-14 03:26:25 <gmaxwell> gwillen: but e.g. if you'll transact at 6 blocks (or 2...) they can just take their merry time at least until you notice something is wrong.
173 2014-08-14 03:26:30 <gwillen> ACTION nods
174 2014-08-14 03:26:31 <Luke-Jr> BlueMatt: softfork both probably
175 2014-08-14 03:26:54 <BlueMatt> Luke-Jr: that is still a terrible solution
176 2014-08-14 03:27:00 <gmaxwell> gwillen: If there is non-trivial hashpower that will also follow the fork the challenge is somewhat easier, and the cost lower.
177 2014-08-14 03:27:06 <gwillen> ACTION nods
178 2014-08-14 03:27:07 <Luke-Jr> BlueMatt: yes, or we can hardfork both
179 2014-08-14 03:27:08 <Luke-Jr> :P
180 2014-08-14 03:27:09 <gmaxwell> but in that case it's also likely to get noticed.
181 2014-08-14 03:27:35 <gwillen> I wish there were a 'bitcoin observatory' that were watching for such things actively
182 2014-08-14 03:27:51 <gwillen> like there are several of for BGP
183 2014-08-14 03:29:13 <gmaxwell> Well I have a pager that pages me on reorgs or forks of depth 3 that my node sees. But unfortunately in an attack there is no promise that anyone but the victim sees the weirdness.
184 2014-08-14 03:29:46 <Luke-Jr> gmaxwell: does it page you on invalid blocks with good PoW?
185 2014-08-14 03:29:51 <gwillen> well, a real 'observatory' would ideally get live feeds from all the major pools/merchants/exchanges
186 2014-08-14 03:29:57 <gwillen> in exchange for giving them alerts, presumably
187 2014-08-14 03:30:09 <gwillen> so it would see anything that anybody important sees
188 2014-08-14 03:30:16 <Luke-Jr> gwillen: just wait until pools are no longer relevant :P
189 2014-08-14 03:30:22 <davec> BlueMeatt: I think having an optional build tag that also causes btcd to also verify against the .so is a fine idea for detecting and reporting issues
190 2014-08-14 03:30:22 <Luke-Jr> (in this respect)
191 2014-08-14 03:30:25 <gmaxwell> Luke-Jr: it's based on the existing invalid fork detected code in bitcoind, which is a bit buggy.
192 2014-08-14 03:30:25 <gwillen> Luke-Jr: oh?
193 2014-08-14 03:30:36 <jrick> gmaxwell: run nodes for each implemetnation (bitcoind, btcd, whatever else) and check forks that way?
194 2014-08-14 03:31:07 <jrick> oh you said attack, maybe that's not what you were talking about
195 2014-08-14 03:31:24 <BlueMatt> davec: yea, and without that there should be a "WARNING: Could not load libbitcoinscript.so, DO NOT USE for consensus-critical applications"
196 2014-08-14 03:31:33 <BlueMatt> davec: I plan on trying to convince mike to add that for bitcoinj
197 2014-08-14 03:31:35 <gmaxwell> jrick: yes, but that has a N fold scalability loss (or worse, assuming some are more efficient than tohers)â it's something that I could do, it's not something that would be a good recommendation for merchants. .. would just create more pressure to use coinbase or other centeralized payment processors. :(
198 2014-08-14 03:31:45 <BlueMatt> (which may very well be optionally using the java-based script execution for now)
199 2014-08-14 03:32:00 <gmaxwell> But I like that above proposal for multiple script engines, ... that would have considerably less overhead.
200 2014-08-14 03:32:40 <jgarzik> build val lib into Moxie ;p
201 2014-08-14 03:33:20 <gmaxwell> A while back I'd pondered that the state-abstracted validation engine should really be bytecoded, and then implementations would just have to make their bytecoin engines agree, which might actually be possible to formally prove that they do. But thats a lot of worth to design.
202 2014-08-14 03:33:23 <BlueMatt> gmaxwell: which?
203 2014-08-14 03:33:30 <gmaxwell> jgarzik: jinx
204 2014-08-14 03:33:53 <jgarzik> :)
205 2014-08-14 03:34:28 <gmaxwell> BlueMatt: I meant the discussion above about being able to link multiple, isolated, validation engines. I dunno if you saw my observation in wizards a week weeks back tat was relevant to this:
206 2014-08-14 03:34:48 <wumpus> BlueMatt: I really like your .so idea, one could load multiple script engine .so's using dlopen and compare the outputs, even those of multiple bitcoind versions
207 2014-08-14 03:35:05 <BlueMatt> wumpus: first it needs trimmed down a lot....
208 2014-08-14 03:35:14 <BlueMatt> but, yea
209 2014-08-14 03:35:14 <wumpus> BlueMatt: why would it? just get a big machine :p
210 2014-08-14 03:35:36 <BlueMatt> heh, ok
211 2014-08-14 03:35:53 <BlueMatt> ACTION does plan on upgrading to haswell-ep and throwing in some insane amount of memory
212 2014-08-14 03:36:14 <wumpus> I don't mean using it for everything, but for some specific purposes such as testing tools, it's nice to be able to
213 2014-08-14 03:36:15 <gmaxwell> 17:46 < gmaxwell> sipa: a potentially interesting observation: per node obfscuation of data stored in databases might serve the purpose of better isolating the database behavior from being consensus normative.
214 2014-08-14 03:36:19 <gmaxwell> 17:47 < gmaxwell> sipa: e.g. if the database had some crazy bug that made it lose some records, if the records have per node scrambling then its unlikely that consensus splits would fall along version lines.
215 2014-08-14 03:36:23 <gmaxwell> 17:47 < sipa> good point yes
216 2014-08-14 03:36:25 <gmaxwell> 17:50 < gmaxwell> (oblivious ram is the insane logical conclusion of that argument⦠but perhaps even the simple version is useful)
217 2014-08-14 03:37:01 <gmaxwell> wumpus: just get a big machine is a centralization pressure.
218 2014-08-14 03:37:22 <wumpus> gmaxwell: hardware is cheap, developers aren't
219 2014-08-14 03:38:11 <gmaxwell> development costs are O(1) with users, hardware is O(N). Developers are cheap, hardware isn't. It's just we don't have the right tools to get people for paying for technology instead of hardware. :)
220 2014-08-14 03:38:20 <wumpus> gmaxwell: needing a lot of implementation work is also a centralization pressure
221 2014-08-14 03:39:05 <wumpus> anyhow I don't feel like arguing, I think bluematt did a great job by actually doing this and it can only improve from here
222 2014-08-14 03:39:14 <gmaxwell> Things were somewhat hopeless on that front since day one. :)
223 2014-08-14 03:44:09 <davec> completely agree. I know that multiple consensus critical effort and proving equality is nearly impossible today with the current state, but if we don't work toward making it a reality, it won't ever happen. I'd rather face it head on than waiting for disaster to strike
224 2014-08-14 03:46:07 <gmaxwell> It would be nice to see more testing infrastructure. The fact that matt created the block tester and found a lot of previously unknown corner cases in bitcoind (and created tests for them) increased my confidence of his work a lot, though it still had forking bugs found on several occasions. Unfortunately proving two programs in different languages do the same thing in any formal sense is basically beyond what the state of the science is ...
225 2014-08-14 03:46:13 <gmaxwell> ... right now.
226 2014-08-14 03:47:13 <jrick> I thought the llvm bytecode idea was great
227 2014-08-14 03:47:37 <jrick> but then again, there could also be a gcc and llvm compiler difference that creates a forking situation
228 2014-08-14 03:47:47 <jrick> if bitcoind hit undefined state
229 2014-08-14 03:50:11 <wumpus> well, compile it using llvm, compile it using gcc, load both :)
230 2014-08-14 03:50:33 <jrick> I assume that static analysis has been done on bitcoind to ensure that unconfirmed state can't be hit?
231 2014-08-14 03:50:45 <jrick> because basically at that point the compiler can insert code to eat puppies
232 2014-08-14 03:50:47 <wumpus> llvm bytecode is not portable between CPU architectures, so it doesn't bring that much compared to binary blobs, you'd indeed need to compile to something like moxie or another VM instruction set that is completely specified
233 2014-08-14 03:51:11 <wumpus> jrick: yes, there have been various static analysis done on bitcoind
234 2014-08-14 03:51:17 <jrick> nice
235 2014-08-14 03:51:33 <jrick> erm, s/unconfirmed/undefined/
236 2014-08-14 03:52:11 <jrick> too used to writing unconfirmed for wallet stuff..
237 2014-08-14 03:52:31 <wumpus> see for example https://github.com/bitcoin/bitcoin/issues/4601
238 2014-08-14 03:53:31 <BlueMatt> did gmaxwell leave?
239 2014-08-14 03:53:36 <BlueMatt> oh
240 2014-08-14 03:53:44 <BlueMatt> e of the science is ...
241 2014-08-14 03:53:44 <BlueMatt> <gmaxwell> [03:46:07] It would be nice to see more testing infrastructure. The fact that matt created the block tester and found a lot of previously unknown corner cases in bitcoind (and created tests for them) increased my confidence of his work a lot, though it still had forking bugs found on several occasions. Unfortunately proving two programs in different languages do the same thing in any formal sense is basically beyond what the stat
242 2014-08-14 03:53:45 <BlueMatt> no its not
243 2014-08-14 03:53:49 <wumpus> also many people ran cppcheck and clang analysis on it, not that static analysis is ever exhaustive
244 2014-08-14 03:53:53 <gmaxwell> freenode bounced, I haven't seen anything since the last thing I said.
245 2014-08-14 03:54:01 <BlueMatt> there exist tools today to analyze program behaviour in this way
246 2014-08-14 03:54:08 <BlueMatt> esp being able to compile everything to llvm
247 2014-08-14 03:54:18 <BlueMatt> which is becoming more and more possible these days
248 2014-08-14 03:54:55 <jrick> gmaxwell: http://sprunge.us/UXGP
249 2014-08-14 03:54:56 <jrick> http://sprunge.us/UXGP
250 2014-08-14 03:55:02 <jrick> oops double paste
251 2014-08-14 03:55:03 <gmaxwell> BlueMatt: the closest thing I've seen is a proof that x86 code met a seperate formal specification. Or compcert which has formal proofs that the output implements the input.
252 2014-08-14 03:55:31 <wumpus> how is being able to compile everything to llvm a panacea?
253 2014-08-14 03:56:26 <gmaxwell> But I've never seen anyone do work on "<program a> and <program b> implement the same thing" in the limit it's not actually possible. (though it's probably possible for many real programs)
254 2014-08-14 03:56:46 <gmaxwell> (esp with a and b being written in different languages)
255 2014-08-14 03:56:51 <BlueMatt> gmaxwell: hmmm, I thought I came across something like that based on llvm bytecode
256 2014-08-14 03:57:04 <BlueMatt> (since everything can be compiled to llvm these days)
257 2014-08-14 03:57:08 <gmaxwell> maybe! I'd like to see that paper.
258 2014-08-14 03:57:26 <jrick> highlight me too if you find it later :)
259 2014-08-14 03:57:56 <wumpus> I mean, everything can be compiled to x86 too, that doesn't help comparison either
260 2014-08-14 03:57:59 <BlueMatt> gmaxwell: hmmmm, maybe I was wrong, but I had a discussion with someone a year or so ago and they mentioned this
261 2014-08-14 03:58:17 <BlueMatt> wumpus: x86 is a crazy insane thing, llvm is at least somewhat, remotely, reasonable
262 2014-08-14 03:58:26 <BlueMatt> gmaxwell: let me see
263 2014-08-14 03:58:48 <Arnavion> There was something on HN recently that decompiled x86 to LLVM bytecode
264 2014-08-14 03:58:48 <wumpus> BlueMatt: sure, llvm is lots easier to understand for humans, but I'm not sure it changes the theoretical problem
265 2014-08-14 03:59:05 <BlueMatt> wumpus: it doesnt(ish), but it makes it simpler to write code for it
266 2014-08-14 03:59:10 <BlueMatt> Arnavion: heh, nice
267 2014-08-14 03:59:13 <gmaxwell> So one of the seL4 papers implement a proof that the x86 output of gcc 4.5.something at -O1 also met the sel4 formal specification.
268 2014-08-14 03:59:41 <Arnavion> It was this: http://blog.trailofbits.com/2014/08/07/mcsema-is-officially-open-source/
269 2014-08-14 04:00:04 <jrick> BlueMatt: oh btw, thought I'd mention it later than never, but it's just google's go implementation that requires stack swapping when calling into c libraries. gccgo doesn't need it
270 2014-08-14 04:00:14 <jrick> so, implemntation detail, not language
271 2014-08-14 04:00:40 <BlueMatt> jrick: hmm, yea, ok
272 2014-08-14 04:02:33 <gmaxwell> BlueMatt: the stack swapping issue is IIRC why rust dropped split stacks in the rust implementation.
273 2014-08-14 04:02:53 <BlueMatt> thats an annoying issue
274 2014-08-14 04:03:58 <gmaxwell> well it's a platform C ABI issue, in theory the C abi for a platform could be setup so that stacks worked the split stacks way everywhere. I had though there was some inititive to support that in GCC but I dunno where its gone.
275 2014-08-14 04:04:00 <wumpus> Arnavion: I do know about the S2E framework, one of their tools disassembles instructions using qemu to an intermediate format which is then converted to LLVM bytecode for symbolic execution, but it's nowhere near perfect and I'm not sure how it could be used to prove that two programs do the same
276 2014-08-14 04:05:06 <jrick> gmaxwell: hmm am I wrong about gccgo then? I thought it didn't need the swap
277 2014-08-14 04:05:46 <gmaxwell> The attraction of the moxie stuff mentioned previously is that the simulator is so simple its relatively easy to be confident that two implementations are functionally equal. (and in languages where the tools exist, e.g. ada, C, ocaml, you could prove a moxie simulator implementation faithfully implemented some formal specification)
278 2014-08-14 04:06:07 <gmaxwell> jrick: I think it does, I think it just doesn't use splitstacks. But I have not followed go that closely.
279 2014-08-14 04:06:25 <gmaxwell> if my belief is correct then it means you get some huge stack overheads from threading.
280 2014-08-14 04:07:28 <jrick> Gccgo supports splitting goroutine stacks as the gc compiler does, but currently only on x86 (32-bit or 64-bit) and only when using the gold linker (on other processors, each goroutine will have a large stack, and a deep series of function calls may run past the end of the stack and crash the program).
281 2014-08-14 04:07:33 <jrick> well there
282 2014-08-14 04:07:37 <jrick> but this is getting a bit off topic
283 2014-08-14 04:07:45 <gmaxwell> hm!
284 2014-08-14 04:08:04 <gmaxwell> then I dunno how it could call into C code without overhead, but ::shrugs:: indeed. :)
285 2014-08-14 04:11:53 <wumpus> gmaxwell: moxie is a better choice than llvm because it's a platform independent, fully specified instruction set
286 2014-08-14 04:12:20 <wumpus> I mean, in theory then, a lot more tools exist for llvm
287 2014-08-14 04:13:46 <wumpus> but then you have to handle llvm-for-x86-32, llvm-for-x86-64, llvm-for-arm32, etc which are all slightly different flavors
288 2014-08-14 04:15:15 <wumpus> although the differences between those are in the typing and the available instructions, not how the instructions are defined, AFAIK
289 2014-08-14 04:39:57 <earlz> Is there a configure flag or some such to not compile tests? and/or why are tests compiled by default anyway
290 2014-08-14 04:40:25 <gmaxwell> because anyone who builds bitcoin should run them
291 2014-08-14 04:40:49 <gmaxwell> (otherwise miscompilation may have very unwelcome effects for you)
292 2014-08-14 04:40:49 <wumpus> hmm then again, to be useful, the moxie-compiled version would then have to be the only version that is executed; if the actual bitcoind's in the field still have their consensus code compiled to the native architecture, any result derived from the moxie version would be questionable
293 2014-08-14 04:41:26 <earlz> So you're saying we should only use reference binaries, even using an x86 emulator on odd architectures? lol
294 2014-08-14 04:41:27 <wumpus> yes, always run at least src/test/test_bitcoin, it's critical for this project
295 2014-08-14 04:41:33 <gmaxwell> wumpus: With a bytecode you could say 'this blob is the normative specification of the system' precisely analogous to specifying the ECC generator.
296 2014-08-14 04:41:49 <gmaxwell> earlz: this has nothing to do with x86.
297 2014-08-14 04:41:50 <wumpus> gmaxwell: yes, that would have to be done, then
298 2014-08-14 04:42:19 <wumpus> gmaxwell: and interpreters would have to be proven to actually implement moxie correctly, they can't get too creative with optimizations
299 2014-08-14 04:42:34 <wumpus> otherwise the point is defeated, too
300 2014-08-14 04:43:18 <BlueMatt> gmaxwell: googleing around for formal equivalence checking llvm does turn up some people working in this space, not sure if anything is plug-and-play easy, but it looks possible
301 2014-08-14 04:43:23 <gmaxwell> wumpus: yes, but since the interpreter is straighforward and simple there is a clear path to doing that, and a lot of work has been done wrt formal verification of simple machines.
302 2014-08-14 04:44:35 <earlz> would it be a decent idea to run the tests as part of the build system?
303 2014-08-14 04:44:38 <wumpus> gmaxwell: sure - I can see how a simple interpreter would be more feasible to verify formally than the general case. But maybepeople start throwing JIT's and such into the mix...
304 2014-08-14 04:44:42 <wumpus> earlz: no
305 2014-08-14 04:44:48 <wumpus> earlz: though you can do 'make check'!
306 2014-08-14 04:44:56 <earlz> was not aware of that
307 2014-08-14 04:44:58 <gmaxwell> earlz: because of cross compiling you can't always run what you built.
308 2014-08-14 04:45:11 <earlz> ah, good point
309 2014-08-14 04:45:18 <gmaxwell> perhaps the build process should tell you "now run make check" at the end. :)
310 2014-08-14 04:46:35 <wumpus> heh, yes that'd work
311 2014-08-14 04:50:32 <wumpus> so yes, adding the tests as part of the build system is a decent idea, but not in the trivial way
312 2014-08-14 04:50:39 <BlueMatt> gmaxwell jrick: http://spinroot.com/spin/Workshops/ws12/spin2012_submission_10.pdf references it, and a few people cite http://www.pgbovine.net/PhD-memoir/ucklee-cav-2011.pdf
313 2014-08-14 04:50:50 <BlueMatt> both of which appear to be not released, but one could email them/look further
314 2014-08-14 04:51:08 <BlueMatt> also, that was after cursory google'ing, before doing anything one might google further
315 2014-08-14 04:53:49 <gmaxwell> hm, now you're reminding me that I read a paper on some super optimizer that used that kind of formal verification that candidate optimizatons were correct.
316 2014-08-14 04:54:08 <wumpus> BlueMatt: emailing them may be a good idea in any case, if anyone is doing research in this, it would maybe make them aware that bitcoin is a good guinea pig for this :-)
317 2014-08-14 04:54:50 <BlueMatt> gmaxwell: there seem to be a few papers analysing whether optimizations were correct
318 2014-08-14 05:01:32 <earlz> bignum.h was moved to only for testing?
319 2014-08-14 05:01:53 <wumpus> earlz: yes, it's only used to compare to now
320 2014-08-14 05:02:17 <wumpus> the script interpreter has its own type now, scriptnum
321 2014-08-14 05:03:28 <cfields> wumpus: everything is ready to go, i'm just having trouble getting travis to light it up for us
322 2014-08-14 05:03:31 <earlz> ugh always making it harder to mine a new genesis block. I prefer testing on a private super-low difficulty network
323 2014-08-14 05:03:35 <jrick> BlueMatt: thanks, reading about FAuST now, looks neat
324 2014-08-14 05:03:37 <cfields> i'm very annoyed
325 2014-08-14 05:04:22 <wumpus> cfields: ohhh still administrative issues, that's a pity
326 2014-08-14 05:04:37 <BlueMatt> i thought travis had no verification step?
327 2014-08-14 05:04:54 <cfields> so, i'm beginning to consider other possibilities. I figured I'd give em to the end of the week, then start working on something else
328 2014-08-14 05:05:44 <cfields> BlueMatt: i'm working with them to get us setup on some dedicated workers. we're not their typical case
329 2014-08-14 05:05:57 <BlueMatt> too much cpu for us?
330 2014-08-14 05:06:17 <cfields> BlueMatt: nah, we need caching enabled
331 2014-08-14 05:06:39 <cfields> that takes our build times from ~50min to ~2
332 2014-08-14 05:07:28 <BlueMatt> caching of what? why not just leave the dependencies as a zip somewhere outside travis and make the build script call out and get it at start
333 2014-08-14 05:07:34 <BlueMatt> waste their bw until they add that feature
334 2014-08-14 05:08:00 <cfields> well, that's pretty close to the backup plan
335 2014-08-14 05:08:05 <cfields> heh, they actually recommend that
336 2014-08-14 05:08:19 <BlueMatt> well, if they want us to eat their bw, we can accommodate
337 2014-08-14 05:09:04 <wumpus> faust? the signal processing language?
338 2014-08-14 05:09:08 <jrick> BlueMatt: so, FAuST looks like it just generates a graph of all possible states, if I'm reading this paper correctly, comparing node reachability between both test programs
339 2014-08-14 05:09:24 <BlueMatt> jrick: I skimmed, it may not be ideal
340 2014-08-14 05:09:40 <cfields> BlueMatt: as for the libtool lib, i suppose i really don't have a good argument. I'm just always really nervous about exposing half-baked apis.
341 2014-08-14 05:09:42 <BlueMatt> note, ofc, that halting problem rears its head here, so its not technically possible, its only possible to approximate
342 2014-08-14 05:09:49 <BlueMatt> and with that you can get good enough (tm)
343 2014-08-14 05:09:49 <jrick> right
344 2014-08-14 05:09:53 <cfields> BlueMatt: i understand that it's not really exposed though.
345 2014-08-14 05:09:57 <wumpus> BlueMatt: we need a pretty specific setup
346 2014-08-14 05:10:17 <BlueMatt> cfields: half-baked? how is it half-baked
347 2014-08-14 05:10:21 <BlueMatt> its minimal
348 2014-08-14 05:10:25 <BlueMatt> but its not half-baked
349 2014-08-14 05:11:20 <cfields> BlueMatt: well, one of my concerns, there's no means (that i see?) of accounting for script behavior changes
350 2014-08-14 05:11:34 <cfields> no in-api versioning, that is
351 2014-08-14 05:12:04 <BlueMatt> script behavior changes???
352 2014-08-14 05:12:09 <BlueMatt> we dont do those
353 2014-08-14 05:12:24 <gmaxwell> BlueMatt: like you add p2sh support, how do you make sure someone isn't linking to an old version of the code?
354 2014-08-14 05:12:34 <BlueMatt> gmaxwell: so versioning?
355 2014-08-14 05:13:02 <gmaxwell> and some genius installs this thing as a system lib, and users are running foowallet on debian shipping a copy from 1974?
356 2014-08-14 05:13:04 <cfields> gmaxwell: yes, that was the example i had in mind :)
357 2014-08-14 05:13:19 <gmaxwell> probably as simple as adding a call that returns the version and then just being able to test at runtime.
358 2014-08-14 05:13:29 <Luke-Jr> gmaxwell: libtool does that..
359 2014-08-14 05:13:34 <cfields> no.
360 2014-08-14 05:14:00 <BlueMatt> gmaxwell: isnt that the point of libtool versioning?
361 2014-08-14 05:14:07 <sipa> gmaxwell, BlueMatt, cfields: versioning is done through flags of optional script features; make the library croak if you pass an unknown flag
362 2014-08-14 05:14:37 <Luke-Jr> sipa: that might be better, since libtool versioning can only track one thing
363 2014-08-14 05:14:46 <sipa> because any functional change will always only be for a subsey of the chain, the default should never change anyway
364 2014-08-14 05:15:28 <BlueMatt> yea, thats my point, so you can do libtool versioning to track if you're running a copy that supports the latest soft-forks?
365 2014-08-14 05:15:40 <Luke-Jr> BlueMatt: but what about stable branches? ;)
366 2014-08-14 05:15:41 <wump> libtool versioning may still make sense for other version changes, such as new methods
367 2014-08-14 05:15:55 <BlueMatt> Luke-Jr: applications should refuse to build on debian.
368 2014-08-14 05:15:59 <Luke-Jr> â¦
369 2014-08-14 05:16:03 <sipa> lol
370 2014-08-14 05:16:13 <Luke-Jr> BlueMatt: so everyone is required to run Windows? ;)
371 2014-08-14 05:16:25 <BlueMatt> Luke-Jr: no, everyone is required to run a vm
372 2014-08-14 05:16:53 <sipa> does a sony rootkit suffice?
373 2014-08-14 05:16:58 <BlueMatt> yes
374 2014-08-14 05:17:26 <gmaxwell> sipa: good call wrt flags.
375 2014-08-14 05:17:33 <Luke-Jr> BlueMatt: libbitcoinscript should just be in Debian volatile
376 2014-08-14 05:17:51 <Luke-Jr> BlueMatt: same category as antivirus definitions and spam detection stuff
377 2014-08-14 05:17:51 <sipa> i fon't understand the problem
378 2014-08-14 05:18:20 <cfields> BlueMatt: regardless of how it's solved, it needs to be solved before exposing the lib. you only get one chance at an api :)
379 2014-08-14 05:18:37 <Luke-Jr> cfields: pfft libtool versioning works just fine for API breakage
380 2014-08-14 05:18:38 <wumpus> unless you use API versioning
381 2014-08-14 05:18:42 <wumpus> right
382 2014-08-14 05:19:01 <wumpus> libbitcoin-script.so.1 libbitcoin-script.so.2 can have breaking API changes
383 2014-08-14 05:19:53 <wumpus> API version should be decoupled from 'bitcoin network' version though, so the flags system makes a lot of sense for that
384 2014-08-14 05:20:02 <Luke-Jr> agreed
385 2014-08-14 05:20:03 <gmaxwell> Anyone ever run a number on how often a pubkey is reused within a transaction?
386 2014-08-14 05:20:08 <cfields> mm, i really don't like libtool versioning for this
387 2014-08-14 05:20:18 <cfields> can't put my finger on why though. i'll think on it tonight
388 2014-08-14 05:20:38 <Luke-Jr> cfields: this is precisely what libtool versioning exists for (ABI and API breaks)
389 2014-08-14 05:20:46 <gmaxwell> speaking of apis.
390 2014-08-14 05:21:06 <wumpus> you *need* some kind of API versioning, you might think you're setting it in stone right now, but things always change
391 2014-08-14 05:21:30 <Luke-Jr> I actually named libblkmaker with -0.1 in the base name just in case
392 2014-08-14 05:21:32 <BlueMatt> I dont want to return a SUCCESS/ERROR/INVALID_CALL tuple
393 2014-08-14 05:21:37 <Luke-Jr> so now libblkmaker 0.4 still says libblkmaker-0.1 :P
394 2014-08-14 05:21:45 <BlueMatt> wumpus: yes, hence libtool versioning.......
395 2014-08-14 05:22:21 <BlueMatt> s/a...tuple/from a ...enum/
396 2014-08-14 05:22:29 <sipa> why not?
397 2014-08-14 05:22:39 <BlueMatt> because...simple api? I dunno
398 2014-08-14 05:22:46 <BlueMatt> feels better to me to just do a libtool version
399 2014-08-14 05:22:58 <sipa> i think these are independent
400 2014-08-14 05:23:01 <gmaxwell> sipa: so libsecp256k1 does not completely uniformly follow the C norm of assignment like argument order. e.g. foo(*output, const*input). secp256k1_ecdsa_sign is wrong, secp256k1_ecdsa_recover_compact is wrong, and secp256k1_ecdsa_privkey_export
401 2014-08-14 05:23:11 <BlueMatt> sipa: you suggested failing on invalid flags?
402 2014-08-14 05:23:18 <sipa> BlueMatt: yes
403 2014-08-14 05:23:21 <BlueMatt> so you'd need to return a third value, INVALID_CALL
404 2014-08-14 05:23:27 <BlueMatt> you dont really just want to return false for all of that
405 2014-08-14 05:23:27 <sipa> BlueMatt: yes
406 2014-08-14 05:23:33 <Luke-Jr> BlueMatt: bcscript_compatiblity_check(BCSCRIPT_NEWEST_FEATURE)
407 2014-08-14 05:23:34 <sipa> indeed
408 2014-08-14 05:23:52 <Luke-Jr> run it at startup, fail there
409 2014-08-14 05:24:04 <sipa> BlueMatt: i'd use api changes for features of the library, not of the scripting language
410 2014-08-14 05:24:20 <sipa> BlueMatt: like if we want one fay support for batch verification
411 2014-08-14 05:24:28 <BlueMatt> sipa: but clearly if someone is linking libbitcoinscript.so.$BEFORE_P2SH, they're doing it wrong and should face build errors
412 2014-08-14 05:24:30 <cfields> BlueMatt: at least include some sort of version define so that client apps/libs can know what they're linking against, then?
413 2014-08-14 05:24:33 <wumpus> sipa: exactly
414 2014-08-14 05:24:56 <Luke-Jr> BlueMatt: then use macros
415 2014-08-14 05:24:58 <sipa> or add a function "return supported flags"
416 2014-08-14 05:24:59 <BlueMatt> cfields: yea, that goes in when script/bitcoinscript.h moves to a proper include/bitcoinscript.h
417 2014-08-14 05:25:01 <sipa> fixed
418 2014-08-14 05:25:16 <sipa> you need to chrck whether the p2sh flag is supporyed befote using it
419 2014-08-14 05:25:31 <BlueMatt> which should happen before release, but before merge? meh
420 2014-08-14 05:25:31 <gmaxwell> sipa: also, any thoughts on moving the static state out of a global and making a secp256k1 context that one shelps around as a first argument to every function?
421 2014-08-14 05:25:41 <sipa> gmaxwell: why?
422 2014-08-14 05:25:55 <cfields> gmaxwell: i already tried that :)
423 2014-08-14 05:25:58 <cfields> he didn't go for it :)
424 2014-08-14 05:26:22 <sipa> i understand there is a cleansiness argument
425 2014-08-14 05:26:42 <sipa> but there is 0 point in ever having multiple copies, and it's not a tiny data structure
426 2014-08-14 05:27:12 <Luke-Jr> #define BCSCRIPT_COMPATIBILITY() do { extern __bcscript_<newestfeature>; __bcscript_<newestfeature> = 1; } while(0)
427 2014-08-14 05:27:20 <Luke-Jr> then change <newestfeature> when necessary
428 2014-08-14 05:27:30 <wumpus> a context structure would make sense if you ever intend to ever add mutable state
429 2014-08-14 05:27:36 <gmaxwell> sipa: It's common, and makes it much harder to fail to initilize. If in the future you wanted to add a cache of WNAF Q precomputation, it would be a more modest API change (as the cache would be mutable state).
430 2014-08-14 05:27:51 <BlueMatt> Luke-Jr: sure, this is what I was suggesting
431 2014-08-14 05:27:56 <BlueMatt> well, +/-
432 2014-08-14 05:28:13 <gmaxwell> E.g. if you were doing an application where public keys were reused a lot and could arrange to verify signatures in pubkey sorted order.
433 2014-08-14 05:28:14 <Luke-Jr> yeah, I guess the __ prefix is a no-no
434 2014-08-14 05:28:33 <sipa> gmaxwell: good point about that
435 2014-08-14 05:28:47 <Luke-Jr> gmaxwell: still no reason to copy it ;)
436 2014-08-14 05:28:53 <BlueMatt> well, ok, whatever, y'all can decide...I prefer the simpler case of putting a version in the .h file and moving the .h to include/ and installing it (which would need to happen before its a "released" library anyway)
437 2014-08-14 05:28:54 <Luke-Jr> gmaxwell: could have an empty context
438 2014-08-14 05:28:56 <gmaxwell> (caching would have ~0 overhead, since you'd just keep one entry, and the check would only need to compare the pubkey and early terminate if they're not the same)
439 2014-08-14 05:29:04 <BlueMatt> but it doesnt need to be in the .so
440 2014-08-14 05:29:26 <sipa> BlueMatt: putting it in the .h doesn't work; it would causr unexpected failure when running with the wtong .so
441 2014-08-14 05:29:27 <wumpus> global mutable contexts are annoying (see opengl), if it's just a lump of read-only data I don't see the point
442 2014-08-14 05:30:03 <Luke-Jr> wumpus: heh, see DirectFB? :p
443 2014-08-14 05:30:07 <sipa> as i said: keep versioning of the library distinct from versioning script features
444 2014-08-14 05:30:09 <BlueMatt> sipa: no, when you build against the .h it decides which version to link against at build-time, allowing you to link against whatever you want and then it will only run with that version
445 2014-08-14 05:30:17 <wumpus> Luke-Jr: I've never actually used directfb
446 2014-08-14 05:30:22 <gmaxwell> sipa: in that case you probably want a _init() that gives you an immutable context, and then a _thread_init(immutable_context*) the gives you the mutable one. To be excessively snazzy the functoins could take either (with sutiable typedef adventures) and check which one they got. (first word of the context should be a canary value that tells you it's initilized and which one it is)
447 2014-08-14 05:30:32 <Luke-Jr> wumpus: *everything* is a somestruct->function(somestruct, â¦)
448 2014-08-14 05:31:12 <sipa> BlueMatt: how do you make it only run with that version?
449 2014-08-14 05:31:15 <cfields> Luke-Jr: raii hides that away quite easily
450 2014-08-14 05:31:24 <wumpus> Luke-Jr: so c-based polymorphism like glib and friends
451 2014-08-14 05:31:37 <Luke-Jr> sipa: see my #define earlier?
452 2014-08-14 05:31:38 <sipa> BlueMatt: that's even overly restrictive... linking eith a newer version is perfectly fine
453 2014-08-14 05:31:47 <Luke-Jr> wumpus: yeah, but typing it out all the time is annoying
454 2014-08-14 05:32:15 <gmaxwell> wumpus: in large applications (e.g. like a browser) it can be quite difficult to be sure a global-stated object is initilizated without passing around a proxy context in any case. And for secp256k1 there is at least one optimization that wants mutiable state. (verification for things that reuse pubkeys a lot can be made MUCH faster if there is a one entry cache)
455 2014-08-14 05:32:18 <Luke-Jr> sipa: if there's a newer version, you probably want to upgrade the rest ;)
456 2014-08-14 05:32:31 <cfields> BlueMatt: what about deciding at build-time? It'll just build against whichever is symlinked
457 2014-08-14 05:32:35 <sipa> Luke-Jr: probably, yes
458 2014-08-14 05:32:45 <sipa> Luke-Jr: but not certainly
459 2014-08-14 05:33:02 <Luke-Jr> sipa: so just add dummy ints instead of renaming one?
460 2014-08-14 05:33:08 <sipa> bleh
461 2014-08-14 05:33:21 <sipa> ACTION likes a runtime function that returns supported flags
462 2014-08-14 05:33:28 <cfields> sipa: yes please :)
463 2014-08-14 05:33:32 <Luke-Jr> ACTION does too
464 2014-08-14 05:33:40 <cfields> i figured that would be the popular opinion around here
465 2014-08-14 05:33:53 <sipa> and then use versioning for library api/feature changes, but not for script features
466 2014-08-14 05:34:01 <cfields> exactly
467 2014-08-14 05:34:12 <gmaxwell> I like that.
468 2014-08-14 05:34:40 <Luke-Jr> wumpus: btw, mostly unrelated to this ⦠did you ever look into porting BCCoreGUI to libbitcoin?
469 2014-08-14 05:35:03 <wumpus> gmaxwell: agreed, but there is some elegance to simple functions, instead of 'class methods', as well, especially if there doesn't really need to be state
470 2014-08-14 05:35:10 <Luke-Jr> if it has everything needed for a SPV wallet, might be a good chance to combine it with BlueMatt's libbitcoinscript to get a SPV BCCoreGUI
471 2014-08-14 05:35:26 <wumpus> Luke-Jr: not really
472 2014-08-14 05:35:37 <sipa> gor SPV you don't need libbitcoinscript...
473 2014-08-14 05:36:20 <gmaxwell> sipa: more on the high end, on NUMA machines you will likely want to have multiple copies of the global state, since memory access to remote nodes is expensive.. though perhaps thats a bit of an exotic optimization.
474 2014-08-14 05:36:20 <Luke-Jr> sipa: you do if you want to display 0conf without being totally spoofable
475 2014-08-14 05:36:50 <sipa> and i would be opposed to putting signing/utility code in it... i like very much that it it is *purely* consensus code
476 2014-08-14 05:36:55 <gmaxwell> Alternatively, all the precomputed stuff in secp256k1 could just be static data in the text segment.
477 2014-08-14 05:37:04 <Luke-Jr> sipa: yeah, I'd assume libbitcoin has that already
478 2014-08-14 05:37:28 <sipa> Luke-Jr: fair enough, though that's trivially bypassable
479 2014-08-14 05:37:57 <sipa> gmaxwell: and change libsecp256k1 from 30kB to 700 kN? :)
480 2014-08-14 05:38:05 <Luke-Jr> sipa: bypassable?
481 2014-08-14 05:38:06 <gmaxwell> sipa: SPV needs libbitcoinscript if someday fraud alerts are something spv clients do. (hey, satoshi mentioned itâ so it's okay to bring that up outside of #-wizards, right? :) )
482 2014-08-14 05:38:33 <sipa> no that is not a typo, i clearly mean kilonewton
483 2014-08-14 05:38:43 <gmaxwell> sipa: yep! :P ... yea, okay, thats a point.
484 2014-08-14 05:39:15 <wumpus> Luke-Jr: but I'm not sure what building on top of libbitcoin would add; we already have everything needed to make a SPV client too
485 2014-08-14 05:39:23 <Luke-Jr> wumpus: not in library form
486 2014-08-14 05:39:32 <cfields> wumpus: i'll try to catch you tomorrow ~8h to discuss a few c-i options. I really don't want that to stall.
487 2014-08-14 05:39:36 <cfields> nnite
488 2014-08-14 05:39:43 <sipa> Luke-Jr: first send to an invalid script, then to a normal one
489 2014-08-14 05:39:44 <wumpus> ok nite cfields
490 2014-08-14 05:39:47 <sipa> cya
491 2014-08-14 05:40:12 <sipa> gmaxwell: i like the simplicity of it
492 2014-08-14 05:40:17 <Luke-Jr> sipa: so the SPV client fetches the inputs' txs..
493 2014-08-14 05:40:32 <sipa> Luke-Jr: hiw?
494 2014-08-14 05:40:40 <sipa> *_how
495 2014-08-14 05:40:44 <wumpus> Luke-Jr: ok, true, but porting over the RPC interface as well as the GUI to a completely different library like libbitcoin could be just as much work as implementing a new wallet
496 2014-08-14 05:40:53 <sipa> agree
497 2014-08-14 05:40:59 <Luke-Jr> sipa: surely if it received its own transaction, the peer that sent it has the input's too
498 2014-08-14 05:41:20 <sipa> not arbitrarily deep
499 2014-08-14 05:41:23 <wumpus> Luke-Jr: which is pointless, as SPV wallets already exist, may as well slap a bitcoind-compatibile-ish RPC interface on top
500 2014-08-14 05:41:32 <Luke-Jr> sipa: right now, we do..
501 2014-08-14 05:41:33 <gmaxwell> hah. it's kind of annoying that CHECKSIG returns a value and there isn't, instead a CHECKSIGEQUALSVALIDATE which takes the return value as an additional input.
502 2014-08-14 05:41:45 <Luke-Jr> wumpus: all existing SPV wallets suck.
503 2014-08-14 05:41:59 <gmaxwell> makes it hard to hoist ECDSA validation checking out of the script processing.
504 2014-08-14 05:42:18 <wumpus> Luke-Jr: making them not suck is probably less work than starting from scratch
505 2014-08-14 05:42:31 <Luke-Jr> wumpus: dunno, I like Bitcoin Core GUIâ¦
506 2014-08-14 05:43:05 <gmaxwell> Since some clown can write scripts that are only valid if ECDSA fails, so you can't just assume it passes, make a note of the verification you need to perform, and consider the transaction invalid later if it fails that check.
507 2014-08-14 05:43:19 <sipa> gmaxwell: an alternative is spliyting mutable and immuyable state, leave the immutable part as is, and have an optiinal pointer tona mutable one passed to verify
508 2014-08-14 05:43:36 <sipa> sorry , phone typing
509 2014-08-14 05:43:43 <wumpus> Luke-Jr: I like it as well
510 2014-08-14 05:43:51 <BlueMatt> my issue is that you want to ensure that a softfork requires application developers to rewrite their application
511 2014-08-14 05:44:02 <sipa> BlueMatt: no it does not
512 2014-08-14 05:44:04 <Luke-Jr> reminder: at some point, it'd be nice to split that Step function :p
513 2014-08-14 05:44:10 <sipa> BlueMatt: not as long as it is nit a hard fork
514 2014-08-14 05:44:22 <gmaxwell> sipa: Thats not crazy, regardless, we should probably add a canary to the state structs... to at least make it fail safe against foolish users who could call it without initilization.
515 2014-08-14 05:44:54 <wumpus> Luke-Jr: but seeing how people are always complaining about it, I've kind of lost motivation to work on it, not that I have much time to work on GUIs these days
516 2014-08-14 05:44:59 <sipa> agree with the step function, but let's punt that?
517 2014-08-14 05:45:09 <Luke-Jr> sipa: ?
518 2014-08-14 05:45:22 <BlueMatt> sipa: the point of using a libbitcoinscript is that you're doing a full validation thing, and then you really do need to follow soft forks
519 2014-08-14 05:45:22 <sipa> until all other script changes are over
520 2014-08-14 05:45:27 <BlueMatt> otherwise your stuff is broken
521 2014-08-14 05:45:50 <sipa> BlueMatt: i agree with encouraging, not with forcing
522 2014-08-14 05:46:15 <Luke-Jr> sipa: just seemed like it would "automatically" solve the alternate verification stuff
523 2014-08-14 05:46:32 <sipa> alternate verification?
524 2014-08-14 05:46:37 <BlueMatt> sipa: well, I think you should force them to change code, even if the change they make is "fuck it, I'll fix this later"
525 2014-08-14 05:48:40 <sipa> BlueMatt: and people will just not upgrade the library to avoid the incompatibility
526 2014-08-14 05:49:07 <BlueMatt> hopefully their os ships the library so they have to
527 2014-08-14 05:49:22 <sipa> i agree with what you're saying, but it's the wrong place to enforce that imho
528 2014-08-14 05:49:24 <Luke-Jr> __attribute__((deprecated))
529 2014-08-14 05:49:25 <Luke-Jr> :p
530 2014-08-14 05:49:45 <Luke-Jr> of course, that means you have to force a recompile
531 2014-08-14 05:50:03 <gmaxwell> this has all kinda of gone a little too forward looking, how about the library exist before you lament people being slow to upgrade it. :P
532 2014-08-14 05:50:13 <Luke-Jr> bcscript_check_version(expected) if (expected < current) sleep(30);
533 2014-08-14 05:50:20 <sipa> loi
534 2014-08-14 05:50:35 <BlueMatt> Luke-Jr: naaa, sleep(60*60)
535 2014-08-14 05:50:42 <BlueMatt> make their users complain like hell about slow startup times
536 2014-08-14 05:51:16 <Luke-Jr> puts("Your software is outdated, please upgrade it. Type 'ignore' to continue (automatic after 2 minutes)");
537 2014-08-14 05:51:26 <gmaxwell> while(1){bcscript_check_version(expected); if < whine(); sleep(); dlopen_magic();} :P
538 2014-08-14 05:51:56 <wumpus> Luke-Jr: or just auto-download the update from the web :P
539 2014-08-14 05:52:07 <gmaxwell> "How do you ensure the the software is current? Exponential backoff."
540 2014-08-14 05:54:14 <wumpus> anyhow the problem with slow-to-update people already exists right now, some update scenarios may become easier if you can just drop in another scripting engine instead of having to upgrade the entire program
541 2014-08-14 05:57:44 <BlueMatt> so...force a call to bitcoinscript_init(), which returns flags which are supported, otherwise calls to bitcoinscript functions will assert() and crash your program
542 2014-08-14 05:57:47 <BlueMatt> ?
543 2014-08-14 05:58:01 <BlueMatt> then at least hopefully people wont skip the call to _init
544 2014-08-14 05:59:01 <sipa> BlueMatt: even better, pass the flags you require to exist to the init function
545 2014-08-14 05:59:08 <wumpus> good idea to have a function to retur the supported caps, not sure the call should be called init if it doesn't initialize any state
546 2014-08-14 06:00:20 <wumpus> otherwise you get the same kind of problems gmaxwell was talking about above, about initialization order in large programs
547 2014-08-14 06:00:26 <gmaxwell> well script really should initilize state anyways, since you probably want a static scratchpad for execution setup. Ideally script should just always allocate the worst case memory usage so you cannot fail at runtime.
548 2014-08-14 06:00:44 <gmaxwell> (and script needs more state than you can reasonably put on the stack)
549 2014-08-14 06:01:06 <gmaxwell> hm actually it doesn't, not current versions at least.
550 2014-08-14 06:01:27 <gmaxwell> But from a structural perspective you'd want to handle future soft-forks that don't without changing the pai.
551 2014-08-14 06:01:30 <gmaxwell> er api.
552 2014-08-14 06:01:39 <wumpus> yeah, things start to get unreasonable to put on the stack very quickly
553 2014-08-14 06:02:03 <gmaxwell> Right now script needs on the order of 260k peak state, plus the transaction itself, I believe.
554 2014-08-14 06:02:06 <wumpus> taking into account platforms with very limited stack sizes especially for threads
555 2014-08-14 06:02:54 <wumpus> so I suppose a scratchpad 'context' makes sense then, you'd have to allocate multiple to do multithreaded verification?
556 2014-08-14 06:03:05 <sipa> yup
557 2014-08-14 06:04:40 <gmaxwell> would also allow the context to carry around things like an ECDSA cache.
558 2014-08-14 06:07:07 <BlueMatt> sipa: so...bitcoinscript_init(FLAGS_I_USE), which returns flags which you really need to implement if you didnt pass them, then you can if(bitcoinscript_init(1)) log("WARNING: NEED TO UPGRADE");
559 2014-08-14 06:07:10 <BlueMatt> ?
560 2014-08-14 06:07:18 <wumpus> (different from secp256k1 where, if it had contexts, different threads would be able to share one context, and would usually also be encouraged to)
561 2014-08-14 06:07:33 <BlueMatt> gmaxwell: yea, having a large context is kinda gross, but may be needed at some point :(
562 2014-08-14 06:07:57 <BlueMatt> still, I deliberately avoided having an ECDSA cache in the script execution because that should be application-level (like the whole script cache is in bitcoin core)
563 2014-08-14 06:08:06 <BlueMatt> again, my preference is minimal, minimal, minimal
564 2014-08-14 06:09:26 <sipa> BlueMatt: yup, and then fail to work afterwards :)
565 2014-08-14 06:09:39 <sipa> sure, i'm not convinced about the ecdsa cache
566 2014-08-14 06:09:44 <gmaxwell> BlueMatt: dynamic memory allocation in the middle of script execution is hardly 'minimal'. :) It means excution could randomly fail based on perfectly valid pertubations of the system state.
567 2014-08-14 06:10:05 <gmaxwell> And yea, the cache was an example, it's hard to put it externally, because to use it safely it needs to be savvy of script internals.
568 2014-08-14 06:10:15 <sipa> but making scriot execution work without memory allocation would be ReallyNice(tm)
569 2014-08-14 06:11:08 <BlueMatt> sipa: you just said you didnt want it to fail to work?
570 2014-08-14 06:11:26 <BlueMatt> gmaxwell: do we do any dynamic memory allocation now?
571 2014-08-14 06:11:31 <gmaxwell> 0_o
572 2014-08-14 06:11:31 <sipa> tons
573 2014-08-14 06:11:33 <gmaxwell> yes.
574 2014-08-14 06:11:39 <gmaxwell> like ... every operation.
575 2014-08-14 06:11:44 <sipa> the script stack is dynamic
576 2014-08-14 06:11:53 <BlueMatt> in EvalScript
577 2014-08-14 06:11:53 <gmaxwell> (thank you C++ from hiding these details so that people aren't even aware of them!)
578 2014-08-14 06:11:53 <sipa> the items on the stack are dynamic
579 2014-08-14 06:11:55 <wumpus> it would also be a large break from how most of bitcoin is programmed, we literally use local vectors and such about everywhere
580 2014-08-14 06:12:02 <BlueMatt> i havent read it in forever
581 2014-08-14 06:12:22 <sipa> wumpus: yeah, unfortunately
582 2014-08-14 06:12:29 <gmaxwell> fortunately it appears easy to know the upper bound.
583 2014-08-14 06:12:36 <BlueMatt> ouch, yea, havent read that in forever
584 2014-08-14 06:12:37 <BlueMatt> yea
585 2014-08-14 06:12:57 <gmaxwell> (since the number of operations are limited and the size of objects is limited)
586 2014-08-14 06:12:57 <wumpus> anyhow, changing the scripting engine to be more memory friendly should be orthogonal to having a exporting library, and its interface
587 2014-08-14 06:13:00 <sipa> BlueMatt: if you rely on a feature that the library does not have, yes it should fail
588 2014-08-14 06:13:16 <sipa> though just querying what is supported is nicer
589 2014-08-14 06:13:26 <sipa> wumpus: ack
590 2014-08-14 06:13:28 <BlueMatt> sipa: yea, sure, but I meant if you dont use a feature that the library thinks you should
591 2014-08-14 06:13:29 <wumpus> I think it's important to make the current scripting engine available, if it is changed for this purpose, it kind of defeats the purpose
592 2014-08-14 06:13:30 <BlueMatt> ie p2sh version 2
593 2014-08-14 06:13:47 <sipa> BlueMatt: i don't think that is enforcable
594 2014-08-14 06:13:54 <sipa> it can be a hint
595 2014-08-14 06:13:59 <BlueMatt> sipa: yes, that is what I was suggesting
596 2014-08-14 06:14:03 <sipa> ok
597 2014-08-14 06:14:06 <BlueMatt> I took your comment to be otherwise
598 2014-08-14 06:14:10 <BlueMatt> misread, sorry
599 2014-08-14 06:14:34 <gmaxwell> wumpus: thats fair enough, but the interface should be setup to be aligned with that kind of goal even if its not changed to it right away.
600 2014-08-14 06:15:02 <wumpus> gmaxwell: ack
601 2014-08-14 06:15:05 <sipa> it will need a change in api
602 2014-08-14 06:15:10 <sipa> but that's fine
603 2014-08-14 06:15:15 <BlueMatt> sure, ofc, I thought you suggested as well a context, per-thread
604 2014-08-14 06:15:18 <BlueMatt> which is....ehhhh
605 2014-08-14 06:15:34 <sipa> yes
606 2014-08-14 06:15:35 <gmaxwell> also, if there is no dynamic memory allocation, I can finally run the damn thing inside klee and get 100% branch coverage tests machine generated.
607 2014-08-14 06:16:46 <gmaxwell> right now sorting out the thousands of superfluous branches related to dynamic memory management is too much work to do manually.
608 2014-08-14 06:17:22 <wumpus> BlueMatt: it could in principle manage a pool of contexts internally, if it's just for scratch space
609 2014-08-14 06:17:25 <BlueMatt> gmaxwell: so...are you suggesting a call to bitcoinscript_init per-thread and make users keep around state now?
610 2014-08-14 06:17:57 <BlueMatt> wumpus: yes, one should do that, but some idiot will write in a language that blindly spins up threads and will end up with 1000 threads to check 1000 transactions
611 2014-08-14 06:17:59 <wumpus> especially if it's just a static data structure
612 2014-08-14 06:18:01 <gmaxwell> BlueMatt: for all I care from an api perspective they could happily bitcoinscript_init(); bitcoinscript_verify(); every use.
613 2014-08-14 06:18:35 <BlueMatt> gmaxwell: huh? so allocate the script memory per-run? now I'm confused
614 2014-08-14 06:18:45 <BlueMatt> or keep around a thread-local variable that depends on your threading system used???
615 2014-08-14 06:18:52 <sipa> help no
616 2014-08-14 06:18:56 <gmaxwell> BlueMatt: then they _should_ fail, becuase when they later got a block with a bunch of high memory using transactions they _would_ fail, and thats exactly the kind of failure that should happen up front instead of intermittently as a consensus failure.
617 2014-08-14 06:18:58 <wumpus> BlueMatt: possible, but hard to clean up when the thread goes away
618 2014-08-14 06:19:26 <gmaxwell> BlueMatt: you just gave a perfect example of why not dynamically allocating memory is important.
619 2014-08-14 06:19:47 <BlueMatt> gmaxwell: no, I agree why we shouldnt dynamically allocate memory
620 2014-08-14 06:19:52 <BlueMatt> my question is what your alternative is here
621 2014-08-14 06:20:02 <wumpus> well it must be dynamically allocated at some point
622 2014-08-14 06:20:06 <wumpus> just not for every check
623 2014-08-14 06:20:17 <BlueMatt> if you're suggesting that users *must* keep their own thread-local scratch space managed themselves, I think thats a terrible api
624 2014-08-14 06:20:20 <gmaxwell> There is just a context that you initialize... one per thread if you're doing parallel validation.
625 2014-08-14 06:20:23 <wumpus> I hope you're not proposing like automotive C to put everything in the data segment statically :-)
626 2014-08-14 06:20:52 <wumpus> BlueMatt: that sounds like the openssl api!
627 2014-08-14 06:21:12 <gmaxwell> BlueMatt: it's just a validator context. It's no different that a zillion other things do. (including codecs, for example)
628 2014-08-14 06:21:22 <sipa> contexts are fine imho
629 2014-08-14 06:21:38 <wumpus> yes, an explicit context is fine
630 2014-08-14 06:21:52 <sipa> whether you use threadlocal storage or stacks or whatever at the application level doesn't matter
631 2014-08-14 06:21:55 <gmaxwell> and if you have multiple threads, then, yes, you need multiple contexts. And if some genuis spins up 1000 threads, he'll have 1000x the memory usage right off the bat: which is correct because that was his peak memory usage in reality.
632 2014-08-14 06:22:16 <gmaxwell> genius*
633 2014-08-14 06:22:30 <wumpus> you could even make your function accept NULL for the context, in which case it allocates it at the beginning and throws it away afterwards, for people that like dynamic allocation on every call
634 2014-08-14 06:23:11 <sipa> it will likely still be more efficient than now
635 2014-08-14 06:23:31 <wumpus> well this is just about the API... initially the context will be a dummy object :-)
636 2014-08-14 06:24:07 <wumpus> as we currently allocate everything on-the-fly no matter what
637 2014-08-14 06:24:31 <sipa> i'm fine with not having contexts for now
638 2014-08-14 06:24:37 <sipa> until we have a real use for them
639 2014-08-14 06:25:04 <gmaxwell> as I said before, best discussed when there is actually a library. it's not like the api is set in stone before it even exists.
640 2014-08-14 06:25:29 <wumpus> yeah
641 2014-08-14 06:25:41 <sipa> yeah
642 2014-08-14 06:26:31 <gmaxwell> (though I suppose it's good to know what people think the (eventual) requirements are!)
643 2014-08-14 06:27:04 <wumpus> well if you know for sure you're going to do something it makes sense to take it into account in the API already, if not it results in overdesign
644 2014-08-14 06:30:07 <BlueMatt> now I'm confused...this whole discussion started because cfields wanted to have something that was stable before we merge the library...so then we went off and planned for 10 years down the road, and now we're saying we should merge before we decide?
645 2014-08-14 06:30:41 <wumpus> you cannot plan for 10 years down the road, period
646 2014-08-14 06:31:00 <BlueMatt> I agree, hence why I wanted to move forward without any of this complication...
647 2014-08-14 06:31:15 <gmaxwell> ah I had no idea you were that far along yet.
648 2014-08-14 06:31:25 <wumpus> yes, just move forward
649 2014-08-14 06:33:09 <wumpus> you've heard everyone's opinion now, the rest is up to you
650 2014-08-14 06:34:38 <sipa> well what matters is that any api that is intended to be public can remain stable for a whilr... though obviously not 10 years
651 2014-08-14 06:35:10 <sipa> inreally like the "only expose serialized structures" for that
652 2014-08-14 06:35:13 <sipa> *i really
653 2014-08-14 06:35:32 <sipa> the only conplication seems how to deal with soft forking changes
654 2014-08-14 06:37:11 <sipa> i personally wouldn't mind first making the core changes necessary for the separation, and only doing the actual library/buildsystem/api later
655 2014-08-14 06:37:18 <sipa> but as you already have that working...
656 2014-08-14 06:39:38 <BlueMatt> gmaxwell: huh? the pull request creates a functional and useful library already?
657 2014-08-14 06:40:14 <BlueMatt> gmaxwell: the included .h file needs to move to an include/ directory before release
658 2014-08-14 06:40:23 <BlueMatt> so it can change, but the .so works fine as it is now
659 2014-08-14 06:41:02 <gmaxwell> I see this, I didn't realize that. So I was in the mode of "why don't you go finish something before we debate the API". :P
660 2014-08-14 06:41:27 <wumpus> going in circles :)
661 2014-08-14 06:41:51 <BlueMatt> wumpus: ok..what do you want to merge?
662 2014-08-14 06:42:11 <BlueMatt> me? I like it as-is and would require breaking .so compat every time plus a version field in the include file
663 2014-08-14 06:42:34 <BlueMatt> but the include file is separate anyway
664 2014-08-14 06:42:59 <BlueMatt> so its whether we want stuff in-library or not
665 2014-08-14 06:43:03 <BlueMatt> at least for release 1
666 2014-08-14 06:43:22 <wumpus> BlueMatt: I haven't looked at the implementation in detail yet, but interface-wise I'm fine with it as it is now
667 2014-08-14 06:48:07 <wumpus> BlueMatt: in any case, merging it would not equal 'an official release of the API'
668 2014-08-14 06:48:58 <BlueMatt> wumpus: I was referring to release as in releasing next bitcoin core version
669 2014-08-14 06:49:29 <BlueMatt> but I figure get the .so in a stable state for merge
670 2014-08-14 06:49:36 <BlueMatt> and then add features/trim down api/etc
671 2014-08-14 06:50:28 <wumpus> but yes a version constant in the header as well as a call to query the API version would make sense, to avoid serious breakage when the wrong .h is used with the wrong .so
672 2014-08-14 06:52:41 <wumpus> remember that the library is also meant to be load dynamically, so you need a way to know the loaded object is compatible with the interface that you expect
673 2014-08-14 06:53:12 <BlueMatt> libtool?
674 2014-08-14 06:53:21 <BlueMatt> but, ok
675 2014-08-14 06:53:35 <wumpus> ie, when you use python ctypes you load the .so but don't use the header
676 2014-08-14 06:53:49 <BlueMatt> I'll add a feature/version check thing but not a context thing yet
677 2014-08-14 06:53:53 <wumpus> well sure you can use the file name for signaling the version
678 2014-08-14 06:54:33 <wumpus> although it seems easy to break/misplace
679 2014-08-14 06:54:44 <wumpus> if it is the only mechanism used
680 2014-08-14 06:55:28 <BlueMatt> sure, yea
681 2014-08-14 06:55:36 <BlueMatt> esp when you have multiple versions
682 2014-08-14 06:55:40 <BlueMatt> (os+dev or so)
683 2014-08-14 06:56:23 <wumpus> well - the idea is that you can have multiple script interpreters that expose the same API version, but are implemented differently
684 2014-08-14 06:56:55 <gmaxwell> BlueMatt: next time you update your relay client, can you make it log the block hash along with the percent of txn saved? I'd do per miner stats on hitrates.
685 2014-08-14 07:00:11 <wumpus> anyhow - I'm fine with a method that just returns an integer which is The API Version, it would in some cases also be useful to be able to query for some description ("this is the script interpreter from bitcoin core 0.10.3") etc but that's just for logging/diagnostic purposes not to actually match against
686 2014-08-14 09:30:03 <Lloydy> Does anyone in here use Cloudflare on a really dynamic website? I'm wanting to use it for the SSL Protection, but when I turn it on my properly working website goes to hell in a hand basket :(
687 2014-08-14 09:32:31 <sipa> offtopic
688 2014-08-14 09:33:39 <Lloydy> sorry sipa I'm new here - can you suggest the proper channel?
689 2014-08-14 09:34:01 <cdecker> #cloudflare?