1 2014-10-21 00:00:09 <coryfields_> it'll just depend on all of core. i don't see that as a problem, do you?
2 2014-10-21 00:00:27 <coryfields_> (the result is arch/compiler/linker specific, if you're discussing the end-library-object)
3 2014-10-21 00:00:51 <sipa> it's a small example, and it may not be relevant enough here
4 2014-10-21 00:01:12 <sipa> but it's what i mean by cross-cutting: you have a separate in terms of knowledge domain, and in terms of layering on top of it
5 2014-10-21 00:01:13 <coryfields_> for ex in that case, almost all linkers would exclude the irrelevan't .o's as if they weren't there
6 2014-10-21 00:01:31 <coryfields_> sure. i don't think there's any way to avoid that, it'll happen one way or the other
7 2014-10-21 00:01:38 <sipa> that's interesting, but it would be nice to also have that guarantee reflected in your structure
8 2014-10-21 00:01:45 <coryfields_> the alternative is to avoid building convenience libs, and rebuild the objects each time
9 2014-10-21 00:02:00 <sipa> i don't understand that
10 2014-10-21 00:02:25 <coryfields_> which is also an option, i just don't believe it to be the "most correct". but i could be easily swayed if there was a need
11 2014-10-21 00:02:37 <sipa> why isn't it possible to build the different components of core (once each), and have whatever uses it link it whichever of those are needed?
12 2014-10-21 00:02:44 <coryfields_> a convenience lib lets you build a set of sources with a specific set of flags
13 2014-10-21 00:02:59 <coryfields_> for ex, scripting will explicitly be missing boost includes, to ensure that it builds without it
14 2014-10-21 00:03:06 <coryfields_> *script verification
15 2014-10-21 00:03:10 <sipa> that is interesting yes
16 2014-10-21 00:03:31 <coryfields_> say there's another component that's explicitly missing another dep (say openssl) for similar reasons
17 2014-10-21 00:03:50 <coryfields_> if there's a convenience library for each, then it wraps up nicely and you're all done
18 2014-10-21 00:03:54 <sipa> but is there no way to have one without the other?
19 2014-10-21 00:04:02 <coryfields_> but if they overlap, you may need to build them once with one set, and once with another
20 2014-10-21 00:04:13 <sipa> have the building with separate flags, but not turn them into actual libs?
21 2014-10-21 00:04:32 <coryfields_> ah crap, i've gtg again. sorry, crazy day
22 2014-10-21 00:54:12 <berndj> michagogo: ./config.status --version will tell you how you configured (but i ^R through history to find things like that too)
23 2014-10-21 01:02:26 <berndj> jtimon, C++ code tends to have more "code" in header files that's more error-prone than just a bunch of #defines and types like in idiomatic C code, is that the sort of breakage you're talking about?
24 2014-10-21 01:04:51 <berndj> because make -j starting to build both streams of code is exactly what -j is supposed to do, even if you end up with multiple streams ending in error
25 2014-10-21 01:09:35 <phantomcircuit> berndj, tends to but doesn't have to
26 2014-10-21 01:10:23 <berndj> i'm looking at src/script.h and seeing a looot of codey stuff :)
27 2014-10-21 01:10:24 <jtimon> I would like to have a generic libscript that doesn't depend on core in the future (say to sign alerts or other things, I would like to see script-based signatures replace regular signatures in the future) but this is probably too far fetched
28 2014-10-21 01:10:48 <phantomcircuit> berndj, it would be easy but potentially dangerous to move that all to script.cpp
29 2014-10-21 01:11:08 <jtimon> my question is, why does it consistently build server before common? it's not dependency and it's not alphabetic order
30 2014-10-21 01:11:19 <berndj> phantomcircuit, C++ definitely encourages more codey content in headers than C does. inline methods and whatnot
31 2014-10-21 01:11:28 <phantomcircuit> sure
32 2014-10-21 01:11:43 <jtimon> can't lib_server just depend on lib_common in the makefile?
33 2014-10-21 01:11:47 <berndj> jtimon, only with -j2 though? or even without (implicit -j1)?
34 2014-10-21 01:12:33 <jeremyrubin> During fork resolution, what is the full set of rules used to select block a or b
35 2014-10-21 01:12:46 <berndj> jtimon, that won't solve your problem; expressing that libbitcoin_server.a depends on libbitcoin_common.a doesn't imply that the objects *inside* the one depend on the other lib
36 2014-10-21 01:12:55 <jeremyrubin> (/ are there any links)
37 2014-10-21 01:12:59 <berndj> and i think expressing such dependencies would be Wrong (tm)
38 2014-10-21 01:13:00 <sipa> berndj: that's not his problems
39 2014-10-21 01:13:15 <sipa> berndj: he just wants to improve the order in which things are built
40 2014-10-21 01:13:26 <jtimon> -j2 doesn't have anything to do with this, it actually starts compiling the different objects in server in parallel
41 2014-10-21 01:13:37 <jeremyrubin> Thinking of a couple solutions to the "majoirty is not enough:" paper
42 2014-10-21 01:13:39 <sipa> so that the changed part gets built first, as it's more likely to be the part that fails
43 2014-10-21 01:14:08 <jtimon> I don't like having too many things in the headers either, but that will take time to fix
44 2014-10-21 01:15:00 <jtimon> ok, can't all objects in server depend on lib_common.a then?
45 2014-10-21 01:15:15 <berndj> sipa, anything that's out of date is "likely to fail" - it's never been compiled (according to timestamps); that one file has been touched more recently doesn't make it more fragile than another that hasn't been compiled
46 2014-10-21 01:16:41 <berndj> jtimon, you can do that, but that's expressing a false situation: the *objects* don't depend on the *library*; they depend on specific files that the library also depends on, and these dependencies are already pulled into the makefile via GCC's dependency discovery
47 2014-10-21 01:18:10 <jtimon> then I don't understand why these dependencies discovered by gcc aren't taken into account, why main is built before core
48 2014-10-21 01:19:35 <berndj> i have a feeling we're talking past each other. if you have a log that illustrates the problem, can i see it?
49 2014-10-21 01:35:27 <jtimon> berndj just make
50 2014-10-21 01:36:03 <jtimon> I'm currently comiling with "make libbitcoin_common.a check -j2" and I'm not going to clean because you do't believe me
51 2014-10-21 01:36:33 <berndj> it isn't that i don't believe you, it's that i'm not sure i understand what (mis)behaviour you mean
52 2014-10-21 01:37:35 <jtimon> https://travis-ci.org/bitcoin/bitcoin/jobs/38554554
53 2014-10-21 01:38:33 <jtimon> CXX libbitcoin_server_a-main.o is before CXX libbitcoin_common_a-chainparams.o
54 2014-10-21 01:39:32 <jtimon> it's not really misbehabiour but it would make more sense to me the other way around (given that things in libbitcoin_server_a-main.o depend on things in libbitcoin_common_a) but not the other way around
55 2014-10-21 01:40:33 <berndj> do you have a line number there for me?
56 2014-10-21 01:41:29 <jtimon> at 586 it starts compiling objects
57 2014-10-21 01:41:44 <berndj> ok got it
58 2014-10-21 01:42:33 <jtimon> IMO it should be util, common, server, wallet, but it does server, common, util, wallet
59 2014-10-21 01:43:20 <jtimon> and it's clearly not alphabetic order so I would like to know what order this is
60 2014-10-21 01:44:09 <berndj> some random topological sort order :-/ make is just finding nodes in the dependency graph that depend only on up-to-date targets and then making those
61 2014-10-21 01:44:43 <jtimon> I suspect is the noinst_LIBRARIES (makefile.am:39)
62 2014-10-21 01:45:57 <berndj> now you could add a fake dependency on libbitcoin_common.a to all the *.o files, but that will soft-break other people's massively parallel builds (they won't get full parallelism)
63 2014-10-21 01:54:58 <coryfields_> berndj: making one build depend on another would make one set build before another
64 2014-10-21 01:55:40 <coryfields_> "a: b" would do what jtimon wants. In make-speak, that means all a's depend on a finished b
65 2014-10-21 01:56:36 <berndj> coryfields_: making libfoo.a depend on libbar.a won't guarantee that bar.cc gets compiled before foo.cc, only that foo.o won't get linked into libfoo.a before bar.o gets linked into libbar.a
66 2014-10-21 01:57:44 <jtimon> coryfields_ wouldn't changing the order on makefile.am make it too? I mean there has to be a reason why server is ALWAYS built before common
67 2014-10-21 01:58:15 <berndj> jtimon, in the -j1 case, that's probably true
68 2014-10-21 01:58:20 <coryfields_> berndj: sorry, that's incorrect. "foo.a: bar.a" means "i can't start foo.a until bar.a exists." that means that it doesn't descend into foo.a's objects until that point.
69 2014-10-21 01:58:23 <coryfields_> try it and see.
70 2014-10-21 01:58:48 <coryfields_> that's how the dependency-tree is built.
71 2014-10-21 01:59:15 <coryfields_> jtimon: more likely because bitcoind or bitcoin-qt or _something_ depends on "server.a common.a" in that order
72 2014-10-21 02:01:31 <coryfields_> jtimon: Makefile.am:262
73 2014-10-21 02:01:45 <coryfields_> reverse those, it'll likely change to the outcome you're expecting
74 2014-10-21 02:02:41 <coryfields_> but that may cause a link error. binutils insists on a link order because it only traverses once when building the map
75 2014-10-21 02:04:23 <berndj> coryfields_: i get "making a.o" first with http://pastebin.com/XPB9quB1 as a makefile
76 2014-10-21 02:05:10 <berndj> "making a.o" comes later if i add "a.o: libb" though
77 2014-10-21 02:15:30 <coryfields_> berndj: we're speaking of automake/libtool convenience libs here, not generic make targets
78 2014-10-21 02:16:01 <berndj> meh, i'm clearly just adding noise at this point
79 2014-10-21 02:18:07 <coryfields_> berndj: but it looks like in the case of an automake convenience lib, you're correct. in the case of libtool, one will depend on the other
80 2014-10-21 02:19:13 <berndj> coryfields_, i'm saying that simply adding "libbitcoin_server.{a,la}: libbitcoin_common.{a,la}" won't be enough to prevent the objects in _server from being built before those in _common
81 2014-10-21 02:19:52 <berndj> you'd have to make the *objects* (not the *library*) of the one lib depend on the other lib
82 2014-10-21 02:19:58 <jtimon> coryfields_ it's actually noinst_LIBRARIES in 39 that determines it, could I change that?
83 2014-10-21 02:20:02 <berndj> which imho is Wrong (tm)
84 2014-10-21 02:20:14 <coryfields_> berndj: yes, i'll concede there. looks like libtool poisoned my thinking on that
85 2014-10-21 02:21:56 <jtimon> coryfields_ reversing in 262 didn't work but reverting in 39 did
86 2014-10-21 02:22:17 <coryfields_> jtimon: yea, but it's still no guarantee
87 2014-10-21 02:22:48 <coryfields_> jtimon: translation: i don't care if you change
88 2014-10-21 02:29:40 <jtimon> yeah, I know it's a long term soution but you're working on one, it's just convenience, great
89 2014-10-21 02:33:23 <coryfields_> going with berndj's suggestion, adding: "$(libbitcoin_server_a_OBJECTS) : | $(libbitcoin_common_a_OBJECTS)" should do it as well, but guaranteeing the order
90 2014-10-21 02:33:42 <berndj> yeah, i just stumbled on order-only prereqs now
91 2014-10-21 02:33:48 <coryfields_> berndj: my apologies for that, i was very much mistaken.
92 2014-10-21 02:34:39 <berndj> it smells like a gnu make-ism, but i don't know if you care about non-gnu makes
93 2014-10-21 02:34:52 <coryfields_> we're pretty deep in 'em already
94 2014-10-21 02:35:36 <jtimon> ok, guaranteeing the order is better, for sure
95 2014-10-21 02:35:46 <coryfields_> i'm not proposing that change, btw. i'd rather not add depends rules that aren't really necessary for anything
96 2014-10-21 03:13:09 <coryfields_> berndj: aha, there's the culprit
97 2014-10-21 03:13:18 <coryfields_> berndj: fyi, automake does something non-obvious
98 2014-10-21 03:13:51 <coryfields_> all-am: Makefile $(LIBRARIES) $(PROGRAMS)
99 2014-10-21 03:14:22 <coryfields_> so, setting the order for the binaries is correct, but they have to be in the correct order where jtimon mentioned as well, for the general "make" or "make check" case.
100 2014-10-21 04:21:31 <justanotheruser> 00:20 <+Luke-Jr> justanotheruser: it's being "developed" in multiple branches
101 2014-10-21 04:21:34 <justanotheruser> is there a master I mean
102 2014-10-21 04:22:06 <Luke-Jr> justanotheruser: the main master branch?
103 2014-10-21 04:23:09 <justanotheruser> oh that makes sense. I was thinking bitcoind may just keep its consensus code rather than use the library
104 2014-10-21 04:25:12 <visiteur_1> if you are well off and healthy please consider to make a donation to a poor guy and his family and please spread it :) https://secure.changa.co.ke/myweb/share/3050
105 2014-10-21 08:20:53 <jessep2> hi folks. will the -blocknotify command fire when a -reindex is done (i.e. for every block)?
106 2014-10-21 08:21:16 <kyuupichan> A question about MAX_BLOCK_SIZE: it seems that in main.cpp where this is enforced, it is checked by serializing the block and checking its size. Is there a reason why it's done this expensive way, rather than e.g. just checking the payload size of the "block" command? Presumably they are one and the same?
107 2014-10-21 08:21:52 <jessep2> and a related question: will the -walletnotify command fire for all transactions upon a rescan/reindex/zapwallettxes?
108 2014-10-21 08:24:41 <phantomcircuit> kyuupichan, because that would be more dangerous
109 2014-10-21 08:24:46 <bad_duck> is header first synchronization in bitcoin-core 0.9.3 ? or should I get src frong github to test this ?
110 2014-10-21 08:24:52 <kyuupichan> phantomcircuit: How so?
111 2014-10-21 08:25:25 <sipa> kyuupichan: it would be incorrect
112 2014-10-21 08:25:44 <sipa> kyuupichan: the size is not the size of the packet, but the size of the minimal serialization of the data
113 2014-10-21 08:26:16 <kyuupichan> sipa: Interesting. Where is there room for ambiguity?
114 2014-10-21 08:26:16 <sipa> so you can use a non-minimally encoded block (for example for the transaction count) which is slightly above 1M bytes, but when correctly encoded, it is below
115 2014-10-21 08:26:20 <sipa> which is valid
116 2014-10-21 08:26:28 <kyuupichan> I see.
117 2014-10-21 08:26:53 <kyuupichan> The ambiguous encoding being varints?
118 2014-10-21 08:26:56 <sipa> yes
119 2014-10-21 08:27:02 <sipa> all of them, actually
120 2014-10-21 08:27:08 <kyuupichan> How nasty
121 2014-10-21 08:27:33 <kyuupichan> I assume there are already blocks in the chain which are not minimally encoded?
122 2014-10-21 08:27:42 <sipa> that question doesn't make sense
123 2014-10-21 08:27:53 <sipa> blocks are not defined by their encoded, but by their contents
124 2014-10-21 08:28:04 <sipa> the encoding is just how you transfer it from one peer to another
125 2014-10-21 08:28:16 <sipa> the hash of a block is for example also always computed on the minimal form
126 2014-10-21 08:28:52 <kyuupichan> Is the only ambiguity in varints? Or scripts too?
127 2014-10-21 08:29:40 <sipa> well scripts have their own share of ambiguities (see bip62), but as scripts (from the point of view of block rules) are just byte arrays, those do not contribute to the size issue
128 2014-10-21 08:30:00 <sipa> changing the script would modify the transaction, and the block it is in
129 2014-10-21 08:30:17 <sipa> so no, the encoding is purely varints
130 2014-10-21 08:30:35 <sipa> specifically, the transaction count, and the txin and txout counts inside every transaction
131 2014-10-21 08:31:01 <kyuupichan> So just to make sure I understand. The hash of a block needn't match the hash of the data as sent in a block command?
132 2014-10-21 08:31:07 <sipa> indeed
133 2014-10-21 08:32:54 <kyuupichan> Sorry the hash of a block is the hash of the header which is constant
134 2014-10-21 08:34:00 <sipa> right, but inside the header is the merkle root
135 2014-10-21 08:34:12 <sipa> which is computed from the transaction hashes
136 2014-10-21 08:34:23 <bad_duck> hmm, I tried to download bitcoin-core from here : http://bitcoin.sipa.be/builds/headersfirst/bitcoin-gf38f7af0/
137 2014-10-21 08:34:48 <bad_duck> but all I get is "Segmentation fault" when launching bitcoin-qt (both 32+64bit), any idea ?
138 2014-10-21 08:34:48 <sipa> and those transaction hashes are defined by their minimal encoding, not by how they are sent over the wire
139 2014-10-21 08:34:58 <sipa> bad_duck: anything in debug.log?
140 2014-10-21 08:35:22 <bad_duck> sipa: in my ~/.bitcoin ? nope
141 2014-10-21 08:35:35 <bad_duck> I trashed the folder to have a clean install
142 2014-10-21 08:35:40 <sipa> bad_duck: i suggest compiling master yourself
143 2014-10-21 08:35:49 <bad_duck> but the 0.9.3 official build is running fine
144 2014-10-21 08:35:49 <sipa> there may be library issues etc
145 2014-10-21 08:36:32 <bad_duck> sipa: seems that what I'll have to do I to test header first sync
146 2014-10-21 08:36:58 <jessep2> my first question sounds like a no: !IsInitialBlockDownload() (which apparently incl
147 2014-10-21 08:37:35 <sipa> jessep2: indeed
148 2014-10-21 08:37:48 <sipa> reindexing would otherwise pretty much be a forkbomb for your system
149 2014-10-21 08:38:00 <sipa> (spanwing thousands of processes)
150 2014-10-21 08:38:18 <jessep2> sipa: yes, that makes sense
151 2014-10-21 08:38:35 <jessep2> (not forkbombing during an intiial download/reindex that is)
152 2014-10-21 08:39:03 <sipa> ACTION afk
153 2014-10-21 08:39:25 <kyuupichan> sipa: Thanks for your help. So it's actually the transactions whose correct hash (the one in the merkle tree) needn't match the hash of the bytes received on the wire.
154 2014-10-21 08:40:14 <kyuupichan> How unfortunate.
155 2014-10-21 09:34:44 <Luke-Jr> wumpus: if by chance you're merging the submitblock fix tonight, ping me here and I'll be glad to rebase the next 2 on top (or anything else you need)
156 2014-10-21 09:44:12 <wumpus> Luke-Jr: you're registering the StateCatcher before the ProcessBlock call, and unregistering afterwards - is this guaranteed to work, or can it be possible that the signal arrives later/asynchronously?
157 2014-10-21 09:47:08 <wumpus> ie, will the signal be raised from the same thread, just deeper in the call tree?
158 2014-10-21 09:48:31 <Luke-Jr> wumpus: I'm not familiar with boost signals, but in theory we have the cs_main lock held the whole time
159 2014-10-21 09:48:44 <Luke-Jr> so it *shouldn't* be processing blocks except in our own thread
160 2014-10-21 09:50:07 <wumpus> ok that makes sense - so the signal won't be raised from a script validation thread, for example?
161 2014-10-21 09:51:52 <Luke-Jr> wumpus: it's only called from ConnectTip, which should only be called with cs_main held
162 2014-10-21 09:51:54 <wumpus> I can take a look myself too, it just *looks* a bit weird to me to register a signal handler for the duration of one call, for the purpose of getting some information from deeper in the call chain, usually you'd want to register a signal for information that arrives asynchronously
163 2014-10-21 09:52:20 <Luke-Jr> sipa wanted to do it this way, and I can see why (it's much simpler)
164 2014-10-21 09:52:27 <wumpus> so in that case you want to wait for the signal to arrive for the RPC to return
165 2014-10-21 09:52:34 <Luke-Jr> no
166 2014-10-21 09:52:44 <Luke-Jr> the signal will occur during ProcessBlock, deep in the reorg code, or not
167 2014-10-21 09:52:49 <Luke-Jr> both cases are handled
168 2014-10-21 09:52:56 <wumpus> ok
169 2014-10-21 09:53:17 <Luke-Jr> if the block isn't checked, we return "inconclusive"
170 2014-10-21 09:53:50 <wumpus> do we want to return the text "inconclusive" or should that have a status code?
171 2014-10-21 09:54:39 <wumpus> ah we do the same for "rejected"
172 2014-10-21 09:54:41 <wumpus> ok
173 2014-10-21 09:54:50 <Luke-Jr> historically, we've been returning "orphan" for true orphan blocks that we can't check
174 2014-10-21 09:55:07 <Luke-Jr> yes, the return value is a string stating why it was rejected
175 2014-10-21 09:55:24 <wumpus> or ::null when it was accepted
176 2014-10-21 09:55:32 <Luke-Jr> right
177 2014-10-21 09:56:25 <wumpus> 5105 is merged
178 2014-10-21 09:58:46 <bad_duck> How can I verify that I'm using header first sync ? (I git clone from github master this morning)
179 2014-10-21 09:59:20 <wumpus> if you have cloned from master this morning, you will be using headers first sync
180 2014-10-21 10:01:22 <bad_duck> ok thx, btw it seems nice -> Statup time : 30 mn ago, 2years and 9 week behind, downloading a full connection speed (2 MB/s)
181 2014-10-21 10:01:35 <wumpus> but if you really want to check you have the right tree, see if 84d13ee appears in the history of your tree
182 2014-10-21 10:03:17 <bad_duck> wumpus: got it (Merge pull request #5095) so it must be ok (and seems to be), thx
183 2014-10-21 10:31:27 <CodeShark> wumpus: have you had a chance to look at 5107 again?
184 2014-10-21 10:34:38 <wumpus> CodeShark: yes, but it will take some time to verify all the locks are properly pushed down ; I'd have preferred an intermediate commit that just puts the locks at the beginnig of each function
185 2014-10-21 10:35:17 <wumpus> before trying to push them down even further
186 2014-10-21 10:35:31 <CodeShark> yeah, I was considering that - there's only one function where I think there might be an issue
187 2014-10-21 10:35:54 <CodeShark> getinfo
188 2014-10-21 10:36:05 <wumpus> getmininginfo is a bit weird, I don't think there is an advantage to getting the lock twice
189 2014-10-21 10:36:22 <CodeShark> yeah, I realize it's a recursive mutex
190 2014-10-21 10:36:36 <CodeShark> I was a little afraid of rpc functions that call other rpc functions
191 2014-10-21 10:36:52 <wumpus> to release it for one push_back
192 2014-10-21 10:36:55 <wumpus> you shouldn't
193 2014-10-21 10:37:28 <CodeShark> then with getinfo, the conditionals on pwalletMain make it harder to use RAII
194 2014-10-21 10:37:52 <wumpus> ugh yes that's ugly
195 2014-10-21 10:37:54 <wumpus> :D
196 2014-10-21 10:38:23 <wumpus> then again
197 2014-10-21 10:38:38 <wumpus> getinfo is going to be deprecated, exactly for reasons like this
198 2014-10-21 10:40:10 <wumpus> what about other calls that conditionally use the wallet?
199 2014-10-21 10:40:20 <wumpus> validateaddress, signtransaction come to mind
200 2014-10-21 10:42:40 <wumpus> they need a similar construction
201 2014-10-21 10:42:59 <CodeShark> validateaddress - yes
202 2014-10-21 10:43:07 <CodeShark> but you mean signrawtransaction, right?
203 2014-10-21 10:43:11 <wumpus> yes
204 2014-10-21 10:44:11 <wumpus> it's annoying to have the same calls do a different thing based on whether there is a wallet or not, but for now, we're stuck with that API
205 2014-10-21 10:44:12 <CodeShark> the way I'd rather do it is to just start placing the locks around the wallet calls that are already in the #ifdef
206 2014-10-21 10:44:34 <wumpus> indeed
207 2014-10-21 10:44:36 <CodeShark> we place an RAII lock at the beginning of the function, then an RAII lock within a code block inside the #ifdef
208 2014-10-21 10:44:45 <CodeShark> but...
209 2014-10-21 10:44:46 <wumpus> do be careful for the case where pWallet is NULL
210 2014-10-21 10:44:49 <CodeShark> right
211 2014-10-21 10:44:59 <CodeShark> that's the annoying thing that makes it a pain in the ass
212 2014-10-21 10:45:00 <wumpus> so you'd need to put the lock in if(pWallet) as well
213 2014-10-21 10:45:18 <CodeShark> yes, absolutely
214 2014-10-21 10:45:43 <CodeShark> can we do LOCK(a); LOCK(b);?
215 2014-10-21 10:45:49 <wumpus> sure
216 2014-10-21 10:46:02 <wumpus> (besides the obvious, that you take the locks always in the same order)
217 2014-10-21 10:46:59 <CodeShark> hmm, validateaddress doesn't seem to check whether pwalletMain is NULL
218 2014-10-21 10:47:02 <wumpus> but that's easy; everything requires cs_main, wallet functions require cs_main and cs_wallet
219 2014-10-21 10:47:17 <wumpus> CodeShark: yes it is nasty, uses some ternary construction :-)
220 2014-10-21 10:47:22 <CodeShark> oh...
221 2014-10-21 10:47:28 <CodeShark> it gets short-circuited
222 2014-10-21 10:47:39 <CodeShark> lol
223 2014-10-21 10:47:54 <CodeShark> it would be better to wrap all those conditionals into a single if (pwalletMain)
224 2014-10-21 10:48:28 <wumpus> I guess; anyhow, go with the solution that is most obviously correct for now
225 2014-10-21 10:48:48 <wumpus> hmm though RAII is important: this code uses exceptions
226 2014-10-21 10:48:51 <CodeShark> someone tried to get clever with the && operator - but in this context it only obfuscates
227 2014-10-21 10:49:23 <wumpus> yes, well if you write it in a way that's clearer that's fine
228 2014-10-21 10:50:29 <wumpus> just be sure that it still handles the same cases (not 100% sure whether they all get tested in rpc tests)
229 2014-10-21 10:50:51 <CodeShark> looks like only the "ismine" field gets populated regardless
230 2014-10-21 10:51:08 <wumpus> generally the without-wallet cases have worst test coverage
231 2014-10-21 10:51:10 <CodeShark> "iswatchonly" and "account" only get populated if pwalletMain is not null
232 2014-10-21 10:51:12 <wumpus> worse*
233 2014-10-21 10:51:38 <wumpus> ok
234 2014-10-21 10:52:38 <CodeShark> hmm, if we put "ismine" after "iswatchonly" and "account" it's easier to take it out of the conditional
235 2014-10-21 10:52:50 <CodeShark> will it break anything if we change the field order?
236 2014-10-21 10:52:57 <wumpus> no, just update the help as well
237 2014-10-21 10:53:56 <wumpus> people are not supposed to depend on the JSON field order, that's why we return objects and not lists :)
238 2014-10-21 10:55:06 <CodeShark> yeah, it's unfortunate the same thing doesn't apply to JSON-RPC calls :)
239 2014-10-21 10:55:16 <wumpus> agreed
240 2014-10-21 10:55:31 <wumpus> if I were to redesign the RPC, I'd use named parameters all over the place
241 2014-10-21 10:55:46 <CodeShark> I've debated this issue for my own APIs - whether it's really that important to comply with JSON-RPC or just use named parameters everywhere
242 2014-10-21 10:56:26 <CodeShark> you can still use the method and id fields of JSON-RPC
243 2014-10-21 10:56:31 <CodeShark> but just change params to an object
244 2014-10-21 10:56:43 <wumpus> then again we're stuck with this and it works, there's no pressing reason to change it, unless we do some big non-backwards-compatible "JSON API 2.0"
245 2014-10-21 10:57:08 <wumpus> but I doubt it's useful, it will just give people a lot of work
246 2014-10-21 10:57:41 <wumpus> yes if you were to design a new API that's a good consideration; all current JSON API wrappers will expect bitcoin-style positional arguments
247 2014-10-21 10:57:44 <CodeShark> the main disadvantage as things are right now is greater difficulty in extending existing commands - but the main disadvantage to changing it is that it breaks most existing apps
248 2014-10-21 10:58:14 <CodeShark> ultimately, the path forward is probably to support both approaches in future releases
249 2014-10-21 10:58:30 <CodeShark> but that can wait :p
250 2014-10-21 10:58:43 <wumpus> it's mostly the wallet API that requires 'extending' current calls sometimes, the node API has been extremely stable with mostly just added commands/return fields
251 2014-10-21 10:59:23 <wumpus> ... after splitting off the wallet to a new project, people can innovate whatever they want there
252 2014-10-21 10:59:33 <CodeShark> a really annoying thing about the parameter list with default parameter thing is that you cannot arbitrarily select which parameters are to use the defaults
253 2014-10-21 10:59:51 <CodeShark> i.e. omitted parameters use defaults
254 2014-10-21 10:59:59 <wumpus> yes
255 2014-10-21 11:00:21 <CodeShark> but this is also a problem with C++ :)
256 2014-10-21 11:00:40 <wumpus> on the other hand -- maybe API versioning is better then adding 'defaulted' arguments
257 2014-10-21 11:01:00 <CodeShark> the only way around it in C++ is to have a special overloaded version for each case
258 2014-10-21 11:01:45 <wumpus> make the caller select an API version; so if you have to introduce a new argument, add the argument in a new version which requires it, then deprecate the old version over time
259 2014-10-21 11:01:46 <CodeShark> and even then, C++ can only resolve the call by type
260 2014-10-21 11:02:48 <wumpus> right
261 2014-10-21 11:02:49 <CodeShark> there are two approaches to API version - either a switchable modeâ¦or perhaps better, prefixed method names
262 2014-10-21 11:03:16 <Luke-Jr> ⦠or a new URI
263 2014-10-21 11:03:26 <CodeShark> yes - we could use a different port altogether
264 2014-10-21 11:03:27 <Luke-Jr> but IIRC a long time ago we decided when we break compat, it will just break XD
265 2014-10-21 11:03:39 <wumpus> no, no new port, a new URI would be fine
266 2014-10-21 11:03:45 <CodeShark> right
267 2014-10-21 11:03:53 <CodeShark> anyhow, these issues can wait
268 2014-10-21 11:04:24 <wumpus> there is a mechanism now to add new APIs to the same HTTP port (see for example https://github.com/bitcoin/bitcoin/pull/2844)
269 2014-10-21 11:04:44 <wumpus> yes, I want to wait at least after splitting off the wallet, as the requirement for a wallet interface is completely different from a node interface in the first place
270 2014-10-21 11:05:14 <wumpus> wallet interface is authenticated, stateful, confidential, whereas the node interface is mostly just information requests and some sysadmin commands
271 2014-10-21 11:05:28 <CodeShark> alright - so then the http server will have to check HTTP path and use that along with the method name in dispatching
272 2014-10-21 11:06:18 <wumpus> I'm in favor of moving pure information requests to REST HTTP style (as jgarzik's pull does)
273 2014-10-21 11:07:41 <CodeShark> the http server should also have an authentication layer, as you say - where you can assign permission levels or groups to different commands
274 2014-10-21 11:08:04 <CodeShark> and ideally, allow the association of access keys with different permission levels, groups, or specific users
275 2014-10-21 11:08:14 <wumpus> nah, not for bitcoin core
276 2014-10-21 11:08:18 <wumpus> I don't want to handle that
277 2014-10-21 11:08:35 <CodeShark> well, bitcoin core should probably not have any account management functionality :p
278 2014-10-21 11:08:38 <wumpus> there's mr sysadmin that controls the node, and the rest that can just do harmless things like request block contents
279 2014-10-21 11:08:59 <wumpus> right, that must go
280 2014-10-21 11:09:12 <CodeShark> so yeah, just two levels
281 2014-10-21 11:09:17 <CodeShark> or three, I suppose
282 2014-10-21 11:09:28 <wumpus> just two
283 2014-10-21 11:09:29 <CodeShark> admin, info, and no access
284 2014-10-21 11:09:53 <CodeShark> right now we just have admin and info on one side and no access on the other
285 2014-10-21 11:10:01 <wumpus> for no access just block access to the port
286 2014-10-21 11:10:10 <wumpus> ie, using binding or a firewall
287 2014-10-21 11:10:20 <wumpus> or rpcallowip
288 2014-10-21 11:10:28 <CodeShark> we already prevent access via username, password, and allowed ips
289 2014-10-21 11:10:42 <CodeShark> but we do not distinguish between admin access and info access
290 2014-10-21 11:11:04 <wumpus> yes, so you're right, there are already three levels
291 2014-10-21 11:11:46 <CodeShark> we could have two separate access keys (username/password combinations)
292 2014-10-21 11:11:47 <wumpus> hosts that can't connect the port or are not in allowip -> get the door shut on them, hosts only in allowip -> can make r/o requests, with username/password -> can do r/w requests
293 2014-10-21 11:13:07 <CodeShark> and then the wallet functionality could have its own access levels - lol
294 2014-10-21 11:13:22 <CodeShark> anyhow...
295 2014-10-21 11:13:27 <wumpus> I suppose so - but not in bitcoin core, we want to move away from that
296 2014-10-21 11:13:39 <CodeShark> yes, I was being at least a little facetious
297 2014-10-21 11:14:07 <CodeShark> anyhow, let's fix these locks and merge 5107
298 2014-10-21 11:15:56 <CodeShark> as a separate note, how do you prefer to handle these edits? keep the git history in chronological order as we make these changes and then do a final rebase before merging? or doing rebases as we make these changes?
299 2014-10-21 11:16:57 <wumpus> during the review it's better to make chronological commits
300 2014-10-21 11:17:04 <wumpus> then a squash before the merge
301 2014-10-21 11:17:43 <CodeShark> ok - I still think it can be useful to break up a pull request into more than one commit - but after at least a few reorders and fixups
302 2014-10-21 11:18:34 <wumpus> oh sure, it can make sense if you have different concerns isolated in different commits, but chronological commits can usually be squashed
303 2014-10-21 11:19:00 <CodeShark> well, for instance, the lock thing was what my first commit in this pull request addressed - the signals came after that
304 2014-10-21 11:19:19 <CodeShark> so if we fix the lock thing, I'd do it chronologically, then reorder the commits and make this one a fixup
305 2014-10-21 11:19:33 <wumpus> sometimes it's not entirely straightforward, but try to avoid having 'fix the fix' style commits on merge :-)
306 2014-10-21 11:19:42 <CodeShark> lol
307 2014-10-21 11:19:58 <wumpus> which are fine during review
308 2014-10-21 11:20:02 <CodeShark> no eighteenth amendment? :)
309 2014-10-21 11:21:11 <wumpus> lol I had to google that one (I'm Dutch) :)
310 2014-10-21 11:21:31 <wumpus> github alcohol lock :D
311 2014-10-21 11:23:09 <CodeShark> too bad we can't rebase the constitution
312 2014-10-21 11:24:46 <btcdrak> ^
313 2014-10-21 11:29:30 <CodeShark> wumpus: I'm not sure I entirely get the signrawtransaction logix
314 2014-10-21 11:30:07 <CodeShark> doesn't seem to access pwalletMain directly
315 2014-10-21 11:30:28 <CodeShark> I see a call to EnsureWalletIsUnlocked - and then it seems to use some global objects
316 2014-10-21 11:32:59 <X00M> A question: What to do with old mining hardware?
317 2014-10-21 11:33:13 <CodeShark> harvest it for precious metals
318 2014-10-21 11:33:17 <X00M> Im talking about ASIC miners
319 2014-10-21 11:33:25 <X00M> isnt a litecoin conversion possible?
320 2014-10-21 11:33:30 <CodeShark> no
321 2014-10-21 11:33:44 <CodeShark> litecoin uses an entirely different hashing algorithm
322 2014-10-21 11:33:53 <X00M> Its all about an algorithm
323 2014-10-21 11:33:59 <X00M> Why no one implements it for ASIC miners
324 2014-10-21 11:34:06 <Luke-Jr> X00M: keep mining
325 2014-10-21 11:34:09 <CodeShark> lol
326 2014-10-21 11:34:23 <X00M> its not profitable anymore
327 2014-10-21 11:34:35 <X00M> difficulty rate is ++
328 2014-10-21 11:35:04 <X00M> why wouldnt it be possible to create the litecoin alghoritm for old ASIC mining system
329 2014-10-21 11:35:14 <CodeShark> because of the AS
330 2014-10-21 11:35:17 <CodeShark> in ASIC
331 2014-10-21 11:35:20 <X00M> Is this a prosessor limitation?
332 2014-10-21 11:35:56 <X00M> Maybe ALU limitation.
333 2014-10-21 11:36:06 <CodeShark> ASICs are optimized to do one thing - and only that one thing wellâ¦and are not designed to be easily reprogrammable
334 2014-10-21 11:36:58 <X00M> I wonder why there is no ASM - startup
335 2014-10-21 11:37:19 <X00M> Maybe a community that reprogramms those old systems.
336 2014-10-21 11:37:30 <X00M> its an absolute waste..
337 2014-10-21 11:37:37 <Luke-Jr> X00M: it's securing the network
338 2014-10-21 11:37:44 <Luke-Jr> it was never supposed to be profitable
339 2014-10-21 11:37:52 <CodeShark> X00M: you are aware of what the AS in ASIC stands for, right?
340 2014-10-21 11:37:57 <X00M> Nice anwser.
341 2014-10-21 11:38:01 <X00M> Yes Asynchronus
342 2014-10-21 11:38:04 <CodeShark> no
343 2014-10-21 11:38:07 <Luke-Jr> ACTION facepalms
344 2014-10-21 11:38:15 <wumpus> CodeShark: signrawtransaction can use the wallet's keystore to 'sign from'
345 2014-10-21 11:38:20 <Luke-Jr> Application-Specific Integrated Circuit
346 2014-10-21 11:38:37 <X00M> ACTION gotschooled
347 2014-10-21 11:39:35 <X00M> Memory on those board are not PROM
348 2014-10-21 11:39:43 <wumpus> when an algorithm is implemented in hardware you can't just 'reprogram' it, you'd have to reorganise the chip's path physically for that, not really realistic
349 2014-10-21 11:39:43 <X00M> You can append
350 2014-10-21 11:40:08 <X00M> Wumpus, you can request circurtry mappings from vendor
351 2014-10-21 11:40:24 <wumpus> maybe with a reaaaly small soldering iron :-) but no kidding, it's impossible, you'd just have to design a new ASIC and produce it
352 2014-10-21 11:40:45 <Luke-Jr> X00M: there is no memory on them
353 2014-10-21 11:40:46 <Luke-Jr> at all
354 2014-10-21 11:40:59 <Luke-Jr> the chip consists entirely of circuits to do mining
355 2014-10-21 11:41:51 <Luke-Jr> X00M: if you want to send me your miners, I'd be glad to take them off your hands..
356 2014-10-21 11:42:39 <X00M> Luke - the miner has an OS on it..
357 2014-10-21 11:42:56 <wumpus> that's just the management firmware
358 2014-10-21 11:42:56 <X00M> I got a jupiter from KNC for free
359 2014-10-21 11:43:08 <X00M> yes wumpus
360 2014-10-21 11:43:14 <X00M> Ithink so.
361 2014-10-21 11:43:32 <wumpus> you could reprogram that, but it's be pretty pointless, the mining ciruitry it drives is fixed
362 2014-10-21 11:43:50 <wumpus> it would be like reprogramming the firmware in your microwave to think it's a refrigator :-)
363 2014-10-21 11:43:59 <X00M> wumpus: haha
364 2014-10-21 11:44:09 <Luke-Jr> X00M: you can repurpose the controller, sure
365 2014-10-21 11:44:32 <X00M> Could I create an high preformance videocard out of my old miner.
366 2014-10-21 11:44:36 <Luke-Jr> no
367 2014-10-21 11:44:43 <X00M> haha
368 2014-10-21 11:44:48 <X00M> was to good to be true.
369 2014-10-21 11:44:56 <wumpus> no, you cannot do anything with it, except double-sha256 with the goal if creating small hashes
370 2014-10-21 11:45:10 <CodeShark> or harvest it for precious metals :p
371 2014-10-21 11:45:25 <X00M> CodeShark was right.
372 2014-10-21 11:45:30 <X00M> scrap metal
373 2014-10-21 11:45:35 <X00M> What a waste!!!
374 2014-10-21 11:45:41 <wumpus> yes you could try to recycle for materials, but I'm not sure how much is possible there with electronics these days, a lot just ends up in landfills
375 2014-10-21 11:46:24 <CodeShark> extracting gold from ICs can be cheaper than mining it from the earth - but it's only really cost-effective if you do it at large scale
376 2014-10-21 11:46:26 <Luke-Jr> actually
377 2014-10-21 11:46:34 <Luke-Jr> I'm repurposing some of my 65nm BFL miners
378 2014-10-21 11:46:38 <wumpus> it's not like you can (profitably) recycle the silicon and create new wafers from it to print new chips
379 2014-10-21 11:46:42 <wumpus> yes the gold can
380 2014-10-21 11:46:51 <CodeShark> silicon is one of the most abundant elements on the surface of earth
381 2014-10-21 11:46:52 <X00M> Did people even think about re-usability
382 2014-10-21 11:46:55 <Luke-Jr> getting chilly here lately
383 2014-10-21 11:46:57 <X00M> its kind of strange,..
384 2014-10-21 11:47:01 <Luke-Jr> they'll make good heaters maybe
385 2014-10-21 11:47:09 <CodeShark> the only thing that makes wafer silicon "precious" is its purity and how it's doped
386 2014-10-21 11:48:19 <Luke-Jr> X00M: actually, you said those are KnC miners? KnC stuff tends to be quiet, so probably fits my heating project better than BFL stuff - maybe I can pay shipping if you send me it :p
387 2014-10-21 11:48:20 <X00M> https://bitcointalk.org/index.php?topic=456033
388 2014-10-21 11:48:32 <CodeShark> but gold CAN be profitably extracted from ICs
389 2014-10-21 11:48:37 <X00M> Luke-Jr: yes it was KNC.
390 2014-10-21 11:48:41 <X00M> Its a Jupiter
391 2014-10-21 11:48:59 <Luke-Jr> X00M: find out how much shipping to Florida is (33523)
392 2014-10-21 11:49:23 <Luke-Jr> oh, you're .nl - probably too expensive :<
393 2014-10-21 11:49:35 <X00M> your heating project?
394 2014-10-21 11:49:37 <Luke-Jr> doesn't it get cold in .nl?
395 2014-10-21 11:49:44 <Luke-Jr> X00M: yes, I'm using miners to heat my house :P
396 2014-10-21 11:49:45 <X00M> Wel Luke-Jr, it does
397 2014-10-21 11:49:46 <Luke-Jr> old miners*
398 2014-10-21 11:49:50 <X00M> Hahaha
399 2014-10-21 11:49:55 <X00M> Nice project
400 2014-10-21 11:50:11 <X00M> but is this heating more efficient / cost effective than traditional heating
401 2014-10-21 11:50:14 <X00M> ?
402 2014-10-21 11:50:25 <Luke-Jr> I would think so, since it's mining bitcoins too
403 2014-10-21 11:50:32 <Luke-Jr> bitcoin-subsidised heating
404 2014-10-21 11:50:49 <X00M> if you say old mining hardware.. what you mean?
405 2014-10-21 11:50:53 <X00M> what kind of hashes
406 2014-10-21 11:51:11 <Luke-Jr> I was thinking the asicminer cubes, BFL stuff, etc
407 2014-10-21 11:51:14 <CodeShark> the stone age was the first silicon revolution :p
408 2014-10-21 11:51:28 <Luke-Jr> BFL stuff is noisy though, so not ideal
409 2014-10-21 11:51:55 <X00M> KNC is also noisy..
410 2014-10-21 11:52:03 <X00M> BUT - generate good heat
411 2014-10-21 11:52:07 <Luke-Jr> really? my Saturn was quiet
412 2014-10-21 11:52:11 <Luke-Jr> compared to BFL at least
413 2014-10-21 11:52:14 <Luke-Jr> BFL sounds like a jet
414 2014-10-21 11:52:17 <X00M> I had a jupiter
415 2014-10-21 11:52:20 <X00M> same fans?
416 2014-10-21 11:52:25 <X00M> no idea
417 2014-10-21 11:52:28 <Luke-Jr> Jupiter is just a double-Saturn AFAIK
418 2014-10-21 11:52:51 <Luke-Jr> maybe we shoudl take this to #bitcoin-mining
419 2014-10-21 11:52:54 <Luke-Jr> not very dev-topic
420 2014-10-21 11:52:54 <X00M> Its just sad..
421 2014-10-21 11:53:02 <CodeShark> wumpus: still not seeing how signrawtransaction accesses the wallet keystore - the only wallet-specific call I see is EnsureWalletIsUnlocked
422 2014-10-21 11:53:17 <CodeShark> oh, duh
423 2014-10-21 11:53:19 <CodeShark> NVM
424 2014-10-21 11:53:20 <X00M> yes Luke-Jr .
425 2014-10-21 11:54:02 <CodeShark> does EnsureWalletIsUnlocked need to be wrapped?
426 2014-10-21 11:55:14 <CodeShark> or can we just place the lock before assigning the keystore variable?
427 2014-10-21 11:55:40 <wumpus> CodeShark: trying to find out
428 2014-10-21 11:55:54 <wumpus> EnsureWalletIsUnlocked calls pwalletMain->IsLocked()
429 2014-10-21 11:56:42 <CodeShark> so it assumes that pwalletMain is not NULL
430 2014-10-21 11:56:45 <wumpus> trying to find the implementation of that, ah in Crypter
431 2014-10-21 11:57:02 <wumpus> yes it doesn't aquire the lock itself
432 2014-10-21 11:57:11 <wumpus> so why not add a wallet lock to EnsureWalletIsUnlocked
433 2014-10-21 11:57:29 <wumpus> and indeed, don't call it if pwalletMain is NULL
434 2014-10-21 11:58:03 <CodeShark> does this mean a potential crash scenario with the way things are?
435 2014-10-21 11:59:26 <CodeShark> and is it safe to lock cs_wallet just for the IsLocked operation, then unlock, then lock again to assign keystore?
436 2014-10-21 11:59:46 <wumpus> I wouldn't dare to say, the reason the RPC locks are so coarse-grained is exactly because it is huge amounts of work and error-prone to audit everything in detail
437 2014-10-21 12:00:09 <CodeShark> what if another thread locks the wallet between the call to EnsureWalletIsUnlocked and the assignment of keystore?
438 2014-10-21 12:00:10 <wumpus> there used to be lots of threading issues with RPC, so please err on the side of taking too much locks instead of taking too few
439 2014-10-21 12:00:34 <CodeShark> seems like we need to keep the lock throughout
440 2014-10-21 12:00:38 <wumpus> CodeShark: that can happen, and will just raise an exception
441 2014-10-21 12:01:03 <CodeShark> so that's acceptable behavior?
442 2014-10-21 12:01:52 <wumpus> there are other places where the wallet can become unlocked during an operation, see https://github.com/bitcoin/bitcoin/issues/3673
443 2014-10-21 12:02:04 <wumpus> although I'd prefer to not introduce more, if possible
444 2014-10-21 12:02:43 <wumpus> as said - this first commit should ideally just provide course locking
445 2014-10-21 12:03:10 <wumpus> not too much changes from before; later on, if performance requires it, we can always push the locks down further for specific calls
446 2014-10-21 12:03:11 <CodeShark> yeah - problem is that we need to check whether pwalletMain is NULL - and only lock if it isn't
447 2014-10-21 12:03:38 <CodeShark> so RAII is only natural in situations where we can wrap entire blocks
448 2014-10-21 12:04:03 <wumpus> or we'd need a conditionally-lock/unlock-wallet RAII.. ugh :p
449 2014-10-21 12:04:47 <CodeShark> the correct way to do it would be to have the wallet class itself manage its own mutex
450 2014-10-21 12:05:01 <hearn> another possibility is to have the wallet be "owned" by a particular thread, as in the actor model. other threads request data by scheduling closures into it. with c++11 that's probably not too painful anymore
451 2014-10-21 12:05:08 <CodeShark> and expose all atomic operations in the wallet interface
452 2014-10-21 12:05:16 <wumpus> anyhow - if you do it that way, lock the wallet for ensurewalletisunlocked, then lock it again after, make sure to test that the race condition doesn't cause a crash
453 2014-10-21 12:05:45 <wumpus> raising an exception (ie, keystore is locked) is fine
454 2014-10-21 12:06:11 <wumpus> that's why #3673 isn't a critical issue, it makes the calls fail, but doesn't cause any harm apart from that
455 2014-10-21 12:06:22 <hearn> bitcoinj has endless fun with these issues too, actually i was just narrowing the wallet lock there because some users started seeing big UI lag when the wallet became huge
456 2014-10-21 12:06:47 <wumpus> hearn: yes, an actor based model would have been best
457 2014-10-21 12:06:52 <hearn> there's probably no truly elegant or perfect way to manage concurrency in bitcoin libraries
458 2014-10-21 12:07:22 <CodeShark> it's probably not worth trying to retrofit the bitcoind wallet to support this model
459 2014-10-21 12:07:28 <wumpus> I'd rather just get rid of wallet functionality in bitcoin core
460 2014-10-21 12:07:29 <wumpus> right CodeShark
461 2014-10-21 12:08:07 <hearn> yeah
462 2014-10-21 12:08:11 <CodeShark> it wasn't designed to be very scalable - so why waste time trying to squeeze out maximum concurrency? :)
463 2014-10-21 12:08:27 <wumpus> right
464 2014-10-21 12:08:36 <wumpus> that just gives people the wrong idea
465 2014-10-21 12:08:41 <Luke-Jr> Bitcoin-Qt should be ported to another wallet first tho
466 2014-10-21 12:08:47 <hearn> well you can get away with coarser locking in Core because it only runs on desktops, and people expect old-looking desktop apps to randomly freeze up for seconds at a time :)
467 2014-10-21 12:09:02 <CodeShark> my main concern at this point is more with code cleanliness than performance
468 2014-10-21 12:09:08 <hearn> the standards are lower. if you try that on Android the OS pops up a crash dialog as punishment :(
469 2014-10-21 12:09:44 <wumpus> yes, the UI thread in android should just be the UI thread, and not do computations and such
470 2014-10-21 12:09:56 <wumpus> that's a very sensible design anyhow
471 2014-10-21 12:10:27 <CodeShark> I find it annoying when an Android app hangsâ¦and then once it does start responding, I get prompted whether I want to wait for it or close it
472 2014-10-21 12:10:27 <hearn> it's great ..... until the UI needs information from the backend :) like "what is my current receive address" or "what is my balance"
473 2014-10-21 12:10:32 <wumpus> especially if you want to do animations, you can't have that the UI thread is blocked even a few MS
474 2014-10-21 12:10:37 <CodeShark> I'd like it if I got asked WHILE it was hanging whether I want to close it
475 2014-10-21 12:10:50 <CodeShark> chrome does the same thing to me
476 2014-10-21 12:10:56 <hearn> CodeShark: it pops up after a few seconds, if you try and interact with it. so that sounds like you just got unlucky. if the app was truly stopped, it'd still appear
477 2014-10-21 12:10:58 <wumpus> indeed - you're forced to use message passing and signals
478 2014-10-21 12:11:11 <wumpus> worker threads
479 2014-10-21 12:11:29 <hearn> bitcoinj supports direct calling from the UI thread, but sometimes aschildbach still has to load things from a worker thread
480 2014-10-21 12:11:39 <hearn> for cases where it'd be too slow otherwise. typically this is only an issue with huge wallets.
481 2014-10-21 12:11:47 <hearn> because some things are unoptimised.
482 2014-10-21 12:11:49 <wumpus> firefox is really bad in this regard, if javascript gets too heavy (happens often on github), then the whole thing will just hang
483 2014-10-21 12:11:57 <wumpus> ubuntu sometimes detects that and makes the window grey
484 2014-10-21 12:12:10 <hearn> yeah. firefox is going the same way as chrome with separated renderer processes, so that should get fixed at some point
485 2014-10-21 12:13:04 <hearn> in Lighthouse i'm using a combination of actor model, locking and asynchronously replicated collections. you can get ObservableList, ObservableMap, ObservableSet etc from the backend, and register listeners on them or wire them directly to the UI. but they are guaranteed to only be updated on the UI thread. the backend just writes to these collections as normal.
486 2014-10-21 12:13:18 <hearn> then the collection implementations know how to replicate deltas from backend to frontend thread
487 2014-10-21 12:13:36 <hearn> it's worked out very nicely. the code is very clean. the observable collection implementation is very tricky but i didn't write that :)
488 2014-10-21 12:13:44 <hearn> now i wish bitcoinj itself worked that way. well, maybe one day ...
489 2014-10-21 12:14:56 <CodeShark> signrawtransaction is a bit of a challenge to do correctly with RAII
490 2014-10-21 12:15:44 <wumpus> right, your table model has to cope with the case where the user requests data but it isn't available yet, it's a bit unfortunate that qt doesn't support this or at least requires implementing all the logic yourself
491 2014-10-21 12:16:50 <wumpus> CodeShark: hmm - what about splitting function into an inner function that takes a keystore
492 2014-10-21 12:17:04 <wumpus> CodeShark: then in the outer function there is a pWallet path and an other path
493 2014-10-21 12:17:12 <CodeShark> wumpus: right - that's sensible
494 2014-10-21 12:17:13 <wumpus> both use the same inner function
495 2014-10-21 12:18:28 <CodeShark> actually, I only think keystore is used once after being assigned
496 2014-10-21 12:18:46 <CodeShark> so we could just wrap that statement in the if (pwalletMain)
497 2014-10-21 12:19:18 <wumpus> ok
498 2014-10-21 12:19:29 <CodeShark> howeverâ¦that statement is inside a loop
499 2014-10-21 12:20:04 <wumpus> hah, it's never easy
500 2014-10-21 12:20:11 <CodeShark> perhaps it's best to put the entire loop inside an inner function
501 2014-10-21 12:20:18 <wumpus> yes
502 2014-10-21 12:24:24 <CodeShark> argh - I'm almost tempted to just put everything in a try clause, enter a critical section, then leave the critical section in the catch clause and rethrow :p
503 2014-10-21 12:25:08 <CodeShark> it's ugly code - but it would at least preserve the existing behavior without having to delve into the logic
504 2014-10-21 12:26:52 <wumpus> well, in that case I'd prefer to add a lock-if-not-null RAII, it's ugly but not that ugly
505 2014-10-21 12:27:27 <CodeShark> the objective of 5107 was NOT to optimize concurrencyâ¦it was simply to push the locks down the call stack so that rpcserver doesn't have to know about them
506 2014-10-21 12:27:38 <wumpus> that's what I said
507 2014-10-21 12:28:10 <wumpus> but maybe now you understand why it wasn't done before :-)
508 2014-10-21 12:28:14 <CodeShark> lol
509 2014-10-21 12:28:52 <wumpus> it's annoying that there are methods that conditionally use the wallet anyway, ideally they would be split into a wallet and non-wallet method
510 2014-10-21 12:29:07 <CodeShark> absolutely
511 2014-10-21 12:29:33 <wumpus> (but that's what gave me the idea of the inner function...)
512 2014-10-21 12:30:04 <CodeShark> I think lock-if-not-null RAII is the best way to go for this merge, IMHO
513 2014-10-21 12:30:19 <CodeShark> preserves behavior, accomplishes the first objective
514 2014-10-21 12:30:32 <wumpus> yes, though it clutters the sync.h with something we'd rather not have
515 2014-10-21 12:30:52 <wumpus> but yes maybe it's the best option...
516 2014-10-21 12:31:01 <wumpus> at least it's very straightforward
517 2014-10-21 12:31:20 <wumpus> could always be removed later
518 2014-10-21 12:31:39 <wumpus> it's temporary :P
519 2014-10-21 12:32:20 <CodeShark> how would such a macro work? short-circuit?
520 2014-10-21 12:32:58 <wumpus> a class that aquires a critical section in the constructor and releases it in the destructor
521 2014-10-21 12:33:04 <wumpus> just with a if(pBla) around both
522 2014-10-21 12:33:23 <CodeShark> right
523 2014-10-21 12:33:54 <CodeShark> we can safely assume that pwalletMain won't be assigned in between, right?
524 2014-10-21 12:34:45 <CodeShark> or worse, destroyed
525 2014-10-21 12:35:09 <wumpus> yes
526 2014-10-21 12:35:46 <wumpus> mind that pwalletMain itself it not protected by any lock - it's supposed to stay the same until shutdown (which first tears down RPC)
527 2014-10-21 12:37:46 <CodeShark> so then usage would be something like CONDITIONAL_LOCK(pwalletMain, pwalletMain->cs_wallet);
528 2014-10-21 12:38:21 <wumpus> dunno if it makes sense to pass pwalletMain itself - just CONDITIONAL_LOCK(pwalletMain ? &pwalletMain->cs_wallet : 0)
529 2014-10-21 12:39:10 <CodeShark> ugh - lol
530 2014-10-21 12:42:08 <CodeShark> well, either way, it's certainly better than if (pcmd->threadsafe) ⦠else if (!pwalletMain) ⦠else ...
531 2014-10-21 12:42:46 <wumpus> yes - it's just that, a first step toward pushing down the lock further
532 2014-10-21 12:42:59 <wumpus> without doing anything remotely tricky or risky
533 2014-10-21 12:45:23 <CodeShark> ok, bedtime for me - I'll work on it tomorrow
534 2014-10-21 12:46:08 <CodeShark> nite
535 2014-10-21 12:46:13 <wumpus> night!
536 2014-10-21 13:41:19 <chmod755> is there a tool to search for bitcoin payments of a certain value (e.g. 1.2345678 BTC)
537 2014-10-21 13:43:06 <elichai2> chmod755, try at #bitcoin this is not the right channel
538 2014-10-21 13:45:59 <earlz> really? I would've figured that question to be on-topic
539 2014-10-21 13:47:40 <Dr-G> ^^ I'm actually looking for a parsed transaction database too
540 2014-10-21 13:48:05 <earlz> In theory, you could skim though all outputs block-by-block... but it'd be horribly slow (ie, take more than 3 hours)
541 2014-10-21 13:48:25 <wumpus> abe https://github.com/bitcoin-abe/bitcoin-abe or https://github.com/bitpay/insight can probably do things like that
542 2014-10-21 13:48:38 <earlz> yea
543 2014-10-21 13:49:10 <earlz> If nothing else, I'm sure Toshi could handle it http://toshi.io
544 2014-10-21 13:49:18 <earlz> but it also takes 200G of diskspace heh
545 2014-10-21 13:50:28 <Dr-G> do you have that thing running?
546 2014-10-21 13:51:16 <wumpus> 20G of diskspace is peanuts
547 2014-10-21 13:51:49 <earlz> no.. you'd also have to sync it up without a bootstrap.dat
548 2014-10-21 13:52:04 <earlz> If I had an extra harddrive, I'd pop an instance up but eh
549 2014-10-21 14:33:42 <sinetek> heya, i'd like to know, is bitcoin using wuille's fast secp256k1 code?
550 2014-10-21 14:43:46 <rdponticelli> sinetek: Not yet
551 2014-10-21 14:55:03 <helo> sinetek: afaik, bitcoin core will start out by using sipa's secp256k1 code for signing only initially
552 2014-10-21 14:55:29 <sinetek> ok
553 2014-10-21 14:55:39 <helo> (someone will correct me if i'm wrong)
554 2014-10-21 14:56:03 <sinetek> I'm writing my own codebase, will check his lib out shortly
555 2014-10-21 16:10:23 <gmaxwell> sinetek: What purpose is your codebase for?
556 2014-10-21 16:15:36 <sinetek> gmaxwell: I have technical objections with the official one, and want to explore experimental things that I doubt would ever get pulled in the official codebase
557 2014-10-21 16:15:45 <sinetek> namely no Boost and no OpenSSL
558 2014-10-21 16:16:16 <sinetek> so I guess my goal is more portability
559 2014-10-21 16:16:23 <sinetek> :-)
560 2014-10-21 16:17:05 <gmaxwell> sinetek: Thats regrettable.
561 2014-10-21 16:17:10 <sinetek> how come?
562 2014-10-21 16:17:59 <sinetek> in what sense?
563 2014-10-21 16:18:46 <gmaxwell> sinetek: because bitcoin is a consensus system, in the consensus parts all full nodes must behave exactly the same or they'll fork relatively to each other, which can cause funds loss (for the users, or even third parties depending on how widespread the use is). It's insanely tricky to get exact agreement right, ... in some cases due to bugs Bitcoin Core hasn't even agreed with itself (same version). The history of full node ...
564 2014-10-21 16:18:52 <gmaxwell> ... reimplementors is a history of implementations that fork over and over again.
565 2014-10-21 16:19:24 <sinetek> gmaxwell: I'm fully aware of this, and something I am still debating
566 2014-10-21 16:19:34 <sinetek> ie, making a new coin I guess?
567 2014-10-21 16:19:41 <gmaxwell> As far as boost and openssl goes... we're almost out of openssl, but elimiating is probably unsafe due to the aformentioned reasons; which is why we haven't switched out of it in bitcoin core yet (and are moving for some block rules changes which will make it safer)
568 2014-10-21 16:19:44 <sinetek> I would prefer that to accidentally forking BTC for sure.
569 2014-10-21 16:20:34 <gmaxwell> And boost is also going away in Bitcoin, but more slowly as we'll need to switch to C++11 to finish it, which breaks compatibility with older compilers.
570 2014-10-21 16:22:08 <gmaxwell> sinetek: as far as a 'new coin', well thats pretty cruddy: You'll copy our design and go compete with us with it? To what end? Money is useful almost purely due to network effects, the more fragmented cryptocurrency is the less valuable it is for everyone. Why own foo coin if maybe barcoin will come out tomorrow and be the next big thing and make your foo coins lose their value?
571 2014-10-21 16:22:24 <Arnavion> sinetek: How is not using boost going to make it "more portable" ?
572 2014-10-21 16:22:34 <gmaxwell> Sounds like you already agree with the direction we have with respect to some of the dependencies. :)
573 2014-10-21 16:22:39 <Arnavion> Alternatively, how is using boost make the current implementation less portable?
574 2014-10-21 16:23:17 <Apocalyptic> Arnavion, every additional dependency damages portability
575 2014-10-21 16:23:27 <sinetek> boost doesn't run on many platforms..
576 2014-10-21 16:23:31 <sinetek> Windows RT for one
577 2014-10-21 16:23:40 <Apocalyptic> as the dependency in question may not run/be available on your platform
578 2014-10-21 16:24:43 <Arnavion> That's just a matter of time
579 2014-10-21 16:24:49 <Arnavion> Some of boost does already work on RT
580 2014-10-21 16:25:30 <gmaxwell> Our motivations for moving away from boost are a bit different, boost is a whole lot of code, some of it kinda skechy, which is really not created with the purpose of being used inside a consensus system. Most of the parts we use are also redundant with C++11 (though, those parts are also the more portable ones which probably already work fine on random platforms)
581 2014-10-21 16:25:45 <Arnavion> Yes, that is a much more sensible reason
582 2014-10-21 16:26:15 <gmaxwell> What likely doesn't work is ASIO and threading stuff, which we don't have any plans to replace right now. (Though I'd personally like to not be using boost ASIO, it's impossible to debug).
583 2014-10-21 16:27:25 <gmaxwell> sinetek: in any case, if you're targeting windows RT you're presumably not targeting a bitcoin full node. You might want to consider using bitcoinj ... which apparently can be transpiled to run natively (hive wallet does this IIRC).
584 2014-10-21 16:27:47 <gmaxwell> Running a novel non-full-node implementation has far fewer risks today.
585 2014-10-21 16:29:35 <sinetek> I'm making it in C99, no dependency on C++/Java
586 2014-10-21 16:30:39 <Apocalyptic> nice sinetek, I always wanted to see a pure C full node
587 2014-10-21 16:30:49 <sinetek> well me too, that's why I started this
588 2014-10-21 16:31:09 <gmaxwell> There are several already.
589 2014-10-21 16:31:23 <sinetek> sure, but none suit my purpose
590 2014-10-21 16:31:57 <Apocalyptic> gmaxwell, last time you linked me to one of jgarzik's work, but it came out that there was some dependencies
591 2014-10-21 16:32:11 <Apocalyptic> I don't know of any other
592 2014-10-21 16:32:27 <gmaxwell> sure, glib, libevent, and openssl.
593 2014-10-21 16:33:38 <sinetek> also Java doesn't run on WinRT
594 2014-10-21 16:33:51 <sinetek> not that WinRT is my #1 target anyway
595 2014-10-21 16:34:19 <hearn> gmaxwell: i experimented with transpilation but never finished the work needed
596 2014-10-21 16:34:24 <hearn> gmaxwell: hive uses JNI
597 2014-10-21 16:34:37 <gmaxwell> Ah, sad.
598 2014-10-21 16:34:38 <hearn> sinetek: trying to write a bitcoin node in C is suicide. strongly advise you use a safe language.
599 2014-10-21 16:34:55 <sinetek> heh
600 2014-10-21 16:35:07 <gmaxwell> sinetek: I was talking about transpiled, source to source translation. (Sounds weird, sure,... but the HTML parser in firefox is done this way.)
601 2014-10-21 16:35:45 <gmaxwell> hearn: language safty is the least of the issues. Bitcoin isn't usually hard for language safty reasons compared to other network protocols. :)
602 2014-10-21 16:35:51 <hearn> you can transpile bitcoinj and the supporting libraries to C++11, with three major caveats
603 2014-10-21 16:35:56 <sinetek> I'm sure you can't transpile the java stdlib though
604 2014-10-21 16:36:02 <hearn> 1) you need to use the boehm gc, of course
605 2014-10-21 16:36:19 <sinetek> sounds more complicated than actually writing the thing
606 2014-10-21 16:36:29 <coryfields_> sinetek: fyi, your goals are the same as mine, and i'm actively working to get them into core (and the changes have been welcomed so far)
607 2014-10-21 16:36:29 <hearn> 2) the runtime stubs need to be filled out, e.g. anywhere java has a "native" method that would call into the JVM, you get an empty C++11 method that you get to fill out. for things like threading, file io, etc, so bind it to whatever platform api you are using there
608 2014-10-21 16:36:37 <coryfields_> sinetek: so i'd suggest you help with that effort instead
609 2014-10-21 16:36:49 <gmaxwell> sinetek: hah. You're really underestimating how hard implementing a consnsitent full node is. :)
610 2014-10-21 16:36:51 <hearn> 3) the generated code lacks array bounds checking, so is insecure
611 2014-10-21 16:37:20 <hearn> oh, and of course, the generated code is huge. but that's because the libraries functionality is huge. a good dead code elimination pass in your linker is required ....
612 2014-10-21 16:37:21 <gmaxwell> hearn: ah (3) is kind of an ouch for java code not written with this target in mind.
613 2014-10-21 16:37:31 <sinetek> coryfields_: Yes, with the caveat of removing the C++ dependency
614 2014-10-21 16:37:50 <hearn> gmaxwell: yeah would need to upgrade the transpiler to add the bounds checks, i think
615 2014-10-21 16:37:55 <coryfields_> sinetek: to what end?
616 2014-10-21 16:38:20 <gmaxwell> sinetek: It's my hope that we get libconsensus to eventually be C only, and not the whole program... but thats a longer road.
617 2014-10-21 16:38:43 <gmaxwell> sinetek: also odd to hear you talk about C99 and windows RT... MSVC is not a C99 compiler, and only supports parts of C99. :(
618 2014-10-21 16:39:10 <gmaxwell> (MSVC is a C++ compiler and seems to only support C99 features so long as they're a strict subset of C++)
619 2014-10-21 16:39:18 <coryfields_> winrt seems like a strange target to even consider in such a conversation
620 2014-10-21 16:39:41 <coryfields_> not that the end-result shouldn't be portable, but still..
621 2014-10-21 16:40:32 <Arnavion> gmaxwell: They did add a lot of C99-only features that aren't also in C++
622 2014-10-21 16:40:44 <Arnavion> gmaxwell: Advertised as "enough to compile ffmpeg and other popular OSS"
623 2014-10-21 16:41:14 <gmaxwell> since when? ffmpeg uses a crazy script to convert their c99 to c89+alloca. :)
624 2014-10-21 16:41:24 <sinetek> yeah some C features are ill supported :(
625 2014-10-21 16:41:37 <sinetek> no compiler supports C11 at all afaik
626 2014-10-21 16:42:04 <gmaxwell> sinetek: in any case, your help in improving bitcoin core's portability would be welcome (if you have the patience to go about it in the careful manner needed); otherwise, you have my condolences in advance for your many forking failures with your illadvised reimplementation of consensus rules. :)
627 2014-10-21 16:42:31 <gmaxwell> sinetek: huh? https://gcc.gnu.org/projects/cxx0x.html
628 2014-10-21 16:42:40 <Arnavion> gmaxwell: Since VS 2013
629 2014-10-21 16:42:57 <Arnavion> >FFmpeg can be built with MSVC 2012 or earlier using a C99-to-C89 conversion utility and wrapper, or with MSVC 2013 and ICL natively.
630 2014-10-21 16:43:03 <Arnavion> From https://www.ffmpeg.org/platform.html#Microsoft-Visual-C_002b_002b-or-Intel-C_002b_002b-Compiler-for-Windows
631 2014-10-21 16:43:10 <sinetek> gmaxwell: yes? that's C++
632 2014-10-21 16:43:33 <gmaxwell> sinetek: ah, sorry, misread.
633 2014-10-21 16:43:58 <gmaxwell> (I thought you were saying that in response to our comments that we planning on replacing boostisms with C++11)
634 2014-10-21 16:44:31 <gmaxwell> Arnavion: Cool.
635 2014-10-21 16:44:57 <hearn> sinetek: more generally, you are very likely to have buffer overflows or memory corruptions of various kinds of you do it in C