1 2014-07-28 00:17:08 <sipa> jgarzik: bitcoin-tx stripped is still 850 kB here, crazy
2 2014-07-28 00:17:28 <sipa> there's a ton of util code that's not all that generic really
3 2014-07-28 00:17:34 <gmaxwell> 0_o
4 2014-07-28 00:17:59 <sipa> as the script code isn't split between validation and signing, it does include a full script interpreter too...
5 2014-07-28 00:18:26 <sipa> which pulls in all hash algorithms
6 2014-07-28 00:19:10 <Luke-Jr> sipa: we should just make .so's and get it over with. we don't need to keep an ABI..
7 2014-07-28 00:19:48 <sipa> Luke-Jr: i don't really care about the binary size itself; i'm surprised by how much code we're still depending on
8 2014-07-28 00:20:11 <Luke-Jr> sipa: did the linker remove unused code/
9 2014-07-28 00:20:13 <Luke-Jr> ?
10 2014-07-28 00:21:06 <sipa> i think our current library definitions for util and core are pretty well following dependency boundaries already
11 2014-07-28 00:21:11 <jgarzik> sipa, add -flto
12 2014-07-28 00:21:42 <jgarzik> it will still be too-large, though, agreed on that point
13 2014-07-28 00:24:06 <jgarzik> sipa, -ffunction-sections might help with C++ too
14 2014-07-28 00:25:27 <sipa> jgarzik: same size
15 2014-07-28 00:25:39 <sipa> 4 kB larger, actually
16 2014-07-28 00:26:14 <jgarzik> sipa, this is size(1) or ls(1) ? :)
17 2014-07-28 00:26:22 <sipa> ls
18 2014-07-28 00:26:25 <sipa> after strip
19 2014-07-28 00:28:34 <sipa> 848 with flto + function sections
20 2014-07-28 00:28:57 <jgarzik> huh
21 2014-07-28 00:29:06 <jgarzik> surprising
22 2014-07-28 00:29:53 <sipa> flto may result in cross-object inlining, which increases size :)
23 2014-07-28 00:30:18 <jgarzik> sipa, -Os
24 2014-07-28 00:30:31 <jgarzik> sipa, for linker and compiler invocation both
25 2014-07-28 00:31:03 <jgarzik> sipa, and size(1) is always more accurate
26 2014-07-28 00:31:14 <sipa> how do i do that with ./configure?
27 2014-07-28 00:31:34 <sipa> ah, LDFLAGS
28 2014-07-28 00:31:55 <jgarzik> sipa, CXXFLAGS="-Os" LDFLAGS="-Os" ./configure
29 2014-07-28 00:31:56 <jgarzik> iirc
30 2014-07-28 00:32:31 <jgarzik> ACTION is pretty brain dead after 8-hour-long meeting on satellites
31 2014-07-28 00:32:40 <Luke-Jr> usually the ./configure goes first?
32 2014-07-28 00:32:48 <Luke-Jr> eg ./configure CXXFLAGS="-Os" LDFLAGS="-Os"
33 2014-07-28 00:33:05 <jcorgan> jgarzik: i thought the first two hours were interesting :)
34 2014-07-28 00:33:30 <jgarzik> Luke-Jr, that's how you set environment variables before an invocation. the latter approach (configure SYM SYM) depends on the program taking some action with those arguments. not as reliable.
35 2014-07-28 00:33:44 <jgarzik> Luke-Jr, FOO=BAR ./program
36 2014-07-28 00:33:49 <jgarzik> sets FOO=BAR in ./program's environment
37 2014-07-28 00:33:55 <sipa> ah, i didn't pass -flto to the linker of course
38 2014-07-28 00:34:00 <sipa> 590 kB
39 2014-07-28 00:34:47 <Luke-Jr> jgarzik: seems to me that's more likely than it reading the environment, but maybe both work
40 2014-07-28 00:35:21 <jgarzik> Luke-Jr, "FOO=BAR ./configure" always worked since day one. "./configure FOO=BAR" is new and doesn't work with all configure scripts
41 2014-07-28 00:36:02 <Luke-Jr> hm
42 2014-07-28 00:36:05 <sipa> -Os + lto + function sections: 492 kB
43 2014-07-28 00:36:25 <jgarzik> sipa, still seems large for what it does
44 2014-07-28 00:36:33 <jgarzik> ACTION blames templates ;p
45 2014-07-28 00:37:21 <sipa> ACTION blames including: hash function, script evaluation, rpc protocol (?!), netbase (!?!?), ...
46 2014-07-28 00:37:23 <berndj-blackout> jgarzik, FOO=BAR ./configure didn't/doesn't preserve the variable into config.status. whether you AC_SUBST([FOO]) also matters
47 2014-07-28 00:38:03 <jgarzik> -ffunction-sections permits splitting foo.o up into one section per function, as the name implies. which permits the linker to pick apart a .o and only use what is needed.
48 2014-07-28 00:38:12 <jgarzik> open question whether it works well/at all on C++
49 2014-07-28 00:39:14 <sipa> HMAC_SHA512 is included, which shouldn't be needed
50 2014-07-28 00:39:28 <sipa> it's only for bip32 style derivation, which is invoked from nowhere
51 2014-07-28 00:40:11 <sipa> huh
52 2014-07-28 00:40:22 <sipa> only the constructor and the finalizer are included, but not the ::Write method
53 2014-07-28 00:41:34 <sipa> crap, my internal clock is still on eastcoast time it seems :(
54 2014-07-28 00:52:45 <sipa> jgarzik: stripped sizes of included .o files, sorted: bitcoin-tx 341kB, util 212kB, script 161kB, rpcprotocol 161kB, netbase 88kB, ...
55 2014-07-28 00:53:27 <sipa> util should be broken up in much smaller pieces (most of which bitcoin-tx doesn't need), script needs to be broken up in script-core, script-eval and script-sign, ...
56 2014-07-28 00:53:35 <sipa> no idea what we need rpcprotocol for
57 2014-07-28 00:53:36 <sipa> or netbase
58 2014-07-28 02:00:27 <BlueMatt> sipa: oh, looks like no one pinged me and I entirely missed a release :(
59 2014-07-28 02:00:28 <BlueMatt> wow thats bad
60 2014-07-28 02:04:06 <cfields> jgarzik: fyi, ./configure FOO=BAR is almost always preferred, as it is preserved between invocations
61 2014-07-28 02:05:14 <cfields> oh, heh, exactly what berndj-blackout said :)
62 2014-07-28 02:06:15 <cfields> jgarzik: also, ffunction-sections needs to be used with gc-sections, otherwise the extras won't be stripped
63 2014-07-28 02:32:23 <gmaxwell> BlueMatt: I may be remembering incorrectly, but I believe that was just a trivial respin for osx fixes, still probably good to catch up, but I don't think it matters.
64 2014-07-28 02:33:06 <BlueMatt> ok, well that makes me feel a bit better
65 2014-07-28 04:33:48 <MBeth> hi. can someone tell me what's the point of the superfast bitcoin-only tor exit node? https://twitter.com/sigwinch/status/493541355709546496
66 2014-07-28 04:37:06 <gmaxwell> MBeth: appears to have no point; if you note: it doesn't have the exit flags, so tor nodes won't use it as an exit (I'm even unable to get tor here to manually exit via it). The traffic stats you're seeing also include relay bandwidth not exit bandwidth.
67 2014-07-28 04:38:06 <gmaxwell> The system appears to be connecting out to many (all?) ipv4 reachable nodes and just listening without relaying transactions. It's lying about what software it's running, presumably to monitor transaction origins (similar to what bc.i doesâ perhaps its even operated by bc.i)
68 2014-07-28 04:38:58 <MBeth> hm, still very weird
69 2014-07-28 04:39:02 <MBeth> thank you!
70 2014-07-28 04:39:11 <gmaxwell> I'd guess the tor exitness is intended to monitor tor using bitcoin users too perhaps to intercept bitcoinj traffic, but its not working as an exit. Alternatively the running of an exit might be an intended cover for why its making so many outbound connections, but that cover fails because of the lack of an exit flag and the detectably weird behavior.
71 2014-07-28 04:39:42 <gmaxwell> It's not new that people have run spy nodes that attempt to monitor the origins of all transactions. amusingly, on my own systems I'd already iptables banned that subnet due to crappy behavior from it in the past.
72 2014-07-28 04:39:53 <MBeth> well my first assumption was that it is somehow designed to listen on darkmarket tx
73 2014-07-28 04:49:24 <earlz> So, I'm looking at how to create a non-standard transaction with bitcoinjs-lib... anyone have any hints on how to go about that? I'm not seeing how to manually specify opcodes and such in a transaction using it
74 2014-07-28 04:50:50 <MBeth> gmaxwell: thank you. I'm really just curious why someone would do this
75 2014-07-28 04:51:49 <MBeth> the general weirdness of it, combined with the pulled blackhat talk would make for a wonderful conspiracy theory
76 2014-07-28 06:47:09 <wumpus> BlueMatt: 0.9.2.1 you mean? that was released a few hours after 0.9.2, with a macosx font fix and a rare crash on exit fixed, no big deal
77 2014-07-28 07:35:50 <SomeoneWeird> wumpus: people were complaining it wasn't up to date
78 2014-07-28 07:43:16 <wumpus> people are always complaining
79 2014-07-28 07:48:49 <SomeoneWeird> that they are
80 2014-07-28 08:04:14 <jcorgan> wumpus: understand your comments on #4594--will update in the AM
81 2014-07-28 08:46:25 <copain_reac> i'm having trouble building bitcoind on osx mavericks using brew
82 2014-07-28 08:46:39 <copain_reac> getting "./json/json_spirit_value.h:20:10: fatal error: 'boost/config.hpp' file not found"
83 2014-07-28 08:47:17 <copain_reac> despite boost being installed correctly with brew
84 2014-07-28 08:48:31 <drizztbsd> copain_reac: set the correct -I
85 2014-07-28 08:49:10 <copain_reac> drizztbsd: i tried "make -I/usr/local/Cellar/boost/1.55.0_2"
86 2014-07-28 08:49:18 <copain_reac> didn't appear to help
87 2014-07-28 08:49:28 <drizztbsd> it's not the correct way
88 2014-07-28 08:49:31 <drizztbsd> see ./configure --help
89 2014-07-28 08:49:44 <copain_reac> oh
90 2014-07-28 08:49:52 <copain_reac> i'll try that tyvm
91 2014-07-28 08:49:54 <drizztbsd> hint: CXXFLAGS
92 2014-07-28 08:50:07 <drizztbsd> and LDFLAGS for the -L part
93 2014-07-28 08:50:41 <copain_reac> yeah, i tried with that too, my problem was that i wasn't passing the options at the correct moments :)
94 2014-07-28 09:26:45 <copain_reac> so i tried this "BOOST_ROOT=/usr/local/Cellar/boost/1.55.0_2 ./configure --with-boost"
95 2014-07-28 09:26:49 <copain_reac> didn't work either
96 2014-07-28 09:45:58 <mn3monic> hello, any advice to track confirmations on non-wallet transactions ?
97 2014-07-28 09:59:06 <sipa> use a watch-only wallet
98 2014-07-28 13:52:06 <maraoz> anyone got example tx input scripts for all types? (p2pkh, p2pk, p2sh, regular multisig).
99 2014-07-28 13:52:59 <helo> maraoz: i'd poke around the tests in src/test/
100 2014-07-28 13:55:32 <maraoz> thx helo
101 2014-07-28 13:55:38 <helo> data/ in there has some json files with transaction data
102 2014-07-28 13:57:09 <melvster> yay trezor has shipped http://satoshilabs.com/news/2014-07-28-trezor-shipped/
103 2014-07-28 13:57:18 <melvster> anyone here going to the launch party?
104 2014-07-28 14:03:31 <hearn> depends when it is - i might g
105 2014-07-28 14:03:33 <hearn> go
106 2014-07-28 14:26:19 <hearn> wumpus: ping
107 2014-07-28 14:36:00 <ribasushi> do bitcoin devs have a particular opinion on the gocoin client?
108 2014-07-28 14:36:15 <ribasushi> (if there is a forum thread I've missed - links welcome)
109 2014-07-28 14:59:18 <ken01203> anyone not busy able to msg me, need some help with upgrading a btc clone wallet to 9.2.
110 2014-07-28 15:07:47 <jcorgan> wumpus: updated #4594 from your feedback
111 2014-07-28 15:47:48 <earlz> So, how does one construct a non-standard transaction, like a any-can-spend transaction ro some such?
112 2014-07-28 15:48:08 <earlz> I know the pool to broadcast it at, but not how to actually get the hex for it
113 2014-07-28 15:50:21 <sipa> use a hex editor? :0
114 2014-07-28 15:50:49 <sipa> (sorry, not a very useful answer, but you'll need some tool for building transactions, and if they're sufficiently non-standard, you'll likely need to do it manually entirely)
115 2014-07-28 15:52:32 <earlz> is there no library to help construct them? Like to specify opcodes to put in and all?
116 2014-07-28 15:54:59 <jrick> http://godoc.org/github.com/conformal/btcscript#ScriptBuilder will let you build the scripts, and then it's just adding them to inputs or outputs to the tx you're creating
117 2014-07-28 15:56:40 <earlz> go? odd lol
118 2014-07-28 15:56:51 <earlz> guess it's an excuse to learn some of it
119 2014-07-28 15:57:14 <sipa> i'm sure BitcoinJ has something similar
120 2014-07-28 15:57:22 <jrick> yeah wouldn't surprise me
121 2014-07-28 15:57:43 <sipa> and there's python libraries for sure too
122 2014-07-28 15:58:12 <sipa> bitcoin core will soon have a bitcoin-tx tool which can do some transaction building
123 2014-07-28 15:58:41 <drizztbsd> sipa: why? :P
124 2014-07-28 15:58:59 <sipa> mostly as an exercise to modularize our code
125 2014-07-28 16:00:44 <earlz> sipa: is there any in-progress code for that?
126 2014-07-28 16:01:00 <sipa> yes, https://github.com/bitcoin/bitcoin/pull/4332
127 2014-07-28 16:01:25 <earlz> it'd be nice if signing was easier than "just follow this 19 step guide" lol
128 2014-07-28 16:02:01 <sipa> bitcoin-tx can sign
129 2014-07-28 16:02:12 <sipa> but only if it recognizes the script, obviously
130 2014-07-28 16:02:53 <earlz> yea, that still doesn't seem like it'll allow you to use op codes without looking up the hex for it
131 2014-07-28 16:02:56 <jrick> why does a tx creation tool care about the network?
132 2014-07-28 16:02:59 <earlz> need like a script assembler heh
133 2014-07-28 16:03:06 <sipa> jrick: it doesn't?
134 2014-07-28 16:03:13 <sipa> earlz: it has a script assembler
135 2014-07-28 16:03:22 <jrick> sipa: that tool in the pr has -regtest and -testnet options
136 2014-07-28 16:03:27 <sipa> jrick: ah, for addresses
137 2014-07-28 16:04:04 <jrick> I suppose, but you could gather the intended network from just the address
138 2014-07-28 16:04:11 <jrick> and bail if you start crossing networks
139 2014-07-28 16:04:24 <sipa> true, though the base libraries don't work that way
140 2014-07-28 16:04:34 <sipa> but yes, it wouldn't be too hard
141 2014-07-28 16:04:43 <davec__> blows me away some people still think Go is "odd". Everything indicates it is steadily replacing Python. Ohoh shows it is now over 10% of all new open source projets, etc.
142 2014-07-28 16:04:55 <davec__> Ohloh*
143 2014-07-28 16:04:55 <wumpus> hearn: pong
144 2014-07-28 16:05:26 <wumpus> jcorgan: great! thanks for reviving the zeromq work
145 2014-07-28 16:05:37 <wumpus> davec__: it is odd, but so is c++
146 2014-07-28 16:05:42 <drizztbsd> zeromq? for what?
147 2014-07-28 16:06:59 <wumpus> drizztbsd: notification of block/transactions
148 2014-07-28 16:08:15 <jcorgan> PR #4594 adds publication of validated blocks and transactions over a ZMQ PUB socket
149 2014-07-28 16:11:27 <jcorgan> regarding the zmq dependency, i'll make it so that 1) if libzmq-dev is not detected, the feature is disabled, 2) have a --disable-zmq to manually disable it, and 3) all conditionalized code would be isolated to zmqports.cpp.
150 2014-07-28 16:11:40 <wumpus> jcorgan: ACK
151 2014-07-28 16:12:33 <drizztbsd> we should remove bdb dependence :P
152 2014-07-28 16:12:42 <sipa> there's --disable-wallet for that
153 2014-07-28 16:12:43 <wumpus> drizztbsd: try configure --disable-wallet
154 2014-07-28 16:12:54 <wumpus> jcorgan: I'm thinking it should be with[out]-zeromq
155 2014-07-28 16:12:56 <drizztbsd> ok, but you don't support wallet.dat
156 2014-07-28 16:13:07 <wumpus> jcorgan: as that's usually used for switching on/off usage of libraries
157 2014-07-28 16:13:12 <sipa> wallets are highly overrated anyway
158 2014-07-28 16:13:26 <wumpus> then use an external wallet
159 2014-07-28 16:13:38 <jcorgan> wumpus: sure. i'll have to bone up on my autotools though :)
160 2014-07-28 16:13:54 <drizztbsd> what is the utility of bitcoin-core without wallet?
161 2014-07-28 16:14:14 <sipa> drizztbsd: core network router
162 2014-07-28 16:14:20 <drizztbsd> oh ok, only p2p
163 2014-07-28 16:14:23 <jcorgan> a trusted source for blockchain data?
164 2014-07-28 16:14:42 <sipa> p2p and rpc
165 2014-07-28 16:14:53 <jcorgan> (and now zmq :)
166 2014-07-28 16:15:15 <sipa> imho, that should be an external program as well
167 2014-07-28 16:15:30 <sipa> but let's not block useful optional functionality right now because of that wish
168 2014-07-28 16:20:18 <wumpus> jcorgan: well IMO zeromq is the best tool for this job, you could do other things like add a listen socket that publishes events in some format, or websocket, but zeromq is quickly becoming a standard for low-level notification (and all those other things could be built on top)
169 2014-07-28 16:22:40 <home_jg> <jcorgan> regarding the zmq dependency, i'll make it so that 1) if libzmq-dev is not detected, the feature is disabled, 2) have a --disable-zmq to manually disable it, and 3) all conditionalized code would be isolated to zmqports.cpp.
170 2014-07-28 16:22:43 <home_jg> << that's fine
171 2014-07-28 16:23:00 <wumpus> drizztbsd: it provides some extra security for SPV wallets that connect to it, for example your own node won't lie by omission :)
172 2014-07-28 16:23:22 <home_jg> wumpus, sipa: I just disagree with situations that obviously create bitrot. Signals and conditional linking do not contribute to bitrot, as the code is always built [if the lib is present on system] by default.
173 2014-07-28 16:23:33 <home_jg> (and obviously create ugly code, like ifdefs)
174 2014-07-28 16:23:54 <sipa> home_jg: i agree, but i find having bitcoin core be low-dependecy more important
175 2014-07-28 16:23:56 <home_jg> ENABLE_WALLET is really our only ugly-yet-unavoidable ifdef. Not concerned that bitrot will occur there, as long as Bitcoin-Qt remains alive etc.
176 2014-07-28 16:24:11 <wumpus> home_jg: IMO your viewpoint on this is too extreme. Obviousl I agree that the code shouldn't be full of levels of ifdefs
177 2014-07-28 16:24:20 <home_jg> sipa, optional dependencies are fine
178 2014-07-28 16:24:26 <home_jg> wumpus, what specifically is too extreme?
179 2014-07-28 16:24:36 <home_jg> sipa, ie. miniupnp etc.
180 2014-07-28 16:24:36 <sipa> i don't understand you standpoint in that case
181 2014-07-28 16:24:49 <sipa> optional dependencies imply not-always-compiled code
182 2014-07-28 16:26:32 <home_jg> sipa, yes. enabling by default is sufficient, provided that some other bitcoin devs have zmq-dev pkg on their system.
183 2014-07-28 16:27:17 <wumpus> home_jg: I've tried to reduce #ifdef WALLET by a lot, there's still some places it is required, for example RPC calls that act differently depending on whether the wallet is enabled or not
184 2014-07-28 16:27:27 <wumpus> home_jg: over time we need to split those up
185 2014-07-28 16:28:06 <gmaxwell> I'm generally not a fan of code which silently compiles in different ways which result in functional differences. "Why isn't X workting?" "Oh, you need to start over".
186 2014-07-28 16:28:06 <home_jg> wumpus, agree, though in general that conditional compilation is the lowest priority [for the reason that nearly all the main devs do compile the wallet code]
187 2014-07-28 16:28:43 <wumpus> home_jg: I use builds with and without wallet in different places
188 2014-07-28 16:29:06 <sipa> home_jg: i always use --disable-wallet :)
189 2014-07-28 16:29:07 <home_jg> wumpus, nod. that is precisely why it is low priority.
190 2014-07-28 16:29:09 <gmaxwell> (for ZMQ it's probably the least of all evils; ... but e.g. we should never have 'library' features in our own libraries where the library's user visible features are based on compiletime detection roulette)
191 2014-07-28 16:29:14 <drizztbsd> wumpus: for electrum too?
192 2014-07-28 16:29:35 <sipa> drizztbsd: if they want to
193 2014-07-28 16:29:39 <home_jg> gmaxwell, sure. But IME the outcome is -worse- if the alternative is simply long lived, rarely compiled code buried inside #ifdef
194 2014-07-28 16:29:48 <wumpus> home_jg: the parts of AppInit2() that depend on ENABLE_WALLET should ideally be split off to functions that are no-ops with wallet disabled, I think it's the worst offender
195 2014-07-28 16:29:56 <home_jg> signals are a fine solution to the problem
196 2014-07-28 16:30:00 <home_jg> wrappers are not the only solution
197 2014-07-28 16:30:20 <gmaxwell> home_jg: Right, though in the long term, perhaps we'll just take ZMQ as a dependency.
198 2014-07-28 16:30:31 <sipa> home_jg: what we see with -DDEBUG_LOCKORDER for example...
199 2014-07-28 16:30:33 <home_jg> optional dependency, like miniupnp
200 2014-07-28 16:31:12 <sipa> while we're at it, maybe deprecate -blocknotify etc when ZMQ is in, and provide a python tool that can run commands based on these...
201 2014-07-28 16:31:14 <home_jg> sipa, that's one reason why the kernel went to runtime code rewriting. you can adjust lock behavior at boot in modern kernels.
202 2014-07-28 16:31:27 <gmaxwell> sipa: yea, my thoughts too. Eventually ZMQ could replace that.
203 2014-07-28 16:31:36 <home_jg> sipa, in general, "sockets > executing a program"
204 2014-07-28 16:31:42 <sipa> oh i fully agree
205 2014-07-28 16:31:43 <home_jg> no forking for network notifications
206 2014-07-28 16:31:52 <home_jg> N listeners
207 2014-07-28 16:32:09 <gmaxwell> (if you note, I expressed horror on the introduction of those forking things)
208 2014-07-28 16:32:20 <wumpus> yes, I agree, all the -notifies should be replaced by ZMQ
209 2014-07-28 16:32:25 <gmaxwell> (esp wallet notify, block seemed, "confined" enough)
210 2014-07-28 16:32:36 <home_jg> exec sucks, ZMQ is good
211 2014-07-28 16:32:42 <home_jg> unclear is ZMQ is best path
212 2014-07-28 16:32:58 <home_jg> good as option; as sole path forward?
213 2014-07-28 16:33:00 <gmaxwell> so in any case, I think auto is fine now, but as a direction it should move towards mandatory assuming that zmq support matures and is widely used.
214 2014-07-28 16:33:08 <wumpus> I'd also like to allow command submission by zmq at some point, instead of through HTTP
215 2014-07-28 16:33:15 <home_jg> gmaxwell, zmq low level protocol still sucks ;p
216 2014-07-28 16:33:18 <wumpus> but it obviously needs to be authenticated then
217 2014-07-28 16:33:30 <gmaxwell> because compile-time-feature-roulette is bad, but less bad than untested code.
218 2014-07-28 16:33:30 <home_jg> do _not_ use ZMQ for anything non-public
219 2014-07-28 16:33:47 <home_jg> if you want to add authentication, don't use ZMQ
220 2014-07-28 16:34:10 <wumpus> ok, never mind then
221 2014-07-28 16:34:15 <wumpus> only use it for notification
222 2014-07-28 16:34:39 <home_jg> yah, not security. blockchain/tx/etc. notification, one-to-many, it's great
223 2014-07-28 16:35:06 <gmaxwell> home_jg: would it be better if later versions of ZMQ added reasonable auth? (I have no opinion on ZMQ, beyond past expierence that toolkits for things like this are usually awfulâ but also that -notify is horiffic)
224 2014-07-28 16:35:25 <wumpus> well at least zmq is standardized
225 2014-07-28 16:35:26 <home_jg> gmaxwell, you have to rev the low level protocol
226 2014-07-28 16:35:36 <home_jg> analogy: hard fork
227 2014-07-28 16:35:39 <wumpus> I would be against rolling anything ourselves here
228 2014-07-28 16:35:44 <sipa> i thought authentication was an explicit non-goal for zm
229 2014-07-28 16:35:46 <sipa> i thought authentication was an explicit non-goal for zmq
230 2014-07-28 16:35:56 <gmaxwell> :(
231 2014-07-28 16:36:20 <gmaxwell> wumpus: I think its standardized in the sense of widely used and compatible with itself.
232 2014-07-28 16:36:40 <wumpus> there's this curvecp stuff for zmq that implements authentication AFAIK
233 2014-07-28 16:36:44 <home_jg> Well, if not rev the low level protocol, you can treat it as an IP tunnel
234 2014-07-28 16:37:00 <wumpus> gmaxwell: yes
235 2014-07-28 16:37:01 <home_jg> add your own message ordering guarantees, security guarantees, MITM guarantees
236 2014-07-28 16:37:03 <jcorgan> sipa: iirc, zmq 4.x is adding authentication, but i say we avoid all of that
237 2014-07-28 16:37:06 <gmaxwell> I'd sure love for there to be a faster interface than the rpc for things like getblock / getrawtransaction / etc. right now collecting data from the blockchain via rpc is painfully slow.
238 2014-07-28 16:37:17 <home_jg> yeah, rpc sucks for binary
239 2014-07-28 16:37:23 <wumpus> gmaxwell: IMO zeromq would be great for that
240 2014-07-28 16:37:36 <wumpus> also, requesting that data doesn't even need to be authenticated
241 2014-07-28 16:37:51 <gmaxwell> It's true, just not stuff you normally want to serve to the internet.
242 2014-07-28 16:37:55 <wumpus> getblock gettransaction etc is public information
243 2014-07-28 16:38:02 <home_jg> I think zmq is fine for public information
244 2014-07-28 16:38:03 <wumpus> you don't want to serve zeromq to the internet
245 2014-07-28 16:38:13 <wumpus> isn't it local-only by default?
246 2014-07-28 16:38:26 <jcorgan> it's whatever you specify
247 2014-07-28 16:38:40 <jcorgan> default is not to listen at all
248 2014-07-28 16:39:10 <wumpus> well having an option to serve over the network is fine, for people with a local network that know what they're doing
249 2014-07-28 16:39:17 <gmaxwell> whats general software security like for ZMQ? if I give a malicious party access and they send random bits in, are they going to scribble over your process ram?
250 2014-07-28 16:39:23 <jcorgan> let me get this current stuff reworked as discussed, so people can actually test it
251 2014-07-28 16:39:30 <wumpus> but an open port over the internet is not good w/ ZMQ
252 2014-07-28 16:39:31 <jcorgan> gmaxwell: no
253 2014-07-28 16:39:44 <home_jg> <jcorgan> regarding the zmq dependency, i'll make it so that 1) if libzmq-dev is not detected, the feature is disabled, 2) have a --disable-zmq to manually disable it, and 3) all conditionalized code would be isolated to zmqports.cpp.
254 2014-07-28 16:39:45 <home_jg> ACK
255 2014-07-28 16:39:49 <wumpus> gmaxwell: well last time I read into ZMQ it certainly wasn't battle-hardened enough to serve stuff over the internet
256 2014-07-28 16:39:59 <gmaxwell> jcorgan: ± bugs of course, okay, good to hear that its at least not 'intentionally' insecure.
257 2014-07-28 16:40:08 <home_jg> jcorgan, signals are a good approach, as sipa indicated.
258 2014-07-28 16:40:15 <home_jg> signals gives us the ability to make the core code generic
259 2014-07-28 16:40:16 <jcorgan> one step at a time
260 2014-07-28 16:40:31 <home_jg> and hook in specialized features (-blocknotify | 0mq)
261 2014-07-28 16:40:31 <wumpus> I do think we should choose one (out-of-process) notification mechanism
262 2014-07-28 16:41:03 <jcorgan> we're over complicating this, i think
263 2014-07-28 16:41:17 <home_jg> with signals, who cares? You mark the events in core code, then let users write plugins.
264 2014-07-28 16:41:22 <wumpus> the others should be built on top, with external processe that adapt the event from one type to another
265 2014-07-28 16:41:51 <jcorgan> ACTION leaves to go update the code before it completely morphs away from him :)
266 2014-07-28 16:42:15 <home_jg> with signals, you can (a) build in a home grown mechanism, (b) use existing -blocknotify for compatibility, (c) use 0mq, ...
267 2014-07-28 16:42:43 <home_jg> wean users off of -blocknotify, then delete it, without touching core code.
268 2014-07-28 16:42:44 <gmaxwell> home_jg: your choice of relatively untested bad solutions! :P
269 2014-07-28 16:42:56 <gmaxwell> but I do like it as a migration out path, I suppose.
270 2014-07-28 16:43:27 <home_jg> I do wonder how much call there is for notifications?
271 2014-07-28 16:43:34 <home_jg> Many people seem to do the half-a-node thing.
272 2014-07-28 16:43:58 <home_jg> ie. you already have a notification socket, if you simply connect to port 8333
273 2014-07-28 16:44:38 <home_jg> also, related, notification interface in general does not tell you much if anything about a reorg
274 2014-07-28 16:45:01 <home_jg> every user of the interface must include code to divine the existence of a reorg
275 2014-07-28 16:45:19 <gmaxwell> home_jg: well, it's used to trigger things like listsinceblock.
276 2014-07-28 16:45:22 <jcorgan> so we can define and add an event topic
277 2014-07-28 16:46:33 <jcorgan> but if i rework the PR to deal with the dependency handling, can we get this "simplest thing that could possibly work" merged before we design ourselves into something that won't happen for a long time?
278 2014-07-28 16:47:20 <home_jg> jcorgan, sure
279 2014-07-28 16:47:55 <home_jg> I think the main change I want right now (which need not block the 0mq PR) is specifically to turn this into a signal,
280 2014-07-28 16:48:08 <home_jg> boost::replace_all(strCmd, "%s", hashNewTip.GetHex());
281 2014-07-28 16:48:08 <home_jg> boost::thread t(runCommand, strCmd); // thread runs free
282 2014-07-28 16:48:08 <home_jg> if (!strCmd.empty()) {
283 2014-07-28 16:48:08 <home_jg> std::string strCmd = GetArg("-blocknotify", "");
284 2014-07-28 16:48:31 <home_jg> and get that out of main.cpp
285 2014-07-28 16:49:26 <home_jg> hmmmmmmm
286 2014-07-28 16:49:37 <jcorgan> with the zmq port, you just connect and subscribe to the "BLK" topic, and get them sent to you
287 2014-07-28 16:49:38 <home_jg> 0mq v4 has a new wire level protocol to go along with the crypto.
288 2014-07-28 16:49:52 <home_jg> promising
289 2014-07-28 16:49:56 <wumpus> jcorgan: just go on with this, we're talking about the future not now
290 2014-07-28 16:50:04 <jcorgan> thanks :)
291 2014-07-28 16:50:07 <wumpus> home_jg: yep, based on curvecp afaik
292 2014-07-28 16:50:19 <wumpus> home_jg: agree on replacing that with a signal
293 2014-07-28 16:50:45 <gmaxwell> "oh god, another cryptographic protocol to review ... otoh, thank god its not TLS"
294 2014-07-28 16:52:04 <wumpus> home_jg: and yes - better notification on reorgs would be great
295 2014-07-28 16:52:17 <wumpus> with zmq it's easy to add topics
296 2014-07-28 16:52:37 <gmaxwell> jcorgan: will you be checking in a simple python example listener too?
297 2014-07-28 16:52:51 <wumpus> gmaxwell: yes, he has one in contrib/...
298 2014-07-28 16:53:24 <home_jg> The main issue there is trying to figure out the best data to return, in the event of a reorg. I like the idea of providing "connect" and "disconnect" stats, ie. return the number of blocks disconnected + connected since last -blocknotify.
299 2014-07-28 16:53:30 <gmaxwell> Is there a way for something that has RPC access to find out where the ZMQ port is?
300 2014-07-28 16:53:39 <jcorgan> no, good idea
301 2014-07-28 16:54:12 <gmaxwell> (I'm thinking of what the bitcoind-ncurses tool would want)
302 2014-07-28 16:54:40 <home_jg> or return the number of block hashes "impacted" since last -blocknotify, thereby returning the block hashes from the last branchpoint, for both weak and strong chains
303 2014-07-28 16:55:15 <wumpus> home_jg: new head hash/height, and hash/height of highest block shared between old and new states
304 2014-07-28 16:56:04 <home_jg> hmm, I suppose returning the block header versus just the block hash would give you reorg detection too
305 2014-07-28 16:56:30 <home_jg> hmm, you'll have that anyway, if you query the block after getting the hash. might just save one RTT.
306 2014-07-28 16:57:56 <home_jg> jcorgan, +1 re discovering zmq port. please don't add it to "getinfo" kitchen sink though. I think there is a "getnetworkinfo" or somesuch.
307 2014-07-28 16:58:13 <jcorgan> just let me know what you want
308 2014-07-28 17:00:03 <gmaxwell> We should probably add a comment to getinfo source to recommend people avoid adding things there.
309 2014-07-28 17:00:43 <jcorgan> btw, the zmq endpoint specifier is actually a URL like string, e.g., tcp://127.0.0.1:28322, and doesn't need to be a TCP port, but could also be an IPC transport or "in process shared memory" transport. So I'll just return the string as is in the RPC call.
310 2014-07-28 17:01:14 <gmaxwell> jcorgan: yea, ZMQ "url" sounds right to me.
311 2014-07-28 17:01:49 <jcorgan> -zmqpub=FOO => return "FOO" in RPC
312 2014-07-28 17:02:51 <wumpus> jcorgan: just use the opaque identifier that zmq uses
313 2014-07-28 17:02:59 <jcorgan> right
314 2014-07-28 17:03:42 <wumpus> gmaxwell: yes, getinfo is locked
315 2014-07-28 17:04:20 <wumpus> at this point until unspecified future it exists just for backwards compatiblity
316 2014-07-28 17:06:44 <wumpus> adding it to getnetworkinfo is OK with me
317 2014-07-28 17:07:32 <wumpus> although it is stretching 'network' a bit far, I don't know where to add it otherwise, it's really meta-information about notifications
318 2014-07-28 17:07:34 <user_asdasd> anyone not busy able to msg me, need some help with upgrading a btc clone wallet to 9.2. prolbably a simple question to some
319 2014-07-28 17:08:17 <wumpus> user_asdasd: we don't support altcoins here
320 2014-07-28 17:08:18 <gmaxwell> wumpus: or just "getzmqurl" no need to batch it in with other things.
321 2014-07-28 17:08:24 <wumpus> gmaxwell: +1
322 2014-07-28 17:08:33 <wumpus> gmaxwell: that may be best
323 2014-07-28 17:09:46 <jcorgan> ACK