1 2015-08-08 00:01:04 <phantomcircuit> wumpus, what do you think about simply inserting the CNode* into vNodes and cleaning it up with the normal disconnect logic?
  2 2015-08-08 00:01:18 <phantomcircuit> everything is written to handle hSocket == INVALID_SOCKET correctly
  3 2015-08-08 00:02:08 <wumpus> phantomcircuit: makes sense, I think
  4 2015-08-08 00:02:42 <wumpus> phantomcircuit: there doesn't seem to be a pressing reason to bypass the normal disconnec tlogic
  5 2015-08-08 00:07:36 <phantomcircuit> wumpus, also i was thinking that possibly the current logic is wrong
  6 2015-08-08 00:07:54 <phantomcircuit> each round of protecting peers removes them from the list of peers to consider
  7 2015-08-08 00:08:26 <phantomcircuit> so for the final round it's looking at 125-8-64-4 peers as eviction candidates rather than the full 125-8
  8 2015-08-08 00:08:56 <phantomcircuit> so it's not really protecting the 8 nodes with the best ping times but rather the best 8 nodes which haven't already been selected for protection
  9 2015-08-08 00:09:13 <phantomcircuit> if more metrics are added later it'll end up protecting all of the nodes...
 10 2015-08-08 00:13:50 <wumpus> hmm yes
 11 2015-08-08 00:25:18 <phantomcircuit> wumpus, would i break anything by sorting vNodes directly?
 12 2015-08-08 00:25:28 <phantomcircuit> i dont think so... but
 13 2015-08-08 00:25:41 <wumpus> why would you? I think working on a copy is a good idea
 14 2015-08-08 00:26:04 <wumpus> keeps the process from interfering with anything else, even accidentally
 15 2015-08-08 00:26:37 <phantomcircuit> oh and there's an example of that pattern just below
 16 2015-08-08 00:34:00 <chris13243> what do the bitcoin wizards think of this: https://github.com/priestc/moneywagon
 17 2015-08-08 00:37:09 <phantomcircuit> chris13243, that this is -dev not -wizards
 18 2015-08-08 00:37:56 <chris13243> i know
 19 2015-08-08 00:38:02 <chris13243> but im not a wizard so i cant post there
 20 2015-08-08 00:38:09 <chris13243> every time i do i get directed here
 21 2015-08-08 00:38:40 <chris13243> most wizards hang out here also anyways
 22 2015-08-08 00:38:45 <OMDPrebel> Well put your Hat on and try again.
 23 2015-08-08 00:42:33 <gmaxwell> phantomcircuit: I don't think thats wrong; it's exactly what I think should be desired-- if more metrics are added then some metrics may need to get their counts reduced.
 24 2015-08-08 00:43:06 <phantomcircuit> gmaxwell, it makes it much more difficult to reason about what's going on also
 25 2015-08-08 00:43:20 <phantomcircuit> changing the order of the protection passes significantly changes the results
 26 2015-08-08 00:44:11 <gmaxwell> phantomcircuit: it makes it easier though to resason about the protection provided. As you can make statements on exactly what resources the attacker needs.  Yes, they do interact, which is why the longer writeup I did gave consideration to the order. :)
 27 2015-08-08 00:47:09 <gmaxwell> phantomcircuit: because the whole goal of multi-metrics is that being good in many ways is much harder and more costly than e.g. totally dominating one particular way. So you don't want the case where someone doing better in one to wipe out another, but since there are multiple dimensions there is no way to make them order independant.
 28 2015-08-08 00:47:29 <gmaxwell> (I think the most you could do like that is randomize the order per node)
 29 2015-08-08 00:47:57 <gmaxwell> (you don't want it to change for a single node though or it could oscillate)
 30 2015-08-08 00:49:41 <phantomcircuit> hmm
 31 2015-08-08 00:50:20 <phantomcircuit> gmaxwell, for some of them you pretty clear want the absolute best peers to be protected; min ping time is a good example of that
 32 2015-08-08 00:50:37 <phantomcircuit> min ping time being exceptionally expensive to gmae
 33 2015-08-08 00:50:38 <phantomcircuit> game*
 34 2015-08-08 00:51:20 <phantomcircuit> others such as the randomly keyed comparison seems to be the same either way
 35 2015-08-08 00:52:05 <gmaxwell> well regardless of the order the minimum pingtime host(s) will be protected.
 36 2015-08-08 00:52:20 <gmaxwell> e.g. if one is protected because its longest uptime (if that came first) .. well it's still protected.
 37 2015-08-08 00:52:55 <phantomcircuit> right
 38 2015-08-08 00:53:24 <gmaxwell> the thing you get is that the Nth best of something may fail to get protected because the slot got taken by something that would have been protected anyways under another criteria.
 39 2015-08-08 00:54:51 <phantomcircuit> currently it's
 40 2015-08-08 00:55:09 <phantomcircuit> 64 oldest connections, 4 keyed peers, 8 best ping times
 41 2015-08-08 00:56:08 <phantomcircuit> and from there the CNetAddr with the most connections is selected
 42 2015-08-08 00:58:02 <gmaxwell> It's generally best to put the oldest connected last so that other criteria aren't 'laundered into it'.
 43 2015-08-08 00:58:13 <gmaxwell> e.g. you end up with 72 best pings.
 44 2015-08-08 00:58:22 <gmaxwell> which would not be so hot for topology. :)
 45 2015-08-08 00:59:21 <phantomcircuit> gmaxwell, yeah i noticed that with that set of metrics the best ping times were like 9 seconds and stuff
 46 2015-08-08 00:59:48 <phantomcircuit> thus the urge to have overlapping sections
 47 2015-08-08 01:00:19 <gmaxwell> nah, just put pings first. (and general stateless criteria before stateful ones)
 48 2015-08-08 01:01:12 <phantomcircuit> gmaxwell, NetGroupKeyed, NodeMinPingTime, NodeTimeConnected
 49 2015-08-08 01:01:32 <phantomcircuit> NetGrouKeyed is only 4 peers and should provide some serious protection
 50 2015-08-08 01:01:43 <phantomcircuit> i cant imagine it's easy to get 1 ip in every /16
 51 2015-08-08 01:02:06 <gmaxwell> actually there are ISPs that sell /8 diverse IP services (for bittorrent jamming)
 52 2015-08-08 01:02:09 <gmaxwell> :(
 53 2015-08-08 01:02:38 <phantomcircuit> sigh
 54 2015-08-08 01:03:18 <gmaxwell> phantomcircuit: as we add more criteria things will get more powerful,  e.g. a metric that prefers peers that relay useful info to you should help prevent degenerate topologies.
 55 2015-08-08 01:08:21 <phantomcircuit> gmaxwell, in that case... code review? :P
 56 2015-08-08 01:28:20 <gmaxwell> Anyone running firefox esp without adblock should consider their ssh key files (though not passphrases) compromised: https://blog.mozilla.org/security/2015/08/06/firefox-exploit-found-in-the-wild/   (continuing my longstanding comment to consider any host that has run a web browser compromised...)
 57 2015-08-08 01:35:54 <phantomcircuit> gmaxwell, :|
 58 2015-08-08 01:36:53 <gmaxwell> https://news.ycombinator.com/item?id=10021865 fun comment
 59 2015-08-08 01:39:31 <phantomcircuit> gmaxwell, fun fact, you can use the gpg keys on a yubikey neo as ssh key
 60 2015-08-08 01:40:25 <gmaxwell> phantomcircuit: I looked before and couldn't tell if it was _really_ doing that or if it was just handing ssh the file.
 61 2015-08-08 01:41:16 <phantomcircuit> gmaxwell, ?
 62 2015-08-08 01:41:51 <gmaxwell> e.g. is it passing the challenge to the yubi, or does the yubi just decrypt an encrypted copy of the private key for it.
 63 2015-08-08 01:42:57 <petertodd> phantomcircuit: yeah, I do that - it's part of the OpenPGP smartcard standard actually, not just yubikeys
 64 2015-08-08 01:42:59 <phantomcircuit> gmaxwell, it seems to be passing the challenge
 65 2015-08-08 01:43:19 <petertodd> gmaxwell: the yubikey makes the signature internally
 66 2015-08-08 01:44:17 <gmaxwell> is there a useful way to avoid having e yubikey tumor hanging off your laptop?  I dunno why no laptop has made resessed usb ports that had enough area around them to accept a small device.
 67 2015-08-08 01:44:29 <petertodd> gmaxwell: get a yubikey neo nano
 68 2015-08-08 01:44:38 <gmaxwell> fits in the usb port?
 69 2015-08-08 01:44:41 <petertodd> yup
 70 2015-08-08 01:44:59 <petertodd> https://www.yubico.com/products/yubikey-hardware/yubikey-neo/
 71 2015-08-08 01:45:36 <gmaxwell> hm. does it still have a button to authorize action?
 72 2015-08-08 01:45:40 <petertodd> yup
 73 2015-08-08 01:45:56 <petertodd> however the OpenPGP code on it doesn't use that button :/
 74 2015-08-08 01:46:12 <gmaxwell> bleh
 75 2015-08-08 01:47:14 <petertodd> agreed
 76 2015-08-08 01:49:18 <phantomcircuit> gmaxwell, the code is open and relatively simple
 77 2015-08-08 01:49:35 <petertodd> phantomcircuit: the code is loaded at the factory and can't be changed
 78 2015-08-08 01:49:36 <phantomcircuit> building the code is difficult but possible
 79 2015-08-08 01:49:50 <phantomcircuit> petertodd, depends on the model number of your neo
 80 2015-08-08 01:50:02 <phantomcircuit> some of them in the wild can load the new key
 81 2015-08-08 01:50:06 <phantomcircuit> new code*
 82 2015-08-08 01:50:12 <petertodd> phantomcircuit: yeah, they briefly shipped some that could, but none since the first ones shipped
 83 2015-08-08 01:50:45 <gmaxwell> gah
 84 2015-08-08 01:51:26 <petertodd> gmaxwell: they had a bug where it wouldn't actually require a pin to be entered, and because of that had to fix the bug by recalling the devices...
 85 2015-08-08 01:55:36 <phantomcircuit> petertodd, yeah you're right
 86 2015-08-08 01:55:52 <phantomcircuit> there's a blog post promising "neo developer" but i dont see any other reference to that
 87 2015-08-08 01:55:57 <petertodd> phantomcircuit: yup
 88 2015-08-08 01:56:42 <petertodd> phantomcircuit: there are some tricky security issues with letting the code be updated granted...
 89 2015-08-08 02:00:22 <phantomcircuit> petertodd, they store the secret separately for each applet and erase the secrets when you update the applet even for the developer edition
 90 2015-08-08 02:00:45 <phantomcircuit> i can understand not having it be a standard thing to give users the key.... but to not provide it to anybody?
 91 2015-08-08 02:00:47 <phantomcircuit> that seems silly
 92 2015-08-08 02:01:22 <petertodd> phantomcircuit: even yubico can't update the devices; they're read-only
 93 2015-08-08 02:01:46 <petertodd> phantomcircuit: if you make then read-write, it does make intercepting devices and backdooring them prior to initialization a lot easier
 94 2015-08-08 02:02:08 <petertodd> phantomcircuit: irrelevant if your adversary is the NSA, but relavant for a lot of lesser adversaries
 95 2015-08-08 07:48:39 <olalonde> What's the default port of -rest server?
 96 2015-08-08 07:49:22 <olalonde> Oh 8332
 97 2015-08-08 07:49:43 <olalonde> Does that mean that rest / json-rpc servers are mutually exclusive?
 98 2015-08-08 08:43:41 <olaa4a> Is it not possible to query a block by height in the REST interface?
 99 2015-08-08 08:45:02 <drazisil> olaa4a: it's not, there's only like 3 rest commands
100 2015-08-08 08:47:01 <olaa4a> Right
101 2015-08-08 08:47:07 <olaa4a> It's possible with the json-rpc though right?
102 2015-08-08 08:48:05 <drazisil> olaa4a: yes, getblockhash. https://bitcoin.org/en/developer-reference#getblockhash
103 2015-08-08 08:48:44 <drazisil> then you would turn around and getblock with the returned hash
104 2015-08-08 08:49:01 <olaa4a> Ok thanks
105 2015-08-08 09:05:03 <phantomcircuit> <petertodd> phantomcircuit: even yubico can't update the devices; they're read-only
106 2015-08-08 09:05:05 <phantomcircuit> im pretty sure that's wrong
107 2015-08-08 09:05:27 <phantomcircuit> they explicitly state that they setup unique management keys... which implies that they have the keys
108 2015-08-08 09:35:44 <olaa4a> Is there a RPC command to get a full block in raw or json format? (including transactions)
109 2015-08-08 09:37:11 <olaa4a> Oh I guess I need to ask for raw format
110 2015-08-08 09:40:11 <sipa> yes
111 2015-08-08 10:07:26 <mjerr> I may end up broadcasting a lot of transactions that are not valid (because the output has been redeemed already) from my application to my local node. Does this work just fine, or should I do background checks, before submitting it to my node? I am using bitcoinj to connect to the node
112 2015-08-08 10:08:34 <sipa> bitcoind won't relay those
113 2015-08-08 10:09:09 <sipa> unless you make your bitcoinj whitelisted
114 2015-08-08 10:09:23 <sipa> which you probably want to, if it rebroadcasts
115 2015-08-08 10:09:27 <mjerr> yea, I thought so, so I will save the work of implementing this myself - and will my node eventually blacklist my app?
116 2015-08-08 10:09:43 <sipa> it will never blacklist localhost
117 2015-08-08 10:10:05 <sipa> but even the valid transactions will only be sent out once
118 2015-08-08 10:10:21 <mjerr> ah perfect. what benefits are there for whitelisting bitcoinj?
119 2015-08-08 10:10:21 <sipa> rebroadcasts won't propagate
120 2015-08-08 10:10:29 <mjerr> ah, even better
121 2015-08-08 10:10:32 <sipa> what i just said
122 2015-08-08 12:41:46 <olaa4a> "Note that since Bitcoin Core v0.10, there are no such orphan blocks anymore, due to a significant change in the download mechanism."
123 2015-08-08 12:41:53 <olaa4a> Anyone knows what that means?
124 2015-08-08 12:42:06 <olaa4a> the client doesn't keep orphan blocks in memory?
125 2015-08-08 12:44:27 <sipa> olaa4a: it is about orphan blocks, not reorganized branches
126 2015-08-08 12:44:43 <sipa> there is a historic ambiguity about what orphan block means
127 2015-08-08 12:44:57 <sipa> originally it meant "block without a known parent"
128 2015-08-08 12:45:36 <sipa> and since 0.10, we always validate headers first and only downloads blocks after the header leading up to it is verified, so we never have blocks without parent
129 2015-08-08 12:46:02 <olaa4a> yeah I was using the term as "block without a known parent"
130 2015-08-08 12:46:14 <olaa4a> Ok thanks.
131 2015-08-08 12:46:18 <sipa> due to a weird misinterpretation, orphan block ended up meaning "block that is not part of the main chain"
132 2015-08-08 12:46:24 <sipa> in many cases
133 2015-08-08 12:46:37 <olaa4a> I usually use "side" blocks for those though this is probably confusign as well
134 2015-08-08 13:22:12 <waxwing> are rpc requests served in a single thread?
135 2015-08-08 13:22:23 <sipa> no
136 2015-08-08 13:22:28 <sipa> by default 4 threads
137 2015-08-08 13:22:56 <sipa> but almost all RPC queries need exclusive access to the blockchain, mempool, or wallet... making them effectively only one-at-a-time
138 2015-08-08 13:23:20 <waxwing> sipa: thanks
139 2015-08-08 13:24:45 <waxwing> why would read requests like gettxout or whatever, need exclusive access?
140 2015-08-08 13:26:01 <sipa> waxwing: because a concurrent modification to the UTXO could lead to a crash otherwise (particular objects being destroyed or moved in memory while another thread is reading them)
141 2015-08-08 13:27:38 <waxwing> sipa: ok, at that low level, yes i see you need exclusivity even for reading. It's not like a database.
142 2015-08-08 13:28:06 <sipa> databases also need exclusive access... they just do it at a more granular level
143 2015-08-08 13:28:27 <waxwing> right
144 2015-08-08 14:19:36 <mjerr> transactions with nlocktime as a timestamp, how much earlier can you broadcast them? and what happens, if another transaction is broadcasted that spends the same output, but has no locktime? do the nodes delete the locked transaction, even though it could be included in the next (X) block(s)?
145 2015-08-08 14:23:40 <sipa> if the locktime is in the future, it's not relayed or mined
146 2015-08-08 14:24:46 <mjerr> not at all? and there is no threshhold, like 10s?
147 2015-08-08 14:25:01 <mjerr> forget the first question
148 2015-08-08 16:53:43 <dgenr8> i've been thinking about submitting a PR that accepts them 10m before lock timestamp, to roughly align with the rule for height-based.  not sure how it would be received.
149 2015-08-08 16:54:22 <dgenr8> as it stands, this is a vector to increase mempool inconsistency
150 2015-08-08 16:59:28 <Luke-Jr> … as if mempools are supposed to be consistent
151 2015-08-08 20:23:39 <bedeho> is there any base58check encoding format for public keys?
152 2015-08-08 20:51:17 <phantomcircuit> waxwing, you still get a significant performance improvement by making concurrent requests though
153 2015-08-08 20:51:43 <phantomcircuit> for lots of the rpc calls the time to encode/decode json is significant (or at least it was before univalue was merged... but sure anymore)
154 2015-08-08 20:51:46 <phantomcircuit> not*
155 2015-08-08 22:31:55 <evoskuil> bedeho: you can base58check encode any data: https://github.com/libbitcoin/libbitcoin-explorer/wiki/bx-base58-encode
156 2015-08-08 22:32:20 <sipa> i assume whether there is a standard format for it
157 2015-08-08 22:32:30 <sipa> and pubkeys are typically hex-encoded, not base58
158 2015-08-08 22:33:06 <evoskuil> bedeho: wrong link, see: https://github.com/libbitcoin/libbitcoin-explorer/wiki/bx-base58check-encode
159 2015-08-08 22:34:53 <Luke-Jr> bedeho: why do you think there should be? afaik there is no standard
160 2015-08-08 22:37:54 <bedeho> evoskuil, thanks for the link! yes I was aware of that, but I was really asking if there were standard version bytes for public keys, to use with base58check encoding.
161 2015-08-08 22:38:55 <bedeho> Luke-Jr, no, not suggesting there should be, but I just couldnt find one, and wanted to be sure
162 2015-08-08 22:39:20 <Luke-Jr> i c
163 2015-08-08 22:40:25 <bedeho> evoskuil, is bx a wallet? or does it just read some sort of standardized format?
164 2015-08-08 22:42:50 <evoskuil> bedeho: It can be used as a wallet, and is a good way to perform advanced operations not exposed by many wallets, but it doesn't maintain a store of information for the user.
165 2015-08-08 22:43:31 <evoskuil> bedeho: it can be full chain (using a local libbitcoin-node/server) or trusting (using a remote libbitociin-server).
166 2015-08-08 22:44:57 <bedeho> ok I see
167 2015-08-08 22:45:07 <Luke-Jr> note there's also the more minimalistic libbase58 for just that (and in C, so no C++ dep) ;)
168 2015-08-08 22:46:24 <bedeho> Luke-Jr, ah yes, I think I have seen that before, didnt know it was you. I already have routines for doing that, but thank, will keep it in mind for another time.
169 2015-08-08 22:47:03 <evoskuil> yeah, definately wouldn't recommend libbitcoin-explorer as a lib for base58. the primary code is an libbitcoin (which is very small by comparison). But it's simple so a dedicated lib would be better if you aren't using libbitcoin.