1 2014-01-26 00:06:00 <jcorgan> sipa: +1
2 2014-01-26 00:10:42 <sipa> jcorgan, midnightmagic: https://github.com/sipa/bips/blob/bip32update/bip-0032.mediawiki
3 2014-01-26 00:31:05 <jcorgan> initial thoughts, pending more detailed reading: better, more distinct terminology throughout==good; using ser/parse/point seems a bit overkill but indeed more explicit
4 2014-01-26 00:31:49 <jcorgan> e.g., i thought k*G was obviously EC multiplication resulting in a point, but point(k) not as obvious
5 2014-01-26 00:32:09 <jcorgan> except by referring to the explicit definition
6 2014-01-26 00:33:23 <jcorgan> but overall this update clarifies many things that I had to puzzle out when reading the earlier version, so bravo
7 2014-01-26 00:49:51 <sipa> jcorgan: thanks
8 2014-01-26 00:51:06 <sipa> indeed, k*G vs point(k) isn't an obvious improvement, i like seeing it a type conversion from scalars to points, though
9 2014-01-26 01:28:40 <jerseykx> ok, got it (my problem from 4h ago) - seems https://en.bitcoin.it/w/images/en/7/70/Bitcoin_OpCheckSig_InDetail.png is not completely correct for P2SH, because bitcoind expects to see the original script before hashing in TxCopy for signing, not the 'OP_HASH160 [20-byte-hash-value] OP_EQUAL' from TxPrev
10 2014-01-26 03:19:56 <jcorgan> sipa: maybe have have the explicit functions, but give them letters, to make the actual formulas look like math and not computer code
11 2014-01-26 03:23:32 <sipa> jcorgan: i'm sure that will make it look more professional... but i'm not sure whether it makes it more readable
12 2014-01-26 03:24:43 <sipa> jcorgan: added a short request for comments on the pullreq
13 2014-01-26 03:24:53 <jcorgan> ok, i'll take it there
14 2014-01-26 04:06:17 <shamoon> how do i find the "size" if a transaction? https://blockchain.info/tx/57e905d87607e8043ea8c369a79c95be0555aabd08df174476c5467eefd55f41
15 2014-01-26 04:07:36 <lianj> its just the bytesize of the serialized tx
16 2014-01-26 04:07:54 <lechuga_> http://fetchtx.info/57e905d87607e8043ea8c369a79c95be0555aabd08df174476c5467eefd55f41?format=html
17 2014-01-26 04:08:10 <lechuga_> yes
18 2014-01-26 04:08:36 <lechuga_> and those are all the individual bytes in yellow
19 2014-01-26 04:10:33 <Luke-Jr> lechuga_: hey, you here in Miami by any chance?
20 2014-01-26 04:16:11 <lechuga_> nope kinda wish i went
21 2014-01-26 04:16:28 <lechuga_> sounds like quite the event
22 2014-01-26 04:18:42 <shamoon> lianj: so just the hex?
23 2014-01-26 04:18:43 <shamoon> of a tx?
24 2014-01-26 04:19:41 <shamoon> 270 characters, so what - one hex digit is 2 bytes?
25 2014-01-26 04:19:46 <shamoon> so half of 270?
26 2014-01-26 04:20:01 <lechuga_> 2 hex digits per byte
27 2014-01-26 04:21:05 <lechuga_> a byte is 8 bits which can have 2^8 different values so a maximum of 255 which in hex is 0xff
28 2014-01-26 04:22:55 <gmaxwell> shamoon: ... no. Just the @#$@ transaction itself.
29 2014-01-26 04:24:43 <shamoon> thanks
30 2014-01-26 07:40:23 <wyager> As I build my own implementation of BIP32 following Electrum's example, I'm frustrated with the apparently random usage of raw binary data and hex-encoded data
31 2014-01-26 10:02:17 <abrkn> anyone able to review my bip32 code? node.js
32 2014-01-26 10:04:12 <abrkn> https://github.com/justcoin/bitcoinjs-lib/blob/bip32/src/hdwallet.js and https://github.com/justcoin/bitcoinjs-lib/blob/bip32/test/hdwallet.js
33 2014-01-26 11:10:38 <wumpus> sipa: indeed, a deterministic wallet wouldn't require a keypool, keeping a counter would be enough in that case... and indeed in the case of a restored old backup it could use the blockchain to determine what keys are 'used' though it's non-exact
34 2014-01-26 11:11:18 <wumpus> sipa: not sure on preserving non-deterministic behavior as an option, it depends on what kind of break we make with the past in other things as well... ie make this a 'wallet 2.0' with different properties
35 2014-01-26 11:12:15 <wumpus> it would have my preference to do that, keep wallet as it is now as legacy and then make a deterministic wallet 2 without keypool and other burdens
36 2014-01-26 11:27:33 <sipa> wumpus: i'm pretty close to integrating bip32 in the current wallet
37 2014-01-26 11:28:02 <sipa> even though you don't technically need a keypool, you still need a lookahead to deal with restored backups
38 2014-01-26 11:28:15 <sipa> and it can easily be reused for that
39 2014-01-26 11:28:49 <sipa> very basic bip32 support is trivial (instead of generating new keypool keys at random, fetch them from a single keychain)
40 2014-01-26 11:29:21 <sipa> it's a bit harder if you want to support multiple chains, extending chains without the private key being unlocked, and dealing correctly with loomahead
41 2014-01-26 11:29:33 <sipa> as then you need multiple keypools
42 2014-01-26 11:31:11 <wumpus> ok, well if you adapt the current wallet I'd recomment keeping support for randomized keys
43 2014-01-26 11:31:17 <sipa> yup
44 2014-01-26 11:32:05 <wumpus> but do change the default to deterministic :)
45 2014-01-26 11:32:12 <wumpus> (at least for new wallets)
46 2014-01-26 11:32:25 <sipa> we'll need some nice gui wizard for setting them up :)
47 2014-01-26 11:33:19 <wumpus> well let me know what you need
48 2014-01-26 11:34:10 <wumpus> so we can try to land this in 1.0
49 2014-01-26 11:39:04 <wumpus> and indeed, a lookahead is needed even for deterministic wallets, but I suppose we can make it bigger if it's no longer necessary to store all the intermediate keys... or is it better to just store all the private keys individually anyway?
50 2014-01-26 11:39:32 <sipa> so, my plan is to *only* store metadata for derived keys
51 2014-01-26 11:39:47 <wumpus> right
52 2014-01-26 11:39:52 <sipa> i have an extended keystore which derives public and private keys on the fly
53 2014-01-26 11:39:57 <sipa> if metadata is known
54 2014-01-26 11:40:13 <sipa> which means you can add a new key without access to the parent private key
55 2014-01-26 11:40:58 <wumpus> that could be useful indeed
56 2014-01-26 11:41:07 <sipa> i still store metadata for all keys, as you want things like birthdate evennfor those
57 2014-01-26 11:42:56 <sipa> and with separated internal and externalnchains, you can have different lookahead
58 2014-01-26 11:43:26 <sipa> as internal keys are alwaysnused immediately (they're for change outputs)
59 2014-01-26 11:45:03 <wumpus> yes, storing metadata makes sense even for deterministic keys, it will be lost when restoring an old backup but that's just a nuisance not catastrophic
60 2014-01-26 11:46:17 <sipa> you need it for labels, payment requests, ... too
61 2014-01-26 11:46:32 <sipa> the only guarantee deterministic wallets provide is not losing money
62 2014-01-26 11:46:50 <sipa> they can't give you lost accounting information back
63 2014-01-26 11:52:12 <wumpus> indeed
64 2014-01-26 12:53:33 <CryptDrift> Is there a project (preferably self hosted) to deal out public bitcoin addresses?
65 2014-01-26 12:56:18 <wallet42> try http://coinpunk.org/
66 2014-01-26 13:02:18 <CryptDrift> wallet42, thanks though I'm looking for something very minimal to deal out address and monitor any new transactions
67 2014-01-26 15:22:54 <sacrelege> hi
68 2014-01-26 15:25:59 <sacrelege> I have a questions about the 'move' call, it doesn't seem described in the man page, what the ['minconf=1'] parameter is doing.
69 2014-01-26 15:26:15 <sipa> it doesn't do anything
70 2014-01-26 15:26:18 <sipa> it's historical
71 2014-01-26 15:26:30 <sipa> originally, move prevented moves if it would result in negative balances
72 2014-01-26 15:26:46 <sipa> so it needed to now at how many confirmations to consider a transaction complete
73 2014-01-26 15:27:11 <sipa> but as preventing it from going negative is inevitable (reorganizations) anyway, that functionality was dropped
74 2014-01-26 15:27:16 <Apocalyptic> sacrelege, the accounts feature is kind of broken, best not to rely on it
75 2014-01-26 15:27:33 <sacrelege> sipa on https://en.bitcoin.it/wiki/Accounts_explained at the end of the page "typical uses" they used: move "userid" "" <amount> 6 "purchased item" just to move only confirmed balances...
76 2014-01-26 15:27:53 <sipa> that's probably very old
77 2014-01-26 15:28:58 <sacrelege> Apocalyptic, sipa <offtopic>: what would be the pref. way to handle/implement account handling for a service?
78 2014-01-26 15:29:23 <sipa> do it yourself
79 2014-01-26 15:29:29 <Apocalyptic> ^
80 2014-01-26 15:29:55 <sipa> keep track of account balances in your own database
81 2014-01-26 15:30:03 <sipa> and only use bitcoind to send and receive transactions
82 2014-01-26 15:30:22 <Apocalyptic> sacrelege, btw I saw you stopped selling those vouchers and went on creating charting apps for gox :(
83 2014-01-26 15:30:42 <sacrelege> Apocalyptic, related to regulators in switzerland :/
84 2014-01-26 15:33:59 <sacrelege> sipa, how to keep track of incoming transactions? listtransactions seems a bit odd and -walletnotify is not merged to all alt-crypto clients yet.
85 2014-01-26 15:34:32 <sipa> on -walletnotify you poll listtransactionssinceblock (or whatever it's called), and update your database
86 2014-01-26 15:35:51 <sacrelege> okay, thanks a lot.
87 2014-01-26 16:30:59 <sacrelege> possible value for 'category' when calling listtransactions are 'move', 'send', 'receive'. 'orphan', 'immature' and 'generate' is that correct or did I miss something ?
88 2014-01-26 16:35:50 <sipa> sounds right
89 2014-01-26 16:49:07 <Ukyo> hey guys. I have a withdraw on record for XXXX amount. I see the amount leaving via address XXX and going turnign into 2 outputs being YYY and ZZZ. ZZZ is the intended destination with the proper amount. The rest went to YYY which is not an account the system has record of creating but validateaddress says "ismine". Can someone explain how this can happen?
90 2014-01-26 16:51:03 <sipa> Ukyo: read this: https://en.bitcoin.it/wiki/Change
91 2014-01-26 17:01:57 <Ukyo> err, I do not see anything about bitcoind generating a new address on its own though
92 2014-01-26 17:03:16 <sipa> so these change addresses are internal
93 2014-01-26 17:03:28 <sipa> they're only used once for privacy reasons
94 2014-01-26 17:03:51 <Ukyo> to be clear, this is an automatic process done by bitcoind that I have no control over, correct?
95 2014-01-26 17:03:56 <sipa> indeed
96 2014-01-26 17:04:01 <Ukyo> Thank you.
97 2014-01-26 17:04:14 <sipa> you're welcome
98 2014-01-26 18:22:44 <TD2> sipa: thanks for the bip32 update!
99 2014-01-26 18:23:04 <TD2> sipa: i was wondering if you'd be willing to run a little python script that does HTTPS-to-DNS for the dns seeds. i'd like to make fetching data from them authenticated
100 2014-01-26 18:46:07 <sipa> TD: there's a changelog entry...
101 2014-01-26 18:46:46 <sipa> TD: i really don't like how stongly wallets already depend on (among others) my dns seeds
102 2014-01-26 18:46:50 <TD> oh yes. bizarre. how did i not see that. typo: conversion
103 2014-01-26 18:47:09 <TD> sipa: nor do i. matt has been writing some code to use addr broadcasts. still, there will always be a need for the initial first-run seed request
104 2014-01-26 18:47:13 <sipa> i've considered turning it off for that reason
105 2014-01-26 18:48:05 <TD> well, you should see load go down in the not so distant future. matt's code still needs some work but he's been funded by the litecoin team to do that in bitcoinj. so it needs work and review and merging, then as people upgrade load will go down
106 2014-01-26 18:48:11 <TD> it's basically a reimplementation of addrman
107 2014-01-26 18:48:27 <sipa> i don't care about load
108 2014-01-26 18:48:52 <TD> sure, i know. what i mean is, you'll see wallets depend on it less
109 2014-01-26 18:49:05 <sipa> that's a good thing
110 2014-01-26 18:49:13 <sipa> but i still wish there were (much) more seeds
111 2014-01-26 18:49:29 <TD> regardless, authenticated access would be useful. in particular for wallets that talk to the p2p network only via tor
112 2014-01-26 18:49:49 <sipa> bitcoind just uses the p2p protocol for that
113 2014-01-26 18:49:53 <TD> otherwise an exit can MITM your entire view of the network (+ you need a custom dns resolver to force TCP via SOCKS)
114 2014-01-26 18:50:18 <sipa> if you specify a seednode when using Tor, it establishes a P2P connection to it, does a getaddr, and disconnects
115 2014-01-26 18:50:21 <TD> starting from what? the hard-coded list of peers?
116 2014-01-26 18:50:32 <sipa> starting from whatever you like
117 2014-01-26 18:50:33 <TD> yes and if you don't?
118 2014-01-26 18:50:46 <sipa> if you don't what?
119 2014-01-26 18:51:06 <TD> if you don't manually pick and specify a seed node
120 2014-01-26 18:51:22 <sipa> it's no different than having a list of hardcoded dns seeds, i mean
121 2014-01-26 18:51:42 <sipa> if you run bitcoind via tor now, it uses those seeds using that trick
122 2014-01-26 18:52:03 <sipa> i mean: no need for a special https protocol for this
123 2014-01-26 18:52:19 <TD> is it possible to run bitcoind via tor with *no* manual setup or configuration?
124 2014-01-26 18:52:26 <TD> no specification of your own seed nodes?
125 2014-01-26 18:52:30 <sipa> yes
126 2014-01-26 18:52:42 <sipa> it just uses the normal dns seeds
127 2014-01-26 18:52:48 <sipa> but instead of doing a dns resolve
128 2014-01-26 18:52:51 <sipa> it just connects to them
129 2014-01-26 18:52:59 <sipa> (where the tor exit node does the resolving)
130 2014-01-26 18:53:08 <sipa> and fetching a getaddr, and disconnects
131 2014-01-26 18:53:12 <TD> ah, the assumption is every DNS seed runs a full node on the same IP?
132 2014-01-26 18:53:17 <sipa> no
133 2014-01-26 18:53:32 <sipa> it connects to a peer the dns seed resolves to
134 2014-01-26 18:53:46 <sipa> so you have an extra indirection step (which is good, actually)
135 2014-01-26 18:53:56 <TD> right, got it. so basically the tor exit picks a random node from the list the seed responds with
136 2014-01-26 18:54:04 <sipa> exactly
137 2014-01-26 18:54:09 <TD> i see
138 2014-01-26 18:54:28 <TD> the tor exit can still MITM you of course .... that's what i was hoping to avoid
139 2014-01-26 18:54:46 <sipa> we could have onion seeds :)
140 2014-01-26 18:55:08 <sipa> where the MITM is avoided by having a hardcoded onion address
141 2014-01-26 18:55:24 <TD> that also works. though i think a 20 line python CGI script that's accessible via HTTPS is probably less overhead for seed operators than running tor
142 2014-01-26 18:55:56 <TD> but possibly i'm wrong about that
143 2014-01-26 18:56:10 <TD> would you/could you run an onion seed?
144 2014-01-26 18:56:15 <sipa> i do
145 2014-01-26 18:56:34 <sipa> but i'd rather not be the *only* hardcoded onion seed
146 2014-01-26 18:56:38 <TD> sure
147 2014-01-26 18:56:52 <TD> so i guess what we need is a list of onion peers that are known to be stable/high uptime and give good answers via getaddr
148 2014-01-26 18:56:53 <TD> hmmm
149 2014-01-26 18:57:05 <TD> the nice thing about the dns seeds is you at least get a fresh list that's been filtered a bit
150 2014-01-26 18:57:34 <TD> your "onion seed" is just a regular peer exported via a hs, right? as i don't see how to do the DNS query via the onion protocol.
151 2014-01-26 18:58:02 <sipa> yeah, it's just a regular bitcoind on a hidden service
152 2014-01-26 18:58:08 <sipa> well, no need for it to be bitcoind
153 2014-01-26 18:58:29 <sipa> but any hidden service bitcoind will work fine
154 2014-01-26 18:59:23 <TD> i wonder about performance. it's much faster to connect to a clearnet site than a hs. hmm.
155 2014-01-26 18:59:37 <TD> also how is the quality of getaddr responses? a dns seed gives nodes that are pretty certain to be up
156 2014-01-26 18:59:43 <TD> getaddr might return IP's that are very stale right?
157 2014-01-26 18:59:56 <sipa> yeah
158 2014-01-26 19:00:13 <sipa> another sign that we're over-relying on dns seeds :)
159 2014-01-26 19:00:36 <TD> hrmmmm. doesn't seem like an easy way to fix that though :)
160 2014-01-26 19:02:00 <TD> i'm still somewhat torn between have a bunch of "sites that translate HTTPS to DNS seed queries" vs just using hs nodes. i worry that it'd take ages to bring up p2p connections with the latter when all the overhead and stale IPs are taken into account
161 2014-01-26 19:02:02 <TD> could experiment though
162 2014-01-26 19:02:07 <TD> devrandom: you reading this?
163 2014-01-26 19:13:08 <TD> sipa: i just tried it. nearly a minute to go from cold to connected to a hidden service :(
164 2014-01-26 19:13:23 <TD> sipa: i'm not sure that's gonna work well for what i want.
165 2014-01-26 19:18:21 <TD> hmm interesting
166 2014-01-26 19:18:25 <TD> second attempt only 33 seconds
167 2014-01-26 19:21:09 <PRab> How many connections attempts did it take to get a successful one?
168 2014-01-26 19:23:09 <TD> third attempt, 5 seconds but that's with cached directory descriptors
169 2014-01-26 19:23:34 <TD> second attempt was after deleting the descriptor cache, so perhaps tor was "warmed up" inside the network or something
170 2014-01-26 19:23:56 <TD> PRab: i'm just timing how long it takes to connect right now. this does not include getaddr and connecting to the actual peers. so it's a lower bound
171 2014-01-26 19:24:33 <PRab> Ooh, oops. I thought you were waiting for the first successful connection from some hardcoded list.
172 2014-01-26 19:25:08 <TD> no. these are the timings to go from "no data locally" to "got a connection to my personal bitcoind hidden service node"
173 2014-01-26 19:25:11 <TD> seems it's rather variable
174 2014-01-26 19:26:06 <TD> when tor is "warmed up" we can go from start to connected in about 6 seconds, it seems. very interesting. that's a lot more like it
175 2014-01-26 19:26:19 <TD> possibly for the mobile case we would always have fresh directory consensus due to the 24 hour syncs
176 2014-01-26 19:26:30 <PRab> got it. Interesting experiment.
177 2014-01-26 19:27:20 <Diablo-D3> hey guys
178 2014-01-26 19:27:40 <Diablo-D3> the btc field in a tx is 32 bit right?
179 2014-01-26 19:27:59 <sipa> do your math
180 2014-01-26 19:28:10 <gribble> 42.94967296
181 2014-01-26 19:28:10 <sipa> ;;calc 2**32 / 100000000
182 2014-01-26 19:28:18 <TD> it's 64 bit
183 2014-01-26 19:28:24 <sipa> ^- that would mean no amounts above 42.94 BTC would be possible...
184 2014-01-26 19:28:33 <Apocalyptic> you couldn't send more than 43 btc otherwise
185 2014-01-26 19:29:45 <TD> sipa: my hidden service node still has no tor connections. i wonder if i set it up right to advertise in addr messages
186 2014-01-26 19:30:10 <sipa> what command line / config flags?
187 2014-01-26 19:31:40 <gmaxwell> we'd talked about that before⦠just not done it. Needed for onlynet=tor config to really work on initial bringup.
188 2014-01-26 19:31:54 <gmaxwell> oops I was in backscroll, was referring to onion seeds.
189 2014-01-26 19:32:46 <TD> in bitcoin.conf i have
190 2014-01-26 19:32:47 <TD> externalip=n5o5kf6ggofn37ne.onion
191 2014-01-26 19:32:47 <TD> listen=1
192 2014-01-26 19:32:48 <TD> discover=1
193 2014-01-26 19:33:31 <sipa> no proxy/tor?
194 2014-01-26 19:33:57 <TD> i don't care about connecting through tor or being hidden. just exposing it as an hs\
195 2014-01-26 19:34:32 <sipa> with -tor you'd also be able to connect to hidden services
196 2014-01-26 19:34:40 <sipa> without using it for normal peers
197 2014-01-26 19:34:58 <sipa> though that doesn't matter for you being able to advertize
198 2014-01-26 19:35:02 <TD> i see. but it should still advertise its hidden service address via getaddr anyway right
199 2014-01-26 19:35:04 <TD> indeed
200 2014-01-26 19:35:50 <TD> i wonder why it's so much faster now than before. perhaps my hidden service was dormant and didn't have any circuits to any rendezvous points or something
201 2014-01-26 19:35:57 <sipa> might be
202 2014-01-26 19:36:43 <TD> do you have your onion address to hand, by any chance?
203 2014-01-26 19:37:08 <sipa> kjy2eqzk4zwi5zd3.onion
204 2014-01-26 19:38:04 <TD> yeah, 6.5 seconds for yours too. i guess indeed mine was asleep in some way
205 2014-01-26 19:38:37 <sipa> which peer are you?
206 2014-01-26 19:38:43 <sipa> i've seen 4 connections the past minute
207 2014-01-26 19:38:58 <TD> the one that didn't announce any version packet. i'm just playing with orchid
208 2014-01-26 19:39:05 <sipa> ah, then i don't know :)
209 2014-01-26 19:39:07 <TD> so the "connect" here does not include version handshaking
210 2014-01-26 19:39:12 <sipa> right
211 2014-01-26 19:43:15 <michagogo> cloud|20:59:22 <TD> i wonder about performance. it's much faster to connect to a clearnet site than a hs. hmm.
212 2014-01-26 19:43:15 <michagogo> cloud|Well, hs has twice as many hops
213 2014-01-26 19:43:47 <michagogo> cloud|(also, if you want another hs to test to, you can use c7nvv7pgmbitcoin)
214 2014-01-26 19:44:16 <sipa> 3 vs 7 hops, iirc
215 2014-01-26 19:44:33 <michagogo> cloud|sipa: Yes, I think that's right
216 2014-01-26 19:44:54 <michagogo> cloud|(the hs and the connector each make a 3-hop circuit to a rendezvous point)
217 2014-01-26 19:45:45 <michagogo> cloud|Ah, right -- so slightly more than twice as many, because of the middle point
218 2014-01-26 19:49:28 <TD> sipa: look in your logs for "test app"
219 2014-01-26 19:50:23 <sipa> TD: seeing it
220 2014-01-26 19:50:41 <TD> sipa: answer: with fresh directory consensus, i can connect to tor in 2.9 seconds, do ssl connect to google.com in 3.7 seconds, version handshake with your node in 9.6 seconds and get a pingpong back in 10.8 seconds
221 2014-01-26 19:51:18 <TD> and it takes about 30 seconds to sync with the tor directories if need be
222 2014-01-26 19:51:25 <sipa> that doesn't sound too bad
223 2014-01-26 19:51:42 <TD> yeah that's ok for seeding. we'd be doing clearnet connects after that anyway
224 2014-01-26 19:51:42 <theorbtwo> Is that ssl connect to google.com over tor, or for comparison?
225 2014-01-26 19:51:49 <TD> theorbtwo: that's over tor
226 2014-01-26 19:52:02 <TD> and those times are cumulative
227 2014-01-26 19:52:20 <TD> i.e. to go from conneted to tor to connected/handshaked with google.com it takes 0.7 seconds, i think
228 2014-01-26 19:53:42 <TD> sorry, my bad. i think the ssl handshake wasn't completing there. i think it's more like ssl handshake after 4.3 seconds
229 2014-01-26 19:57:26 <TD> still need to bridge dns seeds to a hidden service though
230 2014-01-26 20:09:26 <TD> sipa: what do you think to adding a getconnected message to the protocol? so rather than relying entirely on getaddr which may serve a lot of stale peers, you could quickly bootstrap by picking "peers my current peer is connected to". it might also be useful to map the structure of the nework
231 2014-01-26 20:09:56 <sipa> s/useful/dangerous/ ?
232 2014-01-26 20:10:10 <TD> *shrug* dangerous for what? the connections are semi-random anyway
233 2014-01-26 20:11:06 <sipa> i don't like the network structure being blatantly obvious -- it would provide invaluable information if you would want to split the network
234 2014-01-26 20:11:28 <sipa> also, i think it would increase clustering
235 2014-01-26 20:12:07 <TD> do a 3-hop walk or something like that would basically be like a decentralised network crawl
236 2014-01-26 20:13:28 <TD> the problem i need to solve is, how can i get connected to a nice spread of p2p nodes, in only a few seconds
237 2014-01-26 20:13:32 <jaakkos> that sounds incredibly dangerous
238 2014-01-26 20:13:43 <TD> can someone please give me an *actual* attack that this would allow?
239 2014-01-26 20:15:15 <jaakkos> obtaining topology information helps in various attacks...
240 2014-01-26 20:15:27 <TD> like?
241 2014-01-26 20:16:06 <sipa> seems we should just improve the quality of addresses kept in addrman
242 2014-01-26 20:16:25 <TD> unless nodes are crawling each other all the time, which would massively add bandwidth usage, how can addrman keep addresses fresh?
243 2014-01-26 20:16:37 <michagogo> cloud|TD: A strategic DoS attack could split the network
244 2014-01-26 20:16:46 <TD> michagogo|cloud: it's a mesh. how would that work?
245 2014-01-26 20:16:54 <sipa> not crawling, but rotating a peer every few minutes should help a lot
246 2014-01-26 20:16:57 <jaakkos> well, for instance you could try to stop a peer from communicating with everyone else but you
247 2014-01-26 20:17:11 <jaakkos> then if they accept 0-confirmation, it becomes trivial to double spend
248 2014-01-26 20:17:11 <michagogo> cloud|TD: You could still split off clusters
249 2014-01-26 20:17:11 <sipa> i've done simulations a year ago or so, just never got to implementing it
250 2014-01-26 20:18:17 <TD> jaakkos: a peer will add new connections if its existing ones time out, won't it?
251 2014-01-26 20:19:44 <TD> sipa: so then getaddr would return recently used addresses instead of active addresses?
252 2014-01-26 20:20:25 <sipa> it always returns both
253 2014-01-26 20:20:48 <sipa> but usually the set of untried addresses is much larger and up to date than the set of known good addresses
254 2014-01-26 20:20:55 <sipa> just because we rarely make connections ourself
255 2014-01-26 20:22:04 <TD> alright. so how about this plan
256 2014-01-26 20:22:35 <TD> we find a bunch of people running long-term stable onion nodes. we ask them to apply an experimental patch (or get it into 0.9.1 and ask them to enable it) that makes a node rotate its peers constantly so it has a big fresh set
257 2014-01-26 20:23:00 <TD> then to securely bootstrap over tor, you would connect to these nodes and getaddr them. issue: getaddr response does not distinguish between "tried recently" and "untried", does it
258 2014-01-26 20:23:23 <sipa> indeed, it doesn't
259 2014-01-26 20:23:35 <TD> perhaps it could
260 2014-01-26 20:23:46 <sipa> there are some timestams
261 2014-01-26 20:23:50 <TD> if it's careful to exclude currently used peers
262 2014-01-26 20:23:52 <sipa> maybe you can derive something from that
263 2014-01-26 20:27:35 <TD> hm. it's a bit hard to figure out what the timestamps represent, from looking at the code
264 2014-01-26 20:28:03 <devrandom> ACTION is reading backscroll
265 2014-01-26 20:28:58 <devrandom> TD: I tend to agree with sipa that getaddr should be improved so that it returns a good set rather than having a special patch for it
266 2014-01-26 20:29:33 <sipa> I've wanted to implement peer rotation for a long time - i think it's a generally good thing for the network
267 2014-01-26 20:29:35 <devrandom> since having a good set is good for all peers, not just peers that need an initial seed
268 2014-01-26 20:29:57 <TD> sure. i'm just trying to figure out the shortest path to getting tor working
269 2014-01-26 20:30:08 <TD> i'm not against better p2p management in the c++ side in general
270 2014-01-26 20:30:24 <sacrelege> as far as I understand a transaction can have multiple outputs going to the same address (including same amount & time / timereceived ). If I want to store the output of 'bitcoind listransaction' into a Database (MySQL), how can I identify/differentiate if the transaction was already seen/stored or it is a new one?
271 2014-01-26 20:30:40 <devrandom> TD: you don't think that existing getaddr to a set of known onion nodes would be Good Enough for now?
272 2014-01-26 20:31:19 <sipa> (the idea was: assign each outgoing connection a score based on (time_connected)**(-0.8), tweak based on which peers have recently given us good blocks/..., and every N minutes create a new connection even if we are already at the limit, and if that succeeds, disconnect the peer with the lowest score)
273 2014-01-26 20:31:41 <sipa> that results in some long-time connections to remain open, and some connections that cycle regularly
274 2014-01-26 20:31:45 <sipa> but in a random fashion
275 2014-01-26 20:32:18 <TD> devrandom: we can try it once matt's addr tracking code is finished/merged. but i suspect we'd waste a lot of time trying to connect to nodes that have gone offline
276 2014-01-26 20:32:34 <sipa> s/lowest score/proportional to score/
277 2014-01-26 20:32:48 <devrandom> does current getaddr sort by lastseen?
278 2014-01-26 20:32:54 <TD> devrandom: no
279 2014-01-26 20:32:57 <sipa> i doubt that
280 2014-01-26 20:33:07 <TD> sipa: scoring connections like that is useful for anti-DoS as well
281 2014-01-26 20:33:24 <devrandom> does the getaddr packet contain the lastseen at least?
282 2014-01-26 20:33:24 <TD> oh, maybe not. the score would only be for outbound connections i guess
283 2014-01-26 20:33:34 <sipa> TD: well the score can be kept for all of course
284 2014-01-26 20:33:35 <TD> devrandom: no. it contains a timestamp but i can't figure out what it represents :)
285 2014-01-26 20:33:57 <devrandom> hmmm
286 2014-01-26 20:34:48 <devrandom> TD: so maybe having the onion seeds patched with an improved getaddr would be a good short-term solution
287 2014-01-26 20:35:30 <sipa> i don't care that much about improving getaddr - it should be the result of having a good set of addresses in general
288 2014-01-26 20:35:46 <sipa> every node benefits from that
289 2014-01-26 20:36:16 <devrandom> right, I think TD wants a short term solution for SPV clients over Tor
290 2014-01-26 20:36:34 <devrandom> so a simple patch as a stop-gap...
291 2014-01-26 20:36:57 <sipa> you're not going to improve getaddr much anyway, i think
292 2014-01-26 20:37:07 <TD> well, "short term". it'd be nice if we could get it within a bitcoinj release cycle, which is usually measured in months
293 2014-01-26 20:37:42 <TD> probably making peers effectively crawl each other by having peers drop and reconnect as sipa suggests would result in getaddr results becoming much more heavily biased in favour of active nodes
294 2014-01-26 20:38:08 <TD> and then we could replace dns seeding with getaddring a hidden service node. though i'm not a fan of the 2x speed penalty for using hidden services.
295 2014-01-26 20:38:16 <TD> i wish tor supported encrypted services
296 2014-01-26 20:38:39 <sipa> ?
297 2014-01-26 20:38:48 <devrandom> for the onion seeds, the patch could be something that just sorts the current connections above all other addresses
298 2014-01-26 20:39:01 <TD> "encrypted services" is the name of a tor improvement proposal where you get an onion address, but the endpoint isn't hidden
299 2014-01-26 20:39:12 <TD> so it's faster but less anonymous
300 2014-01-26 20:39:43 <sipa> hmm
301 2014-01-26 20:39:47 <TD> devrandom: then everyone would end up hitting the same set of peers
302 2014-01-26 20:39:54 <devrandom> just realized that
303 2014-01-26 20:39:56 <TD> devrandom: the current dns seeds do a good job of spraying the load evenly
304 2014-01-26 20:40:23 <devrandom> yeah, I think sipa's suggestion is the minimum to get a good spread
305 2014-01-26 20:40:35 <devrandom> actually, how do the current seeds generate their list?
306 2014-01-26 20:40:46 <sipa> crawling
307 2014-01-26 20:40:46 <TD> from other nodes addr broadcasts
308 2014-01-26 20:41:19 <TD> oh, sorry
309 2014-01-26 20:41:25 <TD> i misread "seed" as "node" :)
310 2014-01-26 20:41:37 <TD> yeah the dns seeds actually do fast crawls of the network to find nodes that are live and longlived
311 2014-01-26 20:42:06 <devrandom> so maybe a stop-gap would be a special bitcoind that returns getaddr from a file instead of from the builtin?
312 2014-01-26 20:43:10 <TD> the simplest method is what i originally suggested - just add https to the current set of seeds. making the getaddr response of every node competitive with the output of the dns crawlers is a longer term project
313 2014-01-26 20:43:26 <TD> i'd prefer not to entangle tor-for-spv with that project, as they're somewhat orthogonal
314 2014-01-26 20:46:10 <TD> as an aside,seems like we're stable at 9000 nodes at the moment: http://getaddr.bitnodes.io/chart/nodes/?category=v
315 2014-01-26 20:46:18 <TD> up from 6000 in nov
316 2014-01-26 20:46:36 <sipa> my crawler considers 6400 nodes well-reachable
317 2014-01-26 20:47:18 <devrandom> TD: BTW, is there any advantage to https over getaddr, assuming for the sake of argument that the data is the same?
318 2014-01-26 20:47:46 <TD> don't have to use hidden services, exit nodes can't mitm you
319 2014-01-26 20:48:09 <devrandom> so you'd pin the https fingerprint?
320 2014-01-26 20:48:37 <TD> could pin the public key. or just check the hostname in the standard style
321 2014-01-26 20:50:44 <devrandom> OK, both approaches seem reasonable - I don't have a strong preference either way. getaddr over hs seems more future proof protocol wise, but https is very quick to implement
322 2014-01-26 20:51:21 <TD> i'd like both of course :) but if wishes were horses ....
323 2014-01-26 20:51:45 <devrandom> both?
324 2014-01-26 20:52:11 <devrandom> do you mean some seeds over https, some over getaddr?
325 2014-01-26 20:52:43 <TD> no i mean, https for short term so we can launch sooner, and make getaddr work better for the longer run so eventually we can get rid of the custom DNS protocol
326 2014-01-26 20:53:04 <TD> devrandom: sipa also suggested another hack - connect to nodes using the DNS seeds as the hostnames, via a bunch of unrelated exit nodes
327 2014-01-26 20:53:14 <TD> devrandom: they should pick a random (?) ip from the dns response packet
328 2014-01-26 20:53:39 <TD> 5 exits connect to dns seed hostnames 5 times should == 5 separate connections
329 2014-01-26 20:53:44 <TD> if one exit is bad, only one connection is bad
330 2014-01-26 20:53:50 <TD> this could be implemented with no seeder changes at all
331 2014-01-26 20:54:28 <devrandom> oh, I see... basically gets a random IP from the DNS response
332 2014-01-26 20:55:07 <TD> yeah
333 2014-01-26 20:55:13 <TD> you tell tor to connect to "dnsseed.whatever.com"
334 2014-01-26 20:55:19 <TD> as if you were connecting to an ordinary website
335 2014-01-26 20:55:23 <devrandom> right
336 2014-01-26 20:55:48 <TD> we could do that for v1 instead of https. and then work on better p2p connection health/getaddr in parallel
337 2014-01-26 20:56:10 <devrandom> one last idea in the getaddr/hs direction - could write a simple python script that only implemented getaddr and served from a file
338 2014-01-26 20:56:20 <TD> where does the file come from?
339 2014-01-26 20:56:33 <devrandom> from the seed list
340 2014-01-26 20:56:48 <devrandom> or actually, could even be from a DNS lookup to the seed DNS
341 2014-01-26 20:57:17 <devrandom> i.e. it could do the DNS lookup when it got the getaddr req
342 2014-01-26 20:57:49 <TD> yes. that's sort of like the https proxy idea
343 2014-01-26 20:58:04 <TD> as i said, hs has a 2x performance penalty and we don't really care about hiding the services anyway
344 2014-01-26 20:58:18 <TD> so i'd rather not pay that if we don't have to ...... how about we prototype the "5 exits for 5 seeds" idea and see what happens
345 2014-01-26 20:58:33 <TD> there seem to be one or two bugs in orchid anyway so a bunch of things need to be done on the java side anyway
346 2014-01-26 20:58:51 <devrandom> as to hs being slow - in the long term, could have a single-hop hidden service - that would be about the same as the "encrypted service"
347 2014-01-26 20:59:43 <devrandom> TD: which release would this be for?
348 2014-01-26 20:59:45 <TD> yes that's the encrypted service proposal indeed.
349 2014-01-26 20:59:53 <TD> well i'd hope we could get it in, perhaps off by default, for 0.12
350 2014-01-26 21:00:06 <TD> but it depends how fast things move of course. the only defining feature for 0.12 imho should be HD wallets
351 2014-01-26 21:00:22 <devrandom> OK... feel free to assign to me, I should be able to do it in Feb
352 2014-01-26 21:01:10 <TD> great. i'll be away for a lot of feb btw
353 2014-01-26 21:01:17 <TD> so just forge ahead and see how well you can make it work
354 2014-01-26 21:01:39 <devrandom> SG
355 2014-01-26 23:19:52 <Polyatomic> bitcoind 0.8.6 Saucy1: How do I connect to, public.au.relay.mattcorallo.com
356 2014-01-26 23:21:05 <sipa> -addnode=public.au.relay.mattcorallo.com
357 2014-01-26 23:22:17 <Emcy_> saucy1?
358 2014-01-26 23:22:21 <Polyatomic> beauty thanks sipa.
359 2014-01-26 23:27:39 <shesek> all those unknown blocks by 129.132.x.x looks suspicious... a pool that's trying to hide the fact that its coming from a pool? https://blockchain.info/pools?timespan=4days
360 2014-01-26 23:28:16 <brisque> shesek: if you wanted to hide your pool from blockchain.info, you'd just not accept their incoming transactions.
361 2014-01-26 23:29:12 <brisque> shesek: that's a pool named Discus Fish anyway.
362 2014-01-26 23:29:56 <shesek> how can you tell?
363 2014-01-26 23:30:10 <sipa> the first rule of bitcoin is: do not rely on b.i
364 2014-01-26 23:30:21 <brisque> the coinbase contains "ä¸å½©ç¥ä»é±¼" (discus fish)
365 2014-01-26 23:31:58 <brisque> they're a massive pool, so I don't know why blockchain.info hasn't updated whatever regular expressions they use to identify blocks.
366 2014-01-26 23:32:24 <TD> it's not public i guess
367 2014-01-26 23:32:38 <brisque> sure it is. www.f2pool.com
368 2014-01-26 23:33:32 <TD> ah