1 2013-10-25 00:13:09 <super3> good day
2 2013-10-25 01:51:49 <elevatioN> hey i got a message saying 10.0.something.something connected to your device
3 2013-10-25 01:52:15 <elevatioN> is that a hacker
4 2013-10-25 01:53:07 <elevatioN> can someone please help me
5 2013-10-25 01:59:56 <MC1984> what?
6 2013-10-25 02:00:00 <MC1984> thats a private ip range any way
7 2013-10-25 02:04:23 <Apocalyptic> MC1984, so ?
8 2013-10-25 02:04:31 <Apocalyptic> could be someone on his LAN
9 2013-10-25 02:15:08 <MC1984> if you got a hacker physically inside your netowrk you dun goofed
10 2013-10-25 02:26:09 <elevatioN> what if they hacked someone on my network
11 2013-10-25 02:26:13 <elevatioN> like my girlfriend
12 2013-10-25 02:26:18 <elevatioN> her computer is loaded with shit
13 2013-10-25 02:36:47 <lianj> elevatioN: wrong channel, sorry
14 2013-10-25 03:30:12 <csjones> jgarzik around?
15 2013-10-25 03:44:52 <BlueMatt> gavinandresen: why is your linked mempool stuff not doing the mempool-limit and watch thrown-away txn for fee info stuff?
16 2013-10-25 03:45:52 <BlueMatt> imho its a bit cleaner, offers some good memory-limiting stuff and its less abuse-prone (not that the existing stuff is, really)
17 2013-10-25 04:03:02 <gavinandresen> BlueMatt: which "linked mempool stuff" ?
18 2013-10-25 04:03:12 <BlueMatt> s/linked/limited/
19 2013-10-25 04:03:33 <gavinandresen> BlueMatt: I haven't written a memory-limited mempool yet....
20 2013-10-25 04:03:47 <BlueMatt> ACTION did
21 2013-10-25 04:03:51 <BlueMatt> long time ago, though
22 2013-10-25 04:04:24 <gavinandresen> The smart fee code indirectly limits the mempool, but that is really a side effect of making the relay policy more sane
23 2013-10-25 04:05:07 <gavinandresen> ⦠and the problem with watching thrown-away transactions is you don't get a very good sample, because thrown-away transactions aren't relayed and don't propagate across the network very far
24 2013-10-25 04:05:18 <gavinandresen> (well, you don't get a good sample quickly)
25 2013-10-25 04:05:54 <BlueMatt> gavinandresen: in any case, it seems to me like it makes far more sense to only look at the set of the top N txn that are highest priority/fee (separately) and are not getting into blocks and just look at those for fee/prio limits
26 2013-10-25 04:06:21 <BlueMatt> (throwing away is nice because you can just look at the ones you didnt throw away)
27 2013-10-25 04:06:35 <gavinandresen> But those top N might not be getting into blocks for some reason we don't know about. LIke they're 1dice addresses and miners decide they don't like 1dice....
28 2013-10-25 04:06:41 <BlueMatt> sorry, misremembering how it was done, the design I did (and rather liked) was to use the thrown away ones to determine relay policy
29 2013-10-25 04:07:49 <Ferroh> bitcoind is occasionally crashing with SIGSEGV in _int_malloc() on my fresh ubuntu install :/
30 2013-10-25 04:07:58 <BlueMatt> gavinandresen: well it makes more sense to me to look at the ones which are sitting in mempool for M blocks
31 2013-10-25 04:07:58 <Ferroh> version 80500
32 2013-10-25 04:08:04 <BlueMatt> Ferroh: installed from where?
33 2013-10-25 04:08:08 <Ferroh> is this common or do i need to actually investigate it
34 2013-10-25 04:08:13 <Ferroh> i compiled it myself bluematt
35 2013-10-25 04:08:56 <BlueMatt> gavinandresen: throw away txn when mempool gets bigger than P*MAX_BLOCK_SIZE, set relay policy based on thrown-away txn, watch your mempool for txn which have been there for M blocks, median those for your minfee-to-spend policy
36 2013-10-25 04:08:58 <gavinandresen> BlueMatt: but if the relay/mempool policy is tuned correctly, there will be very few transactions sitting in the mempool for M blocks.
37 2013-10-25 04:09:23 <gavinandresen> BlueMatt: ⦠or to put it another way: a perfect relay policy would only relay transactions that will make it into the next block or two.
38 2013-10-25 04:09:28 <BlueMatt> gavinandresen: assuming your mempool isnt exactly one block in size you can always have txn sit around and just miss the cut forever
39 2013-10-25 04:09:53 <BlueMatt> gavinandresen: I disagree, a perfect relay policy would take into account the non-constant nature of block creation
40 2013-10-25 04:10:03 <BlueMatt> (ie if 6 blocks are created in 5 minutes, we want to be able to fill those)
41 2013-10-25 04:10:17 <gavinandresen> BlueMatt: In any case "there isn't one right way to do it" and I hope that wallet start competing to have the Bestest, Fastest, Cheapest fee estimation code
42 2013-10-25 04:10:24 <BlueMatt> so you'd need a mempool that acts as a buffer for N blocks of txn
43 2013-10-25 04:11:09 <BlueMatt> gavinandresen: considering lots of wallets are going spv and cant do the calculation, the realistic solution is gonna be to put the rpc output of bitcoind's getminfee on servers and people will trust those...
44 2013-10-25 04:11:20 <BlueMatt> so bitcoind should get it pretty close imho
45 2013-10-25 04:11:42 <gavinandresen> Well, in my empirical testing the algorithm I'm using does get it pretty close
46 2013-10-25 04:12:03 <BlueMatt> gavinandresen: the current approach of just median txn sitting in mempool and use that to create relay policy seems really fuzzy to me
47 2013-10-25 04:12:14 <gavinandresen> ⦠and gets it pretty close pretty quickly (e.g. after 10 or so blocks it has a good idea of what is being accepted)
48 2013-10-25 04:12:15 <BlueMatt> and doesn't limit mempool size as strictly as Id like, mostly
49 2013-10-25 04:12:37 <BlueMatt> ideally, mempool would be very strictly limited in size imho
50 2013-10-25 04:13:04 <gavinandresen> BlueMatt: agreed, mempool should be a small multiple of the block size.
51 2013-10-25 04:13:18 <BlueMatt> absolutely
52 2013-10-25 04:13:24 <Luke-Jr> ACTION ponders how it'd react if miners started giving higher priority to users of compressed keys
53 2013-10-25 04:13:34 <gavinandresen> BlueMatt: and correction: the "current approach" is median of txns that recently left the mempool and entered blocks.
54 2013-10-25 04:13:36 <BlueMatt> gavinandresen: hence why I ask why not do that now instead of going half-way with fee stuff?
55 2013-10-25 04:14:00 <phantomcircuit> Ferroh, that's not good
56 2013-10-25 04:14:10 <BlueMatt> gavinandresen: wat? I was under the impression it was agreed a /long/ time ago that fee policy should, under no circumstances, be based on txn that get into blocks
57 2013-10-25 04:14:15 <phantomcircuit> Ferroh, can you run in gdb to get a stack trace?
58 2013-10-25 04:14:46 <BlueMatt> Luke-Jr: doesn't the priority calculation sort of do that?
59 2013-10-25 04:14:56 <BlueMatt> not to say priority algorithm works right now...
60 2013-10-25 04:14:56 <gavinandresen> BlueMatt: why not limit mempool size now? Because I think there are higher priority things to work on
61 2013-10-25 04:14:56 <Luke-Jr> BlueMatt: on one end
62 2013-10-25 04:15:02 <Ferroh> phantomcircuit, the error only occurs once per few hours
63 2013-10-25 04:15:59 <gavinandresen> BlueMatt: making fees more sane is a very high priority for me now because what we have now will become increasingly broken as average block size approaches the soft 500K limit.
64 2013-10-25 04:16:08 <BlueMatt> gavinandresen: ehh, its actually surprisingly easy...
65 2013-10-25 04:16:21 <jgarzik> csjones, around, only to say I'm disappearing [into Windows... cursed printer software]
66 2013-10-25 04:16:22 <BlueMatt> gavinandresen: give me some time
67 2013-10-25 04:16:35 <BlueMatt> jgarzik: hah, that sucks
68 2013-10-25 04:16:40 <jgarzik> csjones, email jgarzik@bitpay.com if you have a question or something
69 2013-10-25 04:16:57 <jgarzik> BlueMatt, indeed... hopefully a one time thing
70 2013-10-25 04:17:16 <gavinandresen> BlueMatt: in any case, getting the blockchain tester working properly again is a really high priority right now, too
71 2013-10-25 04:17:18 <BlueMatt> gavinandresen: in any case, averaging txn that get into blocks can be easily replaced with top-N-in-mempool, no?
72 2013-10-25 04:17:27 <BlueMatt> gavinandresen: yea, Ive been looking at that off-and-on
73 2013-10-25 04:17:43 <BlueMatt> s/averaging/medianing/
74 2013-10-25 04:17:53 <gavinandresen> BlueMatt: sure, how the estimate is made should be easy to swap out
75 2013-10-25 04:18:14 <BlueMatt> any chance you can replace the look-at-blocks with look-at-mempool?
76 2013-10-25 04:18:35 <BlueMatt> look-at-blocks is too easy to attack if you're a miner and want to just fill your blocks with send-to-self-high-fee txn
77 2013-10-25 04:18:48 <BlueMatt> especially if you have some hashpower and make an agreement with others
78 2013-10-25 04:19:03 <gavinandresen> BlueMatt: the code I wrote does not "look at blocks"
79 2013-10-25 04:19:25 <BlueMatt> oh, I thought you just said that?
80 2013-10-25 04:19:26 <BlueMatt> <gavinandresen> BlueMatt: and correction: the "current approach" is median of txns that recently left the mempool and entered blocks.
81 2013-10-25 04:19:35 <BlueMatt> left mempool for blocks is fairly similar
82 2013-10-25 04:19:40 <BlueMatt> not identical, sure, but similar
83 2013-10-25 04:19:55 <BlueMatt> mine block, broadcast txn, broadcast block
84 2013-10-25 04:20:01 <gavinandresen> BlueMatt: I addressed the concern about miners trying to influence the algorithm in my dev update blog post
85 2013-10-25 04:20:12 <gavinandresen> executive summary is: they can't.
86 2013-10-25 04:21:04 <BlueMatt> gavinandresen: if you broadcast all your high-fee txn right before you broadcast the block you just mined?
87 2013-10-25 04:21:20 <BlueMatt> gavinandresen: the blog post glosses over pretty much all the detail...
88 2013-10-25 04:21:31 <gavinandresen> if the block gets there first, then the transactions never enter the mempool
89 2013-10-25 04:21:45 <BlueMatt> and if the txn are broadcast seconds before, then they will
90 2013-10-25 04:22:06 <gavinandresen> a finney attack to try to influence 50% median transaction fee?
91 2013-10-25 04:22:09 <BlueMatt> considering the very low fees we see in many txn today, you wouldnt give up much to do it too
92 2013-10-25 04:22:12 <gavinandresen> Mmm. Good luck with that.
93 2013-10-25 04:22:16 <BlueMatt> no, just broadcast in different order?
94 2013-10-25 04:22:22 <BlueMatt> you dont have to finney anything
95 2013-10-25 04:22:29 <BlueMatt> nodes process in-order, so they will always get the txn first
96 2013-10-25 04:22:38 <BlueMatt> (and broadcast the txn to their peers first)
97 2013-10-25 04:22:46 <BlueMatt> so they will get into mempool and then be immediately removed
98 2013-10-25 04:22:58 <gavinandresen> And if your block gets orphaned all of those sweet high transaction fees go to another miner
99 2013-10-25 04:23:23 <BlueMatt> small risk if you have few txn of moderate fee compared to the very low/0 fees we see today
100 2013-10-25 04:23:36 <BlueMatt> and then nodes start paying a moderate fee and it adds up quickly
101 2013-10-25 04:24:02 <BlueMatt> if you have lots of hash power, it would likely make sense, as long as the influence in fees you created is still around when you mine your next block
102 2013-10-25 04:24:59 <BlueMatt> orphan risk is relatively low, as long as you pay fees < orphan rate * prob that you make another block before the influence you create goes away
103 2013-10-25 04:25:16 <BlueMatt> for a miner with like 25% hash power, that looks pretty good to me
104 2013-10-25 04:25:43 <BlueMatt> s/*/+/
105 2013-10-25 04:25:45 <gavinandresen> BlueMatt: sigh. I'll run a back-of-the-envelope to convince you it doesn't make economic sense.
106 2013-10-25 04:26:00 <gavinandresen> ⦠but I'll have to dig out the data I have on the spread of transaction fees being paid today
107 2013-10-25 04:26:48 <BlueMatt> gavinandresen: other thing to keep in mind is a lot of fees today are hard-coded constants because no one write fee calculators
108 2013-10-25 04:26:53 <gavinandresen> With a 50% median as the "this is a reasonable fee to pay" threshold, a 25% miner would have to basically dedicate ALL of their block to high-fee send-to-self transactions to have a significant effect
109 2013-10-25 04:26:58 <BlueMatt> if you add one, all of a sudden people realize they can pay 0 fees and...
110 2013-10-25 04:27:07 <gavinandresen> ⦠and would have to broadcast them all before the block....
111 2013-10-25 04:27:12 <gavinandresen> ⦠which would increase their orphan rate...
112 2013-10-25 04:27:38 <gavinandresen> ⦠all to get higher fees⦠but since they aren't including any transactions in their blocks except pay-to-self they don't get those fees....
113 2013-10-25 04:28:12 <BlueMatt> hmm...yes, it may be hard to pull off given current hard-coded fees
114 2013-10-25 04:28:19 <BlueMatt> not sure if that will stick around, but who knows
115 2013-10-25 04:28:25 <gavinandresen> Even with non-hard-coded fees it will be hard to pull off.
116 2013-10-25 04:28:43 <BlueMatt> non-hard-coded fees could end up at 0
117 2013-10-25 04:28:51 <BlueMatt> depending on implementation and all kinds of things
118 2013-10-25 04:29:22 <gavinandresen> BlueMatt: did you see the "information propagation" paper by cdecker and roger wattenhofer?
119 2013-10-25 04:29:52 <BlueMatt> really, the foundation needs to hire like 5 people to work full-time on bitcoind to get all the things done it needs :(
120 2013-10-25 04:30:16 <BlueMatt> no
121 2013-10-25 04:31:03 <gavinandresen> BlueMatt: I'll find the link⦠but they look at tx/block propagation times, and come up with 80ms delay for each addtional 1K of block size
122 2013-10-25 04:31:08 <BlueMatt> reading it
123 2013-10-25 04:31:59 <BlueMatt> ACTION wonders how much large pools peer directly with each other
124 2013-10-25 04:32:15 <BlueMatt> and how much they publicize their nodes so they can peer with large individual/p2poolish miners
125 2013-10-25 04:32:26 <BlueMatt> seems like there is a very strong incentive to do that
126 2013-10-25 04:32:35 <gavinandresen> Yup.
127 2013-10-25 04:32:52 <BlueMatt> wasnt jgarzik working on that or so?
128 2013-10-25 04:33:04 <gavinandresen> Most interesting result from that paper might be them single-handedly cutting the orphan block rate
129 2013-10-25 04:33:20 <gavinandresen> yeah, a 'backbone network' was something jgarzik was talking about long ago....
130 2013-10-25 04:33:35 <BlueMatt> can we just do that?
131 2013-10-25 04:33:51 <gavinandresen> BlueMatt: blockchain.info might be doing that already....
132 2013-10-25 04:33:54 <BlueMatt> I mean foundation can afford a vps or three for a few nodes and then just work out contracts between others
133 2013-10-25 04:34:09 <BlueMatt> really its a matter of convincing others to do it
134 2013-10-25 04:34:25 <BlueMatt> and then getting up a few nodes that peer into that network of highly-connected peers that are public
135 2013-10-25 04:35:37 <gavinandresen> Feels like there is a grant to be given there...
136 2013-10-25 04:35:51 <BlueMatt> ACTION raises his hand, Ive been wanting to do that forever
137 2013-10-25 04:36:05 <BlueMatt> well, should probably see if jgarzik made any progress first
138 2013-10-25 04:36:24 <gavinandresen> Some sort of "there's a high-speed backbone that is different from the Internet because we somehow keep out the DoS'ers and only allow trusted nodes to join and contribute traffic"
139 2013-10-25 04:36:42 <gavinandresen> I have no idea how you decide who is trusted enough to put data on it, though.
140 2013-10-25 04:37:06 <BlueMatt> well, you have different levels of "you connect to all others in the core of like 10 groups, essentially just miners"
141 2013-10-25 04:37:22 <BlueMatt> to "you connect to some edge nodes that are in the core just for second level"
142 2013-10-25 04:37:32 <gavinandresen> Anyway, I brought up block propagation times because I'm still convinced that puts a floor on transaction fees, so fees won't go to zero.
143 2013-10-25 04:37:33 <BlueMatt> to "you are public and mining on p2pool and connect to second level edge nodes"
144 2013-10-25 04:38:08 <BlueMatt> hmm, well Ive always doubted the fee-goes-to-0 thing, but if you implement it marginally wrong, its quite possible because most of fees are just controlled by software recommendations, realistically
145 2013-10-25 04:38:41 <BlueMatt> and if it does go to 0, I think gaming may happen, hell miners would have a very strong incentive to group together and do it
146 2013-10-25 04:38:51 <BlueMatt> or at least marginal such that fees are <<< block reward
147 2013-10-25 06:01:49 <BlueMatt> gavinandresen: good call on getting the block tester to work on master, seems there is a regression hiding here
148 2013-10-25 06:43:05 <BlueMatt> any core devs awake?
149 2013-10-25 06:56:21 <petertodd> better than nothing :P
150 2013-10-25 06:59:30 <BlueMatt> them with push, be I referring to
151 2013-10-25 06:59:45 <wumpus> yes
152 2013-10-25 07:00:13 <petertodd> BlueMatt: I shouldn't reveal the extend of my 1337 HAX0R skills in a public channel
153 2013-10-25 07:00:23 <BlueMatt> lol
154 2013-10-25 07:27:36 <warren> https://github.com/bitcoin/bitcoin/commit/28f6b8dbad27f0dac72daca6f1bfe41d7e701908
155 2013-10-25 07:27:43 <warren> I'm having trouble finding the PR that included this.
156 2013-10-25 07:29:30 <Luke-Jr> warren: it didn't go through one
157 2013-10-25 07:30:28 <warren> o_O
158 2013-10-25 07:30:59 <warren> I backported it to 0.8.5 a while ago, and now I can't figure out how I identified it.
159 2013-10-25 07:36:42 <CodeShark> I'm trying to make sense of the merkleblock traversal encoding - the flags aren't really making sense
160 2013-10-25 07:37:35 <BlueMatt> CodeShark: heh, good luck there
161 2013-10-25 07:37:41 <BlueMatt> where are you stuck?
162 2013-10-25 07:40:30 <CodeShark> so say we traverse nodes a, b, c, d, e, f, and g and the flags (in hex) are 1d
163 2013-10-25 07:40:44 <CodeShark> and four hashes are stored
164 2013-10-25 07:41:11 <CodeShark> so 0x1d = 00011101 (in binary)
165 2013-10-25 07:41:37 <CodeShark> the traversal begins with the rightmost bit, correct?
166 2013-10-25 07:42:00 <CodeShark> so a = 1, b = 0, c = 1, d = 1, e = 1, f = 1, g = 0
167 2013-10-25 07:42:33 <CodeShark> and the initial two bits are ignored, yes?
168 2013-10-25 07:43:14 <CodeShark> a is the merkle root, then we move to the left and store a 0 which means the first stored hash corresponds to this node, yes?
169 2013-10-25 07:44:53 <CodeShark> I'd like to put a couple specific examples in the documentation
170 2013-10-25 07:44:55 <BlueMatt> sounds right
171 2013-10-25 07:45:10 <BlueMatt> sounds like a good idea
172 2013-10-25 07:47:01 <CodeShark> BlueMatt: have you implemented an SP
173 2013-10-25 07:47:05 <CodeShark> *SPV client?
174 2013-10-25 07:47:19 <BlueMatt> Ive worked a lot on bitcoinj, but thats still like 95% mike
175 2013-10-25 07:47:21 <BlueMatt> if not more
176 2013-10-25 07:50:32 <BlueMatt> lots of the bloom filtering code is mine, though the partial merkle tree stuff is sipa
177 2013-10-25 07:51:07 <CodeShark> I guess I'll just go through the bitcoind source
178 2013-10-25 07:52:25 <BlueMatt> bitcoinj is possibly more readable
179 2013-10-25 07:52:28 <BlueMatt> if you're a java person
180 2013-10-25 07:52:38 <CodeShark> I'm far more familiar with the bitcoind source
181 2013-10-25 08:00:12 <thermoman> hi
182 2013-10-25 08:01:24 <thermoman> i'm running a 0.7.2 client before a 0.8.5 client and now the 0.7.2 client did ban the 0.8.5 client
183 2013-10-25 08:01:44 <thermoman> the 0.8.5 client (child) uses connect=<IP from 0.7.2 client(A)>
184 2013-10-25 08:02:09 <thermoman> how can i whitelist the (child) on (A)?
185 2013-10-25 08:02:26 <thermoman> 10/24/13 16:08:55 ERROR: CTransaction::CheckTransaction() : vin empty
186 2013-10-25 08:02:26 <thermoman> 10/24/13 16:08:55 ERROR: CTxMemPool::accept() : CheckTransaction failed
187 2013-10-25 08:02:41 <thermoman> Misbehaving: <IP>:53081 (20 -> 30)
188 2013-10-25 08:02:51 <thermoman> onnection from <IP>:53092 dropped (banned)
189 2013-10-25 08:08:51 <BlueMatt> thermoman: better question - why did the 0.8.5 client relay that transaction?
190 2013-10-25 08:18:24 <thermoman> BlueMatt: we have a 0.7.2 (A) acting as proxy, and 2 childs running 0.7.2 and 2 childs running 0.8.5.
191 2013-10-25 08:18:38 <thermoman> our dev used one of the 0.7.2 childs to send out transaction - which always worked
192 2013-10-25 08:18:56 <thermoman> yesterday he choose to send out transactions via one of the 0.8.5 childs
193 2013-10-25 08:18:58 <BlueMatt> well you sent out a broken transaction (are you using a modified node?)
194 2013-10-25 08:19:08 <BlueMatt> bitcoind wont let you do that by default afaik
195 2013-10-25 08:19:12 <thermoman> instead of the 0.7.2 child - and oops there are these error messages with empty vin
196 2013-10-25 08:19:22 <thermoman> nothing modified
197 2013-10-25 08:19:30 <thermoman> it's stock compiled from source
198 2013-10-25 08:19:32 <BlueMatt> then you done found a bug
199 2013-10-25 08:19:37 <BlueMatt> how did he send that tx?
200 2013-10-25 08:19:43 <thermoman> i'll ask him
201 2013-10-25 08:21:28 <thermoman> he used two different methods: some transactions are grouped (10 tx in one) and some are send individually
202 2013-10-25 08:21:39 <thermoman> we have a partial tx in the error message
203 2013-10-25 08:21:47 <thermoman> but he can't find that tx in our database
204 2013-10-25 08:21:54 <BlueMatt> you mean eg sendrawtransaction rpc?
205 2013-10-25 08:22:01 <thermoman> nope
206 2013-10-25 08:22:07 <thermoman> he doesn't use sendraw
207 2013-10-25 08:22:20 <BlueMatt> regular sendmoney sent a tx with 0 vin?
208 2013-10-25 08:22:23 <BlueMatt> wat?
209 2013-10-25 08:23:03 <thermoman> he uses sendmany or sendfrom
210 2013-10-25 08:23:19 <thermoman> but he doesn't know at this state which of the two caused this
211 2013-10-25 08:23:32 <thermoman> because he can't link the tx that failed to our log
212 2013-10-25 08:23:48 <thermoman> 10/24/13 16:08:55 received getdata for: tx 9f6cc3544bcb4358295b
213 2013-10-25 08:24:04 <thermoman> thats from the log before the client got banned
214 2013-10-25 08:24:18 <thermoman> 10/24/13 16:08:55 ERROR: CTransaction::CheckTransaction() : vin empty
215 2013-10-25 08:24:19 <thermoman> 10/24/13 16:08:55 ERROR: CTxMemPool::accept() : CheckTransaction failed
216 2013-10-25 08:24:34 <BlueMatt> what does the client that sent the transaction show, is the real question?
217 2013-10-25 08:24:37 <thermoman> 10/24/13 16:08:55 Misbehaving: <IP>:53081 (20 -> 30)
218 2013-10-25 08:24:55 <thermoman> you mean look in debug.log of the child?
219 2013-10-25 08:25:19 <CodeShark> try to get the raw transaction
220 2013-10-25 08:26:17 <CodeShark> sendmany and sendfrom should never have an empty vin unless either you've modified bitcoind or there's a bug
221 2013-10-25 08:30:54 <thermoman> modified bitcoind = negative
222 2013-10-25 08:31:05 <thermoman> how do i get the raw transaction?
223 2013-10-25 08:31:21 <CodeShark> use getrawtransaction on the client that sent
224 2013-10-25 08:31:43 <thermoman> but then again we need to know what TX caused that
225 2013-10-25 08:32:07 <CodeShark> have you sent many transactions since from this client?
226 2013-10-25 08:33:04 <thermoman> the client says at the time this happened on the proxy node:
227 2013-10-25 08:33:08 <thermoman> 2013-10-24 16:08:55 Relaying wtx 19fc786f155c9d5f468bc844430e62c3d59cef56eaa2a27bdfdf16df672ca0b5
228 2013-10-25 08:33:08 <thermoman> 2013-10-24 16:08:55 ResendWalletTransactions()
229 2013-10-25 08:33:11 <thermoman> 2013-10-24 16:08:55 Relaying wtx a7eb2485ca9600f99ed4bcd778cb6cdcb51c11fdbd587c440e8850b8de3679c1
230 2013-10-25 08:33:14 <thermoman> 2013-10-24 16:08:55 received getdata for: tx d21633ba23f70118185227be58a63527675641ad37967e2aa461559f577aec43
231 2013-10-25 08:33:17 <thermoman> .
232 2013-10-25 08:42:44 <CodeShark> here's the output of my program: http://pastebin.com/C5MUjzy9
233 2013-10-25 08:42:55 <CodeShark> the flags are 5d01
234 2013-10-25 08:43:06 <CodeShark> however, only nine nodes should be traversed'=
235 2013-10-25 08:45:26 <BlueMatt> hey, there's a sipa
236 2013-10-25 08:45:34 <sipa> where?
237 2013-10-25 08:45:56 <BlueMatt> ACTION is lazy and was just trying to get a response so gribble could deliver later tell's
238 2013-10-25 08:46:18 <CodeShark> sipa, I'm struggling a bit going over the documentation and source for the partial merkle tree - I'd like to add a couple concrete examples to the documentation to leave no doubt as to how the data is formatted
239 2013-10-25 08:46:35 <sipa> CodeShark: please do! :)
240 2013-10-25 08:46:50 <CodeShark> the tree is traversed depth-first in-fix?
241 2013-10-25 08:47:30 <CodeShark> that's to say, the flag bits correspond to the nodes traversed in-fix?
242 2013-10-25 08:48:00 <sipa> prefix
243 2013-10-25 08:48:13 <CodeShark> I posted the output of my program here: http://pastebin.com/C5MUjzy9
244 2013-10-25 08:48:28 <sipa> you output a flag before descending into the children
245 2013-10-25 08:48:44 <CodeShark> basically, I create a bloom filter with two elements (each corresponding to a pubkey hash)
246 2013-10-25 08:49:00 <CodeShark> I ask specifically for a full block, then ask for the filtered block
247 2013-10-25 08:49:15 <CodeShark> the two transactions that match the filter have indices 4 and 7 in the block
248 2013-10-25 08:49:53 <CodeShark> so the flags are 5d01
249 2013-10-25 08:50:27 <CodeShark> there are only nine nodes traversed, so presumably the first 7 bits of the second byte are irrelevant
250 2013-10-25 08:52:53 <CodeShark> perhaps I can use mathematica to create some pretty graphics :)
251 2013-10-25 08:59:22 <TD> good morning
252 2013-10-25 09:03:52 <BlueMatt> morning TD
253 2013-10-25 09:03:56 <BlueMatt> oh, shit, Im up this late?
254 2013-10-25 09:03:57 <BlueMatt> fuck
255 2013-10-25 09:04:07 <TD> haha
256 2013-10-25 09:04:16 <TD> GOOD MORNING ! :)
257 2013-10-25 09:04:24 <BlueMatt> indeed, sun in like an hour
258 2013-10-25 09:05:05 <TD> prime hacking time over. hit the sack ... btw thanks to you and petertodd for the discussion on the script stuff
259 2013-10-25 09:05:18 <BlueMatt> thanks to him for auditing things
260 2013-10-25 09:05:28 <BlueMatt> nooo, must finish task at hand
261 2013-10-25 09:08:32 <CodeShark> ok, so the tree basically looks like this: http://blockhawk.net/images/PartialMerkleTree.pdf
262 2013-10-25 09:08:46 <TD> yes, indeed. i'll mention that in the release notes for the next release.
263 2013-10-25 09:08:57 <BlueMatt> TD: btw, any chance to look over nonetty any time recently?
264 2013-10-25 09:09:01 <CodeShark> the nodes are numbered in traversal order, the included hashes are shown
265 2013-10-25 09:09:16 <BlueMatt> TD: if not, I may have some time to bang on the update you suggested
266 2013-10-25 09:09:21 <TD> i've been focused on my end-of-november deadline
267 2013-10-25 09:09:29 <TD> so other than rebasing i didn't do anything lately.
268 2013-10-25 09:09:36 <BlueMatt> oh, end-of-november already? shit
269 2013-10-25 09:09:38 <BlueMatt> have fun
270 2013-10-25 09:09:38 <TD> also i've been pushing the micropayments work through to completion
271 2013-10-25 09:09:50 <TD> https://github.com/mikehearn/PayFile
272 2013-10-25 09:09:57 <CodeShark> so how do the flags 5d01 correspond to this tree?
273 2013-10-25 09:09:59 <TD> basically there's two bugs/edge cases left, and I want to rename CLOSE to SETTLE in the protocol
274 2013-10-25 09:10:11 <TD> because it was getting way too confusing to have multiple kinds of close/unbind/settle floating around under the same name
275 2013-10-25 09:10:27 <TD> then i reckon our first demo app will be ready for release! i got myself an apple codesigning cert the other day in anticipation
276 2013-10-25 09:10:57 <BlueMatt> yea, I kinda went overboard on the whole an-attacker-might-do-this-so-legitimate-nodes-should-too-as-a-part-of-the-protocol design stuff....
277 2013-10-25 09:11:05 <TD> i/we need to decide how to handle the case where the client tries to settle the channel without having sent enough money to make it valid yet.
278 2013-10-25 09:11:12 <TD> s/valid/settleable/
279 2013-10-25 09:11:19 <CodeShark> sipa, you still there?
280 2013-10-25 09:11:24 <TD> at the moment the server throws an exception, which is fine, but then the client and server get de-synced and stuff gets messy.
281 2013-10-25 09:11:43 <TD> so, yeah .... possibly changing the protocol to demand the min-settleable payment up front as part of the initiate phase
282 2013-10-25 09:12:02 <BlueMatt> throw-away-all-money makes sense to me
283 2013-10-25 09:12:12 <BlueMatt> as long as the server rate-limits things (which it should anyway)
284 2013-10-25 09:12:25 <TD> well, yes, then it gets harder to implement as you have to watch out for "repeat offenders"
285 2013-10-25 09:12:38 <TD> demanding an up-front payment also works, but then you have to specify it as part of the higher level app protocol
286 2013-10-25 09:12:50 <BlueMatt> you have to rate limit anyway, and gavin is working on killing dust stuff recently
287 2013-10-25 09:14:26 <TD> why do you have to rate limit?
288 2013-10-25 09:14:42 <TD> if you're charging money for a service, it's inherently anti-DoS-proofed to some extent by the fact that the attacker will run out of money
289 2013-10-25 09:14:43 <BlueMatt> because dos?
290 2013-10-25 09:15:02 <BlueMatt> create-close-without-paying dos
291 2013-10-25 09:15:04 <BlueMatt> that is
292 2013-10-25 09:15:08 <TD> if you do work and throw away the money, THEN you have to do extra anti-DoS code to make the attacker burn other valuable resources, like IP addresses
293 2013-10-25 09:15:22 <TD> well yes, exactly. that's why i'm saying, maybe you shouldn't be able to create without paying.
294 2013-10-25 09:15:40 <BlueMatt> ahh, ok
295 2013-10-25 09:15:41 <TD> it's also an easy programming error to make. right now the API is basically: open a channel. make payments of whatever amount you like. settle.
296 2013-10-25 09:15:45 <BlueMatt> well Im too tired to reason reasonably
297 2013-10-25 09:15:57 <TD> so the natural thing to do is like, attach "download next chunk of file data" to "increment payment"
298 2013-10-25 09:16:18 <TD> but then with that totally natural and intuitive design you introduce a bug because if the file is too small, or the user cancels the download too fast -> crash
299 2013-10-25 09:16:29 <BlueMatt> wait, why does it crash
300 2013-10-25 09:16:33 <BlueMatt> doesnt it throw money away
301 2013-10-25 09:16:40 <BlueMatt> and it could just make the user wait until the time expires
302 2013-10-25 09:16:40 <TD> i mean the server throws an exception
303 2013-10-25 09:16:47 <BlueMatt> then its inherinently anti-dos
304 2013-10-25 09:17:03 <TD> well but then the end user gets randomly whacked with money locked up, just for the "crime" of downloading a small file instead of a big one.
305 2013-10-25 09:17:15 <BlueMatt> well, it should just say "fuck it", you get no payment, user spent some effort to lock up money, but thats their payment
306 2013-10-25 09:17:50 <TD> sigh. this dust thing is such a massive pain in the ass.
307 2013-10-25 09:17:58 <BlueMatt> its going away, apparently
308 2013-10-25 09:18:03 <TD> no, i wish
309 2013-10-25 09:18:10 <TD> it just gets even more fucking complicated. it's going to float.
310 2013-10-25 09:18:25 <TD> and each node will have a different estimate of what the dust limit actually is
311 2013-10-25 09:18:54 <TD> is gavinandresen around?
312 2013-10-25 09:19:16 <BlueMatt> well, it already does float, in the sense that if you send crap you (maybe) wont be accepted into a block
313 2013-10-25 09:19:19 <BlueMatt> (in the future)
314 2013-10-25 09:19:27 <BlueMatt> (now miners take everything so it doesnt matter)
315 2013-10-25 09:19:43 <BlueMatt> but thats limited anyway
316 2013-10-25 09:20:30 <CodeShark> could someone explain to me how the flag bits correspond to the tree nodes? http://blockhawk.net/MerkleBlock/
317 2013-10-25 09:20:30 <TD> i wonder how many more doublings of the exchange rate we need until the cost of the 200 bytes or whatever a utxo output costs is lower than 1 satoshi
318 2013-10-25 09:21:03 <TD> CodeShark: you consume a bit each time you step through the tree, iirc
319 2013-10-25 09:21:10 <TD> CodeShark: sipa can explain it better, he designed it :)
320 2013-10-25 09:21:21 <CodeShark> yes, I get the general idea but I would like to know specifically how it works in this example
321 2013-10-25 09:21:35 <CodeShark> I can't seem to get the bits to match up with the node traversal the way I've numbered it
322 2013-10-25 09:21:41 <BlueMatt> TD: what we need is a million merchants worth a few mil each who run nodes for the good of bitcoin
323 2013-10-25 09:21:47 <BlueMatt> (and mine a bit each)
324 2013-10-25 09:21:50 <BlueMatt> then we're golden
325 2013-10-25 09:21:52 <TD> heh
326 2013-10-25 09:22:20 <TD> ok, well, how about this. a server can already specify a minimum amount that it's willing to tolerate being put into a channel
327 2013-10-25 09:22:32 <CodeShark> I would like to know if 1) the node ordering I've given is the correct traversal order and 2) how the bits in 0x5d01 correspond to the nodes
328 2013-10-25 09:22:40 <TD> why can we not just make that the min payment that has to be presented as the refund instead of the current "all back to me" refund?
329 2013-10-25 09:22:48 <BlueMatt> CodeShark: no idea, test it?
330 2013-10-25 09:23:04 <CodeShark> sipa was just in here and claimed it's a depth-first prefix traversal
331 2013-10-25 09:23:20 <CodeShark> which if I understand correctly, it means I've numbered the nodes in the order of traversal starting at 0 and ending at 8
332 2013-10-25 09:23:50 <CodeShark> problem is I can't seem to get the flag bits to match up to the nodes correctly
333 2013-10-25 09:23:55 <BlueMatt> TD: hmm...sounds like fun, as long as the client app has some concept of what it wants to spend up front
334 2013-10-25 09:24:05 <BlueMatt> TD: wait, why is there a min amount right now?
335 2013-10-25 09:24:13 <TD> because you put it there!
336 2013-10-25 09:24:21 <BlueMatt> Im sure I had a reason
337 2013-10-25 09:24:21 <TD> remember the minValue negotiation that takes place during INITIATE?
338 2013-10-25 09:24:23 <TD> haha
339 2013-10-25 09:24:30 <TD> perhaps we just re-discovered what that reason was
340 2013-10-25 09:24:41 <TD> actually i was wondering about that too, as it adds some error cases that have to be handled ....
341 2013-10-25 09:24:43 <BlueMatt> right now Im only really incredibly tired (and a wee bit drunk), so I cant say I remember why
342 2013-10-25 09:24:51 <TD> ok
343 2013-10-25 09:24:59 <TD> well it'll be a few days until i circle back around to this
344 2013-10-25 09:25:28 <BlueMatt> ok, well send me mail and Ill respond when I can think through everything and make a well-reasoned argument as to why I was right all-along
345 2013-10-25 09:25:36 <TD> ok. it was probably an anti-DoS thing
346 2013-10-25 09:25:37 <BlueMatt> and fuck reality, I must've been right, right?
347 2013-10-25 09:25:42 <TD> like preventing you from creating lots of tiny channels
348 2013-10-25 09:25:46 <BlueMatt> probably
349 2013-10-25 09:25:48 <TD> you define your own reality matt
350 2013-10-25 09:26:24 <BlueMatt> "I reject to reality and substitute my own" -Adam Savage
351 2013-10-25 09:26:50 <TD> CodeShark: the entire block only had 8 transactions in it?
352 2013-10-25 09:26:54 <CodeShark> yes
353 2013-10-25 09:27:02 <CodeShark> I purposely chose a relatively small block
354 2013-10-25 09:27:24 <BlueMatt> CodeShark: is this example dumped from real values?
355 2013-10-25 09:27:27 <CodeShark> yes
356 2013-10-25 09:28:45 <BlueMatt> CodeShark: that absolutely looks right to me, assuming the byte-order is right
357 2013-10-25 09:28:53 <CodeShark> the filter matched the transactions at the nodes I've numbered 5 and 8 (which correspond to transactions 4 and 7 zero-indexed in the block)
358 2013-10-25 09:29:09 <BlueMatt> ok, then that must be right
359 2013-10-25 09:29:18 <CodeShark> so the flags are 0101110100000001
360 2013-10-25 09:29:27 <CodeShark> which bit corresponds to which node?
361 2013-10-25 09:30:14 <CodeShark> node 0 has bit true or false?
362 2013-10-25 09:30:37 <BlueMatt> order (based on your indexes) appears to be, 101110101
363 2013-10-25 09:30:41 <BlueMatt> then 0-padded
364 2013-10-25 09:30:51 <CodeShark> the documentation says the bit indicates whether the node "is the parent of at least one matched leaf txid (or a matched txid itself)
365 2013-10-25 09:31:01 <BlueMatt> so, starting right per byte, byte order flips it around here
366 2013-10-25 09:31:02 <TD> it's a little-endian value
367 2013-10-25 09:31:50 <CodeShark> ok, so using the notation (node, bit) we have (0, true), (1, false), (2, true)
368 2013-10-25 09:32:02 <CodeShark> but 2 is not a parent of a matched leaf txid
369 2013-10-25 09:32:19 <TD> to decode you run the traversal in reverse, right
370 2013-10-25 09:32:51 <BlueMatt> am I looking at a different diagram? 2 is the parent of 3 and 6 which both have children
371 2013-10-25 09:32:54 <BlueMatt> which are matched
372 2013-10-25 09:32:59 <CodeShark> yes
373 2013-10-25 09:33:35 <CodeShark> is it "ancestor" rather than "parent"?
374 2013-10-25 09:33:37 <BlueMatt> so 0 is true as 2 is matched, 1 is false as it has no children, 2 is true as 3 and 6 match, 4 is true as it is a match, 5 is not...
375 2013-10-25 09:33:43 <BlueMatt> ancestor
376 2013-10-25 09:33:46 <TD> the bit means "parent of match" unless we're at the leaf node
377 2013-10-25 09:33:55 <CodeShark> parent or ancestor?
378 2013-10-25 09:34:02 <BlueMatt> ancestor
379 2013-10-25 09:34:10 <BlueMatt> or parent of something with a 1-bit
380 2013-10-25 09:35:22 <CodeShark> so (0, true), (1, false), (2, true), (3, true), (4, true), (5, false), (6, true), (7, false), (8, true)
381 2013-10-25 09:35:44 <TD> BlueMatt: the bitcoinj code for this is rather confusing because you expect a method named "recursiveExtractHashes" to recurse down the tree from the top.
382 2013-10-25 09:35:53 <TD> BlueMatt: in fact it walks up the tree from the bottom
383 2013-10-25 09:36:02 <TD> what's more it has this code:
384 2013-10-25 09:36:03 <TD> byte[] left = recursiveExtractHashes(height-1, pos*2, used, matchedHashes).getBytes(), right;
385 2013-10-25 09:36:03 <TD> / otherwise, descend into the subtrees to extract matched txids and hashes
386 2013-10-25 09:36:16 <TD> but in fact it subtracts one from the height, going UP the tree, not descending
387 2013-10-25 09:36:35 <BlueMatt> TD: blame sipa, I copied the C++ and tweaked syntax until it compiled
388 2013-10-25 09:36:45 <CodeShark> lol
389 2013-10-25 09:36:45 <TD> i think that's probably what CodeShark should do.
390 2013-10-25 09:36:48 <BlueMatt> cheating, I know, but I was pressed for time and lazy
391 2013-10-25 09:37:09 <TD> ACTION blames sipa
392 2013-10-25 09:37:21 <BlueMatt> also, I see your up/down confusion and flip your tree
393 2013-10-25 09:37:23 <BlueMatt> (or you)
394 2013-10-25 09:37:29 <CodeShark> why not just establish a canonical node labeling and a structure to indicate which nodes in the tree are hashes?
395 2013-10-25 09:37:47 <CodeShark> then let the implementation traverse as it pleases
396 2013-10-25 09:37:47 <sipa> CodeShark: sorry, battery died
397 2013-10-25 09:37:49 <BlueMatt> CodeShark: too much effort?
398 2013-10-25 09:38:02 <BlueMatt> sipa: waaatttt? no, we have magic these days, betteries never die
399 2013-10-25 09:38:13 <CodeShark> sipa, so I've posted an example of a merkleblock at http://blockhawk.net/MerkleBlock/
400 2013-10-25 09:38:19 <TD> CodeShark: originally the entire merkle tree was provided. sipa suggested this representation as it was more compact.
401 2013-10-25 09:38:41 <sipa> BlueMatt: i plugged in my phone at night, but didn't realize the cable wasn't plugged into the wall socket...
402 2013-10-25 09:38:47 <CodeShark> I'm not sure I'm understanding how the flag bits correspond to the nodes in the tree
403 2013-10-25 09:39:05 <sipa> CodeShark: do a depth-first search over the nodes in the tree
404 2013-10-25 09:39:06 <BlueMatt> wooo, data usage for last month: 15GB: suck it unlimited date
405 2013-10-25 09:39:21 <CodeShark> sipa: so is the node numbering from 0 to 8 in the correct order of traversal?
406 2013-10-25 09:39:23 <BlueMatt> sipa: ouch...I did that last monday...
407 2013-10-25 09:39:35 <BlueMatt> TD: again: lazy
408 2013-10-25 09:39:47 <TD> there's a lot to be said for simplicity
409 2013-10-25 09:39:50 <sipa> every time you encounter a node that either has 1) leafs (=txids) beneath it that need to be included or 2) level-0 (=txid) themself, output a bit
410 2013-10-25 09:40:11 <TD> you mean output a 1 bit
411 2013-10-25 09:40:16 <sipa> no, output a bit
412 2013-10-25 09:40:26 <sipa> wait
413 2013-10-25 09:40:32 <sipa> i'll need to re-read it myself
414 2013-10-25 09:40:46 <sipa> yes, that's correct
415 2013-10-25 09:41:04 <sipa> the bit balue indicates whether the nodes has children to be included or not
416 2013-10-25 09:41:07 <sipa> value
417 2013-10-25 09:41:20 <sipa> and 0 bits indicate 'this branch is pruned, don't descend further'
418 2013-10-25 09:41:27 <CodeShark> so my first question is: is the node ordering from 0 to 8 in the tree in the correct order of traversal?
419 2013-10-25 09:41:48 <TD> the BIP really needs a lot more detail on how this works
420 2013-10-25 09:42:06 <sipa> CodeShark: looks right
421 2013-10-25 09:42:06 <TD> it has a single paragraph which doesn't give enough detail. like, it says "traverse the tree in depth first order", but is that left to right or right to left?
422 2013-10-25 09:42:15 <sipa> TD: agree
423 2013-10-25 09:42:31 <CodeShark> ok, so then my second question: does each numbered node correspond to exactly one bit in the flags?
424 2013-10-25 09:42:42 <BlueMatt> TD: thats only an issue for people in non-left-to-right languages who dont usually think left-to-right
425 2013-10-25 09:42:50 <sipa> CodeShark: yes
426 2013-10-25 09:43:09 <sipa> CodeShark: which leafs are to be included?
427 2013-10-25 09:43:41 <CodeShark> ok, so then my third question: what is the correspondence? we grab each byte left-to-right and read each bit right-to-left
428 2013-10-25 09:44:01 <CodeShark> the first byte is 0x5d = 01011101
429 2013-10-25 09:44:20 <TD> the whole bit array is LE
430 2013-10-25 09:44:24 <CodeShark> this means that node 0 corresponds to 1, node 1 corresponds to 0, node 2 corresponds to 1, etc...?
431 2013-10-25 09:44:49 <sipa> bits 0,2,3,6 are certainly 1
432 2013-10-25 09:44:55 <sipa> bit 1 is certainly 0
433 2013-10-25 09:45:11 <sipa> for bits 4,5,7,8 it depends on whether that txid is to be included
434 2013-10-25 09:45:16 <TD> huh, that's not LE
435 2013-10-25 09:46:05 <CodeShark> nodes 4 and 7 match the filter
436 2013-10-25 09:46:08 <CodeShark> err
437 2013-10-25 09:46:12 <CodeShark> nodes 5 and 8
438 2013-10-25 09:46:15 <CodeShark> err
439 2013-10-25 09:46:17 <TD> sipa: in CPartialMerkleTree::TraverseAndExtract it talks about "descending" but subtracts from the height each time, which starts at the max height of the tree.
440 2013-10-25 09:46:19 <CodeShark> nodes 4 and 8
441 2013-10-25 09:46:26 <TD> sipa: so the comment doesn't appear to correspond with what the code does
442 2013-10-25 09:46:27 <CodeShark> nodes 4 and 8 match the filter
443 2013-10-25 09:46:53 <BlueMatt> TD: you're upside-down, stand on your head!
444 2013-10-25 09:47:07 <TD> /o\
445 2013-10-25 09:47:20 <sipa> in any case, the low bits go first
446 2013-10-25 09:47:21 <CodeShark> ok, so we assign a 1 bit to a node if it either matches the filter or is an ancestor of a node that matches the filter?
447 2013-10-25 09:47:36 <sipa> whether you want to call that little-endian or big-endian depends on your perspective :)
448 2013-10-25 09:47:44 <sipa> CodeShark: bingo
449 2013-10-25 09:47:51 <CodeShark> and we assign 0 to a node if its hash is included in the list but it does not match the filter?
450 2013-10-25 09:48:11 <CodeShark> (or is not a leaf)
451 2013-10-25 09:48:21 <thermoman> hi
452 2013-10-25 09:48:25 <sipa> ehhh yes, thought i would put it otherwise: you output a hash for every node that has bit 0, or is a leaf
453 2013-10-25 09:48:40 <TD> that's written from the perspective of an ENCODER.
454 2013-10-25 09:48:47 <TD> but codeshark is thinking from the perspective of a DECODER
455 2013-10-25 09:48:51 <sipa> oh
456 2013-10-25 09:48:56 <CodeShark> so the only way to know whether a 1 or 0 bit node is to be included is by the height?
457 2013-10-25 09:48:56 <TD> that's one reason this protocol is so confusing
458 2013-10-25 09:49:01 <TD> (apart from the LE/BE confusion)
459 2013-10-25 09:49:06 <BlueMatt> thermoman: figure out what was up?
460 2013-10-25 09:49:15 <sipa> CodeShark: yup
461 2013-10-25 09:49:22 <thermoman> CodeShark: our proxy node has "vin empty" also from external bitcoin clients
462 2013-10-25 09:49:30 <TD> the BIP is written the same way. it's explains how to encode, but leaves you on your own for decoding
463 2013-10-25 09:49:41 <TD> and then the code itself suffers the same problem. the comments are written as if encoding, even when decoding
464 2013-10-25 09:49:43 <thermoman> BlueMatt: not yet - still investigating
465 2013-10-25 09:49:53 <CodeShark> what most confused me is "parent" vs. "ancestor"
466 2013-10-25 09:50:12 <CodeShark> for instance, I'm not sure I'd call 2 a parent of 4
467 2013-10-25 09:50:19 <sipa> i wouldn't either
468 2013-10-25 09:50:25 <sipa> is the BIP written that way?
469 2013-10-25 09:50:35 <BlueMatt> who wrote that paragraph?
470 2013-10-25 09:50:38 <BlueMatt> TD or sipa?
471 2013-10-25 09:50:39 <CodeShark> the BIP (and the comments in both bitcoind and bitcoinj) say parent
472 2013-10-25 09:50:40 <sipa> me, probably
473 2013-10-25 09:50:43 <BlueMatt> ACTION gets the pitchfork ready
474 2013-10-25 09:50:56 <sipa> ACTION gets pitchfork()ed
475 2013-10-25 09:51:12 <BlueMatt> also, thanks to TD and sipa for writing the BIP when I was off not working on important things
476 2013-10-25 09:51:20 <TD> well, I just copied the language in the code into the BIP
477 2013-10-25 09:51:27 <BlueMatt> ACTION gets to write the code and let the smarter people write the docs
478 2013-10-25 09:51:28 <CodeShark> I figured, TD :)
479 2013-10-25 09:51:28 <TD> because frankly I didn't understand it either. so this is how we ended up in this situation
480 2013-10-25 09:51:30 <BlueMatt> wait...how did that happen
481 2013-10-25 09:51:36 <TD> basically nobody understands this except sipa
482 2013-10-25 09:51:45 <sipa> crap!
483 2013-10-25 09:51:50 <sipa> ACTION gets hit by a bus
484 2013-10-25 09:51:52 <BlueMatt> bus factor == 1
485 2013-10-25 09:52:07 <sipa> i'll try to come up with a more detailed explanation
486 2013-10-25 09:52:12 <thermoman> did you hear? we have "vin empty" also from other clients
487 2013-10-25 09:52:14 <TD> well we can always just translate the code from C++ into whatever language people need, as was done for the rest of bitcoin :)
488 2013-10-25 09:52:27 <BlueMatt> thermoman: again, unmodified node? strange.
489 2013-10-25 09:52:30 <sipa> i believe i promised that when the BIP was written, i'd rewrite that section
490 2013-10-25 09:52:33 <thermoman> yes, unmodified
491 2013-10-25 09:52:42 <sipa> thermoman: sure, we've been seeing vin empty's for a while, without explanation
492 2013-10-25 09:52:52 <BlueMatt> TD: hey, thats what bitcoinj's script engine is...hasnt failed yet
493 2013-10-25 09:52:55 <BlueMatt> oh, wait, yes it has
494 2013-10-25 09:53:02 <TD> sipa: can i take you up on that promise? a worked example would help a lot
495 2013-10-25 09:53:11 <sipa> TD: poke me
496 2013-10-25 09:53:14 <BlueMatt> ACTION shouldn't be allowed to touch code anymore...Ive only cause, what, 3 bitcoind vulns?
497 2013-10-25 09:53:15 <CodeShark> I'll put together a couple more examples and perhaps we can add these tree diagrams to the documentation
498 2013-10-25 09:53:15 <TD> sipa: also correcting the comments in the code :) where it talks about "descending" it should talk about "ascending"
499 2013-10-25 09:53:16 <sipa> (if i forget)
500 2013-10-25 09:53:19 <TD> ok
501 2013-10-25 09:53:43 <sipa> thermoman: my guess is a mapWallet[somehash] call somewhere that instanciates a empty CTransaction, which gets broadcasted
502 2013-10-25 09:54:02 <sipa> i'm sort of ignoring wallet stuff for now though, working on other things
503 2013-10-25 09:54:17 <BlueMatt> can we just remove wallet altogether and move on?
504 2013-10-25 09:54:19 <CodeShark> I'm going to try a slightly more complex example - and see if my understanding of the flag bits corresponds to it
505 2013-10-25 09:54:26 <BlueMatt> let others deal with wallet
506 2013-10-25 09:54:28 <BlueMatt> like TD
507 2013-10-25 09:54:56 <sipa> so yes, when decoding, you read a bit, and if it is either 0, or a leaf node, you read a corresponding hash
508 2013-10-25 09:55:34 <sipa> if it's a 1 and not a leaf node, you descend into its children, to compute their hash
509 2013-10-25 09:55:38 <sipa> and use that
510 2013-10-25 09:55:56 <CodeShark> so the representation tries to optimize for space rather than logic in decoding
511 2013-10-25 09:56:03 <sipa> yes, absolutely
512 2013-10-25 09:56:09 <sipa> it's designed to be very small
513 2013-10-25 09:56:14 <thermoman> can i ignore these empty vins for now and just unban the child node or is there anything to be concerned with?
514 2013-10-25 09:56:25 <BlueMatt> its just a tree traversal, a horribly undocumented tree traversal
515 2013-10-25 09:56:41 <BlueMatt> (and revere-confusingly documented tree traversal)
516 2013-10-25 09:56:43 <sipa> thermoman: probably yes
517 2013-10-25 09:57:10 <BlueMatt> thermoman: double-check your bin is trusted, and then just unban and have fun
518 2013-10-25 09:57:17 <sipa> CodeShark: in particular, it's designed to have near-0 overhead for a full-block match
519 2013-10-25 09:57:45 <CodeShark> sipa: I think I get it - writing another slightly more complex example now
520 2013-10-25 09:58:01 <thermoman> BlueMatt: bin?
521 2013-10-25 09:58:14 <BlueMatt> s/bin/binary you're running/
522 2013-10-25 09:58:24 <thermoman> it's self compiled from source
523 2013-10-25 10:11:21 <tgs3> bus are so last century, in democracy we say dronn'ed
524 2013-10-25 10:11:37 <BlueMatt> bugs == unintended features
525 2013-10-25 10:20:32 <dobry-den> What stops the current client from download blocks from multiple peers in parallel?
526 2013-10-25 10:22:45 <wumpus> dobry-den: knowledge of which blocks exist, see https://github.com/bitcoin/bitcoin/pull/2964
527 2013-10-25 10:33:58 <dobry-den> oh, duh. node simply doesnt know
528 2013-10-25 10:36:03 <dobry-den> why cant a node grab the latest hash in the inv-vect and use it to start downloading the next chunk from another node?
529 2013-10-25 10:39:00 <BlueMatt> dobry-den: good code?
530 2013-10-25 10:39:19 <BlueMatt> dobry-den: https://github.com/bitcoin/bitcoin/pull/2964
531 2013-10-25 10:58:27 <feddy3> would this site be helpful to you devs? http://fetchtx.info/ ?
532 2013-10-25 11:01:47 <sipa> i doubt it, to be honest; you can just ask your own node for a transaction
533 2013-10-25 11:02:22 <feddy3> I was hoping the dissection display would be nice
534 2013-10-25 11:02:41 <sipa> oh, i didn't recheck it since the last time you posted it
535 2013-10-25 11:02:54 <feddy3> Been working on it a little bit
536 2013-10-25 11:03:05 <feddy3> The next step is to parse the script into a human-readable form
537 2013-10-25 11:03:18 <wumpus> looks pretty nice
538 2013-10-25 11:03:28 <feddy3> eventually, it could also verify signatures
539 2013-10-25 11:03:45 <feddy3> and adding a pushtx page would be trivial but I'm not sure I want that traffic
540 2013-10-25 11:04:25 <wumpus> I think it can be pretty useful to people trying to understand the block chain format, especially if you aadd script parsing
541 2013-10-25 11:04:31 <michagogo> cloud|What's the txid ("edbc..." maybe?) of that broken pushdata TZ?
542 2013-10-25 11:04:40 <michagogo> cloud|Tx*
543 2013-10-25 11:04:53 <feddy3> ebc9fa1196a59e192352d76c0f6e73167046b9d37b8302b6bb6968dfd279b767 this one?
544 2013-10-25 11:04:57 <michagogo> cloud|Yes
545 2013-10-25 11:05:09 <michagogo> cloud|Thanks
546 2013-10-25 11:05:21 <sipa> feddy3: oh, that looks really nice indeed!
547 2013-10-25 11:05:30 <feddy3> really?
548 2013-10-25 11:05:31 <sipa> sorry, previous time i looked it just showed the hex code
549 2013-10-25 11:05:41 <feddy3> sipa: yeah, last time I mentioend was probably a bit premature
550 2013-10-25 11:06:15 <michagogo> cloud|Ooh!
551 2013-10-25 11:06:27 <michagogo> cloud|That seems like something really handy...
552 2013-10-25 11:06:59 <feddy3> michagogo: especially once I finish with the script parsing, it'd be useful (I think) to analyze them and the data contained in them
553 2013-10-25 11:07:07 <michagogo> cloud|feddy3: FR: paste in a raw transaction to dissect
554 2013-10-25 11:07:21 <CodeShark> sipa: a slightly more complex example: http://blockhawk.net/MerkleBlock/index2.html
555 2013-10-25 11:07:24 <sipa> agree with michagogo|cloud, that sounds useful
556 2013-10-25 11:07:42 <feddy3> ahh pasting shouldn't be difficult
557 2013-10-25 11:07:51 <feddy3> all the dissection is happening in the client
558 2013-10-25 11:07:56 <sipa> CodeShark: looks right
559 2013-10-25 11:07:58 <michagogo> cloud|Ah, really?
560 2013-10-25 11:08:00 <michagogo> cloud|Cool
561 2013-10-25 11:08:21 <michagogo> cloud|(Watch out for things like gmaxwell's <alert> tx
562 2013-10-25 11:08:32 <feddy3> ahh, cool. will do
563 2013-10-25 11:08:39 <feddy3> i gotta step afk. gym time:)
564 2013-10-25 11:08:46 <feddy3> thanks for the feedback!
565 2013-10-25 11:08:48 <sipa> feddy3: whatever you do, don't show decoded binary data without html quoting it
566 2013-10-25 11:09:05 <feddy3> why?
567 2013-10-25 11:09:15 <michagogo> cloud|Er, <script>alert(), I mean
568 2013-10-25 11:09:16 <feddy3> embedded images and things?
569 2013-10-25 11:09:29 <sipa> feddy3: and javascript
570 2013-10-25 11:09:39 <feddy3> raw tx data is never added to the document
571 2013-10-25 11:09:55 <michagogo> cloud|feddy3: there's a certain transaction where until recently, if you went to the bc.i page, an alert box popped up
572 2013-10-25 11:10:17 <CodeShark> sipa, the way I'd define the node bits then is 1 iff the node lies along a path to a matched leaf (including the leaf itself)
573 2013-10-25 11:10:24 <michagogo> cloud|gmaxwell put JavaScript into a script, which bc.i decoded and displayed
574 2013-10-25 11:10:29 <feddy3> haha nice
575 2013-10-25 11:10:33 <feddy3> yeah, they show the raw tx contents
576 2013-10-25 11:11:12 <michagogo> cloud|Something like "if I were an attacker, you'd have lost your coins"
577 2013-10-25 11:11:26 <feddy3> afk:)
578 2013-10-25 11:11:53 <sipa> CodeShark: sounds good
579 2013-10-25 11:11:59 <michagogo> cloud|;;Google JavaScript inside bitcoin transaction gmaxwell
580 2013-10-25 11:12:21 <gribble> JavaScript Library to Sign Transaction - Bitcoin Forum: <https://bitcointalk.org/index.php?topic=264127.0>; 1564 - BitcoinStats: <http://bitcoinstats.com/irc/bitcoin-dev/logs/2013/09/10>; 1568 - BitcoinStats: <http://bitcoinstats.com/irc/bitcoin-dev/logs/2013/07/03>
581 2013-10-25 11:13:48 <sipa> CodeShark: and descending happens in case of a non-leaf 1 bit
582 2013-10-25 11:14:00 <CodeShark> sipa: right
583 2013-10-25 11:14:05 <sipa> CodeShark: and hash outputting happens in case of non-descending
584 2013-10-25 11:14:23 <CodeShark> we keep descending until we either hit a 0 or hit a leaf
585 2013-10-25 11:16:59 <sipa> indeed
586 2013-10-25 11:17:10 <CodeShark> the depth can be calculated as ceil(log_2(nTx)) - and we can optimize the implementation of this as needed
587 2013-10-25 11:17:44 <CodeShark> although surely this calculation is not a bottleneck :P
588 2013-10-25 11:18:57 <sipa> i think the only (and that's perhaps even trivial) optimization, is how you determine whether a node has matched leafs under it
589 2013-10-25 11:19:16 <sipa> in particular, don't recalculate it for every node :)
590 2013-10-25 11:30:36 <dobry-den> does a node only write to the addr_from host:port and not through the host:port that connected to its socket?
591 2013-10-25 11:38:06 <sipa> not sure what you mean?
592 2013-10-25 11:38:16 <sipa> you're referring to the field in the version message?
593 2013-10-25 11:41:42 <CodeShark> time for bed - thanks for the help, sipa
594 2013-10-25 11:42:08 <CodeShark> tomorrow I'll try to finish implementing filtered block sync
595 2013-10-25 11:45:01 <dobry-den> sipa: I've connected to my bitcoind's socket, I write to it, but it never writes anything back
596 2013-10-25 11:46:00 <CodeShark> dobry-den: I would recommend posting the exact raw data you're sending
597 2013-10-25 12:11:51 <dobry-den> Here's the hex I'm trying to send https://www.refheap.com/20165
598 2013-10-25 12:13:39 <sipa> dobry-den: is there any data in debug.log appearing in bitcoind when you send it?
599 2013-10-25 12:13:45 <sipa> passing -debug may help
600 2013-10-25 12:19:08 <dobry-den> sipa: it reports that it accepted connection, the connection timeout
601 2013-10-25 12:19:40 <sipa> do you have message length, checksum, magic bytes, ... right?
602 2013-10-25 12:19:46 <dobry-den> yes
603 2013-10-25 12:20:19 <dobry-den> it also passes bitcoinj and bitcoin-ruby's versionmessage test suites
604 2013-10-25 12:21:05 <dobry-den> at least parsing it. so i'm guessing im missing something obvious
605 2013-10-25 12:26:41 <sipa> dobry-den: i think your size is wrong
606 2013-10-25 12:26:54 <sipa> the size doesn't include the header afaik
607 2013-10-25 12:31:42 <dobry-den> yeah, it's just the payload
608 2013-10-25 12:31:50 <dobry-den> 103
609 2013-10-25 12:34:17 <sipa> you're right
610 2013-10-25 12:50:07 <dobry-den> sipa: wow
611 2013-10-25 12:50:36 <dobry-den> sipa: just had to (.flush buffer)
612 2013-10-25 12:50:41 <dobry-den> kill me
613 2013-10-25 12:50:49 <sipa> /kill dobry-den
614 2013-10-25 12:51:09 <sipa> what language is that?
615 2013-10-25 12:51:27 <dobry-den> im writing a client with clojure
616 2013-10-25 12:51:33 <sipa> cool
617 2013-10-25 12:56:23 <dobry-den> im storing the blockchain in datomic (http://www.datomic.com/) so i can query it with datalog.
618 2013-10-25 13:00:08 <dobry-den> it's like a temporal database that models the blockchain
619 2013-10-25 13:28:54 <jgarzik> TD, (re email) BitPay has a prelim server version of PP. I am working on OSX builds of Bitcoin-QT from HEAD, to enable the client side.
620 2013-10-25 13:29:03 <TD> cool
621 2013-10-25 13:29:06 <jgarzik> TD, people might test more, if there were semi-official test builds
622 2013-10-25 13:29:11 <TD> yeah
623 2013-10-25 13:29:18 <TD> we could really use automated nightlies of bitcoin-qt
624 2013-10-25 13:29:41 <TD> how can devs access your prelim server implementation?
625 2013-10-25 13:30:21 <gmaxwell> sipa: You around?
626 2013-10-25 13:31:45 <michagogo> cloud|sipa: didn't know you were a freenode staffer :-P
627 2013-10-25 13:39:04 <Ry4an> the automated nightlies could default to testnet perhaps
628 2013-10-25 13:43:19 <sipa> gmaxwell: i am
629 2013-10-25 13:43:38 <sipa> michagogo|cloud: huh?
630 2013-10-25 13:43:44 <sipa> michagogo|cloud: ah :)
631 2013-10-25 13:44:00 <ecoloco> whats happend to the bitcoin rate? do you think its gonna bounce up again?
632 2013-10-25 13:44:40 <Ry4an> ecoloco: wrong channel
633 2013-10-25 13:45:13 <kjj> there is some natural variability around the bitcoin rate, but it averages out well in the long run. 25 per 10 minutes, on average, for the next few years
634 2013-10-25 13:45:31 <Ry4an> kjj: :)
635 2013-10-25 13:45:32 <sipa> kjj: i expect he's talking about the exchange rate
636 2013-10-25 13:45:41 <Ry4an> sipa: I expect kjj knew that and was being clever
637 2013-10-25 13:46:05 <Ry4an> ecoloco: try #bitcoin-market if you want to talk exchange rates to other currencies
638 2013-10-25 13:46:10 <sipa> sorry, i need to improve my reading-between-the-lines
639 2013-10-25 13:46:13 <kjj> that is the only bitcoin rate that is on topic in this channel
640 2013-10-25 13:46:24 <sipa> or #bitcoin-pricetalk (if that still exists)
641 2013-10-25 13:47:08 <ecoloco> sipa: thanks
642 2013-10-25 13:48:16 <gmaxwell> sipa: So I'm confused about your comment about it not being urgent to fix the disconnect block problem. The conclusion I had reached was that it could produce forks between git and 0.8.5? Where am I confused?
643 2013-10-25 13:49:11 <sipa> gmaxwell: well people shouldn't be running git head...
644 2013-10-25 13:50:58 <sipa> gmaxwell: and as i understand it, it can't create forks... only cause nodes to fail starting up?
645 2013-10-25 13:51:34 <sipa> hmm, maybe in a reorg it can cause a fork
646 2013-10-25 13:56:03 <thermoman> oh boy
647 2013-10-25 13:56:04 <thermoman> http://media.coindesk.com/2013/10/human-error-bitcoin-loss.png
648 2013-10-25 13:56:57 <lianj> ^^
649 2013-10-25 13:57:00 <edcba> lol
650 2013-10-25 13:57:20 <ecoloco> shit
651 2013-10-25 13:57:24 <ecoloco> :S
652 2013-10-25 13:57:24 <gmaxwell> How do you know that was an error and not some fruitcake intentionally giving away a bunch of coins?
653 2013-10-25 13:57:35 <sipa> Sense?
654 2013-10-25 13:57:56 <ecoloco> painful
655 2013-10-25 13:58:00 <ecoloco> or what?
656 2013-10-25 13:58:01 <gmaxwell> sipa: yea, well shouldn't but they do. (and, scroll up a few lines... "automated nightlies" :) )
657 2013-10-25 13:58:22 <lianj> wasn't there a case where a guy got lucky and the miner gave him like half back
658 2013-10-25 13:58:29 <gmaxwell> sipa: yea, I thought it could make git head refuse to reorg off a block that contains an op_return only transaction.
659 2013-10-25 13:58:53 <sipa> gmaxwell: yup, that seems correct
660 2013-10-25 13:59:25 <gmaxwell> lianj: there have been a couple cases where large fees have been given back, and a couple cases where pools offered to give them back but no one came asking for them.
661 2013-10-25 14:00:09 <gmaxwell> lianj: sendrawtransaction in bitcoin-qt now requires a manual override switch to send something with a fee that large.
662 2013-10-25 14:00:44 <Apocalyptic> gmaxwell, does it in bitcoind as well ?
663 2013-10-25 14:00:59 <gmaxwell> Apocalyptic: they're one and the same.