1 2013-07-22 00:43:35 <Luke-Jr> ACTION ponders if the warning in LevelDB might be related to the corruption issues
2 2013-07-22 00:43:50 <Luke-Jr> probably not, if the file is posix_logger.h
3 2013-07-22 00:59:42 <JyZyXEL> just how unsafe it is to accept 0/unconfirmed as a payment?
4 2013-07-22 00:59:56 <gmaxwell> as payment for what?
5 2013-07-22 01:00:03 <JyZyXEL> for goods and services
6 2013-07-22 01:00:11 <gmaxwell> the details matter.
7 2013-07-22 01:00:42 <JyZyXEL> well i heard in this podcast that people on some festival were accepting bitcoin payments with 0 confirmations
8 2013-07-22 01:00:43 <gmaxwell> it's pretty easy to reverse an unconfirmed payment. If you're operating in a context where thats bad, then thats bad.
9 2013-07-22 01:01:25 <gmaxwell> JyZyXEL: presumably, if they even understood the risks at all, they just planned on eating whatever losses they had.
10 2013-07-22 01:01:32 <gmaxwell> Sometimes thats perfectly reasonable.
11 2013-07-22 01:01:52 <gmaxwell> Often if the txn is reversed you can abort the transaction, and then thats fine too.
12 2013-07-22 01:02:09 <gwillen> JyZyXEL: in general, it's totally unsafe and you shouldn't do it
13 2013-07-22 01:02:11 <JyZyXEL> i guess they were selling food items and such and couldn't really have people waiting there for the confirmations or something :p
14 2013-07-22 01:02:30 <gwillen> JyZyXEL: but like, if you're selling someone access to a website, it doesn't matter, because if they reverse the payment you just cut off their access
15 2013-07-22 01:02:40 <gmaxwell> Or you know you you were transacting with and "wtf, dude, pay up or I call your mom"
16 2013-07-22 01:02:43 <gwillen> JyZyXEL: and if you're selling a doughnut, it's probably so cheap that it doesn't amtter
17 2013-07-22 01:03:26 <gmaxwell> but in cases where you can't eat it, and where you can't abort, and where you can't hunt them down ... and especially if one person could screw you over and over again (e.g. with a bot) then you shouldn't be accepting them.
18 2013-07-22 01:03:56 <gmaxwell> (you probably shouldn't be accepting just 1 in that case either)
19 2013-07-22 01:04:17 <gmaxwell> (esp if the attack could be automated)
20 2013-07-22 01:04:35 <gwillen> in general accepting 0 is not a good idea unless you have a detailed understanding of the specific ways that low-confirm bitcoin transactions can be reversed, OR the transaction is zero-risk for you even if it does get reversed
21 2013-07-22 01:05:04 <JyZyXEL> i'd like to know more how does one reverse it?
22 2013-07-22 01:05:14 <gmaxwell> as often as they want.
23 2013-07-22 01:05:41 <gmaxwell> The question you're asking about is not a random failure. It's not something that has a rate like a failing lightbulb.
24 2013-07-22 01:05:59 <gwillen> gmaxwell: not how often, how
25 2013-07-22 01:06:26 <gwillen> JyZyXEL: with zero confirms, they can do things like sending two conflicting transactions to different nodes in the network
26 2013-07-22 01:06:35 <gwillen> and only one of those can succeed
27 2013-07-22 01:06:49 <gwillen> so if they can get you to see one, and get the other one to succeed, then they can make you think they're paying you, but really pay themselves
28 2013-07-22 01:07:10 <gmaxwell> gwillen: thanks for the reading comprehension help. :P
29 2013-07-22 01:07:14 <gwillen> np :-)
30 2013-07-22 01:07:49 <gmaxwell> Yes, the simplest way is to simply produce two payments and ~concurrently announce.
31 2013-07-22 01:08:08 <gmaxwell> blockchain.info even has a handy integrated tool for doing this attack.
32 2013-07-22 01:08:13 <michagogo> JyZyXEL: This is especially bad if they can find and connect to your node.
33 2013-07-22 01:09:14 <gmaxwell> michagogo: you can actually use doublespend propagations to find the relevant nodes if you can pay a couple times.
34 2013-07-22 01:09:27 <gmaxwell> michagogo: but really doing that isn't supremely important.
35 2013-07-22 01:10:18 <gmaxwell> accepting unconfirmed also exposes you to things like someone paying you with a very low priority txn that will fall out of mempools before getting mined. Makes it pretty easy to reliably replace with one that has a fee.
36 2013-07-22 01:10:34 <michagogo> gmaxwell: Right, but it's still a way to improve the success rates (i.e. the merchant seeing your transaction and not the conflicting one)
37 2013-07-22 01:15:57 <JyZyXEL> pay the merchant with normal fee and then make another competing transaction with higher fee?
38 2013-07-22 01:17:10 <JyZyXEL> since miners cannot include both transactions in to a block they would probably prefer to include the one with higher fee?
39 2013-07-22 01:17:24 <gwillen> JyZyXEL: right now in general it doesn't work quite that way
40 2013-07-22 01:17:42 <gwillen> JyZyXEL: but the same general idea, that you get the merchant to see one transaction, and the miner to mine a different one
41 2013-07-22 01:19:22 <JyZyXEL> if i was a merchant i would have my wallet connected to very popular nodes like blockchain.info or something like that
42 2013-07-22 01:27:44 <michagogo> oops
43 2013-07-22 01:34:18 <gwillen> JyZyXEL: yes, that's an important step to protect yourself as a merchant
44 2013-07-22 01:34:28 <gwillen> JyZyXEL: really you want to be connected to all the major mining pools
45 2013-07-22 01:34:53 <gwillen> JyZyXEL: if you _receive_ the same transaction from each of the major mining pools' nodes, you have a good chance that a conflicting one will not be mined
46 2013-07-22 01:35:08 <gwillen> but I am not an expert so you shouldn't stake any money on my claim; there are more subtleties, undoubtedly
47 2013-07-22 01:35:45 <gmaxwell> gwillen: there is basically no software to do what you're suggesting, and it's not clear how worthwhile it is.
48 2013-07-22 01:36:01 <JyZyXEL> and if you receive two transactions for the same coins, then something fishy is going on? :p
49 2013-07-22 01:36:21 <gmaxwell> The problem is that getting a txn from a mining pool is no guarentee that they're mining it (any mining pool you could connect to runs many nodes, and the important ones aren't exposed)
50 2013-07-22 01:36:38 <gmaxwell> certantly being well connected is helpful though.
51 2013-07-22 01:37:04 <gwillen> JyZyXEL: listen to gmaxwell rather than me
52 2013-07-22 01:37:08 <gwillen> he knows of what he speaks
53 2013-07-22 01:37:13 <MC1984> does going out of your way to peer with "major mining pools" really grant you useful extra protection
54 2013-07-22 01:37:58 <gmaxwell> MC1984: as I was saying, probably not. Being well connected is good, otoh: well connected sometimes blinds you to conflicts, and no widely used software warns you about them either, AFAIK.
55 2013-07-22 01:38:16 <MC1984> simply well connected is fine
56 2013-07-22 01:38:22 <MC1984> a given even
57 2013-07-22 01:39:31 <MC1984> i want to be able to listen on and inject txn thru the jankiest couple of nodes in the netwok and not have it be a problem
58 2013-07-22 01:40:03 <MC1984> eg decently amorphous fuzzball of a network
59 2013-07-22 01:41:40 <JyZyXEL> so if you send two competing transactions (spending the same coins) to very well connected nodes, which ever transaction gets the first confirmation is likely to be the one that wins?
60 2013-07-22 01:42:17 <gwillen> JyZyXEL: yes, and one-confirmation is much much safer than zero
61 2013-07-22 01:42:25 <gwillen> (but still significantly less safe than two)
62 2013-07-22 01:42:50 <gmaxwell> gwillen: well maybe safer! depends on the attack being done. But yes, in the case where two get announced the first to get a confirm is likely the survivor.
63 2013-07-22 01:43:01 <JyZyXEL> would the miners who know about the one confirmation even care about the other transaction anymore?
64 2013-07-22 01:43:11 <gwillen> gmaxwell: well, safer in that it doesn't take a lot in the way of resources or help to attack zero-confirm
65 2013-07-22 01:43:15 <gmaxwell> JyZyXEL: because of this awesome thing called latency.
66 2013-07-22 01:43:22 <gwillen> whereas to successfully attack one-confirm, you need _some_ mining power
67 2013-07-22 01:43:26 <gwillen> for zero-confirm you really don't need any
68 2013-07-22 01:43:29 <gmaxwell> gwillen: no, not so.
69 2013-07-22 01:43:32 <gwillen> no?
70 2013-07-22 01:44:00 <gmaxwell> Say I broadcast two transactions. Miner A gets txn A, miner B gets txn B first.
71 2013-07-22 01:44:20 <gmaxwell> now A produces a block, but 4 seconds later B also produces a block.
72 2013-07-22 01:44:33 <gmaxwell> You see A's confirm.. you now have 1 confirmed.
73 2013-07-22 01:44:36 <gwillen> okay, but now we're talking about attacks that can only be executed with a great deal of luck or patience
74 2013-07-22 01:44:47 <gmaxwell> gwillen: this sequence happens a couple times a day naturally.
75 2013-07-22 01:44:50 <gwillen> whereas zero-confirm attacks do not require nearly as much of either
76 2013-07-22 01:44:51 <gwillen> hm, *nods*
77 2013-07-22 01:45:11 <gmaxwell> gwillen: so if you constantly send two transactions you'll hit 1 and get a reversal every once in a while without any hashpower.
78 2013-07-22 01:45:11 <gwillen> well, but it's not enough for that sequence to happen
79 2013-07-22 01:45:17 <gwillen> you need that sequence to happen AND you need B's later block to win
80 2013-07-22 01:45:37 <gwillen> anyway.
81 2013-07-22 01:45:46 <gmaxwell> Yes, and then B gets another block or C decided to extend B because it heard B first. (not at all unlikely with a 4 second span)
82 2013-07-22 01:46:16 <gmaxwell> gwillen: the exact rate depends on the network, it's not hard to demonstrate though.
83 2013-07-22 01:46:20 <gwillen> ACTION nods
84 2013-07-22 01:47:13 <gmaxwell> if you instead assume the attacker hash access to hashpower (e.g. via something like one of those mining resale services) then thats a different set of risks.
85 2013-07-22 01:47:52 <MC1984> how much does it really cost to pull a finney now
86 2013-07-22 01:49:07 <gmaxwell> I actually think all those hashpower sale services are currently out of business again.
87 2013-07-22 01:49:36 <MC1984> lol
88 2013-07-22 01:50:14 <JyZyXEL> one thing that i like is how the transactions from an orphan block get automatically re-added to the pool of queued transactions so that lessens the risk of having an accidental reversal of payment
89 2013-07-22 01:50:37 <gmaxwell> yes, though they won't bump already queued transactions.
90 2013-07-22 01:51:04 <MC1984> oh github has changed marginally
91 2013-07-22 01:51:54 <MC1984> merged #2670 qt: allow user to choose data directory
92 2013-07-22 01:51:55 <MC1984> nice
93 2013-07-22 01:56:21 <gmaxwell> gwillen: in any case, thinking about low confirm security is hard.
94 2013-07-22 01:56:27 <gwillen> ACTION nods
95 2013-07-22 01:56:28 <gwillen> agreed
96 2013-07-22 01:57:24 <gmaxwell> it's especially a problem for things unattended things that allow depost and withdraw.
97 2013-07-22 01:57:49 <gmaxwell> lots of otherwise infesable attacks??? because they have low success rates??? work fine if you can try over and over again.
98 2013-07-22 02:09:18 <Tabaza> where are the developers?
99 2013-07-22 02:09:21 <Tabaza> i mean
100 2013-07-22 02:09:23 <Tabaza> the serious ones?
101 2013-07-22 02:09:52 <michagogo> Tabaza: In here, many of them
102 2013-07-22 02:10:05 <michagogo> Why?
103 2013-07-22 02:10:31 <Tabaza> i want to develop bitcoin trading/exchange system
104 2013-07-22 02:10:37 <Tabaza> to be secured and reliable
105 2013-07-22 02:10:47 <Tabaza> can y ou highlight some companies who can do the task?
106 2013-07-22 02:17:43 <Tabaza> michagogo?
107 2013-07-22 02:19:29 <TheLordOfTime> Tabaza: patience. it is a virtue.
108 2013-07-22 02:19:38 <TheLordOfTime> michagogo could also be slightly busy so... :P
109 2013-07-22 02:21:08 <Tabaza> yeah
110 2013-07-22 02:21:19 <Tabaza> everyday pass without my system up
111 2013-07-22 02:21:23 <Tabaza> i am losing money :P
112 2013-07-22 02:21:29 <TheLordOfTime> oh geez, it's stamit again
113 2013-07-22 02:21:30 <TheLordOfTime> :/
114 2013-07-22 02:21:57 <TheLordOfTime> this is the only place he can really join to though... and even then he can't do anything last i checked
115 2013-07-22 02:22:01 <TheLordOfTime> yep he can't.
116 2013-07-22 02:27:37 <gjs278> why the hell does freenode disconnect to often jesus
117 2013-07-22 02:38:36 <TheLordOfTime> gjs278: no idea.
118 2013-07-22 02:38:50 <TheLordOfTime> just live with it for a while because it looks like they're doing server maintenance and shuffling
119 2013-07-22 02:40:16 <gjs278> yeah on reconnect my irc client stopped trying because it somehow connected to a server that didn't have ssl on
120 2013-07-22 02:43:39 <TheLordOfTime> that was probably the newest server, maybe.
121 2013-07-22 02:43:47 <TheLordOfTime> but anyways, all looks good now. :)
122 2013-07-22 03:58:15 <Luke-Jr> sipa: ping
123 2013-07-22 04:00:22 <Luke-Jr> nm
124 2013-07-22 04:09:16 <sipa> Luke-Jr: ok?
125 2013-07-22 04:09:38 <Luke-Jr> sipa: your BIP32 tests are doing some fancy things :p
126 2013-07-22 04:09:54 <Luke-Jr> and the BIP32 pull doesn't work with your secp branch <.<
127 2013-07-22 04:10:02 <sipa> i know
128 2013-07-22 04:10:18 <sipa> i'll rebase once it's merged
129 2013-07-22 04:10:45 <sipa> is overloading operator() fancy? :p
130 2013-07-22 04:12:29 <Luke-Jr> sipa: well, basing the next test on the results of the previous
131 2013-07-22 04:13:14 <Luke-Jr> I had #ifndef'd out the derive-based tests, and missed that their result influenced the next test run
132 2013-07-22 04:13:59 <sipa> ah, right
133 2013-07-22 07:25:37 <swulf--> sipa: ever seen this one? "bool LoadExternalBlockFile(FILE*, CDiskBlockPos*)() : Deserialize or I/O error caught during load"
134 2013-07-22 09:50:37 <imton> guys, good morning :)
135 2013-07-22 09:51:05 <imton> sips is there a way to get bitcoind push me when a new tx is made to and address?
136 2013-07-22 09:51:09 <imton> ipa
137 2013-07-22 09:51:10 <imton> sipa
138 2013-07-22 09:51:42 <imton> I want to implement a websockets and I need bitcoind to notify me when a new tx is made to an address instead of polling it
139 2013-07-22 09:52:21 <sipa> -walletnotify
140 2013-07-22 09:53:19 <imton> oh???.
141 2013-07-22 09:53:48 <imton> but??? if I had hundreds of addresses (may be thousands) ??? will it scale?
142 2013-07-22 09:53:53 <sipa> unlikely
143 2013-07-22 09:54:00 <imton> :(
144 2013-07-22 09:54:14 <imton> what do you recommend me?
145 2013-07-22 09:54:22 <sipa> also, afaik, it doesn't support your use case anyway
146 2013-07-22 09:54:33 <sipa> as it only notifies you for transactions to your wallet addresses
147 2013-07-22 09:54:36 <imton> yeah...
148 2013-07-22 09:54:53 <sipa> i think there are better tools around than bitcoind
149 2013-07-22 09:54:57 <sipa> for what you're doing
150 2013-07-22 09:55:00 <sipa> at least for now
151 2013-07-22 09:55:29 <imton> well, I reached a really good implementation yesterday -finally-
152 2013-07-22 09:55:52 <imton> I get all the info from searchfromrawtransactions and mix it with mempool too, I am kind of happy :)
153 2013-07-22 09:56:50 <imton> sipa http://cl.ly/QNUL
154 2013-07-22 09:57:20 <imton> so, first of all thank you for writing that rpc that you hate most :)
155 2013-07-22 09:57:24 <imton> spia
156 2013-07-22 09:57:26 <imton> sipa
157 2013-07-22 10:01:40 <imton> sipa: my idea was to have a service polling the mempool and finding if that txs cotain any of the addresses requested. May be 1 sec delay. will that work?
158 2013-07-22 10:02:21 <imton> sipa: so then i only have 1 service polling not each client refreshing the site so frequently (which they will do any ways)
159 2013-07-22 12:19:50 <jgarzik> mornin'
160 2013-07-22 12:20:40 <sipa> 'fternoon
161 2013-07-22 12:37:43 <jgarzik> sipa, RE #2840, what is cs_main supposed to cover? Just "main message processing"?
162 2013-07-22 12:38:03 <jgarzik> asking "supposed to cover" not "does cover"
163 2013-07-22 12:44:57 <Vinnie_win> You guys ever gonna merge my pull request for leveldb?
164 2013-07-22 12:51:29 <sipa> Vinnie_win: sorry, i've had very little time
165 2013-07-22 12:51:45 <sipa> Vinnie_win: i've created a leveldb repo under bitcoin, but still haven't found the time to learn git subtree
166 2013-07-22 12:52:19 <sipa> jgarzik: cs_main is our main (!) source of non-parallellism really
167 2013-07-22 12:52:30 <sipa> jgarzik: as pretty much everything locks it (apart from the network thread)
168 2013-07-22 12:52:51 <jgarzik> Vinnie_win, ditto sipa
169 2013-07-22 12:53:20 <sipa> Vinnie_win: anyway, there's consensus i think to use that mechanism
170 2013-07-22 12:53:43 <sipa> Vinnie_win: and there are also no roadblocks to upgrading to a newer leveldb
171 2013-07-22 12:55:06 <sipa> jgarzik: i'm working on headers-first syncing (it's currently working perfectly, except it is headers-first-and-then-nothing), and i think i'll find more code that unnecessarily locks cs_main
172 2013-07-22 12:55:52 <sipa> jgarzik: in particular, cs_main protects modifications to mapBlockIndex, vBlockIndexByHeight, the chainstate, and pretty much all main.cpp globals
173 2013-07-22 12:56:43 <jgarzik> sipa, so the public block chain engine
174 2013-07-22 12:56:55 <jgarzik> ACTION kicks OSX autocorrect. blockchain dammit.
175 2013-07-22 13:02:45 <sipa> jgarzik: that blacklist rpc and reduce cs_main are just two things i've found while working on headers-first
176 2013-07-22 13:03:29 <sipa> as that will probably result in more edge-cases in reorganizations
177 2013-07-22 13:08:59 <jgarzik> sipa, headers-first or reverse-headers-first?
178 2013-07-22 13:09:18 <jgarzik> sipa, We still don't have anyway to find (a) best chain or (b) locator from remote P2P node, correct?
179 2013-07-22 13:12:02 <sipa> jgarzik: it uses the 'headers' command to do a forward sync of the best chain, validating syntactic correctness and PoW
180 2013-07-22 13:14:38 <sipa> jgarzik: and then afterwards (well, really simultaneously), starts fetching blocks along this chain (it already knows it is the best one), and connects them
181 2013-07-22 13:14:53 <sipa> advantage is that you can do out-of-order block fetching
182 2013-07-22 13:15:22 <sipa> and don't need checkpoints for efficiently (you can set a rule like "don't validate signatures if buried under 3 months of blocks")
183 2013-07-22 13:16:48 <petertodd> nice: 315ac7d4c26d69668129cc352851d9389b4a6868f1509c6c8b66bead11e2619f turns out SIGHASH_SINGLE is a valid flag, even without a corresponding txout, SignatureHash() printf()'s an error, but it returns 1, which is duitifully checked and the tx goes through...
184 2013-07-22 13:16:48 <sipa> and always know whether you're up-to-date or not
185 2013-07-22 13:16:55 <sipa> jgarzik: and there is no more concept of orphan blocks... you never fetch a block that you can't connect in the first place
186 2013-07-22 13:16:59 <petertodd> bitcoin-ruby doesn't handle this case, so webbtc.com just stopped working for instance
187 2013-07-22 13:17:19 <CodeShar_> +10 for sipa's proposal :)
188 2013-07-22 13:18:55 <CodeShark> +10 for sipa's proposal
189 2013-07-22 13:19:08 <sipa> headers-first or the timestamp thing?
190 2013-07-22 13:19:38 <CodeShark> headers-first plus sliding validation window
191 2013-07-22 13:20:17 <sipa> i have assumed for a long time that this was the accepted way-to-go
192 2013-07-22 13:20:22 <sipa> just never found time to work on it
193 2013-07-22 13:20:30 <CodeShark> ok, then +10 for sipa's attempt at implementing it :)
194 2013-07-22 13:21:14 <sipa> currently is synchronizes headers, and if fed a block, it will download, validate, store and try to connect it
195 2013-07-22 13:21:25 <sipa> but there needs to be some code to decide what block to fetch from where :)
196 2013-07-22 13:22:05 <CodeShark> eventually might be nice to add better prioritization of peers
197 2013-07-22 13:22:26 <sipa> that may not even be needed
198 2013-07-22 13:23:09 <sipa> just maintain how many 'outstanding' blocks you can have for each peer, and distribute the first N blocks that you don't have to them
199 2013-07-22 13:23:38 <sipa> if one peer is holding you up, decrease its max number of blocks
200 2013-07-22 13:23:53 <CodeShark> well, even if you receive invalid block headers, at least you can stop wasting precious resources as soon as you realize it does not connect :)
201 2013-07-22 13:24:19 <sipa> it's really nice to see how simple ProcessBlock & co become :p
202 2013-07-22 13:24:27 <CodeShark> indeed
203 2013-07-22 13:25:01 <jgarzik> sipa, nice
204 2013-07-22 13:26:23 <jgarzik> sipa, it's relevant in my JavaScript work too, because I'm writing header sync in the SPV JS client right now
205 2013-07-22 13:26:36 <sipa> header sync is really easy
206 2013-07-22 13:26:55 <jgarzik> yep
207 2013-07-22 13:27:33 <sipa> and SPV is really header-first sync, with block-sync replaced by some wallet-aware optimized selective to-wallet-only-download
208 2013-07-22 13:27:36 <CodeShark> plus if the logic for building the header tree is separate from the logic for block validation, the header tree code can be readily reused in SPV
209 2013-07-22 13:27:42 <sipa> ^
210 2013-07-22 13:29:16 <petertodd> re: SIGHASH_SINGLE libbitcoin handles that case incorrectly too...
211 2013-07-22 13:29:28 <jgarzik> Yah, I'm making "find best chain" (header sync) an entirely separate process, before wallet (merkle block) sync
212 2013-07-22 13:29:36 <jgarzik> runs first
213 2013-07-22 13:30:45 <sipa> jgarzik: header sync takes 49s for me :)
214 2013-07-22 13:31:25 <CodeShark> the most expensive operations for header sync are network and db insertion
215 2013-07-22 13:32:10 <sipa> it actually does take 20-30% CPU while syncing
216 2013-07-22 13:32:18 <helo> after header sync, chain downloading can be done efficiently ala bittorrent?
217 2013-07-22 13:32:19 <sipa> and all the rest is network latency
218 2013-07-22 13:32:28 <sipa> helo: yes
219 2013-07-22 13:33:06 <gmaxwell> It'll be good when this leaves us looking to merge back in faster sha256 code. :P
220 2013-07-22 13:33:48 <CodeShark> a couple generations down the line, perhaps Intel chips will have sha in hardware :p
221 2013-07-22 13:33:51 <jgarzik> sipa, is your address index merge-ready?
222 2013-07-22 13:34:24 <sipa> jgarzik: it could be, but the latest changes are pretty much untested
223 2013-07-22 13:34:34 <jgarzik> sipa, ok
224 2013-07-22 13:35:59 <jgarzik> sipa, also, RE "easy block explorer", I'm strongly tempted to add a REST interface. Initial target is "GET /rest/tx/$txid" (iff txindex=1) and "GET /rest/block/$blkid", serve as binary|hex|json expansion
225 2013-07-22 13:36:08 <jgarzik> gmaxwell, ^
226 2013-07-22 13:36:26 <jgarzik> any objections?
227 2013-07-22 13:36:38 <petertodd> jgarzik: ACK
228 2013-07-22 13:36:58 <petertodd> jgarzik: Also /rest/searchtx/$pushdata
229 2013-07-22 13:37:16 <jgarzik> sipa, also, RE "easy block explorer", I'm strongly tempted to add a REST interface. Initial target is "GET /rest/tx/$txid" (iff txindex=1) and "GET /rest/block/$blkid", serve as binary|hex|json expansion
230 2013-07-22 13:37:19 <sipa> jgarzik: have you seen overblock?
231 2013-07-22 13:37:21 <jgarzik> gavinandresen, ^ (resend)
232 2013-07-22 13:37:25 <jgarzik> sipa, no
233 2013-07-22 13:37:40 <sipa> jgarzik: it's a python blockexplorer-like webserver using bitcoind RPC
234 2013-07-22 13:37:48 <sipa> using txindex
235 2013-07-22 13:38:05 <sipa> realazthat wrote it; it would be nice if addrindex support got added
236 2013-07-22 13:38:29 <jgarzik> cool
237 2013-07-22 13:38:30 <sipa> https://github.com/realazthat/overblock
238 2013-07-22 13:38:47 <realzies> mmmm
239 2013-07-22 13:38:49 <realzies> hi sipa
240 2013-07-22 13:38:53 <jgarzik> I always read his name as "real ass hat" (though I'm sure he's an awesome fellow)
241 2013-07-22 13:38:59 <realzies> lol
242 2013-07-22 13:39:02 <sipa> i read it as "realize that"
243 2013-07-22 13:39:09 <realzies> real az that :P
244 2013-07-22 13:39:23 <sipa> real as dad?
245 2013-07-22 13:39:34 <realzies> lol
246 2013-07-22 13:40:04 <realzies> I am away from home atm, but I'll talk more about addrindex when I get back
247 2013-07-22 13:40:41 <gmaxwell> I tried the overblock stuff and found it to be pretty neat.
248 2013-07-22 13:42:02 <gmaxwell> I'd encourage that whatever interface we offer something like that on have no access to private data, for risk of some kind of JS injection through block data. :P Certantly would be awesome to have more integrated explorer stuff... that is what the varrious indexes are for. :P
249 2013-07-22 13:42:42 <petertodd> gmaxwell: Good idea, people *will* expose it to the web to others in all sorts of dumb ways, or to themselves.
250 2013-07-22 13:44:56 <CodeShark> you should never be using these things on an instance of bitcoind you also use as a wallet :p;
251 2013-07-22 13:45:08 <CodeShark> there's absolutely no need to
252 2013-07-22 13:45:29 <realzies> heh
253 2013-07-22 13:45:33 <CodeShark> in fact, the vast majority of my bitcoind instances, I never even touch the wallet
254 2013-07-22 13:45:37 <petertodd> Well, a local blockexplorer is useful, so people will do just that...
255 2013-07-22 13:45:47 <CodeShark> I just leave it there because the damn thing won't load without a wallet.dat file
256 2013-07-22 13:45:49 <realzies> yeah it wasn't meant to be "secure", but I am not going to try to introduce vulnerabilities
257 2013-07-22 13:46:04 <gmaxwell> Considering how 'costly' running a bitcoind instance is ... its hard to tell most people that.
258 2013-07-22 13:46:09 <jgarzik> Let's just delete the wallet and go home.
259 2013-07-22 13:46:16 <jgarzik> or shopping
260 2013-07-22 13:46:22 <gmaxwell> I do most of my bitcoind usage on the actual network on something with a wallet that has some coin in it.
261 2013-07-22 13:46:37 <sipa> i'm really in the middle here
262 2013-07-22 13:46:58 <sipa> i dislike all indexing functionality that can be abused to build non-scalable solutions
263 2013-07-22 13:47:15 <sipa> but if it can get people to run more fully-validating nodes because they like a local blockexplorer...
264 2013-07-22 13:47:26 <jgarzik> BitPay is moving away from bitcoind as a wallet, but continues to see bitcoind in the role of Industrial Strength P2P Node </vendor hat>
265 2013-07-22 13:47:39 <jgarzik> i.e. public blockchain engine
266 2013-07-22 13:47:41 <sipa> makes perfect sense
267 2013-07-22 13:47:42 <gmaxwell> I also dislike people depending on centeralized blockexplorers, esp the fact that there is basically only one that people use.
268 2013-07-22 13:48:14 <CodeShark> the bitcoind wallet was NEVER designed nor intended to be used for enterprise app development
269 2013-07-22 13:48:21 <CodeShark> it's clearly a single end-user model
270 2013-07-22 13:48:49 <gmaxwell> And yea, it sucks when we don't get to have missing footguns to encourage people to not use footguns. But there is only so much responsiblity we can take for what crazy things people do. If leaving out an index makex them use the bc.i API or run some squirlly alternative full node, thats worse.
271 2013-07-22 13:49:11 <CodeShark> anyone who uses it for things like running web stores is already abusing bitcoind to build a nonscalable solution
272 2013-07-22 13:49:32 <jgarzik> heh
273 2013-07-22 13:50:13 <jgarzik> part of what sucks about that is a lack of docs for merchants, telling them how to build a secure web store, how to manage keys securely
274 2013-07-22 13:51:12 <gmaxwell> CodeShark: I dunno, its a little less bad than you're giving it credit for. Go rig IsMine to return true on 1% of the total transactions and resync the chain... it does actually work.
275 2013-07-22 13:51:53 <gmaxwell> CodeShark: there are plenty of other reasons to not do that, but with a little profiling and kicking I don't see that there are any basic mechnical scalablity issues with using it to receive/send txn (not accounting, of course)
276 2013-07-22 13:52:14 <sipa> meh
277 2013-07-22 13:52:26 <gmaxwell> CodeShark: you have to rotate out wallets periodically right now if you're a big user, but this isn't a huge burden.
278 2013-07-22 13:52:33 <sipa> even just rpc as an interface makes it pretty bad as a solution
279 2013-07-22 13:52:47 <sipa> -walletnotify is nice but doesn't scale to high transaction rates really
280 2013-07-22 13:52:53 <CodeShark> absolutely - walletpassphrase?! for RPC?
281 2013-07-22 13:53:02 <CodeShark> no access controls (unless you build your own proxy)?
282 2013-07-22 13:53:09 <CodeShark> absolutely horrid concurrency?
283 2013-07-22 13:53:22 <sipa> what concurrency are you talking about? :o
284 2013-07-22 13:53:28 <swulf--> sipa: i added a listtransactions2 call that makes use of bitcoind for processing transactions a tiny bit more convenient
285 2013-07-22 13:53:42 <gmaxwell> CodeShark: access controlls aren't enormously important. And what the heck do you need concurency for? the whole rate of the network is relativly limited.
286 2013-07-22 13:54:27 <gmaxwell> CodeShark: the fine tradition of really solid access control is upheld: run another copy for another security domain. :P
287 2013-07-22 13:54:29 <CodeShark> access controls are immensely important when you don't want to give your web developers full reign over all accounts or want to send alerts when thresholds are exceeded
288 2013-07-22 13:56:09 <gmaxwell> CodeShark: what the heck would you be doing with using a production wallet in development? In practice most businesses _in the world_ do not have enough tech staff that you could firewallet techstaff with wallet access and ones that don't. That isn't to say that there aren't many that do, but I think you're overgeneralizing the needs of the very largest.
289 2013-07-22 13:56:19 <CodeShark> as for concurrency, it could mean a user having to wait several more seconds (at least) before their view updates
290 2013-07-22 13:56:45 <gmaxwell> ...
291 2013-07-22 13:56:50 <CodeShark> and that's assuming low volume
292 2013-07-22 13:57:11 <CodeShark> not all RPC wallet commands send transactions
293 2013-07-22 13:57:20 <gmaxwell> thats nonsense. If you're using a thread per user to access bitcoind you're using a broken design.
294 2013-07-22 13:57:34 <CodeShark> you don't need a thread per user
295 2013-07-22 13:57:43 <CodeShark> you could even use a single thread that handles requests asynchronously
296 2013-07-22 13:57:43 <gmaxwell> You could change bitcoind to accomidate your design, true. Or you could not use one that doesn't work with it.
297 2013-07-22 13:58:03 <gmaxwell> CodeShark: then you wouldn't have the head of line blocking you propose.
298 2013-07-22 13:58:06 <sipa> i think the point is that eventually, you'll not be server account balances or transaction lists from bitcoind on the serving path
299 2013-07-22 13:58:13 <sipa> for any decent volume of traffic
300 2013-07-22 13:58:23 <sipa> even though those are tasks it is designed for
301 2013-07-22 13:58:56 <CodeShark> there's also the issue of separation of signing from verification/receiving
302 2013-07-22 13:59:00 <CodeShark> which I've been harping on for so long :p
303 2013-07-22 13:59:07 <gmaxwell> Certantly the accounting stuff is worthless, it simply fails on a durability perspective if nothing else.
304 2013-07-22 13:59:31 <gmaxwell> I use it for personal notes to simplyify my taxes and thats about all its good for.
305 2013-07-22 13:59:58 <CodeShark> the correct model for a merchant receiving bitcoin payments,
306 2013-07-22 14:00:01 <CodeShark> IMHO
307 2013-07-22 14:00:03 <CodeShark> ...
308 2013-07-22 14:00:04 <CodeShark> is
309 2013-07-22 14:00:19 <CodeShark> running several verification/relay nodes that forward messages over to a message broker
310 2013-07-22 14:00:42 <CodeShark> or a cluster of them
311 2013-07-22 14:00:52 <CodeShark> and being able to subscribe and ack all messages
312 2013-07-22 14:00:57 <gmaxwell> it can be really easy in the model to build exploitable site specific code.
313 2013-07-22 14:01:28 <gmaxwell> Not that that is a killer or anything, but a tradeoff in highly decopled things is that you end up with a lot of gnarly bitcoin logic in multiple tools that way.
314 2013-07-22 14:01:45 <CodeShark> the correct model for a merchant sending bitcoin payments is requesting a signature from a signing-only node sitting behind a firewall (or running on dedicated hardware)
315 2013-07-22 14:02:35 <gmaxwell> IMO if you're doing both these things in realtime, you're a bank-like service, and you're instantly into the realm of very specialized and difficult to secure problems.
316 2013-07-22 14:02:57 <CodeShark> I believe all these tools can become commoditized and simple to use with proper design
317 2013-07-22 14:02:57 <gmaxwell> because there are a bunch of vulnerablities that are impratical to have unless you automatically both send and recieve.
318 2013-07-22 14:04:08 <CodeShark> I believe the correct model for key generation is either deterministic type 2 keys or batch pregeneration and synching of address pools with invoicing servers
319 2013-07-22 14:04:41 <CodeShark> it would be fairly simple to develop tools to ensure synchronization
320 2013-07-22 14:04:54 <CodeShark> or to warn in the event of synchronization failure
321 2013-07-22 14:05:02 <gmaxwell> "commoditized and simple" maybe but there are some considerations. For example, I don't think you can ever exposed site specific code to a low confirmed transaction if that code is to be both simple and safe.
322 2013-07-22 14:05:19 <gmaxwell> s/exposed/expose/
323 2013-07-22 14:05:32 <CodeShark> confirmation count policy depends on application
324 2013-07-22 14:06:03 <CodeShark> there can be general guidelines
325 2013-07-22 14:06:07 <gmaxwell> because otherwise it has to deal with transactions being undone, and this is reasonably difficulty to get right even if you understand the problem. Many people don't understand it, its hard to test, and they may never observe it happen absent an attack.
326 2013-07-22 14:06:35 <CodeShark> undone transactions would produce messages that the message brokers would also queue up
327 2013-07-22 14:06:50 <CodeShark> making it easy to handle this logic on the application side
328 2013-07-22 14:06:51 <gmaxwell> yea, right great, might as well just copy everything on p2p to your message brokers. :P
329 2013-07-22 14:07:05 <CodeShark> the p2p protocol is far too low level for app developers
330 2013-07-22 14:07:17 <gmaxwell> as I said, you walk a fine line with requiring all the app logic to safely reimplement a lot of bitcoin stuff.
331 2013-07-22 14:07:21 <CodeShark> it
332 2013-07-22 14:07:34 <CodeShark> the p2p protocol is great for two things: verification and relay
333 2013-07-22 14:07:51 <CodeShark> it's not designed for high level message brokering
334 2013-07-22 14:08:01 <CodeShark> pub/sub, filtering, topics, etc...
335 2013-07-22 14:08:32 <gmaxwell> I'm sorry I distracted you with that, you missed my point. P2P is bad for those reasons for that purpose, but it's bad for a whole nother set of them:
336 2013-07-22 14:08:56 <CodeShark> there's no reimplementation of bitcoin here
337 2013-07-22 14:09:04 <CodeShark> the bitcoin protocol remains the backbone of the network
338 2013-07-22 14:09:05 <gmaxwell> which is that it's hard to get the bitcoin logic right for even an expert. If you make app developers add a lot of bitcoin specififc logic a lot will get it wrong and be vulnerable as a result.
339 2013-07-22 14:09:25 <CodeShark> there's no need for the application developers to keep track of all the low level intricacies for things like verification
340 2013-07-22 14:09:42 <gmaxwell> CodeShark: if you have to undo a transaction triggered from the network, thats bitcoin specific logic that a lot of people will get wrong.
341 2013-07-22 14:09:43 <CodeShark> they just need to receive messages for when specific types of events occur
342 2013-07-22 14:10:00 <gmaxwell> If you have to identify if a payment is for you??? thats bitcoin specific logic that a lot of people will get wrong.
343 2013-07-22 14:10:28 <CodeShark> no, it's rather simple to use a filtering message broker
344 2013-07-22 14:10:45 <CodeShark> provide it with a list of receiving addresses
345 2013-07-22 14:10:45 <gmaxwell> OK
346 2013-07-22 14:12:23 <CodeShark> use hash trees to verify that the receiving addresses on different message brokers are synchronized
347 2013-07-22 14:12:47 <CodeShark> (this functionality would be built into such products, no need for application developers to deal with the low level implementation specifics)
348 2013-07-22 14:13:56 <CodeShark> application developers should be able to subscribe to messages for events like transaction seen, transaction confirmation count change, etc...
349 2013-07-22 14:14:20 <CodeShark> and message brokers should ensure that they are synched properly
350 2013-07-22 14:14:35 <Diablo-D3> CodeShark: yes but
351 2013-07-22 14:14:37 <Diablo-D3> that'd make sense
352 2013-07-22 14:14:42 <Diablo-D3> so we obviously cant have that
353 2013-07-22 14:14:45 <CodeShark> lol
354 2013-07-22 14:15:17 <sipa> http://abstrusegoose.com/518
355 2013-07-22 14:16:06 <CodeShark> lol
356 2013-07-22 14:16:14 <Diablo-D3> sipa: oh dear god, a xkcd clone
357 2013-07-22 14:16:25 <sipa> a much geekier one
358 2013-07-22 14:18:36 <Diablo-D3> so wait
359 2013-07-22 14:18:39 <Diablo-D3> doe sthat make it funny or not?
360 2013-07-22 14:19:22 <sipa> for some people, probably
361 2013-07-22 14:19:27 <sipa> you know
362 2013-07-22 14:19:31 <Diablo-D3> some people find xkcd funny
363 2013-07-22 14:19:33 <sipa> de gustibus et coloribus
364 2013-07-22 14:20:03 <Diablo-D3> english please
365 2013-07-22 14:21:26 <sipa> Diablo-D3: it's a latin proverb about there's no point discussing differences in taste
366 2013-07-22 15:00:18 <bakingbread> sipa: you forgot to add "non est disputandum"
367 2013-07-22 15:00:45 <sipa> bakingbread: not forgotten; abbreviated :)
368 2013-07-22 15:24:23 <gwillen> does anybody have an order-of-magnitude estimate of the current UTXO set size, or know how one might compute it?
369 2013-07-22 15:25:05 <sipa> ./bitcoind gettxoutsetinfo
370 2013-07-22 15:25:07 <gmaxwell> gwillen: run gettxoutsetinfo on bitcoind (takes a minute)
371 2013-07-22 15:25:10 <gwillen> thanks
372 2013-07-22 15:25:49 <gmaxwell> gwillen: do not that this is just the size of the highly compacted representation seralized. It's larger fully unpacked and/or stuffed into a authentication tree or a database for fast lookups.
373 2013-07-22 15:25:53 <gmaxwell> s/not/note/
374 2013-07-22 15:26:01 <gwillen> ahh, *nods*
375 2013-07-22 15:52:37 <TheUni> bitcoind up and running on android if there's any interest
376 2013-07-22 15:53:40 <gmaxwell> TheUni: really? cool. compiled using the ndk stuff?
377 2013-07-22 15:54:08 <TheUni> yea
378 2013-07-22 15:54:26 <TheUni> seems like it'd be a cheap/quick/easy way to get relay nodes up and running
379 2013-07-22 15:55:14 <TheUni> if there's interest, i could put some time into making it a quick POC apk with a gui that's basically just an on/off switch
380 2013-07-22 15:57:40 <TheUni> only took about an hour of fiddling to get it patched, compiled and running. pretty uninvasive
381 2013-07-22 15:58:52 <gmaxwell> TheUni: how long did it take to sync up?
382 2013-07-22 15:59:21 <TheUni> heh, no clue. just fired it up 10min ago
383 2013-07-22 16:00:31 <gmaxwell> TheUni: whats it at now?
384 2013-07-22 16:00:40 <TheUni> running on a 4core odroid, should be on par with an oldish desktop, i'd think
385 2013-07-22 16:01:14 <TheUni> SetBestChain: new best=00000000069d5d70aad5bf72b8d856c821ea41ee1969d803a31fcff91e633350 height=62363 log2_work=49.883994 tx=69725 date=2010-06-23 19:04:12 progress=0.001289
386 2013-07-22 16:08:30 <gmaxwell> Thats pretty slow. :( and it's not even to the ecdsa hard parts yet. I suppose I should profile on arm at some point.
387 2013-07-22 16:13:18 <jgarzik> w00t. HTTP REST framework working. Now to add /rest/tx/$txid and /rest/block/$blkid
388 2013-07-22 16:13:38 <petertodd> +1
389 2013-07-22 16:18:23 <TheUni> gmaxwell: it came up literally 30min ago, completely unoptimized. haven't even run the tests yet
390 2013-07-22 16:18:49 <TheUni> tbh, i have no idea what it's doing. i can tell from the debug log that it's running, but that's about it
391 2013-07-22 16:19:16 <TheUni> i was hoping for a quick load+crash, didn't expect it to actually do anything useful :)
392 2013-07-22 16:21:17 <gmaxwell> TheUni: K. Bluematt has had it running on odroids on Ubuntu/Linux for some time. Android is a little more of an accomplishment. :P
393 2013-07-22 16:22:43 <TheUni> well like i said, i haven't actually wrapped it in the typical android-isms yet, i'm just launching from cmdline. No sense wasting time into making it a proper deployable apk if there's no interest in it
394 2013-07-22 16:23:05 <TheUni> gmaxwell: got any recommendations for gauging ^^ ?
395 2013-07-22 16:24:07 <gmaxwell> make one once, and then see if people whine at you to update it. :P
396 2013-07-22 16:24:27 <TheUni> haha
397 2013-07-22 16:25:08 <TheUni> well i sure as hell wouldn't download a random bitcoind app, so it'd need to be signed with official keys
398 2013-07-22 16:25:32 <gmaxwell> I'll point out that even if almost no one cares testing the software in weird enviroments can expose serious bugs that we need to also fix for other platforms, so its a good excercise even if people don't use it.
399 2013-07-22 16:26:17 <TheUni> sure, that's why i work on abstractions and ports, i'm pretty obsessive about exactly that point
400 2013-07-22 16:28:12 <TheUni> heh, good proof of that: http://pastebin.com/raw.php?i=bEf1D0SF
401 2013-07-22 16:29:35 <Luke-Jr> gavinandresen: I'll have a patch to fix that gitian error in a few min
402 2013-07-22 16:30:51 <TheUni> Luke-Jr: anything i can do to move the autotools PR along? I have abstraction work that depends on it
403 2013-07-22 16:31:44 <Luke-Jr> TheUni: dunno, that's a question for wumpus :/
404 2013-07-22 16:32:00 <Luke-Jr> wumpus: ping, TheUni would appreciate a review from you :p
405 2013-07-22 16:32:07 <TheUni> Luke-Jr: thanks :)
406 2013-07-22 16:39:15 <wumpus> TheUni: that was easy :)
407 2013-07-22 16:39:40 <TheUni> ?
408 2013-07-22 16:40:10 <wumpus> I think it was already ok with me last time, so you've got your ack
409 2013-07-22 16:40:31 <wumpus> let's just get this in
410 2013-07-22 16:40:53 <TheUni> aha, great. thanks
411 2013-07-22 16:41:01 <TheUni> wumpus: not that this time around, i completely ignored qt-creator
412 2013-07-22 16:41:04 <TheUni> *note
413 2013-07-22 16:41:15 <TheUni> so, that is all on you :)
414 2013-07-22 16:41:35 <wumpus> yeah, I understand...
415 2013-07-22 16:43:02 <TheUni> ok, perfect. I'll close this one and do a final squashed PR with updated docs and working gitian descriptors
416 2013-07-22 16:44:33 <wumpus> great!
417 2013-07-22 16:46:51 <TheUni> will be a bit delayed this time around. in-laws coming in from out of the country for the next month, so no clue how much hacking time i'll have
418 2013-07-22 16:47:20 <wumpus> well if it's just a matter of squashing the commit we could do it too, I suppose
419 2013-07-22 16:48:23 <TheUni> squashing should be straightfoward, it's gitian and docs that still need love
420 2013-07-22 16:48:33 <TheUni> i didn't want to do em til the core work was ack'd
421 2013-07-22 16:53:16 <wumpus> makes sense
422 2013-07-22 16:54:25 <TheUni> will try to find some time in the next week
423 2013-07-22 16:54:29 <TheUni> thanks for taking another look
424 2013-07-22 17:17:48 <jgarzik> and? REST is working. Modulo an interpret-binary-data-as-C-string bug.
425 2013-07-22 17:17:59 <jgarzik> binary, hex and json formats.
426 2013-07-22 17:18:14 <petertodd> +2
427 2013-07-22 17:30:26 <TheUni> jgarzik: mm, as i think about it, android app could be interesting with checkpointing. add a huge number of nodes to the network for relaying unconfirmed transactions. soon as a coin is mined, dump the chain and start over
428 2013-07-22 17:30:42 <TheUni> even a tiny fraction of android devices doing that would add up to a huge number
429 2013-07-22 17:34:41 <TheUni> er, i suppose the lightweight clients do that already though. nm
430 2013-07-22 17:41:04 <jgarzik> REST assured, you are safe to query. https://github.com/bitcoin/bitcoin/pull/2844
431 2013-07-22 17:54:10 <MC1984> dohoho
432 2013-07-22 17:57:59 <gmaxwell> uhhh
433 2013-07-22 17:58:00 <gmaxwell> fuck
434 2013-07-22 17:58:15 <gmaxwell> debian is shipping bitcoind patched to use a system leveldb
435 2013-07-22 17:58:24 <petertodd> ugh
436 2013-07-22 17:58:33 <gmaxwell> and it appears to fork on some of the older history
437 2013-07-22 17:59:25 <petertodd> Can we add something to the Bitcoin source itself to check for the leveldb version, and put a explanatory comment next to that check saying why it's so important to leave leveldb the way it is?
438 2013-07-22 18:01:15 <gwillen> I would encourage you to have someone _extremely patient_ talk to debian about this
439 2013-07-22 18:01:21 <gwillen> it will probably have to be explained slowly several times
440 2013-07-22 18:01:41 <gmaxwell> This actually has been done before.
441 2013-07-22 18:01:51 <gwillen> heh
442 2013-07-22 18:01:58 <TheUni> i suppose there are non-upstream'able patches?
443 2013-07-22 18:02:14 <jgarzik> distros just have a MAJOR compulsion to use system libs
444 2013-07-22 18:02:24 <gmaxwell> For good reasons in general.
445 2013-07-22 18:02:25 <jgarzik> embedded libs drive distro hackers bonkers
446 2013-07-22 18:02:30 <TheUni> as well they should..
447 2013-07-22 18:02:39 <jgarzik> e.g. having to fix XXX copies of zlib, for each zlib bug
448 2013-07-22 18:02:45 <gmaxwell> (Also, I'm beginning to think debian is hopeless, they are carrying a number of patches in addition to the system changes which they've never exposed to us before)
449 2013-07-22 18:03:15 <gmaxwell> TheUni: It isn't a matter of non-upstream'able patches. Its a matter of _fixing_ a bug in any of our network rules centeric code can cause total network failure.
450 2013-07-22 18:03:24 <gwillen> gmaxwell: you may want to just ask them not to carry it at all
451 2013-07-22 18:03:30 <gmaxwell> gwillen: We did.
452 2013-07-22 18:03:32 <gwillen> oh.
453 2013-07-22 18:03:32 <gwillen> that seems like a request they may respect
454 2013-07-22 18:03:39 <jgarzik> IMO this is worth explaining in a blog post
455 2013-07-22 18:03:45 <TheUni> gmaxwell: so an upstream version can never be used, then?
456 2013-07-22 18:03:46 <jgarzik> because it is non-obvious
457 2013-07-22 18:03:52 <jgarzik> (outside of bitcoin, to other devs)
458 2013-07-22 18:04:18 <petertodd> jgarzik: Honestly, put it in a Bitcoin Foundation blog post signed by Gavin.
459 2013-07-22 18:04:27 <jgarzik> indeed
460 2013-07-22 18:04:35 <petertodd> jgarzik: On your blog is good too, but we want to be really official about this so people listen.
461 2013-07-22 18:04:40 <gmaxwell> TheUni: E.g. Say level DB has a bug where any key with a ID that begins with 0xDEADBEEF can not be found. Such bugs can happen here and then in database engines. Say there is a transaction with 0xDEADBEEF as its previx (perhaps even created by a malicious part).
462 2013-07-22 18:04:53 <jgarzik> I'm pretty sure people would let me post on the BF blog, if I ask nicely
463 2013-07-22 18:05:00 <jgarzik> anyway
464 2013-07-22 18:05:11 <gmaxwell> TheUni: if you fix the deadbeef bug, then the network splits into two: fixed nodes and non-fixed nodes. And whichever side ultimately loses gets all its txn reversed.
465 2013-07-22 18:05:13 <petertodd> IMO distros shouldn't even package bitcoind due to the security issues, but they already package other libraries so...
466 2013-07-22 18:05:35 <petertodd> At least the builds we officially release are staticly linked... right?
467 2013-07-22 18:05:37 <TheUni> gmaxwell: is this in theory? or are there documented cases of such fixes/splits?
468 2013-07-22 18:06:02 <petertodd> TheUni: Both, and even if there were no cases, that'd just mean we had been doing our job.
469 2013-07-22 18:06:05 <gmaxwell> TheUni: yes, we've created them ourselves in our own code. Prior fixes in leveldb before we picked it up had this kind of potential too.
470 2013-07-22 18:06:34 <TheUni> gmaxwell: then imo rename it from leveldb. If leveldb can never be dropped in, then it's functionally a different lib for all future points
471 2013-07-22 18:06:38 <petertodd> TheUni: Consensus is just such a different problem than what normal software engineering faces.
472 2013-07-22 18:06:42 <petertodd> TheUni: Good idea!
473 2013-07-22 18:06:44 <gmaxwell> TheUni: None of our dependences promise to keep bugs. Sooo...
474 2013-07-22 18:06:56 <TheUni> a rename in the srctree would discourage packagers from trying to drop it in.
475 2013-07-22 18:07:10 <gwillen> that's a really good idea
476 2013-07-22 18:07:14 <gwillen> fork it and rename the fork bitcoindb
477 2013-07-22 18:07:19 <gmaxwell> TheUni: we do update our leveldb from time to time, but we review and test the changes and gain a degree of confidence that a fork is unlikely.
478 2013-07-22 18:07:22 <gwillen> and make some random irrelevant changes
479 2013-07-22 18:07:30 <gwillen> just to prevent someone from being clever ;-)
480 2013-07-22 18:07:53 <gmaxwell> But obviously a distro level-db package has no reason to go through that kind of review. Fixing a bug is usually a great thing. :)
481 2013-07-22 18:08:17 <Luke-Jr> gmaxwell: that's a distro-specific thing
482 2013-07-22 18:08:19 <petertodd> ACTION wonders how many lines of usermode sourcecode bitcoin depends on
483 2013-07-22 18:08:38 <gmaxwell> petertodd: it's also the case that not all of our depenencies are network critical.
484 2013-07-22 18:08:52 <gmaxwell> (at least assuming 0-3 sigma bugs)
485 2013-07-22 18:09:11 <gmaxwell> e.g. via qt we link zlib, but you're not going to fork the network with zlib changes with any great likelyhood.
486 2013-07-22 18:09:11 <TheUni> gmaxwell: yea, the situation makes sense from both sides. But I also see how the distro guys see it as their job to bring those two sides together. So forcing an incompatibility the reasonable fix imo
487 2013-07-22 18:09:12 <Luke-Jr> Gentoo's bitcoind depends on the exact LevelDB version with the exact configuration we use (ie, no snappy support)
488 2013-07-22 18:09:39 <TheUni> eg, rename the lib, call it 1.0.0, and add a routine that bitcoind depends on
489 2013-07-22 18:09:49 <Luke-Jr> TheUni: that's dumb
490 2013-07-22 18:09:52 <gwillen> Luke-Jr: it's smart
491 2013-07-22 18:09:55 <gmaxwell> TheUni: Agreed, as I said, I'm sympathic of the distro's motivations. One challenge is that getting the distros to understand that we understand and aren't just being an ordinary obnoxious upstream is a problem.
492 2013-07-22 18:10:00 <gwillen> Luke-Jr: it's also dumb, but that doesn't mean it's not smart
493 2013-07-22 18:10:07 <Luke-Jr> TheUni: linking to system leveldb is a good thing, we just need to make sure anyone doing it has a clue
494 2013-07-22 18:10:18 <gmaxwell> TheUni: I've seen distros do a lot of work to undo embedding like that though.
495 2013-07-22 18:10:26 <gmaxwell> So if they don't understand what we're doing even that may not be enough.
496 2013-07-22 18:10:26 <TheUni> Luke-Jr: going by the explanation given above, linking to system leveldb would never be desirable
497 2013-07-22 18:10:28 <Luke-Jr> ie, configure should refuse to do it without jumping through hoops
498 2013-07-22 18:10:35 <gwillen> gmaxwell: yeah, but you make the Right Thing easier than the wrong thing
499 2013-07-22 18:10:35 <petertodd> gmaxwell: Indeed, although pretty much any dependency can be intentially make network crit.
500 2013-07-22 18:10:40 <gmaxwell> Luke-Jr: basically only gentoo is going to do what we need there.
501 2013-07-22 18:10:40 <gwillen> gmaxwell: that doesn't make the wrogn thing impossible
502 2013-07-22 18:10:56 <gmaxwell> gwillen: they already have to patch it to break it out, so I dunno that 'easier' is enough.
503 2013-07-22 18:11:10 <gmaxwell> petertodd: Yes thats why I said 0-3 sigma bugs.
504 2013-07-22 18:11:19 <gwillen> gmaxwell: well, that plus you put specific instructions somewhere obvious, in a comment, explaining why they shouldn't fuck with it
505 2013-07-22 18:11:22 <gwillen> AND you make a blog post
506 2013-07-22 18:11:25 <gwillen> and you yell at them if they try
507 2013-07-22 18:11:26 <gmaxwell> petertodd: zlib could be randomly overwriting memory in a way we depend on, but this is very very unlikely.
508 2013-07-22 18:11:27 <gwillen> defense in depth :-)
509 2013-07-22 18:11:31 <TheUni> gmaxwell: would it be possible to detect such splits in-code. Meaning, have a set of black-listed db entires like some of the blacklisted bitcoin txns?
510 2013-07-22 18:11:35 <gmaxwell> gwillen: I guess it's good now that debian is actually failing.
511 2013-07-22 18:11:46 <Luke-Jr> TheUni: only if we know about them!
512 2013-07-22 18:11:55 <TheUni> if such an entry is detected, shut down with an error that an incompat vers was detected
513 2013-07-22 18:12:01 <gmaxwell> TheUni: only if you can anticipate all possible failures in advance. We could and should have a bit more sanity checking...
514 2013-07-22 18:12:09 <gwillen> gmaxwell: I am asking a friend of mine who's part of debian what he would suggest
515 2013-07-22 18:12:12 <Luke-Jr> gmaxwell: has anyone checked that Debian hasn't setup a procedure where leveldb is never updated without a review for this?
516 2013-07-22 18:12:15 <TheUni> Luke-Jr: sure, but my point is that an upstream build could be detected against an in-tree one that way
517 2013-07-22 18:12:16 <gmaxwell> But we already have some and it's not sufficient. There is a startup time trade off too.
518 2013-07-22 18:12:20 <TheUni> regardless of future changes
519 2013-07-22 18:12:35 <gmaxwell> Luke-Jr: are we the only leveldb use in gentoo?
520 2013-07-22 18:12:50 <Luke-Jr> gmaxwell: doubtful
521 2013-07-22 18:13:22 <gmaxwell> at least when we started down the leveldb route I didn't find anything else packaged that used it.
522 2013-07-22 18:13:27 <gmaxwell> I had hoped that would at least defer the packaging problems.
523 2013-07-22 18:13:53 <gmaxwell> In any case, does anyone have a debian sid install? handy that can do some testing?
524 2013-07-22 18:14:17 <gmaxwell> I still don't know _why_ it's rejecting the chain other than it's something in a april 2013 block and it rejects it during an initial sync.
525 2013-07-22 18:14:34 <TheUni> gmaxwell: does it not take care of itself socially (assuming a non majority of debian users, that is) ?
526 2013-07-22 18:14:52 <gmaxwell> (e.g. we're very fortunate to have dodged a bullet with something that was already in the chain)
527 2013-07-22 18:15:18 <gmaxwell> TheUni: kinda, but anyone on the fork gets robbed. If there is almost noone using debian then obvious "too bad for you, using a weirdo distro build" ... but the ecosystem harm grows if there are more users.
528 2013-07-22 18:15:22 <gmaxwell> (to be clear: has the potential of getting robbed if an attacker notices the fork)
529 2013-07-22 18:15:23 <TheUni> gmaxwell: that's pretty much what i meant...
530 2013-07-22 18:15:33 <TheUni> socially, that debian maintainer will be hanged eventually
531 2013-07-22 18:16:00 <gmaxwell> TheUni: unless it gets to be too many nodes before the problem is noticed.
532 2013-07-22 18:16:13 <Luke-Jr> can't TD get upstream to check for these things? ;)
533 2013-07-22 18:16:20 <gmaxwell> And then the black mark is on bitcoin.
534 2013-07-22 18:16:54 <gmaxwell> Luke-Jr: We could try talking to the leveldb maintainers about what we need from them, I don't believe we have... but it's really asking a lot.
535 2013-07-22 18:16:54 <Luke-Jr> TD: can you get upstream LevelDB to include "doesn't change behaviour at all, even bug fixes" in their process?
536 2013-07-22 18:16:54 <TD> hm?
537 2013-07-22 18:16:54 <TheUni> gmaxwell: other route would be to have a tm policy in place
538 2013-07-22 18:17:03 <TheUni> i've gone down that route before with some other floss software
539 2013-07-22 18:17:08 <TheUni> worth doing if the foundation hasn't done it yet
540 2013-07-22 18:17:20 <gmaxwell> Foundation doesn't hold the mark, MtGOX does.
541 2013-07-22 18:17:22 <gmaxwell> AFAIK.
542 2013-07-22 18:17:22 <petertodd> TheUni: tm policy?
543 2013-07-22 18:17:29 <TD> the behaviour isn't supposed to ever change
544 2013-07-22 18:17:36 <TheUni> ah, that's a problem indeed :\\
545 2013-07-22 18:17:37 <Luke-Jr> ACTION would oppose any kind of implementation-oppressive ??? policy. <.<
546 2013-07-22 18:17:38 <TD> except for fixing serious bugs, of course. did it change?
547 2013-07-22 18:17:54 <Luke-Jr> TD: the point is that fixing serious bugs changing behaviour IS BAD :p
548 2013-07-22 18:17:59 <TheUni> Luke-Jr: it's not oppressive at all.
549 2013-07-22 18:18:10 <gmaxwell> td: its changed in the past, IIRC there was a bug before where it would run out of FDs and return keys not found.
550 2013-07-22 18:18:11 <TD> well, it's not supposed to ever happen. AFAIK since we switched to leveldb there were no such changes
551 2013-07-22 18:18:12 <Luke-Jr> TheUni: yes it is.
552 2013-07-22 18:18:22 <gmaxwell> (in the past meaning before we switched to it)
553 2013-07-22 18:18:25 <TD> well, yes. that was fixed.
554 2013-07-22 18:18:26 <TheUni> Luke-Jr: you find mozilla's policy oppressive?
555 2013-07-22 18:18:29 <Luke-Jr> TheUni: yes.
556 2013-07-22 18:18:36 <Luke-Jr> TheUni: also, not the same thing.
557 2013-07-22 18:18:42 <TD> i mean technically any bug can be behaviour. that's one reason we embed it, so we can carefully review the diffs and make a case by case decision
558 2013-07-22 18:18:48 <Luke-Jr> TheUni: Mozilla isn't using a trademark of "World Wide Web", they're using a brand.
559 2013-07-22 18:18:59 <gmaxwell> td: some background, debian is shipping a copy of bitcoin relinked to system libs. It rejects the chain in an april 2013 block, though we don't know why yet.
560 2013-07-22 18:19:13 <Luke-Jr> TD: can we get upstream to make sure it never changes, even for bugfixes, without a clear prominent disclosure?
561 2013-07-22 18:19:17 <TD> a system leveldb?
562 2013-07-22 18:19:20 <gmaxwell> TD: fortunately since it rejects in the past this isn't a sev 0 security problem.
563 2013-07-22 18:19:23 <gmaxwell> TD: yes.
564 2013-07-22 18:19:43 <TheUni> Luke-Jr: correct. in this case, bitcoind is the brand. not bitcoin.
565 2013-07-22 18:20:01 <gmaxwell> "icecoin"
566 2013-07-22 18:20:01 <TD> well, everyone knows linux packaging is a fucking disaster zone. i spent several years trying to make that community change their ways and they never did. we should just detect linux useragents with a bit of javascript on bitcoin.org and have a big flashing warning telling people not to use distributor packaging
567 2013-07-22 18:20:03 <Luke-Jr> TheUni: oh ok, I thought you meant abusing the Bitcoin trademark
568 2013-07-22 18:20:03 <TheUni> Luke-Jr: if you run a modified firefox, it's not firefox. see iceweasel et all
569 2013-07-22 18:20:13 <Luke-Jr> TheUni: my modified Firefox says Firefox ;)
570 2013-07-22 18:20:18 <TheUni> *et al
571 2013-07-22 18:20:30 <petertodd> Luke-Jr: I'd be OK with a "tm" policy *if* Bitcoin had a core library for validation and block processing, everything else is nowhere near as consensus critical. You'd basically be able to say "luke-jr bitcoin, based on Bitcoin Core(TM)"
572 2013-07-22 18:20:36 <TD> what we could also do is make some custom changes to leveldb (change some API prototypes) so a standard leveldb won't work
573 2013-07-22 18:20:37 <Luke-Jr> TD: that's nonsense.
574 2013-07-22 18:20:39 <gwillen> gmaxwell: by what means was debian asked not to package it? Is there a public request somewhere?
575 2013-07-22 18:20:44 <TD> (unless they also patch the code that uses it, etc
576 2013-07-22 18:20:46 <Luke-Jr> Linux packaging is lightyears ahead of everything else.
577 2013-07-22 18:20:56 <gmaxwell> TD: that was gwillen and TheUni's suggestion. Of course, they're already patching to use the system one.
578 2013-07-22 18:20:58 <gwillen> gmaxwell: I'm told that debian is somewhat anarchic, but there are people who can fulfill such a request, if they see it
579 2013-07-22 18:21:10 <TD> as someone familiar with the software distribution schemes of many OS's i can assure you, linux is by far the worst
580 2013-07-22 18:21:11 <Luke-Jr> petertodd: I wouldn't; someone should be free to implement Bitcoin using their own unrelated code
581 2013-07-22 18:21:27 <TD> gmaxwell: remote attestation ;)
582 2013-07-22 18:21:39 <TD> no, i jest. i think a big flashing warning on the home page is the way to go
583 2013-07-22 18:21:49 <petertodd> Luke-Jr: Nothing stopping them, they just can't use the magic words "Bitcoin Verification Core(TM)"
584 2013-07-22 18:21:55 <TheUni> Luke-Jr: they could do whatever they wanted, they just couldn't call it bitcoind
585 2013-07-22 18:21:57 <TD> asking linux distributors not to package things won't work. they will just ignore us. i know because i've been in this situation before.
586 2013-07-22 18:21:58 <TheUni> ^^
587 2013-07-22 18:22:05 <petertodd> Luke-Jr: heck, call it "Bitcoin Foundation Verification Core(TM)" to be really clear
588 2013-07-22 18:22:13 <gwillen> TD: well, I have a Debian Developer asking me for more information
589 2013-07-22 18:22:27 <gwillen> TD: so if we actually want them to not package it, I want to give that person more information, and see what happens
590 2013-07-22 18:22:27 <TD> what information does he need? tell him to delete his packages and send users upstream to get binaries
591 2013-07-22 18:22:28 <gmaxwell> Okay, so I think a two fold approach makes sense right now: (1) better info to cluestick people, (2) a nice blog post on why this is an unusual concern for bitcoin.
592 2013-07-22 18:22:29 <Luke-Jr> petertodd: it begs to be made irrelevant ;)
593 2013-07-22 18:22:57 <gwillen> TD: he wants me to point him to the request from the actual bitcoin devs
594 2013-07-22 18:22:59 <gwillen> TD: which I am not one of
595 2013-07-22 18:23:07 <gwillen> TD: that's why I'm asking how it was made, and whether it was published
596 2013-07-22 18:23:07 <TD> i can do a post to the bitcoin forum
597 2013-07-22 18:23:14 <Luke-Jr> FWIW, I agree that given Debian's packaging model (needing a new package name for different versions/builds), embedding it is the right solution for them
598 2013-07-22 18:23:18 <TheUni> gmaxwell: do you know if the foundation has discussed such an agreement? I've worked with the SFLC on exactly that, i'd be curious to hear the current status/opinion
599 2013-07-22 18:23:21 <gmaxwell> In the long term there should probably be some certified behavior mark. "Certified by Bluematt(tm)"
600 2013-07-22 18:23:25 <TD> heh
601 2013-07-22 18:23:26 <Diablo-D3> http://boingboing.net/2013/07/22/bhangra-remix-of-daft-punks.html
602 2013-07-22 18:23:33 <petertodd> Luke-Jr: No, if your core validation library is sufficiently small, it will be the part that you can't change anyway without risking forks.
603 2013-07-22 18:24:11 <TD> gwillen: it's common sense, it shouldn't really require a special request. but would it help if i sum up the problems in a public forum post?
604 2013-07-22 18:24:14 <gmaxwell> Then there is an interesting question if nodes transmit "Certified by Bluematt(tm)" over the p2p port, and if peers won't treat non-"Certified by Bluematt(tm)" as full nodes. :P
605 2013-07-22 18:24:27 <gmaxwell> (interesting software freedom question)
606 2013-07-22 18:24:32 <gwillen> TD: I don't want common sense, I want an Official Request from Someone Official
607 2013-07-22 18:24:41 <gwillen> those work much better
608 2013-07-22 18:24:57 <Luke-Jr> TD: it's not common sense, it's a special situation
609 2013-07-22 18:25:06 <TheUni> gmaxwell: indeed. Imo it'd be up to the protocol to sort that out via rules rather than client detection
610 2013-07-22 18:25:15 <Luke-Jr> for every other software package, the request to embed libraries is unreasonable.
611 2013-07-22 18:25:19 <TheUni> otherwise you get into nasty territory (see web browsers)
612 2013-07-22 18:25:20 <petertodd> gmaxwell: Which would actually be really bad, because non-satoshi peers are a good thing re: making sure relay bugs don't cause problems.
613 2013-07-22 18:25:29 <Luke-Jr> gwillen: Bitcoin has no official.
614 2013-07-22 18:25:43 <gmaxwell> I agree that it's special. I've been irritated by my packagers before, but on other projects I've never thought that "remove the program" would be a better solution, but here that doesn't sound unreasonable to me.
615 2013-07-22 18:25:47 <gwillen> Luke-Jr: I understand, but gmaxwell is telling me that A Request was made for debian not to package it
616 2013-07-22 18:25:57 <petertodd> gmaxwell: Provided a peer isn't DoSing you and you have a path to majority hashing power, who cares what they are running?
617 2013-07-22 18:26:01 <TD> it's not really. we've seen similar critical bugs introducing during packaging in openssl, wine and firefox, to give a few notorious examples.
618 2013-07-22 18:26:03 <gwillen> and it appears that the request did not go to the right party
619 2013-07-22 18:26:05 <gmaxwell> Luke-Jr: I already, as an upstream, suggested debian not package it.
620 2013-07-22 18:26:12 <gwillen> gmaxwell: who did you suggest it to, and how?
621 2013-07-22 18:26:15 <Luke-Jr> gmaxwell: gwillen wants the link :P
622 2013-07-22 18:26:40 <Luke-Jr> TD: not similar, no
623 2013-07-22 18:26:53 <gwillen> gmaxwell: e.g. there is http://lists.alioth.debian.org/pipermail/pkg-bitcoin-devel/ , which is the Official List for this package, it appears
624 2013-07-22 18:27:14 <gmaxwell> gwillen: On IRC to one of the people that was supposted to be maintaining it and said that they should probably remove it, because version consistency is important and doesn't really match debian's release cycle.
625 2013-07-22 18:27:26 <gwillen> ah, I see
626 2013-07-22 18:27:32 <Luke-Jr> someone should write up a draft Formal Request of some sort, which we can all edit, and then everyone sign off
627 2013-07-22 18:27:38 <gwillen> so there was never anything that rose to the level of an Official Request
628 2013-07-22 18:27:38 <Luke-Jr> lots of names is good
629 2013-07-22 18:27:49 <petertodd> Hmm... a foundation blog post should also include a user-oriented "DO NOT use distro-packaged bitcoind/bitcoin-qt"
630 2013-07-22 18:27:57 <gwillen> gmaxwell: did anybody actually talk to them about like, not screwing with it?
631 2013-07-22 18:27:58 <nsh> gmaxwell, is it currently possible to using Bitcoin to have a decryption key released at a certain time in the future (as measured by blocks, for instance) in a demonstrable manner?
632 2013-07-22 18:28:02 <Luke-Jr> petertodd: but that's wrong too
633 2013-07-22 18:28:03 <gmaxwell> I suppose not! Sorry, I'm used to working with the maintainer of my xiph.org packages and he is 100% IRC API.
634 2013-07-22 18:28:06 <gwillen> gmaxwell: or just the one offhand comment that they shouldn't package it?
635 2013-07-22 18:28:17 <TheUni> petertodd: s/don't use distro/don't use unverified binaries/g
636 2013-07-22 18:28:19 <petertodd> Along with some explanation of the gitian process and how to verify you have the real thing.
637 2013-07-22 18:28:34 <gwillen> gmaxwell: because a request not to screw with it would probably be a good idea, if nobody's actually requested that of them yet
638 2013-07-22 18:28:39 <Luke-Jr> petertodd: gitian process does not work for Linux really. it makes Ubuntu binaries.
639 2013-07-22 18:28:43 <gwillen> gmaxwell: before a big announcement behind their backs asking people not to use their package
640 2013-07-22 18:28:46 <gmaxwell> petertodd: too bad gitian needs so much state, I'd love to try to get the distros to gitian too. :P
641 2013-07-22 18:28:52 <Luke-Jr> petertodd: and it's no more secure than distros in that sense
642 2013-07-22 18:29:14 <Luke-Jr> why should users trust Ubuntu rather than their own distro?
643 2013-07-22 18:29:23 <petertodd> gmaxwell: Yeah, if they *were* using gitian, we could give totally different advice.
644 2013-07-22 18:29:25 <Luke-Jr> at least trusting their own distro is distributed
645 2013-07-22 18:29:30 <gmaxwell> gwillen: I always think it's best to give people heads up regardless.
646 2013-07-22 18:29:37 <gwillen> gmaxwell: *nods*
647 2013-07-22 18:29:43 <gmaxwell> Even if we're going to flame them, we should send them a prerelease draft.
648 2013-07-22 18:29:57 <gwillen> gmaxwell: but if nobody's tried just like, asking them to leave the baked-in leveldb
649 2013-07-22 18:29:58 <gmaxwell> make sure the public argument is about the stuff where there is disagreement.
650 2013-07-22 18:30:02 <gwillen> gmaxwell: and to reverse whatever else might be an issue
651 2013-07-22 18:30:03 <petertodd> Luke-Jr: Trusting a distro you make yourself is also impractical advice for most people.
652 2013-07-22 18:30:05 <gwillen> someone shold probably try that first
653 2013-07-22 18:30:43 <TheUni> gmaxwell: seems to me there's nothing debian-specific about this issue. It could be a calmly worded open-letter to distros
654 2013-07-22 18:30:50 <TheUni> otherwise the flame-gates open up
655 2013-07-22 18:31:03 <petertodd> TheUni: good point, debian's just the first
656 2013-07-22 18:31:07 <TheUni> (other than the fact that it's debian in this case, that is)
657 2013-07-22 18:31:19 <Luke-Jr> petertodd: masses trusting N different distros is better than masses trusting 1 specific distro
658 2013-07-22 18:32:01 <petertodd> Luke-Jr: Depends on the threat, for consensus it's worse, for wallet security it's better.
659 2013-07-22 18:32:01 <TD> i'm writing up something in a google doc that we could potentially sign
660 2013-07-22 18:32:10 <Luke-Jr> TD: bah, I just started the same!
661 2013-07-22 18:32:15 <TD> :)
662 2013-07-22 18:32:23 <Luke-Jr> TD: share yours for others to view/edit? :p
663 2013-07-22 18:32:35 <petertodd> ...well if there are two google docs it points out the distributed nation of bitcoin...
664 2013-07-22 18:32:39 <petertodd> ACTION ducks
665 2013-07-22 18:32:39 <TD> https://docs.google.com/document/d/1naenR6N6fMWSpHM0f4jpQhYBEkCEQDbLBs8AXC19Y-o/edit?usp=sharing
666 2013-07-22 18:33:53 <petertodd> heh, watching TD type reminds me of my BBS days...
667 2013-07-22 18:34:29 <TD> i wonder what's up with the animals
668 2013-07-22 18:34:30 <Luke-Jr> TD: yours reads very centralized so far :p
669 2013-07-22 18:34:38 <TD> i guess "anonymous user" was too boring...
670 2013-07-22 18:35:08 <petertodd> s/unbundled/replaced with shared libraries/ ?
671 2013-07-22 18:35:19 <gmaxwell> We still need to figure out whats actually breaking in debian now.
672 2013-07-22 18:35:21 <Luke-Jr> "We have seen a version of Bitcoin modified during the Debian distribution process exit the consensus for reasons that are unclear" we have?
673 2013-07-22 18:35:29 <gmaxwell> Being able to point out a _real_ problem will help make it persusaive.
674 2013-07-22 18:35:41 <TD> i think that's just what was stated
675 2013-07-22 18:35:47 <TD> the debian bitcoin failed to process an april block
676 2013-07-22 18:35:52 <TD> but i don't know what diffs are applied
677 2013-07-22 18:36:19 <petertodd> TD: that you don't know is IMO worth mentioning to make it clear how fragile the consensus is
678 2013-07-22 18:36:26 <TD> http://ftp-master.metadata.debian.org/changelogs//main/b/bitcoin/bitcoin_0.8.3-1_changelog
679 2013-07-22 18:37:05 <gmaxwell> http://sources.debian.net/src/bitcoin/0.8.3-1/debian/patches here are the patches.
680 2013-07-22 18:38:51 <TD> they all look pretty innocuous except for the system leveldb
681 2013-07-22 18:38:54 <Luke-Jr> TD: IMO, I would approach this as a "how to use system/packaged leveldb with Bitcoin-Qt" and have the text lead to the conclusion that it's too much effort and the easy route or Just Don't Do It is best <.<
682 2013-07-22 18:39:03 <petertodd> 1007_libmemenv_hurd.patch <- interesting, a patch to the leveldb even though they change it to use the leveldb
683 2013-07-22 18:39:38 <petertodd> Luke-Jr: strongly disagree, leveldb is consensus critical
684 2013-07-22 18:40:06 <petertodd> 2002_libmemenv_debian-ports.patch <- second leveldb patch
685 2013-07-22 18:40:12 <gmaxwell> It's important to note that leveldb isn't the only consensus critcial code we have. I think that it's some of the _riskiest_ consensus critcial code however.
686 2013-07-22 18:40:24 <jgarzik> leveldb consensus is indeed critical -- we have already had fork risk related to database libraries
687 2013-07-22 18:40:33 <jgarzik> let's not reinvent old, solved problems
688 2013-07-22 18:40:44 <gmaxwell> OpenSSL and boost are every bit as critical, but also less likely to change in a way that doesn't fail instantly or fail our unit tests.
689 2013-07-22 18:40:47 <Luke-Jr> petertodd: that's my point
690 2013-07-22 18:41:13 <petertodd> Luke-Jr: and my point is it's damn dangerous, so just don't do it
691 2013-07-22 18:41:44 <gmaxwell> petertodd: I don't think you and luke arguing is likely to be productive here.
692 2013-07-22 18:41:45 <TD> anyone want to comment on the doc?
693 2013-07-22 18:41:49 <TD> or sign it?
694 2013-07-22 18:41:51 <petertodd> reminds me: Partial UTXO mode will trigger more leveldb failures because you'll see more nodes with different amounts of data in the database.
695 2013-07-22 18:42:00 <gmaxwell> TD: where is it?
696 2013-07-22 18:42:01 <petertodd> gmaxwell: good point
697 2013-07-22 18:42:02 <TD> https://docs.google.com/document/d/1naenR6N6fMWSpHM0f4jpQhYBEkCEQDbLBs8AXC19Y-o/edit
698 2013-07-22 18:42:15 <gmaxwell> petertodd: luke is doing something in gentoo which I think is adequately safe, but I don't think anyone else will do it.
699 2013-07-22 18:42:18 <Luke-Jr> TD: I already commented.
700 2013-07-22 18:42:40 <Luke-Jr> TD: IMO, I would approach this as a "how to use system/packaged leveldb with Bitcoin-Qt" and have the text lead to the conclusion that it's too much effort and the easy route or Just Don't Do It is best <.<
701 2013-07-22 18:42:43 <petertodd> gmaxwell: interesting
702 2013-07-22 18:42:57 <TD> well, i'm not sure that gets across the strength of concern. but ok, noted.
703 2013-07-22 18:43:05 <Luke-Jr> educate them on what's involved/necessary, and lead them to come to the conclusion
704 2013-07-22 18:43:32 <Luke-Jr> side effect: we may get people who really want to do it enough, that they contribute to LevelDB update reviews
705 2013-07-22 18:43:33 <petertodd> Luke-Jr: given it should be a short FYI warning, maybe add a link to contact you about how to do it right?
706 2013-07-22 18:43:35 <gmaxwell> petertodd: gentoo's package management is flexible enough to express "must be linked with this version _Exactly_"
707 2013-07-22 18:43:51 <Luke-Jr> petertodd: well, I was thinking of more of a longer document, but maybe
708 2013-07-22 18:43:53 <petertodd> gmaxwell: that changes the situation a lot
709 2013-07-22 18:44:10 <gmaxwell> petertodd: and with the right kicking can happily handle having multiple versions installed.
710 2013-07-22 18:44:13 <petertodd> Luke-Jr: or write it as an appendix? I'm thinking we want to get the point across in the first page (above the fold)
711 2013-07-22 18:44:27 <Luke-Jr> I suppose a shorter document can always be elaborated on later
712 2013-07-22 18:45:48 <gmaxwell> Before sending this we should get a better understanding of why the debian copy is failing.
713 2013-07-22 18:45:56 <TD> does anyone have a log?
714 2013-07-22 18:46:02 <Luke-Jr> erm "a request from the upstream maintainers to not distribute Bitcoin as part of distribution package repositories."
715 2013-07-22 18:46:07 <jgarzik> TD, a doc being edited by luke-jr and petertodd in parallel? I'll wait a bit to sign :) :)
716 2013-07-22 18:46:12 <TD> haha
717 2013-07-22 18:46:14 <gmaxwell> It would look a little silly to implicated leveldb and have it turn out to be GCC or something... though it doesn't really change our message.
718 2013-07-22 18:46:17 <Luke-Jr> packaging bitcoin clients should be expected. just not modifying them.
719 2013-07-22 18:46:39 <petertodd> jgarzik: lol
720 2013-07-22 18:46:48 <TD> i think the temptation will prove too much. besides, you lose the nice auditability of deterministic binaries
721 2013-07-22 18:47:21 <gmaxwell> we might want a note about determinstic binaries in this.
722 2013-07-22 18:47:29 <TD> there's one at the end
723 2013-07-22 18:47:30 <TD> last sentence
724 2013-07-22 18:47:38 <gmaxwell> haha.
725 2013-07-22 18:47:41 <gmaxwell> sorry!
726 2013-07-22 18:47:57 <gmaxwell> petertodd: careful on length.
727 2013-07-22 18:48:01 <TD> the other problem with distributor packaging is that they will attempt to backport security fixes
728 2013-07-22 18:48:12 <Luke-Jr> telling distros not to distribute clients at all is ridiculous. I'd sign against it. :/
729 2013-07-22 18:48:18 <jgarzik> I thought about the doc. My idea for table of contents (doc sections): 1) describe why system lib inclusion is a good thing in general. use zlib as example. 2) describe fork risk. include description of recent hard fork event, and note how database lib played a role. 3) packager recommendations. short, and specific. "do not elide leveldb; be careful with openssl"
730 2013-07-22 18:48:27 <Luke-Jr> TD: well, that's why we have backport branches
731 2013-07-22 18:48:27 <TD> or - they'll get frozen on a single version that might get hardforked off, etc. just generally asking them to ignore all their existing policies for this package won't work reliably, i think
732 2013-07-22 18:48:31 <jgarzik> do not delve too far into other details
733 2013-07-22 18:48:34 <petertodd> Luke-Jr: Can we agree on the please don't until you talk to us?
734 2013-07-22 18:48:35 <jgarzik> short and sweet
735 2013-07-22 18:48:37 <gmaxwell> Luke-Jr: would you agree with something like "don't distribute unless you are able to deal with this"
736 2013-07-22 18:48:46 <gmaxwell> Luke-Jr: like what petertodd said.
737 2013-07-22 18:48:54 <jgarzik> petertodd, +1
738 2013-07-22 18:48:58 <Luke-Jr> jgarzik: +1
739 2013-07-22 18:49:13 <gmaxwell> jgarzik: I like describing why it is important. Show that we do understand the motivations.
740 2013-07-22 18:49:20 <gmaxwell> We care about them too.
741 2013-07-22 18:49:24 <Luke-Jr> gmaxwell: I don't see harm in distributing it unmodified, at least.
742 2013-07-22 18:49:28 <jgarzik> gmaxwell, that's section #1
743 2013-07-22 18:49:37 <jgarzik> gmaxwell, show them we understand the general principle
744 2013-07-22 18:49:47 <jgarzik> gmaxwell, section #2, describe why we deviate from general principle
745 2013-07-22 18:50:00 <jgarzik> gmaxwell, section #3, specific technical recommendations
746 2013-07-22 18:50:02 <Luke-Jr> petertodd: I think encouraging anyone making modifications *in general* to talk to us before public releases, is a good idea.
747 2013-07-22 18:50:10 <petertodd> Luke-Jr: agreed
748 2013-07-22 18:50:12 <jgarzik> section #3 should be very short, very direct
749 2013-07-22 18:50:18 <Luke-Jr> with emphasis on script/blockchain related code
750 2013-07-22 18:50:50 <gmaxwell> I'd add do the recommendations is that we provide an open offer to consult with them. Lets get people talking. It's sad that we're not even aware of the crazy stuff people are doing until we get reports.
751 2013-07-22 18:50:57 <TD> i think "don't distribute it" is pretty short and direct :)
752 2013-07-22 18:51:02 <Luke-Jr> TD: and wrong
753 2013-07-22 18:51:03 <gmaxwell> There is someone competent on this matter in this channel 24/7.
754 2013-07-22 18:51:21 <gmaxwell> TD: it may be too much to ask for and also too much like what someone who is clueless about this stuff demands.
755 2013-07-22 18:51:26 <TD> not being aware of dumb distributor patches is pretty normal. i wasted too many hours debugging bugs introduced by "helpful" packaging back when i worked on wine.
756 2013-07-22 18:51:44 <TD> a marathon debugging session often ended up with the discovery of a surprising patch some unknown guy had silently applied
757 2013-07-22 18:51:45 <petertodd> I think we can remove the last line IMO
758 2013-07-22 18:51:57 <Luke-Jr> TD: that's patching, not distributing.
759 2013-07-22 18:52:15 <TD> well, i know your packages may be an exception, but normally they go hand in hand
760 2013-07-22 18:52:19 <gmaxwell> petertodd: I do think it's good to mention that our normal model has this property that distro packages lack.
761 2013-07-22 18:52:33 <TD> they always find some reason why some change seems harmless
762 2013-07-22 18:52:48 <TD> like the openssl change they thought was a no-op
763 2013-07-22 18:53:16 <sipa> gmaxwell, TD, TheUni: regarding leveldb changes - since some 0.8.x change, we detect database errors as separate from validation errors, so they should just cause a quit (which is perhaps a DoS attack), but it doesn't have chain splitting risk
764 2013-07-22 18:53:23 <petertodd> gmaxwell: good point
765 2013-07-22 18:53:39 <TD> cool
766 2013-07-22 18:54:01 <sipa> and if any leveldb doesn't read back the data as it was written, that is a critical bug
767 2013-07-22 18:54:04 <sipa> for leveldb i mean
768 2013-07-22 18:54:13 <TD> yeah. there have been critical bugs like that in the past though
769 2013-07-22 18:54:27 <sipa> still, it cannot hurt that we follow up on the changes made upstream before adopting them
770 2013-07-22 18:54:27 <TD> the refresh that vinnies pull request is waiting to do fixes one such bug, although bitcoin isn't affected by it
771 2013-07-22 18:54:44 <TD> one can imagine though, that a "simple" change by some packager could have exposed us to it
772 2013-07-22 18:55:00 <petertodd> sipa: but even leveldb being unusually slow for particular data can break consensus for us
773 2013-07-22 18:55:17 <jgarzik> ere is why"
774 2013-07-22 18:55:17 <jgarzik> Please be aware of the distributor context, though! Speaking as one who wandered the corridors of Fedora high level devs... they wage a constant battle against this behavior. Everybody loves to embed+ship libs, and there is a decade-long history of the practice creating security problems. The general tenor of the message should be "we REALLY DO understand why packagers want to ship system libs, but trust us, we have a reason not to, and $h
775 2013-07-22 18:55:49 <jgarzik> Red Hat v. Google constantly fight over Chrome* lib shipping
776 2013-07-22 18:55:52 <gmaxwell> jgarzik++
777 2013-07-22 18:56:32 <gmaxwell> the debian packager for most of my codec stuff is also an upstream contributor now and hangs out in the channels and has lots of stories about REALLY STUPID upstreams.
778 2013-07-22 18:56:33 <petertodd> jgarzik: say something like "In Bitcoin the requirements for consensus even outweight the usual desire to ship patches quickly."
779 2013-07-22 18:56:51 <jgarzik> ACTION handwaves, 90+% of the embedded lib shipping is wrong and hides bugs
780 2013-07-22 18:57:00 <sipa> petertodd: note that we only use libmemenv for the unit tests
781 2013-07-22 18:57:15 <gmaxwell> jgarzik: or even if its right, it still hides bugs. :P
782 2013-07-22 18:57:16 <Luke-Jr> jgarzik: I'd say 99+%!
783 2013-07-22 18:57:53 <gmaxwell> Luke-Jr: some portion is merely harmless.
784 2013-07-22 18:58:02 <gmaxwell> very little is actually good.
785 2013-07-22 18:58:31 <Luke-Jr> gmaxwell: even if it's "harmless", it makes more work for distributors, so harmful in that sense <.<
786 2013-07-22 18:59:22 <gmaxwell> perhaps balanced with upstream bug reporting load though.
787 2013-07-22 18:59:25 <TD> jgarzik: given that chrome has literally pushed seamless auto updates into new realms, you'd think red hat would get over it and let them do what they wanted.
788 2013-07-22 18:59:34 <petertodd> jgarzik: How's the text I added re: "consensus is a security issue"
789 2013-07-22 19:00:02 <Luke-Jr> TD: Google can be as centralized as they want with their own software, but that's a big problem for things like Bitcoin.
790 2013-07-22 19:00:17 <TD> if someone sees a "stupid upstream" the right fix is to do a fork and become a new upstream. not silently fork it behind peoples back at the packaging level
791 2013-07-22 19:00:25 <TD> or contribute patches
792 2013-07-22 19:00:46 <Luke-Jr> TD: embedding libraries (generally) makes one a stupid upstream, but generally not enough reason to fork
793 2013-07-22 19:00:57 <Luke-Jr> packaging-independent auto-upgrade as well
794 2013-07-22 19:01:10 <gmaxwell> TD: Agreed. And good distro packagers aren't silent.
795 2013-07-22 19:02:55 <Luke-Jr> anyhow, I guess the topic is deviating a bit
796 2013-07-22 19:03:03 <Luke-Jr> I suggest we go with jgarzik's outline
797 2013-07-22 19:05:43 <petertodd> Luke-Jr, jgarzik: agreed, current document seems unfocused to me
798 2013-07-22 19:06:03 <gwillen> btw, I would suggest not waiting until the bug is found to specifically contact debian about this
799 2013-07-22 19:06:33 <gwillen> just contact them and advise that 1) their version is incompatible, and 2) they're making some changes, at least one of which has caused a problem in the past, and could they back that one out and maybe also help with deubgging
800 2013-07-22 19:06:46 <gwillen> and by (1) I mean point to the fact that it actually does not work, which is not speculation
801 2013-07-22 19:07:28 <gmaxwell> gwillen: because it forks off early in the chain I do not believe this is as urgent as it might be otherwise.
802 2013-07-22 19:07:36 <gwillen> gmaxwell: that's very fair
803 2013-07-22 19:07:48 <gmaxwell> gwillen: If it were something not yet triggered in the chain I wouldn't be discussing it here.
804 2013-07-22 19:07:50 <gwillen> gmaxwell: otoh that's a lot of potentially frustrated users
805 2013-07-22 19:07:57 <gmaxwell> (because it would be an exploitable vulnerability)
806 2013-07-22 19:07:59 <gwillen> ACTION nod
807 2013-07-22 19:08:10 <TheSeven> hm, looks like I just triggered some primecoin bug... I attempted to swipe several hundred txouts, with default fee settings, and while the resulting transaction is being relayed, it isn't being mined :/
808 2013-07-22 19:08:25 <TheSeven> any general hints on how to fix that kind of mess?
809 2013-07-22 19:09:16 <gmaxwell> TheSeven: salvagewallet your wallet (After a backup) and try again.
810 2013-07-22 19:10:51 <TheSeven> gmaxwell: is there an easy (and fast) way to merge wallets? I just care about the keys, not the txn history etc.
811 2013-07-22 19:11:20 <TheSeven> right now I usually extract the keys using pywallet and import them using RPC commands, but that's terribly slow
812 2013-07-22 19:11:47 <gmaxwell> use the wallet recovery tool on a bunch of concatinated wallets.
813 2013-07-22 19:11:56 <gmaxwell> doesn't work if they're encrypted.
814 2013-07-22 19:12:10 <TheSeven> interesting approach
815 2013-07-22 19:12:56 <TheSeven> I have ~170k privkeys to import, check for funds, swipe and discard from a huge primecoin farm
816 2013-07-22 19:13:12 <TheSeven> takes almost a day through RPC...
817 2013-07-22 19:14:00 <petertodd> jgarzik: How's the current version read? Seems pretty close to your outline.
818 2013-07-22 19:16:12 <Luke-Jr> and memory usage?
819 2013-07-22 19:16:52 <gmaxwell> petertodd: I'd like to be able to say something like we can help provide you with testing procedures to reduce risks... or something because I do think its sounding a bit too much like its saying we're demanding you use the software exactly.
820 2013-07-22 19:17:10 <gmaxwell> petertodd: which may gain us criticism from people who really don't completely grok the hardness of the consensus problem.
821 2013-07-22 19:17:49 <gmaxwell> "Bitcoin is rubbish made by idiots who can't even write software secure enough to handle divergence in clients for their network protcol HAHA!"
822 2013-07-22 19:18:01 <TheUni> it makes sense to me to remove the references to leveldb in the letter, and instead commit a set of packaging instructions in source
823 2013-07-22 19:18:10 <petertodd> gmaxwell: good points, though I'd rather they think bitcoin is rubbish than they ship broken clients
824 2013-07-22 19:18:23 <TheUni> so that they may be updated in the future, and the current requirements are always bundled with a corresponding srctree
825 2013-07-22 19:18:30 <gmaxwell> petertodd: keep in mind that 99.999% of people who will read this are not distro packaging people.
826 2013-07-22 19:19:08 <TheUni> then reference those instuctions in the letter if necessary
827 2013-07-22 19:19:23 <TheUni> otherwise, this is a one-off post that fades away in time
828 2013-07-22 19:19:40 <gmaxwell> I think TheUni has a good suggestion. The instructions can even be "DON'T" but saying in the document that we have instructions for you is an improvement.
829 2013-07-22 19:20:38 <petertodd> TheUni: good point, so "DON'T UNLESS" is page one, "HOW" is page two
830 2013-07-22 19:20:53 <Luke-Jr> "For this reason, it is vital that as much of the network as possible uses the upstream reference implementation unmodified in any way." screams centralization
831 2013-07-22 19:20:57 <TheUni> it could even be added in the man pages, so that distros would have to either patch out the upstream requirements, or violate the requirements listed in 'man'
832 2013-07-22 19:21:02 <gmaxwell> Basically the instructions should be "Distribute exactly." "Failing that, build with the same diligence we do: Make public your activities to the bitcoin community (include the nearly 24/7 reachablity of competent help), run these tests, ..."
833 2013-07-22 19:21:34 <gmaxwell> Luke-Jr: thanks. yea. lets rephrase that a bit.
834 2013-07-22 19:22:00 <Luke-Jr> "uses implementations that have been carefully audited to meet the consensus of the rest of the network" perhaps
835 2013-07-22 19:22:06 <TheUni> but either way, official requirements/recommendations for the client should be bundled with the client itself, not found by googling for press-releases
836 2013-07-22 19:22:13 <petertodd> Luke-Jr: +1
837 2013-07-22 19:22:21 <gmaxwell> Luke-Jr: good, {{sofixit}}
838 2013-07-22 19:22:33 <gmaxwell> Yea, the statement should just reference it, it's already too long.
839 2013-07-22 19:22:59 <jgarzik> petertodd, reviewing
840 2013-07-22 19:28:14 <gmaxwell> I'm writing "So you want to build some Bitcoin(d/qt):", I'll put it up for revision when I have a first pass done in a few.
841 2013-07-22 19:28:20 <jgarzik> petertodd, I would include specific examples (link to recent fork portmortem)
842 2013-07-22 19:28:27 <jgarzik> be specific
843 2013-07-22 19:28:34 <jgarzik> Your readers are smart, technical people.
844 2013-07-22 19:28:43 <jgarzik> They just don't know this area of C.S.
845 2013-07-22 19:29:03 <jgarzik> overall it is still worded like one academic to another, a bit
846 2013-07-22 19:29:13 <TheUni> Headline is obvious as-is: "bitcoin devs blast debian for patching their sources"
847 2013-07-22 19:29:26 <jgarzik> theory is theory. we have practice, we have field experience to back up this theory.
848 2013-07-22 19:29:28 <TheUni> again, i very strongly recommend dropping the Debian references
849 2013-07-22 19:29:34 <jgarzik> +1
850 2013-07-22 19:29:49 <petertodd> jgarzik: Good idea
851 2013-07-22 19:30:47 <petertodd> jgarzik: https://en.bitcoin.it/wiki/BIP_0050 link for march chian fork? got a better one in mind?
852 2013-07-22 19:31:18 <gmaxwell> TheUni: that would be kind of us at least, certatnly "debian fucks up security again by patching shit they don't understand" is a message which, perhaps unfairly, would have a lot of resonance.
853 2013-07-22 19:31:45 <sipa> wait, do we know there are actually problems caused by debian patching to use system leveldb?
854 2013-07-22 19:31:51 <sipa> or is this a guess?
855 2013-07-22 19:32:08 <gmaxwell> sipa: we do not know that it's leveldb! thats why I keep saying we need to figure that out first.
856 2013-07-22 19:32:11 <jgarzik> gmaxwell, <grin> this recalls the grand reddit tradition of wildly rewriting an otherwise staid, professional headline
857 2013-07-22 19:32:18 <TheUni> gmaxwell: yea, if their name is mentioned, all discussion will end. it'll just be a debian packaging circlejerk/flamewar
858 2013-07-22 19:32:28 <jgarzik> +1
859 2013-07-22 19:32:32 <gmaxwell> sipa: we know that debian's sid builds are (apparently reliably?) rejecting the chain.
860 2013-07-22 19:32:41 <gmaxwell> sipa: in some april 2013 block.
861 2013-07-22 19:32:47 <ahmedbodi> doublec: you here?
862 2013-07-22 19:33:15 <gmaxwell> dropping all mention of debian has an advantage of letting us not get distracted by the specific incident.
863 2013-07-22 19:33:28 <TheUni> sipa: you said that there are blacklisted db entries already?
864 2013-07-22 19:33:34 <ahmedbodi> anyone else here who knows anything about bitcoin's merged mining patch?
865 2013-07-22 19:33:35 <sipa> TheUni: ?
866 2013-07-22 19:33:59 <TheUni> gmaxwell, TD, TheUni: regarding leveldb changes - since some 0.8.x change, we detect database errors as separate from validation errors, so they should just cause a quit
867 2013-07-22 19:34:05 <TheUni> er, your quote ^^
868 2013-07-22 19:34:13 <sipa> TheUni: yes
869 2013-07-22 19:34:21 <TheUni> sipa: are those cases in bitcoin_test ?
870 2013-07-22 19:34:22 <sipa> i'm not sure what that has to do with blacklisting?
871 2013-07-22 19:34:31 <TheUni> ah i see, nm
872 2013-07-22 19:34:36 <gmaxwell> TheUni: yes, but only if the _error_ reports as an error.
873 2013-07-22 19:34:52 <sipa> TheUni: i mean errors like failed to read from disk, or errors from the database layer
874 2013-07-22 19:34:56 <TheUni> yes, i misread
875 2013-07-22 19:35:06 <sipa> TheUni: previously they would get the chain that was being connected at that point being marked invalid
876 2013-07-22 19:35:41 <TheUni> but it is possible in theory to add the current known-busted db entries to a blacklist, no? ignoring the fact that it doesn't help in the future
877 2013-07-22 19:35:42 <Luke-Jr> ACTION wonders if the next-test he just released will do IBD successfully <.< http://bitcointroll.org/?topic=260749
878 2013-07-22 19:36:28 <Luke-Jr> anyone know if the Debian issue is reproducable with -reindex?
879 2013-07-22 19:36:34 <sipa> TheUni: there are no "knwon busted entries"
880 2013-07-22 19:36:38 <sipa> TheUni: if there were, we had a problem
881 2013-07-22 19:36:42 <gmaxwell> TheUni: if there were any known. (well other than the genesis block's)
882 2013-07-22 19:37:06 <sipa> Luke-Jr: from what this person was telling on IRC before, it was not deterministic
883 2013-07-22 19:37:13 <Luke-Jr> ugh
884 2013-07-22 19:37:14 <sipa> Luke-Jr: and it sounded much more like a hardware problem to me
885 2013-07-22 19:37:20 <TheUni> gmaxwell: the march split you mentioned above? it's not possible to trace that to a single verifiable entry somewhere?
886 2013-07-22 19:37:26 <TheUni> sipa: ok, that answers my question. thanks.
887 2013-07-22 19:37:29 <Luke-Jr> so it might not even be a Debian fault. definitely let's not blame <.<
888 2013-07-22 19:37:30 <sipa> TheUni: yes, but only for 0.7
889 2013-07-22 19:37:36 <sipa> especially since -par=1 fixes it
890 2013-07-22 19:37:53 <sipa> even with multithreaded verification, db writing is single-threaded
891 2013-07-22 19:38:04 <sipa> so if it were a leveldb problem, i don't see how -par=1 fixes it
892 2013-07-22 19:38:11 <sipa> except by loading the CPU less
893 2013-07-22 19:38:26 <Luke-Jr> that's scary
894 2013-07-22 19:38:42 <sipa> not more scary than other corruption reports we've had
895 2013-07-22 19:38:46 <gmaxwell> sipa: doesn't explain why the failure point is apparently consistent.
896 2013-07-22 19:38:50 <sipa> gmaxwell: is it?
897 2013-07-22 19:39:00 <sipa> it's certainly not the same block
898 2013-07-22 19:39:03 <TheUni> maybe debian users are running on old pentium pros that can't add :)
899 2013-07-22 19:39:21 <sipa> he has reported before that he crashed at some point, and later that he got past it in another run
900 2013-07-22 19:39:23 <TheSeven> haha. looks like I indeed just broke primecoin by accident...
901 2013-07-22 19:39:35 <gmaxwell> sipa: indeed.
902 2013-07-22 19:39:50 <Luke-Jr> TheSeven: not sure anyone cares about primecoin in here
903 2013-07-22 19:40:09 <gmaxwell> TheSeven: if the problem exists in bitcoin please report the bug. :P
904 2013-07-22 19:40:26 <gmaxwell> Luke-Jr: the graveyard of failed altcoins is of mild interest.
905 2013-07-22 19:40:43 <Luke-Jr> gmaxwell: you saw how much primecoin deviated.. :p
906 2013-07-22 19:41:02 <gmaxwell> Luke-Jr: http://www.despair.com/mistakes.html
907 2013-07-22 19:41:10 <Luke-Jr> gmaxwell: (your "I'm not sure what this is, but it doesn't belong here")
908 2013-07-22 19:41:29 <TheSeven> I just thought I might share it... I somehow managed to generate these not-being-mined txns, and it apparently, despite not being mined, dropped network hashrate by a factor of about >10. what the hell.
909 2013-07-22 19:41:40 <sipa> gmaxwell: this is very applicable to unit tests :)
910 2013-07-22 19:42:03 <Luke-Jr> lol
911 2013-07-22 19:44:08 <petertodd> Other than the document being too long for anyone to sign it, IMO the existing version looks great.
912 2013-07-22 19:44:51 <sipa> which document?
913 2013-07-22 19:45:01 <gmaxwell> sipa: https://docs.google.com/document/d/1naenR6N6fMWSpHM0f4jpQhYBEkCEQDbLBs8AXC19Y-o/edit#
914 2013-07-22 19:45:06 <petertodd> sipa: what gmaxwell said
915 2013-07-22 19:46:53 <TheUni> might also drop the extreme part about "must be fixed within hours or the system is threatened"
916 2013-07-22 19:47:08 <TheUni> headline: "one change by distro maintainers could wipe our your life savings"
917 2013-07-22 19:48:10 <petertodd> TheUni: s/survival/functionality/ ?
918 2013-07-22 19:49:31 <petertodd> Removed the 'consistency more important than correctness' line for the same headline generating reasons...
919 2013-07-22 19:49:38 <gmaxwell> aww.
920 2013-07-22 19:49:40 <gmaxwell> :P
921 2013-07-22 19:49:47 <Luke-Jr> having my wife proofread it
922 2013-07-22 19:50:34 <petertodd> gmaxwell: hurt to do that
923 2013-07-22 19:51:50 <TheUni> petertodd: "...could necessitate manual intervention by users and devs alike" or so, is what i had in mind
924 2013-07-22 19:52:07 <gmaxwell> TheUni: it's not just manual intervention.
925 2013-07-22 19:52:18 <gmaxwell> a bad enough consensus fail could actually be the end of bitcoin.
926 2013-07-22 19:52:51 <gmaxwell> e.g. if someone manages a multi doublespend against two large merchants.. it would be really hard to rebuild confidence in bitcoin.
927 2013-07-22 19:53:08 <TheUni> gmaxwell: lots of things could be the end of bitcoin, but stressing them during routine PR is not exactly going to give anyone a warm feeling
928 2013-07-22 19:53:21 <petertodd> TheUni: agreed
929 2013-07-22 19:53:32 <gmaxwell> right, I'm fine with not saying loud things. But we shouldn't say things which underplay the risk either.
930 2013-07-22 19:53:58 <TheUni> need to decide on an end goal: are we stressing what the consequences of not making this change could be? or are we trying to get the change made?
931 2013-07-22 19:54:01 <gmaxwell> E.g. Just say this is important. Don't say "this is important because a couple people might have mildly bad day"
932 2013-07-22 19:54:18 <gmaxwell> (if the consequence is really a lot worse)
933 2013-07-22 19:54:18 <TheUni> cause those are 2 different perspectives, which require their own tones
934 2013-07-22 19:54:18 <warren> Can we consider upgrading the library deps in gitian for bitcoin-0.9?
935 2013-07-22 19:54:28 <sipa> warren: sure
936 2013-07-22 19:55:06 <petertodd> gmaxwell: "is a critical problem that threatens the functionality and security of the system for all users" <- happy with that?
937 2013-07-22 19:55:28 <Luke-Jr> warren: just needs a pullreq and some audits
938 2013-07-22 19:55:30 <TheUni> gmaxwell: not trying to be a contrarian here, just taking the typical PR approach of "never say anything bad about your product"
939 2013-07-22 19:55:37 <warren> hmm, I wonder what versions we should use.
940 2013-07-22 19:55:41 <TheUni> imo that goes a long way
941 2013-07-22 19:55:58 <TheUni> warren: whatever is current in gitian distro should be reference for the rest
942 2013-07-22 19:56:15 <TheUni> ^^ easy rule of thumb. that way all OS's can align
943 2013-07-22 19:56:16 <gmaxwell> TheUni: Bitcoin is risky enough that I feel that we have a bit of a moral obligation to be frank. Obviously we don't want to caused confused panic either, but we should generally error towards being frank.
944 2013-07-22 19:56:26 <petertodd> gmaxwell: agreed
945 2013-07-22 19:56:38 <sipa> i think we should first at least get clarification whether this is a consistent block that triggers the bug
946 2013-07-22 19:57:11 <warren> TheUni: btw, have you submitted autotool and gitian fixes for secp256k1, in addition to autotooling bitcoin master?
947 2013-07-22 19:57:33 <petertodd> sipa: IMO the advice is good and something we should be saying, though I agree that clear evidence of a problem first is better
948 2013-07-22 19:57:39 <gmaxwell> sipa: ignoring the debian thing, knowing that they're patching out our internal leveldb it's probably prudent to say something here.
949 2013-07-22 19:58:04 <sipa> gmaxwell: agree, but if there is no direct threat here, i'd rather not have it linked to it
950 2013-07-22 19:58:07 <gmaxwell> This no longer mentions our specific concerns with debian.. also I believe arch is doing the same thing.
951 2013-07-22 19:58:15 <TheUni> warren: no
952 2013-07-22 19:58:24 <gmaxwell> sipa: agree, but I think thats fixed now.
953 2013-07-22 19:58:42 <warren> If fedora gets past the EC issue, they'll insist on similarly not shipping "redundant" libraries, which will be a problem here...
954 2013-07-22 19:59:00 <TheUni> EC issue?
955 2013-07-22 19:59:09 <gmaxwell> TheUni: redhat strips ECC support from openssl.
956 2013-07-22 19:59:14 <gmaxwell> (though note: they have it in NSS now)
957 2013-07-22 19:59:21 <TheUni> i still say a library rename is in order, btw
958 2013-07-22 19:59:25 <Luke-Jr> warren: all the more reason to swtich to sipa's secp256k1 :D
959 2013-07-22 19:59:30 <petertodd> How about instead of "this requirement is not compatible" say "For instance even simply replacing the included leveldb library with a system library is a change that requires careful auditing."
960 2013-07-22 19:59:31 <sipa> TheUni: to?
961 2013-07-22 19:59:44 <gmaxwell> TheUni: openssl's ECC support mixes up a bunch of probably patented optimizations and clearly not-patented stuff, so they just turn it all off.
962 2013-07-22 19:59:53 <warren> Luke-Jr: yeah, with that at least fedora/RHEL users will be able to build it without replacing their openssl
963 2013-07-22 19:59:59 <Luke-Jr> petertodd: *constant* careful auditing!
964 2013-07-22 20:00:01 <TheUni> sipa: leveldb -> bitcoindb. If leveldb can never be dropped in, calling it leveldb can only be a bad thing
965 2013-07-22 20:00:11 <warren> TheUni: good idea
966 2013-07-22 20:00:12 <sipa> oh, that
967 2013-07-22 20:00:21 <petertodd> Luke-Jr: good point
968 2013-07-22 20:00:22 <sipa> i though you were talking about libsecp256k1 :)
969 2013-07-22 20:00:29 <Luke-Jr> TheUni: leveldb *can* be dropped it, it just requires care
970 2013-07-22 20:00:31 <warren> call the db a "fork".
971 2013-07-22 20:00:32 <TheUni> sipa: hehe, nah, that's all yours :)
972 2013-07-22 20:00:44 <Luke-Jr> TheUni: Gentoo does and will continue to do so, since its packaging system is capable of that care
973 2013-07-22 20:01:00 <warren> TheUni: and people can colloquially call it "bdb"
974 2013-07-22 20:01:01 <warren> oh wait
975 2013-07-22 20:01:02 <TheUni> Luke-Jr: that "care" requires using old versions and/or patching, right?
976 2013-07-22 20:01:09 <TheUni> warren: that's a terrible idea :)
977 2013-07-22 20:01:16 <phantomcircuit> TheUni, requires specific versions
978 2013-07-22 20:01:18 <gmaxwell> TheUni: gentoo's package enviroment is able to enforce a specific version _exactly_.
979 2013-07-22 20:01:20 <Luke-Jr> TheUni: there is a dependency on the specific version bundled, basically
980 2013-07-22 20:01:27 <Luke-Jr> TheUni: with the specific build flags we use
981 2013-07-22 20:01:27 <phantomcircuit> JYNX ON ALL OF YOU
982 2013-07-22 20:01:29 <TheUni> Luke-Jr: right, so it's not a drop-in going forward
983 2013-07-22 20:01:42 <petertodd> Luke-Jr: how's that language?
984 2013-07-22 20:01:44 <TheUni> it's a snapshot of an old version that's not compatible with the current leveldb development, right?
985 2013-07-22 20:01:55 <sipa> TheUni: it is compatible
986 2013-07-22 20:02:09 <Luke-Jr> FWIW, other Gentoo packages using leveldb: dev-libs/replicant net-fs/cvmfs net-misc/elliptics sys-cluster/ceph
987 2013-07-22 20:02:16 <Luke-Jr> petertodd: ?
988 2013-07-22 20:02:34 <petertodd> Luke-Jr: I mean, I changed it to talk about constant auditing re: shared libraries
989 2013-07-22 20:02:44 <warren> "bdb" is compatible too, and it's a major source of headaches for users.
990 2013-07-22 20:02:52 <TheUni> sipa: sorry, poor choice of words. what i meant is this: the version of leveldb produced by gentoo is effectively what would be called "bitcoindb" if it were renamed
991 2013-07-22 20:03:01 <Luke-Jr> petertodd: ah, I'd stress constant more.. it's important that distros realize they will need to audit *ever* *single* *bump* to the lib
992 2013-07-22 20:03:05 <sipa> TheUni: not sure what you mean by that
993 2013-07-22 20:03:06 <warren> TheUni: how about coindb?
994 2013-07-22 20:03:19 <sipa> TheUni: the databases we write can be read/modified by any recent leveldb version
995 2013-07-22 20:03:23 <TheUni> warren: heh, the name is not really the matter at hand :)
996 2013-07-22 20:03:45 <Luke-Jr> TheUni: we will probably upgrade LevelDB for 0.9
997 2013-07-22 20:03:48 <TheUni> sipa: and the latest stable would cause a disagreement in clients, no?
998 2013-07-22 20:03:55 <sipa> TheUni: highly unlikely
999 2013-07-22 20:03:57 <Luke-Jr> TheUni: we don't know if it would
1000 2013-07-22 20:04:00 <sipa> TheUni: but there is a risk
1001 2013-07-22 20:04:14 <Luke-Jr> TheUni: the point is, before upgrading it, *someone* MUST do a careful review
1002 2013-07-22 20:04:37 <Luke-Jr> TheUni: if Debian wants to add that to their process, that's fine. but most likely the easiest option for them is to leave it embedded.
1003 2013-07-22 20:04:46 <TheUni> Luke-Jr: if that's the point, then you can't also make the point that it's generally compatible with the upstream version
1004 2013-07-22 20:04:50 <sipa> yes, i am in favor of upgrading leveldb - i've gone through the changes, and didn't find anything that should affect us, but i didn't look that deep either
1005 2013-07-22 20:04:54 <TheUni> if it requires the "after careful review" qualifier
1006 2013-07-22 20:05:31 <Luke-Jr> TheUni: that is true for EVERY one of our deps, arguably
1007 2013-07-22 20:06:16 <Luke-Jr> who is Cookie Monster?
1008 2013-07-22 20:06:23 <TheUni> Luke-Jr: i'm only arguing that if the general concensus is to require a snapshotted, patched version of leveldb at all times, then we have become maintainers of a fork that closely follows an upstream. and it should be treated as such
1009 2013-07-22 20:06:37 <sipa> it is not about leveldb itself - if there are no bugs in it, there is no problem (and if there is, they have a problem)
1010 2013-07-22 20:06:44 <Luke-Jr> TheUni: we do no patching
1011 2013-07-22 20:06:48 <warren> TheUni: https://github.com/sipa/bitcoin/commits/secp256k1 bitcoin patched for secp256k1. I highly recommend prepping this repo for the autotools change too.
1012 2013-07-22 20:07:03 <sipa> Luke-Jr: we do
1013 2013-07-22 20:07:09 <sipa> Luke-Jr: especially the windows branch
1014 2013-07-22 20:07:18 <Luke-Jr> oh, well, Windows isn't really a concern here..
1015 2013-07-22 20:07:26 <sipa> it absolutely is
1016 2013-07-22 20:07:36 <Luke-Jr> I don't see Microsoft doing a Bitcoin-Qt distro..
1017 2013-07-22 20:07:43 <TheUni> it is if the same source can't be used as a reference across the board
1018 2013-07-22 20:08:12 <TheUni> ok, i've made made my opinion known. you guys can hash it out :)
1019 2013-07-22 20:08:20 <warren> Luke-Jr: you're underestimating the resistance we have from distros who have a hard line stance against using embedded libraries. Declaring our internal db a "fork" with a different name will entirely allow it to sidestep at least fedora's policy.
1020 2013-07-22 20:09:09 <TheUni> sipa: re the above, how do you see secp256k1 being hooked up?
1021 2013-07-22 20:09:19 <TheUni> part of the bitcoin tree? or out of tree?
1022 2013-07-22 20:09:26 <sipa> TheUni: right now, out
1023 2013-07-22 20:09:41 <sipa> there are several people who are interested in it, outside of bitcoin
1024 2013-07-22 20:09:52 <TheUni> sipa: when it hits master, still out?
1025 2013-07-22 20:10:03 <sipa> probably similarly to leveldb
1026 2013-07-22 20:10:10 <TheUni> so.. in :)
1027 2013-07-22 20:10:12 <sipa> maintained out of tree, but with snapshotted versions copied in
1028 2013-07-22 20:10:38 <TheUni> ok. so it'd still need its own buildsystem then
1029 2013-07-22 20:10:43 <warren> sipa: please call the one in bitcoin a 'fork' as merely a policy issue
1030 2013-07-22 20:11:05 <sipa> warren: https://github.com/bitcoin/leveldb
1031 2013-07-22 20:11:17 <warren> nice
1032 2013-07-22 20:11:23 <sipa> TheUni: jgarzik once offerred autotoolsifying libsecp256k1, but never got to it
1033 2013-07-22 20:11:36 <warren> sipa: can we rename it to make it more obviously a "fork"?
1034 2013-07-22 20:11:38 <sipa> i would certainly like a more professional build env for it
1035 2013-07-22 20:11:45 <TheUni> sipa: sure you're not thinking of of me? i offered the same at one point
1036 2013-07-22 20:11:50 <sipa> TheUni: i know
1037 2013-07-22 20:11:54 <TheUni> ok
1038 2013-07-22 20:12:33 <sipa> TheUni: you're very welcome to do so, but i'd rather get bitcoin autotoolsified now :)
1039 2013-07-22 20:12:53 <TheUni> sipa: offer back on the table after bitcoin PR goes in, so that i'm not actively working on 2 at once
1040 2013-07-22 20:13:16 <TheUni> if someone beats me to it, all the better :)
1041 2013-07-22 20:13:32 <sipa> warren: as long as it is 1) backward+forward compatible with leveldb and 2) intended to be useful for projects that are not bitcoin -> no
1042 2013-07-22 20:14:16 <phantomcircuit> lol bernanke
1043 2013-07-22 20:14:16 <sipa> warren: i think so
1044 2013-07-22 20:14:16 <warren> sipa: secp256k1 for bitcoin-0.9 is currently a stretch goal?
1045 2013-07-22 20:14:54 <sipa> (my english fails me here... by stretch goal, you mean something unlikely to happen?)
1046 2013-07-22 20:14:54 <TheUni> sipa: as an option? or on?
1047 2013-07-22 20:14:54 <warren> sipa: I'm just saying I don't want fedora to use its system leveldb, and they will force the issue if the leveldb in bitcoin is named as such.
1048 2013-07-22 20:15:24 <warren> sipa: stretch goal ... if primary objectives are completed and we have enough time to ensure secp256k1 is safe
1049 2013-07-22 20:15:25 <sipa> warren: if gmaxwell's letter isn't convincing enough, we shouldn't falsify the facts
1050 2013-07-22 20:15:44 <gmaxwell> Here is a first rough stab at some guidance that could be checked into git about building binaries for others: https://docs.google.com/document/d/1b1WKdvUl8K8mdNA9yTx2ATg97tpWPEL8NUpWRTob5Tw/edit
1051 2013-07-22 20:15:55 <warren> Is it really a falsification? It's a fork that is tested for this purpose.
1052 2013-07-22 20:15:57 <gmaxwell> please feel free to hack on it some if you're interested.
1053 2013-07-22 20:16:58 <gmaxwell> warren: which may happen at some point or another to be identical to upstream.
1054 2013-07-22 20:16:58 <petertodd> gmaxwell: isn't public
1055 2013-07-22 20:16:58 <sipa> not because it is different
1056 2013-07-22 20:16:58 <sipa> warren: but even if it goes into upstream at some point, we'll still keep out own copy
1057 2013-07-22 20:16:58 <warren> gmaxwell: that'd be perfect
1058 2013-07-22 20:17:03 <phantomcircuit> petertodd, actually it is you just gotta change the url a bit
1059 2013-07-22 20:17:04 <sipa> but because we need controlled changes
1060 2013-07-22 20:17:13 <warren> wait, "it goes into usptream" is what?
1061 2013-07-22 20:17:35 <gmaxwell> Try again: https://docs.google.com/document/d/1b1WKdvUl8K8mdNA9yTx2ATg97tpWPEL8NUpWRTob5Tw/edit
1062 2013-07-22 20:18:00 <warren> sipa: the point here is bitcoin cannot risk arbitrary changes in the system library. Upstreaming things doesn't help this issue.
1063 2013-07-22 20:18:14 <gmaxwell> Have you all read the color of your bits essay?
1064 2013-07-22 20:18:27 <petertodd> gmaxwell: "color of your bits" ?
1065 2013-07-22 20:18:45 <gmaxwell> petertodd: http://ansuz.sooke.bc.ca/entry/23
1066 2013-07-22 20:19:03 <gmaxwell> We need software which has a tested-for-bitcoin-consistency color, which isn't something you can see in any patch.
1067 2013-07-22 20:19:43 <warren> there's precedent for this in distros, they stick to certain versions of old software or ship it separately because it is the only version certified by <authority>
1068 2013-07-22 20:20:00 <TheUni> btw, autotools will allow testing for specific versions of libs, so it's easier to kick out with "this version does not work, do not patch this out"
1069 2013-07-22 20:20:14 <TheUni> doesn't help in the case of leveldb, but would with openssl and others
1070 2013-07-22 20:20:20 <gmaxwell> TheUni: except distros regularly patch packages while keeping the versions the same. :(
1071 2013-07-22 20:20:36 <phantomcircuit> lol debian
1072 2013-07-22 20:20:43 <warren> Please just declare the tested-for-consistency library version as a fork with its own name. distro policy would probably mean it ships as a dynamic library to be used by arbitrary other software that links to it.
1073 2013-07-22 20:20:55 <TheUni> gmaxwell: sure, just saying they'd have to litterally remove the warning telling them not to do what they're doing
1074 2013-07-22 20:21:18 <TheUni> *literally
1075 2013-07-22 20:22:22 <petertodd> gmaxwell: ah, yeah that one's good
1076 2013-07-22 20:22:50 <gmaxwell> TheUni: http://pastebin.mozilla.org/2689635 OpenSSL's patches in fedora. :P
1077 2013-07-22 20:23:16 <Luke-Jr> TheUni: --with-system-leveldb='I have read doc/packaging with SHA256 <hash> and understand the risks'
1078 2013-07-22 20:23:16 <TheUni> gmaxwell: to be far, openssl is a maintainer's nightmare
1079 2013-07-22 20:23:41 <TheUni> gmaxwell: have you tried working with their buildsystem for some exotic arch?
1080 2013-07-22 20:23:42 <warren> TheUni: lots of that is for their government certification. I don't know the story of why things are needed in there.
1081 2013-07-22 20:23:52 <TheUni> put it this way.. i think i'd rather work with boost's bjam
1082 2013-07-22 20:23:56 <gmaxwell> TheUni: 22792 lines of patch. :P
1083 2013-07-22 20:24:45 <TheUni> heh
1084 2013-07-22 20:24:52 <petertodd> gmaxwell: that's like 10x more lines of code than I've written for Bitcoin...
1085 2013-07-22 20:25:38 <TheUni> it's a nightmarish lump of code. I can see what that level of patching would be required
1086 2013-07-22 20:25:40 <warren> It's huge patches, but I wouldn't be surprised if the government and contractors were involved in auditing it.
1087 2013-07-22 20:25:47 <petertodd> gmaxwell: "so you want to" doc looks good to me generally, good to be clear as to exactly what a break of consensus does too
1088 2013-07-22 20:25:57 <petertodd> gmaxwell: *good that you are clear
1089 2013-07-22 20:26:00 <gmaxwell> But as I said before, I think that leveldb has a lot more risk of unnoticed bitcoin specific breakage than openssl.
1090 2013-07-22 20:26:14 <warren> thus, please rename and declare it a fork
1091 2013-07-22 20:27:18 <lianj> did anything change about how txs that depends on other unconfirmed txs that are already in the memory pool are relayed/broadcasted?
1092 2013-07-22 20:27:40 <sipa> lianj: the mempool does not rebroadcast - wallets do
1093 2013-07-22 20:27:53 <sipa> nothing changed wrt that in the past... years afaik
1094 2013-07-22 20:28:17 <phantomcircuit> sipa, he's trying to figure out why chained dependent unconfirmed transactions aren't being relayed
1095 2013-07-22 20:28:26 <gmaxwell> I wish I knew which software was the agressive rebroadcaster.
1096 2013-07-22 20:28:30 <warren> phantomcircuit: aren't being relayed by whom?
1097 2013-07-22 20:28:44 <Luke-Jr> gmaxwell: bitcoin-ruby? :p
1098 2013-07-22 20:28:45 <TheUni> warren: in case you didn't see above, i have bitcoind up and running on android. looking to see if there's any interest in actively working on it
1099 2013-07-22 20:28:52 <phantomcircuit> some % of the network significant enough to cause a propagation issue
1100 2013-07-22 20:28:56 <gmaxwell> there is some node out there which appears to be agressively rebroadcasting its mempool every block. it wastes a bunch of bandwidth.
1101 2013-07-22 20:29:12 <Luke-Jr> gmaxwell: no UA?
1102 2013-07-22 20:29:15 <lianj> not rebroadcast, just relay. hm ok thanks. seeing issues where pushing one tx and another tx that depends on the previous one a second later results in not showing up on blockchain.info/blockexplorer. but if i ask the node i pushed it to it has both inside his memory pool
1103 2013-07-22 20:29:18 <warren> TheUni: that's cool, but perhaps more useful when bitcoind grows the proposed SPV mode discussed on the list.
1104 2013-07-22 20:29:27 <sipa> we may need a negative mempool, i.e. a list of invs that we know about though have deleted ourself
1105 2013-07-22 20:29:46 <TheUni> warren: spv mode? link?
1106 2013-07-22 20:29:49 <petertodd> sipa: absolutely required IMO
1107 2013-07-22 20:29:50 <sipa> which are actively ignored when announced
1108 2013-07-22 20:29:51 <Luke-Jr> sipa: bloom filter?
1109 2013-07-22 20:29:54 <warren> TheUni: follow bitcoin dev mailing list
1110 2013-07-22 20:30:02 <TheUni> crap, back in 30
1111 2013-07-22 20:30:07 <gmaxwell> Luke-Jr: I think it claims to be bitcoin-qt.
1112 2013-07-22 20:30:26 <petertodd> Luke-Jr: IMO bloom not ok given the use of bitcoind to protect non-bitcoind nodes
1113 2013-07-22 20:30:39 <sipa> agree with petertodd
1114 2013-07-22 20:30:40 <petertodd> Luke-Jr: (you can't even not apply it to wallet-related, it'll still break stuff)
1115 2013-07-22 20:30:50 <phantomcircuit> gmaxwell, 4000 * 500 * 36 bytes / 10 minutes ~= 120 KB/s to broadcast 500 transactions via inv (assuming most people already have them)
1116 2013-07-22 20:31:00 <warren> petertodd: how much work will the partial UTXO thing be?
1117 2013-07-22 20:31:02 <petertodd> Luke-Jr: Granted 2^-64 probability is fine, so it can truncate hashes.
1118 2013-07-22 20:31:08 <phantomcircuit> it's like 30 baud per peer
1119 2013-07-22 20:31:09 <petertodd> warren: Lots...
1120 2013-07-22 20:31:23 <petertodd> warren: Still looking at it, though sipa's done some of the work re: header sync.
1121 2013-07-22 20:31:27 <warren> ah
1122 2013-07-22 20:32:00 <midnightmagic> sipa: There's something like it in the transaction non-relay patches floating around, so people (myself included) are already running a sort-of negative mempool mechanism.
1123 2013-07-22 20:32:23 <sipa> actually
1124 2013-07-22 20:32:24 <phantomcircuit> midnightmagic, that could be an issue actually if done poorly
1125 2013-07-22 20:32:33 <petertodd> Luke-Jr: (tuncation requirs per-node seed though at 2^-64, that's not secure re: birthday attack in principle)
1126 2013-07-22 20:32:35 <phantomcircuit> you could essentially permanently blacklist a transaction
1127 2013-07-22 20:32:59 <sipa> i think header-first sync will be useful pretty soon
1128 2013-07-22 20:33:00 <Luke-Jr> well, be careful not to add a memory exhaustion leak
1129 2013-07-22 20:33:01 <midnightmagic> phantomcircuit: that's the idea. at least until some other chump includes it in a block.
1130 2013-07-22 20:33:04 <Luke-Jr> vuln*
1131 2013-07-22 20:33:11 <sipa> i need to figure out some heuristics for downloading still
1132 2013-07-22 20:33:22 <sipa> perhaps i can just do a simple download-from-one-node for now
1133 2013-07-22 20:33:24 <petertodd> Luke-Jr: my idea would be the blacklist would be on disk actually
1134 2013-07-22 20:33:39 <phantomcircuit> petertodd, nope * 1000
1135 2013-07-22 20:33:47 <petertodd> phantomcircuit: rational?
1136 2013-07-22 20:34:32 <phantomcircuit> petertodd, having multiple databases on disk which a remote peer can cause churn on is a bad idea
1137 2013-07-22 20:34:43 <phantomcircuit> conventional hdds deal *very* poorly with that type of access
1138 2013-07-22 20:35:04 <petertodd> phantomcircuit: Hmm... so you figure the UTXO lookup + blacklist lookup is unacceptable basically? One or the other?
1139 2013-07-22 20:35:27 <phantomcircuit> petertodd, unless you enjoy the sound of *click* *click* *click*
1140 2013-07-22 20:36:17 <warren> phantomcircuit: does bitcoind relay orphan tx at all? I've seen mine accept() it but not relay it
1141 2013-07-22 20:36:55 <phantomcircuit> i dont think so but i'd have to check that one
1142 2013-07-22 20:36:56 <warren> I was creating orphan tx's on purpose to test what I thought was bug. It wasn't a bug.
1143 2013-07-22 20:37:29 <sipa> no
1144 2013-07-22 20:37:31 <Luke-Jr> warren: it can't.
1145 2013-07-22 20:37:41 <sipa> only things that enter the mempool are relayed
1146 2013-07-22 20:37:42 <Luke-Jr> warren: relaying orphan tx would make it vulnerable to DoS
1147 2013-07-22 20:37:54 <warren> ok, that's what I thought
1148 2013-07-22 20:38:14 <warren> lianj: you're seeing orphan tx not relay at all after their parents appear?
1149 2013-07-22 20:38:22 <petertodd> phantomcircuit: Hmm... well minimum transaction size is 156 currently, and absolute minimum is ~64, so a in-memory blacklist saves you 20x to 8x roughly. Worth it still I agree.
1150 2013-07-22 20:38:50 <petertodd> phantomcircuit: Probably best to kick nodes if they send us huge numbers of tx's that wind up in the blacklist instead.
1151 2013-07-22 20:38:58 <phantomcircuit> petertodd, yup
1152 2013-07-22 20:39:33 <sipa> with blacklist, you mean what i called negative mempool before?
1153 2013-07-22 20:39:38 <lianj> warren: no, once the parent makes it into a block and we rebroadcast it again it starts to show up on blockchain.ifno aswell, just not while the parent isn't confirmed yet. which is weird because this behavior is new
1154 2013-07-22 20:40:12 <sipa> lianj: sounds like exactly the intended behaviour
1155 2013-07-22 20:40:26 <sipa> lianj: but maybe b.i worked differently before, and somehow it did reach them?
1156 2013-07-22 20:40:50 <sipa> lianj: oh wait
1157 2013-07-22 20:40:58 <sipa> the parent shouldn't need to be confirmed
1158 2013-07-22 20:41:01 <sipa> just known
1159 2013-07-22 20:41:38 <petertodd> lianj: blockchain.info filters out tx's with unconfirmed parents unless the parent was sent to b.i through their /pushtx facility
1160 2013-07-22 20:41:48 <petertodd> lianj: they started doing that after the nLockTime thing
1161 2013-07-22 20:42:06 <gmaxwell> crud, looks like troublemakers have been crapping on the document
1162 2013-07-22 20:43:06 <sipa> gmaxwell: how so?
1163 2013-07-22 20:43:16 <warren> he made it public editable
1164 2013-07-22 20:43:28 <sipa> i see no vandalism?
1165 2013-07-22 20:43:33 <gmaxwell> I removed some of it.
1166 2013-07-22 20:44:29 <lianj> petertodd: thanks, thats a good datapoint, but the locktime thing was months ago. and we are starting to see it just now (1-2weeks(
1167 2013-07-22 20:45:07 <petertodd> lianj: example tx?
1168 2013-07-22 20:45:09 <TheUni> warren: that discussion seems a bit over my head as i'm not yet very familiar with bitcoind's internals. But once in, I can make android happen if that's desired
1169 2013-07-22 20:46:15 <warren> TheUni: will the new autotoolified source allow make to actually reuse objects if unchanged? currently -qt builds don't seem to.
1170 2013-07-22 20:46:20 <lianj> petertodd: hold on, let my look for a current one
1171 2013-07-22 20:46:33 <TheUni> reuse if unchanged?
1172 2013-07-22 20:46:42 <sipa> warren: we do reuse objects if unchanged
1173 2013-07-22 20:46:45 <warren> hmm
1174 2013-07-22 20:46:48 <warren> I wonder why mine isn't
1175 2013-07-22 20:46:59 <warren> bitcoind builds only what changed, but -qt doesn't
1176 2013-07-22 20:47:16 <sipa> oh, don't know about -qt
1177 2013-07-22 20:47:24 <TheUni> warren: not sure what you're asking, but to try to answer your question, yes, it tracks dependencies and only rebuilds what was touched
1178 2013-07-22 20:47:27 <sipa> i build that once a month or so
1179 2013-07-22 20:48:12 <TheUni> warren: also it detects and uses ccache. so 2nd build from clean source takes about 5sec
1180 2013-07-22 20:48:19 <warren> nice
1181 2013-07-22 20:49:02 <warren> TheUni: as soon as our auditor finishes and we release 0.8.3.x I'll help by getting my team to test the autotools PR on multiple platforms
1182 2013-07-22 20:49:16 <phantomcircuit> lianj, have you tried mapping which listening peers accept them?
1183 2013-07-22 20:49:16 <TheUni> great
1184 2013-07-22 20:49:30 <Luke-Jr> is the "Linux distribution packaging and Bitcoin" note done? time to add signatures?
1185 2013-07-22 20:49:35 <TheUni> warren: got it signed-off today, so afaik it's ready to go as soon as i squash and document it
1186 2013-07-22 20:49:41 <warren> TheUni: our devs run 3 different distros, mac and windows
1187 2013-07-22 20:49:47 <Luke-Jr> (and was Gavin's actually added by Gavin?)
1188 2013-07-22 20:49:49 <warren> TheUni: cool
1189 2013-07-22 20:49:50 <phantomcircuit> lianj, should be able to do it by abusing two connections to each listening peer, one of them sends the two transactions, the other listens to see if that peer forwards them
1190 2013-07-22 20:53:53 <petertodd> Luke-Jr: IMO it should be a text document with PGP signatures...
1191 2013-07-22 20:54:24 <Luke-Jr> petertodd: good idea; although a list of names would be nice for publishing on eg blogs and such
1192 2013-07-22 20:54:36 <petertodd> Luke-Jr: yeah, do both
1193 2013-07-22 20:54:57 <warren> bbl
1194 2013-07-22 20:55:56 <lianj> phantomcircuit: petertodd just verified again, like we guessed yesterday that it seems to be a blockchain.info thing/issue only. which is still annoying but closes the confusing
1195 2013-07-22 20:56:08 <petertodd> Luke-Jr: You can concatenate cleartext sigs FWIW: https://litecoin.org/archive/2013/litecoin-0.8.3.x-code-audit-agreement.txt.asc
1196 2013-07-22 21:08:36 <Luke-Jr> petertodd: it doesn't play too nicely with Markdown, however: https://gist.github.com/luke-jr/6058439
1197 2013-07-22 21:09:45 <petertodd> Luke-Jr: annoying
1198 2013-07-22 21:10:14 <petertodd> Luke-Jr: ugh, looooong lines... :)
1199 2013-07-22 21:10:27 <petertodd> Luke-Jr: I think it's fine then for it to just be in text like everything else then
1200 2013-07-22 21:10:39 <gmaxwell> how can we concat clearsigs? I agree this thing ought to be gpg signed.
1201 2013-07-22 21:10:40 <petertodd> Luke-Jr: make up a PDF for the "press" later
1202 2013-07-22 21:10:55 <petertodd> gmaxwell: see how it's done here: https://litecoin.org/archive/2013/litecoin-0.8.3.x-code-audit-agreement.txt.asc
1203 2013-07-22 21:11:00 <petertodd> gmaxwell: turns out it's really easy
1204 2013-07-22 21:11:09 <Luke-Jr> petertodd: markdown can work good enough for raw text
1205 2013-07-22 21:11:11 <Luke-Jr> https://gist.github.com/luke-jr/6058439/raw/0dba9c1ba8f110c8043e67bbde2d2c1c97f3db35/linux-distribution-packaging-and-bitcoin.md
1206 2013-07-22 21:11:24 <petertodd> gmaxwell: just separately clearsign the same document and concatenate them
1207 2013-07-22 21:11:39 <Luke-Jr> petertodd: btw, is this PGP-standard, or a GPG extension?
1208 2013-07-22 21:11:40 <gmaxwell> wow! coool!
1209 2013-07-22 21:12:22 <petertodd> Luke-Jr: dunno actually
1210 2013-07-22 21:12:25 <gmaxwell> Can you insert some text: e.g. Gregory Maxwell <greg@xiph.org>:
1211 2013-07-22 21:12:38 <gmaxwell> so that you can show the names with the signatures?
1212 2013-07-22 21:12:58 <Luke-Jr> http://luke.dashjr.org/tmp/code/20130722-linux-distribution-packaging-and-bitcoin.md 5d9dd4cd3787c540fcb62fb7c720105292d02d0fa8478c4ff31ae528e6aabb80
1213 2013-07-22 21:13:03 <petertodd> gmaxwell: Hmm... maybe you can have a Comment: ? I dunno, you can obvs put it in the version
1214 2013-07-22 21:13:34 <petertodd> Luke-Jr: I can sign it later tonight
1215 2013-07-22 21:15:32 <Luke-Jr> gmaxwell: seems to work: https://gist.github.com/luke-jr/6058439/raw/ab8e6561efddbc6963cf20bba5a9cccdef3a19a3/linux-distribution-packaging-and-bitcoin.md
1216 2013-07-22 21:16:27 <petertodd> Luke-Jr: I strongly recomend wrapping it to email-length lines so we can send it around in emails
1217 2013-07-22 21:16:31 <gmaxwell> sipa: are you happy with that text?
1218 2013-07-22 21:16:39 <petertodd> Luke-Jr: I think 72 is the standard for email?
1219 2013-07-22 21:17:03 <gmaxwell> yea, it should be wrapped to 75. Use fmt.
1220 2013-07-22 21:17:20 <Luke-Jr> petertodd: I don't think wrapping will make emails viable alone
1221 2013-07-22 21:17:49 <gmaxwell> Luke-Jr: no, but it shouldn't be gratitiously long.
1222 2013-07-22 21:18:02 <petertodd> Luke-Jr: point is if the line's don't get wrapped anywhere cut-n-paste generally doesn't break the signatures
1223 2013-07-22 21:18:19 <petertodd> Luke-Jr: --clearsign tolerates whitespace, but not line wrapping
1224 2013-07-22 21:18:52 <Luke-Jr> so 72 or 75?
1225 2013-07-22 21:19:02 <gmaxwell> Luke-Jr: try both?
1226 2013-07-22 21:19:12 <gmaxwell> if 72 looks okay, start that.
1227 2013-07-22 21:19:24 <Luke-Jr> I don't understand the diff between -w and -g
1228 2013-07-22 21:20:11 <Luke-Jr> http://luke.dashjr.org/tmp/code/20130722-linux-distribution-packaging-and-bitcoin.md f8fc92dd32f6ae6065bddc9e3be3718a5922de37daf514b8d49c74a322e1126e
1229 2013-07-22 21:20:13 <petertodd> Luke-Jr: one last suggestion: s/*/_/ looks better, and leads to lines being underlined in a lot of stuff
1230 2013-07-22 21:20:36 <petertodd> Luke-Jr: *'s look less professional and more obtrusive
1231 2013-07-22 21:20:47 <Luke-Jr> http://luke.dashjr.org/tmp/code/20130722-linux-distribution-packaging-and-bitcoin.md bc078c7b05a60a8c008b7b99b4599ac8110ff5b199bc9085125d1d97744d7910
1232 2013-07-22 21:21:58 <petertodd> Luke-Jr: oh, we should say July 22 2013
1233 2013-07-22 21:22:15 <Luke-Jr> then we'll all have to resign on July 23? :P
1234 2013-07-22 21:22:33 <gmaxwell> Can we drop the "hereby" from it?
1235 2013-07-22 21:22:46 <warren> henceforth
1236 2013-07-22 21:22:56 <gmaxwell> it's excessively pretending-to-be-formal, so if no one loves it it should just be removed as it adds nothing.
1237 2013-07-22 21:22:57 <Luke-Jr> "???the undersigned request that distributors???" ?
1238 2013-07-22 21:23:04 <petertodd> gmaxwell: +1
1239 2013-07-22 21:23:19 <gmaxwell> Luke-Jr: or just ask either is fine with me.
1240 2013-07-22 21:23:27 <sipa> Luke-Jr: resign or re-,sign?
1241 2013-07-22 21:23:29 <petertodd> Luke-Jr: lol, nah, I think it's fine if the sigs don't strictly match the stated time
1242 2013-07-22 21:23:47 <sipa> gmaxwell: i'll read it when i get home
1243 2013-07-22 21:23:59 <Luke-Jr> http://luke.dashjr.org/tmp/code/20130722-linux-distribution-packaging-and-bitcoin.md f815dce7aab77eb47c7b7edb26ad8cd766120421e4ab96780c2a350eb91ab5e3
1244 2013-07-22 21:24:30 <petertodd> Luke-Jr: s/2013 July 22
1245 2013-07-22 21:24:44 <petertodd> s/2013 July 22/July 22 2013/ please... we are humans :)
1246 2013-07-22 21:24:54 <Luke-Jr> petertodd: that's so backward!
1247 2013-07-22 21:24:56 <gmaxwell> bleh.
1248 2013-07-22 21:25:04 <gmaxwell> 2013-07-22
1249 2013-07-22 21:25:11 <petertodd> heh, ISO has it's place when there is ambiguity, but that's not ambiguous
1250 2013-07-22 21:25:15 <petertodd> gmaxwell: +1 problem solved
1251 2013-07-22 21:25:31 <Luke-Jr> http://luke.dashjr.org/tmp/code/20130722-linux-distribution-packaging-and-bitcoin.md 4b525d3557a310bda24f081087010e9492c5a2e6f455c76ead9a62f86a1d3ea4
1252 2013-07-22 21:26:27 <Luke-Jr> ACTION quickly adds the date in tonal ??? :p jk
1253 2013-07-22 21:26:40 <gmaxwell> not in blockheight?
1254 2013-07-22 21:26:42 <petertodd> Luke-Kr: s/packages. In/packages. In/ <- double-spaced there
1255 2013-07-22 21:26:51 <Luke-Jr> gmaxwell: that might be interesting, if the timestamp really mattered :P
1256 2013-07-22 21:27:03 <petertodd> Oh, add the most recent blockhash to prove it was created after that date
1257 2013-07-22 21:27:13 <petertodd> And then I'll timestamp it for the window!
1258 2013-07-22 21:27:19 <petertodd> ACTION geeks out
1259 2013-07-22 21:27:27 <Luke-Jr> http://luke.dashjr.org/tmp/code/20130722-linux-distribution-packaging-and-bitcoin.md a6a8d3b9c1c9db28b7ebe760fd617f1730bfb7f28ef3c1026e90cc6644a650fb
1260 2013-07-22 21:28:14 <Luke-Jr> petertodd: blockhash was serious?
1261 2013-07-22 21:28:19 <gmaxwell> we realize this is in en-gb / en-ca right?
1262 2013-07-22 21:28:29 <Luke-Jr> en-gb and en-ca are different?
1263 2013-07-22 21:28:38 <gmaxwell> I mean its not in en-us.
1264 2013-07-22 21:28:49 <Luke-Jr> good, let the English define the English language.
1265 2013-07-22 21:28:57 <Luke-Jr> <.<
1266 2013-07-22 21:29:04 <gmaxwell> Okay, fine with me. Satoshi used en-gb in a number of places.
1267 2013-07-22 21:29:07 <petertodd> Luke-Jr: semi-serious, heck, go for it
1268 2013-07-22 21:29:22 <Luke-Jr> petertodd: watch it get reorg'd..
1269 2013-07-22 21:29:26 <petertodd> Luke-Jr: lol
1270 2013-07-22 21:30:16 <gmaxwell> I'd like to at least get a signoff from jgarzik and sipa (having both expressed opinions earlier) on the final text before we consider it really final.
1271 2013-07-22 21:30:20 <Luke-Jr> http://luke.dashjr.org/tmp/code/20130722-linux-distribution-packaging-and-bitcoin.md 4d45fba38ac4af43765936e17daea8e96bf5f37992fd12ed1a01a12efe26c1a1
1272 2013-07-22 21:30:43 <petertodd> gmaxwell: Nice, Comment: Peter Todd <pete@petertodd.org> works in the signature
1273 2013-07-22 21:30:46 <Luke-Jr> and gavinandresen, since he hasn't even seen it yet
1274 2013-07-22 21:30:53 <Luke-Jr> petertodd: I said that already :P
1275 2013-07-22 21:30:58 <petertodd> oops
1276 2013-07-22 21:31:15 <gmaxwell> petertodd: I think it would be pretty awesome if we could get a whole mob of signatures after the initial noteworthy ones.
1277 2013-07-22 21:31:31 <petertodd> Luke-Jr: hmm, put the hash just under the ==== line I think, looks better that way
1278 2013-07-22 21:31:36 <petertodd> IE date second
1279 2013-07-22 21:31:42 <gmaxwell> e.g. well known experts and then any bitcoin technical folks who want to sign it.
1280 2013-07-22 21:31:48 <petertodd> (date is obvs less important :))
1281 2013-07-22 21:31:51 <petertodd> gmaxwell: for sure
1282 2013-07-22 21:32:18 <petertodd> ACTION needs to implement clearsigned timestamps in OpenTimestamps
1283 2013-07-22 21:32:31 <Luke-Jr> http://luke.dashjr.org/tmp/code/20130722-linux-distribution-packaging-and-bitcoin.md 20130722-linux-distribution-packaging-and-bitcoin.mdd898ff8654e22e648685b67705f7efdeeeefbb0dcd1efecccc7fb61864a3ddd9
1284 2013-07-22 21:32:33 <Luke-Jr> err
1285 2013-07-22 21:32:33 <petertodd> Problem is you really want to timestamp the signatures, not the document
1286 2013-07-22 21:32:43 <Luke-Jr> http://luke.dashjr.org/tmp/code/20130722-linux-distribution-packaging-and-bitcoin.md d898ff8654e22e648685b67705f7efdeeeefbb0dcd1efecccc7fb61864a3ddd9
1287 2013-07-22 21:33:03 <petertodd> Luke-Jr: Hmm, I think we can drop the "Block " thing
1288 2013-07-22 21:33:07 <petertodd> Luke-Jr: just leave the hash
1289 2013-07-22 21:33:17 <petertodd> Luke-Jr: Anyone in bitcoin would understand the reference
1290 2013-07-22 21:33:24 <Luke-Jr> http://luke.dashjr.org/tmp/code/20130722-linux-distribution-packaging-and-bitcoin.md 670f1976548b400572f14fd12ca91aa9db99745c381b0e3d49bb1e89f9c289e9
1291 2013-07-22 21:34:27 <petertodd> Luke-Jr: ha, perfect.
1292 2013-07-22 21:34:38 <petertodd> That I'm worrying about this stuff means the text is solid of course. :)
1293 2013-07-22 21:36:31 <Luke-Jr> :p
1294 2013-07-22 21:37:01 <petertodd> Hmm... extend the ='s to the end of the block hash. Or don't and tell the !@#$$ arts grad to shut the !@#$ up.
1295 2013-07-22 21:37:26 <gmaxwell> If I keep reading this I'm going to start shedpainting the text.
1296 2013-07-22 21:37:35 <gmaxwell> "and direct users to the upstream-provided binaries instead _until they
1297 2013-07-22 21:37:36 <gmaxwell> understand the"
1298 2013-07-22 21:37:37 <jgarzik> ACK 670f1976
1299 2013-07-22 21:37:41 <gmaxwell> they is ambigious.
1300 2013-07-22 21:37:43 <Luke-Jr> http://luke.dashjr.org/tmp/code/bcpatching.png look good for Bitcoin Magazine?
1301 2013-07-22 21:37:56 <gmaxwell> hah.
1302 2013-07-22 21:38:00 <petertodd> gmaxwell: I'm not feeling the shade of gray I'm seeing, can we lighten it a bit?
1303 2013-07-22 21:38:12 <petertodd> Luke-Jr: lol
1304 2013-07-22 21:39:12 <Luke-Jr> side note: the extra newline at the end is to avoid PGP messing up markdown rendering too much
1305 2013-07-22 21:39:24 <petertodd> Luke-Jr: I noticed that, good idea
1306 2013-07-22 21:41:00 <petertodd> bbl
1307 2013-07-22 21:41:13 <phantomcircuit> side side note: i enjoy oreos
1308 2013-07-22 21:41:17 <Luke-Jr> ???
1309 2013-07-22 21:41:27 <Luke-Jr> phantomcircuit: they've got some nasty chemicals in them :P
1310 2013-07-22 21:41:37 <phantomcircuit> Luke-Jr, that's what makes them so delicious
1311 2013-07-22 21:41:43 <Luke-Jr> I know :<
1312 2013-07-22 21:41:54 <gmaxwell> Luke-Jr: I've seen you drink gatoraid. :-/
1313 2013-07-22 21:41:59 <gmaxwell> No place to talk. :P
1314 2013-07-22 21:42:13 <Luke-Jr> gmaxwell: dunno what brand that stuff was, but I did check the chemicals in it!
1315 2013-07-22 21:54:43 <doublec> ahmedbodi: yes
1316 2013-07-22 22:18:33 <ahmedbodi> !seen doublec
1317 2013-07-22 22:18:33 <gribble> doublec was last seen in #bitcoin-dev 23 minutes and 49 seconds ago: <doublec> ahmedbodi: yes
1318 2013-07-22 22:39:28 <maaku> the CTxUndo does not store metadata -- version, coinbase flag, height; where is this information pulled from when the undo data is used?
1319 2013-07-22 22:39:39 <sipa> maaku: the blocks
1320 2013-07-22 22:39:46 <sipa> wait
1321 2013-07-22 22:39:56 <sipa> it does store metadata, afaik
1322 2013-07-22 22:40:08 <sipa> if it's the last output of a transaction being spent
1323 2013-07-22 22:40:44 <maaku> ack you are right
1324 2013-07-22 22:40:45 <maaku> thank you
1325 2013-07-22 22:44:44 <maaku> actually it looks like the height and coinbase flags are always stored. is there a reason for that?
1326 2013-07-22 22:44:52 <maaku> is that because of non-unique txids?
1327 2013-07-22 22:46:53 <sipa> well the fields are always present
1328 2013-07-22 22:47:02 <sipa> but afaik they are 0 and ignored for non-last outputs
1329 2013-07-22 22:47:19 <sipa> the version is optimized away if height=0
1330 2013-07-22 22:47:48 <sipa> (which means either genesis block - for which you don't care about a version - or non-last output)
1331 2013-07-22 22:48:37 <jgarzik> gmaxwell, Luke-Jr, petertodd: Actually, the folks at opensource.com have been poking me for content. It is tempting to post a version of this admonishment there, where it would reach the Fedora community.
1332 2013-07-22 22:49:29 <sipa> maaku: actually, disconnectblock even verifies whether height==0 of an output being reversed corresponds to its idea of whether it's the last one
1333 2013-07-22 22:49:56 <maaku> sipa: thank you
1334 2013-07-22 22:51:11 <gmaxwell> gavinandresen: You cool with the distribution comment message? http://luke.dashjr.org/tmp/code/20130722-linux-distribution-packaging-and-bitcoin.md
1335 2013-07-22 22:52:21 <gavinandresen> gmaxwell: yes. except what is the hex at the beginning?
1336 2013-07-22 22:52:43 <warren> gavinandresen: looks like a petertodd timestamp
1337 2013-07-22 22:52:53 <warren> oh wait, just a block hash?
1338 2013-07-22 22:53:07 <gavinandresen> gratuitous geekiness
1339 2013-07-22 22:53:16 <warren> for the entire doc to be timestamped
1340 2013-07-22 22:53:59 <gavinandresen> It has a date in it, which will be gpg signed... (date is wrong for me, by the way, it is the 23'rd here...)
1341 2013-07-22 22:54:18 <sipa> gavinandresen: yeah, the US is always a bit behind :)
1342 2013-07-22 22:55:30 <maaku> sipa: ah ok, i see. Spend() only sets the metadata if it's the last output, so for all non-final outputs the serialization of nHeight<<1+fCoinBase is 0x00
1343 2013-07-22 22:55:49 <sipa> maaku: right, indeed
1344 2013-07-22 22:56:16 <sipa> it's a weird mix of combinatios of values and conditionals
1345 2013-07-22 22:56:46 <sipa> it was originally desiged to just get an idea of how small a dataset the UTXO set could be reduced to
1346 2013-07-22 22:57:27 <sipa> so there are many of these tricks... many are really overkill and probably not worth the CPU usage
1347 2013-07-22 22:57:40 <maaku> ok that makes sense, and yes it is as compact as you can get
1348 2013-07-22 22:57:49 <sipa> oh far from it
1349 2013-07-22 22:58:10 <sipa> it's sort of the best you can do with byte-level rule-based serializations
1350 2013-07-22 22:58:23 <maaku> yes, that's what I meant ;)
1351 2013-07-22 22:58:28 <sipa> but you could define custom entropy coders for this data :p
1352 2013-07-22 22:59:50 <sipa> but for example something like not storing the number of unspentness bitvector bytes, but reducing it by one whe the first 2 outputs are not set
1353 2013-07-22 22:59:56 <Luke-Jr> gavinandresen: petertodd suggested the block hash as a timestamp; should I remove it and update the day?
1354 2013-07-22 23:00:15 <sipa> the expected effect on the UTXO set is probably less than a byte
1355 2013-07-22 23:00:41 <gavinandresen> Luke-Jr: yes
1356 2013-07-22 23:01:31 <Luke-Jr> http://luke.dashjr.org/tmp/code/20130723-linux-distribution-packaging-and-bitcoin.md 64bc8b77b58eed9c8f1ab1206f6eaac2af164338a67fb597807105f9a4010381
1357 2013-07-22 23:01:58 <gmaxwell> sipa: you okay with the text now too?
1358 2013-07-22 23:02:01 <sipa> gmaxwell: yes
1359 2013-07-22 23:02:15 <sipa> nit: use [1] instead of (1) for the references?
1360 2013-07-22 23:02:35 <sipa> or does that interact badly with markdown formatting?
1361 2013-07-22 23:04:14 <sipa> i don't understand 'in addition to
1362 2013-07-22 23:04:15 <sipa> ensuring that security-critical fixes can be applied once'
1363 2013-07-22 23:04:35 <sipa> do you mean "such as ensuring ..."
1364 2013-07-22 23:05:04 <gmaxwell> sipa: agreed, artifact of editing down from a longer version.
1365 2013-07-22 23:05:10 <sipa> now it sounds as if that ensuring is not part of "variety of good reasons"
1366 2013-07-22 23:05:28 <gmaxwell> (there was a longer list there at one point)
1367 2013-07-22 23:05:35 <maaku> square braces are standard reference format, IIRC (would be square brace at the bottom too)
1368 2013-07-22 23:05:40 <maaku> for markdown
1369 2013-07-22 23:05:58 <sipa> in stack exchange, that would cause the links to be inlined even
1370 2013-07-22 23:06:11 <sipa> not sure if that is standard MD
1371 2013-07-22 23:16:08 <petertodd> s/in addition to/including/
1372 2013-07-22 23:17:38 <petertodd> warren, Luke-Jr: in the literature that's called a random beacon FWIW; proves something was created after a date, although frankly it's mostly geekyness - what's usful re: signatures is proving they were created before some date so if the private keys are compromised/expired you can still validate them
1373 2013-07-22 23:18:18 <petertodd> warren, Luke-Jr: I have a script to put blockhashes in my email sigs for fun
1374 2013-07-22 23:19:06 <petertodd> warren, Luke-Jr: When you think about it, without a proper timestamp a random beacon doesn't prove anything at all.
1375 2013-07-22 23:19:44 <gmaxwell> If only someone ran a globally visible random beacon service that carries validated timestamps. :P
1376 2013-07-22 23:20:26 <petertodd> gmaxwell: I know right? Preferably decentralized, and open
1377 2013-07-22 23:20:51 <gmaxwell> but s'not like we need one for this. :P
1378 2013-07-22 23:23:08 <petertodd> Careful, I'll suggest we put it into the blockchain...
1379 2013-07-22 23:23:45 <Luke-Jr> square brackets on refs then? but that breaks github markdown for the footnotes.. I think, need to check
1380 2013-07-22 23:25:17 <petertodd> https://bitcointalk.org/index.php?topic=260898.0 <- someone
1381 2013-07-22 23:25:25 <petertodd> someone's implementing mike's oracles apparently...
1382 2013-07-22 23:25:56 <gmaxwell> Luke-Jr: also the other copyedit from sipa.
1383 2013-07-22 23:26:04 <petertodd> doesn't understand that sequence numbers are useless - gotta update the wiki on that
1384 2013-07-22 23:28:48 <sipa> an earlier version had testing instructions
1385 2013-07-22 23:28:57 <sipa> perhaps they don't belong in this text
1386 2013-07-22 23:29:09 <sipa> but having them listed in some document in our source tree probably makes sense
1387 2013-07-22 23:30:08 <gmaxwell> sipa: I started writing one for the tree.
1388 2013-07-22 23:30:30 <gmaxwell> Still needs more editing. But I agree.
1389 2013-07-22 23:30:31 <gmaxwell> https://docs.google.com/document/d/1b1WKdvUl8K8mdNA9yTx2ATg97tpWPEL8NUpWRTob5Tw/edit
1390 2013-07-22 23:32:15 <Luke-Jr> what other copyedit from sipa? :/
1391 2013-07-22 23:33:52 <sipa> 02:59:21 < sipa> i don't understand 'in addition to
1392 2013-07-22 23:33:52 <sipa> 02:59:22 < sipa> ensuring that security-critical fixes can be applied once'
1393 2013-07-22 23:33:52 <sipa> 02:59:42 < sipa> do you mean "such as ensuring ..."
1394 2013-07-22 23:33:53 <gmaxwell> 17:57 < sipa> nit: use [1] instead of (1) for the references?
1395 2013-07-22 23:34:00 <gmaxwell> if it's not bad
1396 2013-07-22 23:34:04 <gmaxwell> and 18:11 < petertodd> s/in addition to/including/
1397 2013-07-22 23:34:11 <gmaxwell> oops
1398 2013-07-22 23:34:50 <Luke-Jr> square brackets at the end break github's markdown: https://gist.github.com/luke-jr/6058439
1399 2013-07-22 23:35:25 <Luke-Jr> well, sortof
1400 2013-07-22 23:35:29 <Luke-Jr> it's not unreadable either
1401 2013-07-22 23:36:00 <sipa> you may need to indent them
1402 2013-07-22 23:36:33 <Luke-Jr> that makes it think it's code
1403 2013-07-22 23:36:56 <Luke-Jr> err
1404 2013-07-22 23:36:56 <sipa> ACTION .sleep(21600);
1405 2013-07-22 23:37:12 <Luke-Jr> yeah
1406 2013-07-22 23:37:37 <Luke-Jr> http://luke.dashjr.org/tmp/code/20130723-linux-distribution-packaging-and-bitcoin.md a3e5297322b2fb302a4fb1a112a80bb275b0457adc5f1849cb78e0006ebcdf7d
1407 2013-07-22 23:40:10 <gmaxwell> sipa: can we get a gpg --clearsign on that file from you?
1408 2013-07-22 23:40:59 <gmaxwell> I wanted to do the gpg multisig thing. I think thats neat, and having multiple names on it helps make it clear that its not just a single premadonna complaining that someone might move his cheese.
1409 2013-07-22 23:42:36 <SomeoneWeird> i can sign it! but that won't mean anything :P
1410 2013-07-22 23:43:58 <petertodd> gmaxwell: one catch re: multisig is you'll need to have multiple Hash: lines for every algorithm people use, just so you know
1411 2013-07-22 23:44:13 <petertodd> gmaxwell: probably just SHA1 and SHA256 of course, no big deal
1412 2013-07-22 23:44:25 <SomeoneWeird> ACTION is using DES
1413 2013-07-22 23:44:35 <petertodd> weird
1414 2013-07-22 23:44:37 <petertodd> :P
1415 2013-07-22 23:46:31 <gmaxwell> petertodd: https://people.xiph.org/~greg/try.txt < try to compose with me to see how that works.
1416 2013-07-22 23:47:50 <gmaxwell> the comment trick is neat, though really we could leave the names off, if you care who's signing it you could validate the signatures. :P
1417 2013-07-22 23:48:12 <petertodd> gmaxwell: ha
1418 2013-07-22 23:48:28 <petertodd> gmaxwell: gimmie 40 minutes, just about to leave work, don't have PGP keys here
1419 2013-07-22 23:49:10 <gmaxwell> fair enough, looks like sipa expired so if it'll be tomorrow I think. No biggie, gives some time to work on testing instructions to stick into git.