1 2014-08-15 00:18:24 <Luke-Jr> dgenr8: yes, all mining code in Bitcoin Core is example code. sadly, many pools run it as-is
2 2014-08-15 00:19:59 <Luke-Jr> anyone know why BC Foundation has a committee telling people it's a best practice to REFUSE giving refunds of bitcoin purchases? :/
3 2014-08-15 00:27:43 <BlueMatt> Luke-Jr: from address confusion?
4 2014-08-15 00:29:31 <Luke-Jr> BlueMatt: ?
5 2014-08-15 00:29:51 <BlueMatt> Luke-Jr: ie they're confused about from address not existing and didnt think it through?
6 2014-08-15 00:29:59 <Luke-Jr> it's not hard to ask for a refund address
7 2014-08-15 00:30:04 <Luke-Jr> and payment protocol has it builtin
8 2014-08-15 00:30:34 <dgenr8> Luke-Jr: ah, so in your view CPFP will affect a significant fraction
9 2014-08-15 00:31:29 <Luke-Jr> dgenr8: yes. merging it also means it no longer needs to be maintained out of master, which creates merge conflicts when master incorporates other changes
10 2014-08-15 00:31:54 <dgenr8> Luke-Jr: cuz if it were just example code ... it would be getting more complex and not getting tested much out there
11 2014-08-15 00:33:59 <dgenr8> Luke-Jr: yes, that's why i rebased for you this week :)
12 2014-08-15 00:35:07 <Luke-Jr> dgenr8: thanks
13 2014-08-15 00:35:40 <Luke-Jr> lately I wait to rebase until someone expresses a readiness to merge XD
14 2014-08-15 06:15:28 <wumpus> cfields: sorry for the slowness - that last one works great!
15 2014-08-15 06:59:37 <Luke-Jr> ACTION pokes lechuga_ and dansmith_btc
16 2014-08-15 08:31:00 <dansmith_btc> ACTION pokes Luke-Jr back
17 2014-08-15 09:55:31 <Luke-Jr> aww, no way to build bitcoin-tx without bitcoin-cli? :p
18 2014-08-15 09:57:38 <wumpus> no :)
19 2014-08-15 09:58:31 <Luke-Jr> â¹
20 2014-08-15 09:58:52 <Luke-Jr> guess I have to wait for the new PR before I can figure out how to write the packages for them
21 2014-08-15 09:58:53 <wumpus> and --with-utils will get you future bitcoin utilities as well, if for example one is added for deterministic key generation
22 2014-08-15 09:59:21 <Luke-Jr> wumpus: sounds like the enemy of modularity :p
23 2014-08-15 09:59:34 <Luke-Jr> I suppose I can 'make src/bitcoin-tx' and do the install step by hand
24 2014-08-15 09:59:46 <wumpus> it's different executables -- how much more modular can you get?
25 2014-08-15 09:59:53 <Luke-Jr> wumpus: different git repositories
26 2014-08-15 10:00:00 <Luke-Jr> as it should be <.<
27 2014-08-15 10:00:09 <wumpus> well, people differ in opinion strongly about that
28 2014-08-15 10:00:57 <wumpus> I'd also prefer that though
29 2014-08-15 10:01:07 <gmaxwell> we're a long way from bitcoin-tx being well seperated yet (go look at the size of the binary)
30 2014-08-15 10:02:02 <wumpus> then one parent repository that pulls everything and builds everything, for people that just want everything and want it to work... it can be frustrating to have to check out zillions of repos just to get something to work
31 2014-08-15 10:02:08 <bitnumus> Hi all, i've had a few freezes whilst trying to sync on two different machines, just wondering if i can report what it may be ?
32 2014-08-15 10:02:14 <Luke-Jr> gmaxwell: perhaps, but having it closely linked to configure options doesn't help the situation
33 2014-08-15 10:02:17 <bitnumus> last log is (2014-08-15 09:50:53 ResendWalletTransactions())
34 2014-08-15 10:02:33 <Luke-Jr> wumpus: yes, that's ideal for our current situation (not having shared libraries) IMO
35 2014-08-15 10:03:07 <Luke-Jr> btw, someone submitted a base58 encode in C for BFGMiner - so maybe I should take that and libblkmaker's base58 decoder and make a libbase58
36 2014-08-15 10:03:18 <wumpus> well the shared library is nice for the script interpreter, something that we really want to export to other projects
37 2014-08-15 10:03:47 <wumpus> and for a future library that wraps more of the consensus code
38 2014-08-15 10:05:02 <wumpus> bitnumus: does it still reply to RPC commands?
39 2014-08-15 10:05:36 <bitnumus> sorry just restarted :S
40 2014-08-15 10:05:46 <gmaxwell> bitnumus: how are you defining 'freezes'?
41 2014-08-15 10:05:59 <bitnumus> just stopped syncing
42 2014-08-15 10:06:07 <wumpus> that's normal
43 2014-08-15 10:06:27 <gmaxwell> bitnumus: it would have continued after the next block on the network.
44 2014-08-15 10:06:35 <bitnumus> http://pastebin.com/viKcGq1C
45 2014-08-15 10:06:47 <bitnumus> huh?
46 2014-08-15 10:07:00 <bitnumus> it was 8days behind, catching up from 3 weeks
47 2014-08-15 10:07:11 <bitnumus> stopped for like 10-15minutes
48 2014-08-15 10:07:28 <wumpus> that log looks perfectly normal
49 2014-08-15 10:07:29 <Luke-Jr> bitnumus: Bitcoin in general is still very much a work-in-progress, and syncing is especially delicate. It's normal that it will stop and then start again after a while.
50 2014-08-15 10:07:57 <gmaxwell> bitnumus: if the peer you're pulling from goes unresponsive or disconnects it'll idle until the next block on the network.
51 2014-08-15 10:08:01 <bitnumus> yes i know it looks normal which is why i thought it was odd
52 2014-08-15 10:08:03 <Luke-Jr> bitnumus: fixing that is on the agenda, but needs work and testing
53 2014-08-15 10:08:19 <bitnumus> gmaxwell, i see, thats odd
54 2014-08-15 10:08:44 <bitnumus> maybe write this fact to debug.log ?
55 2014-08-15 10:08:56 <gmaxwell> bitnumus: it's just an artifact of how it works. It just doesn't know that there are more blocks waiting if the peer providing them goes away.
56 2014-08-15 10:09:13 <gmaxwell> bitnumus: what would it write? for all the sync process knows its done.
57 2014-08-15 10:09:21 <jgarzik> wumpus, ACK --with-utils
58 2014-08-15 10:09:26 <bitnumus> hmm, so i was only connected to a single peer, how likely is that ?
59 2014-08-15 10:09:36 <Luke-Jr> bitnumus: it only uses a single peer, ever
60 2014-08-15 10:09:38 <jgarzik> wumpus, I won't have time until next week, so feel free to supercede mine, if you want to PR before then.
61 2014-08-15 10:10:39 <bitnumus> This seems like quite a large reason for long sync times if client needs to wait for a block everytime the single peer its connected to disappears ?
62 2014-08-15 10:11:08 <wumpus> jgarzik: no problem, I'll do it then
63 2014-08-15 10:11:16 <wumpus> bitnumus: it is
64 2014-08-15 10:11:56 <wumpus> ideally you would just let a node running and don't babysit or wait for it too much
65 2014-08-15 10:12:48 <wumpus> as that can be frustrating :)
66 2014-08-15 10:12:59 <bitnumus> call me old fashioned but i still use core client primarily for my coins :)
67 2014-08-15 10:13:17 <gmaxwell> bitnumus: sure but this doesn't have anything to do with the initial sync process.
68 2014-08-15 10:13:20 <bitnumus> its worked for years, don't see a reason to change generally it doesn't take this long to catch up a few weeks
69 2014-08-15 10:13:42 <gmaxwell> ah, usually doesn't get stalled when catching up only a few weeks.
70 2014-08-15 10:13:49 <bitnumus> gmaxwell, its not initial though a largish amount, i just moved house and only fired machine back up today
71 2014-08-15 10:13:59 <bitnumus> "3weeks"
72 2014-08-15 10:13:59 <gmaxwell> in any case, this will hopefully be fixed in 0.10.
73 2014-08-15 10:14:16 <bitnumus> kk, keep up the good work as always :)
74 2014-08-15 10:14:30 <gmaxwell> it just takes a lot of testing to completely replace the synchronization process. That work has basically been going on for a year.
75 2014-08-15 10:15:13 <wumpus> maybe the client should have a 'tilt' switch that tries to poll for the next block manually :p
76 2014-08-15 10:15:14 <bitnumus> i can imagine
77 2014-08-15 10:17:31 <wumpus> or at least, switch sync nodes
78 2014-08-15 10:18:24 <gmaxwell> wumpus: mooted by headers first
79 2014-08-15 10:18:29 <wumpus> 'reset networking'
80 2014-08-15 10:18:54 <jgarzik> RPC "kicksync"
81 2014-08-15 10:18:56 <wumpus> gmaxwell: well, I'm sure even then things can get stuck or in unexpected situations
82 2014-08-15 10:19:01 <wumpus> jgarzik: :D
83 2014-08-15 10:19:24 <Luke-Jr> just make it an undocumented feature of clicking the sync icon in the bottom of the iwndow
84 2014-08-15 10:19:39 <Luke-Jr> then we'll have people advising clicking it 5 years after we remove the code for it
85 2014-08-15 10:19:42 <gmaxwell> wumpus: hm, no by design there shouldn't be any such case.
86 2014-08-15 10:19:47 <wumpus> Luke-Jr: it may or may not actually do something, like the close button in a elevator :)
87 2014-08-15 10:20:17 <Luke-Jr> ACTION wonders when people stopped suggesting -rescan (if ever)
88 2014-08-15 10:20:31 <gmaxwell> well okay, that was too expansive... it might get stuck with _all_ bad peers.
89 2014-08-15 10:20:51 <gmaxwell> Luke-Jr: they still constantly suggest it.
90 2014-08-15 10:21:06 <Luke-Jr> >_<
91 2014-08-15 10:21:14 <gmaxwell> we should probably rename it in the next version, hide it, and replace it with a "if you really need this, please file a bug"
92 2014-08-15 10:21:33 <gmaxwell> because any straggling reasons to rescan are going to turn into serious bugs when people have enabled pruning.
93 2014-08-15 10:21:56 <wumpus> like importing private keys...
94 2014-08-15 10:22:15 <gmaxwell> wumpus: you don't need -rescan when you import a private key though.
95 2014-08-15 10:22:34 <gmaxwell> (I mean you don't need the startup commandline switch)
96 2014-08-15 10:22:44 <wumpus> gmaxwell: well it does the same internaly, or do you just suggest removing the switch?
97 2014-08-15 10:22:46 <wumpus> oh, right
98 2014-08-15 10:22:49 <gmaxwell> Just the switch.
99 2014-08-15 10:23:56 <gmaxwell> There are some cases where rescans are used, sure. But the startup one _I think_ is just wasting people's time when they erroniously think it will solve some problem. And if there really are cases where it does something helpful we need to find out about them, so we can either fix them (if fixable) or add them to the things you can't do with pruning enabled.
100 2014-08-15 10:24:03 <Luke-Jr> btw, considering everyone who doesn't know how to use it correctly anyway is already rebasing/patching in the "address index", is there any (other) harm is just merging it? :/
101 2014-08-15 10:24:11 <wumpus> if -rescan goes away there's always -zapwallettxes
102 2014-08-15 10:24:13 <gmaxwell> Luke-Jr: it's broken
103 2014-08-15 10:24:44 <gmaxwell> wumpus: I _think_ zapwallettxes is less likely to be mistaken for something people should run at random times.
104 2014-08-15 10:24:50 <Luke-Jr> oh
105 2014-08-15 10:25:05 <gmaxwell> Luke-Jr: I mean the patch runs, but it misses some (many?) transactions.
106 2014-08-15 10:25:21 <wumpus> Luke-Jr: I am still convinced that that should not be core functionality
107 2014-08-15 10:25:54 <Luke-Jr> wumpus: it shouldn't, but (if it wasn't broken) everyone is doing it wrong anyway
108 2014-08-15 10:25:56 <wumpus> "let's just merge it" and we'll have even more feature creep, why not merge any kind of index everyone may ever need when we're at it
109 2014-08-15 10:26:02 <Luke-Jr> heh
110 2014-08-15 10:26:10 <wumpus> why not add a custom interpreter to define indexes
111 2014-08-15 10:26:20 <Luke-Jr> that might actually be interestingish
112 2014-08-15 10:26:24 <wumpus> why not change it to a block chain forensic framework?
113 2014-08-15 10:26:26 <gmaxwell> wumpus: dunno if you looked into it but addrindex is not really an address index.
114 2014-08-15 10:26:28 <wumpus> oh wait, those already exist
115 2014-08-15 10:26:42 <gmaxwell> wumpus: they're useless trash usually.
116 2014-08-15 10:26:59 <wumpus> sigh, that's no reason for us to do it thouh
117 2014-08-15 10:27:03 <gmaxwell> the addrindex it indexes every push > 64 bytes long, so it can do arbritary lookups.
118 2014-08-15 10:27:29 <gmaxwell> wumpus: the really extensive dependance on almost-universally-wrong centeralized block explorers is a reason.
119 2014-08-15 10:27:36 <Luke-Jr> wumpus: well, it would make multiwallet support easier I think
120 2014-08-15 10:27:37 <wumpus> oh it's broken, and someone needs it! so let's make it part of bitcoind... like picking up all stray kittens and letting them live in your house
121 2014-08-15 10:27:52 <Luke-Jr> s/easier/not take forever to load wallets/
122 2014-08-15 10:28:28 <gmaxwell> wumpus: it isn't just that, its a hard thing to do well because its integrated with the state of the network. The implementation in bitcoind has only 1.5gb of additional overhead over the txindex because we already have the blocks in any case too.
123 2014-08-15 10:28:46 <Luke-Jr> ACTION wonders how big Electrum's index of it is
124 2014-08-15 10:29:21 <wumpus> yes, it's a hard thing to do well, I hope someone will pick it up as a seperate project and specialize in it
125 2014-08-15 10:29:56 <wumpus> picking up extra things that are hard to do well makes bitcoind very hard to maintain well
126 2014-08-15 10:29:59 <gmaxwell> Its much easier to do well inside bitcoind, fundimentally so because it's mostly using data an unpruned full node already has.
127 2014-08-15 10:30:12 <wumpus> there are many things that are easier to do in bitcoind
128 2014-08-15 10:30:14 <gmaxwell> It's also relatively well isolated.
129 2014-08-15 10:30:30 <wumpus> I just don't want to pick up all that extra code
130 2014-08-15 10:30:44 <wumpus> I'm fine with an extension mechanism, I've even suggested one in some pull request
131 2014-08-15 10:30:52 <dsnrk> you can't do much worse than Insight's address based index.
132 2014-08-15 10:30:54 <wumpus> but as it is right now, I don't want to merge it
133 2014-08-15 10:31:08 <gmaxwell> I don't think it should be merged now in any cas.e
134 2014-08-15 10:31:25 <gmaxwell> Its currently buggy, and I think we must not merge it until pruning is a widely deployed thing.
135 2014-08-15 10:31:38 <wumpus> I'd be fine with a small API in which bitcoind can load modules that do in-process processing, if there is really a high demand for that
136 2014-08-15 10:32:00 <wumpus> though you'd say that handling things in external processes is better for separation and sanity
137 2014-08-15 10:32:22 <gmaxwell> wumpus: I would not really be excited to peper the code with function-pointer distpatches all over the place.
138 2014-08-15 10:32:32 <gmaxwell> right.
139 2014-08-15 10:32:35 <wumpus> gmaxwell: well, it's the one or the other
140 2014-08-15 10:33:05 <gmaxwell> No, it's not. "search for transactions with pushes matching this pattern" is a pretty general functionality which could be provided internally.
141 2014-08-15 10:33:09 <wumpus> I'm not really excited to peper the bitcoin core source with all kinds of indexes that are not required for functioning as a node
142 2014-08-15 10:33:10 <Luke-Jr> {"params": ["1D8P57AQunnGNSHGGbEMyRcakdfnnHCv5W"], "id": 7, "method": "blockchain.address.subscribe"}
143 2014-08-15 10:33:12 <Luke-Jr> wtf, seriously?
144 2014-08-15 10:33:57 <wumpus> the point of it is not trying to be everything, the point is to have basic infastructure
145 2014-08-15 10:35:23 <dsnrk> Luke-Jr: that's from Electrum yes?
146 2014-08-15 10:35:31 <Luke-Jr> dsnrk: yes
147 2014-08-15 10:35:42 <Luke-Jr> dsnrk: I was hoping maybe I could abuse their servers as a scriptPubKey index
148 2014-08-15 10:35:53 <Luke-Jr> but of course they don't do it sanely
149 2014-08-15 10:35:55 <gmaxwell> wumpus: I am also not interested in peppering with all kinds of indexes which are not required. Otoh, this is a pretty general thing, is an ~700 line patch, and could be even more isolated than it is currently.
150 2014-08-15 10:36:22 <dsnrk> Luke-Jr: no, there's not a lot sane in electrum-server. did you notice the IRC based server discovery?
151 2014-08-15 10:36:32 <Luke-Jr> dsnrk: I think I did before
152 2014-08-15 10:36:37 <gmaxwell> wumpus: e.g. there is a middle ground between some kind of dynamically loadable plugin whatever and by omission forcing people to continue to use crap centeralized services.
153 2014-08-15 10:37:28 <gmaxwell> wumpus: and I include with this the belief that its also not a priority, so it should not come before other considerations, and right now there are a lot of those.
154 2014-08-15 10:37:34 <dsnrk> Luke-Jr: mm, lots of questionable choices. Insight makes the same sort of indexes but it's dog slow. don't know what they're doing, but they're doing it wrong.
155 2014-08-15 10:37:58 <wumpus> gmaxwell: sure, the dynamically loadable plugin thing was just a wild idea, I'd also be much more happy if it could be done as an external process
156 2014-08-15 10:38:41 <dsnrk> Luke-Jr: for some reason requests to it scale linearly with the number of transactions, so if you have an address like Satoshi Dice you pretty much kill the interface.
157 2014-08-15 10:38:51 <gmaxwell> dsnrk: any of these things need to be indexing hundreds of millions of records. It actually requires fairly clever work to make that perform well.
158 2014-08-15 10:38:59 <wumpus> gmaxwell: but in general a way in which people can define plugins that can piggy-back on bitcoind's capabilities wouuld be nice
159 2014-08-15 10:39:17 <wumpus> address indexes is just one thing
160 2014-08-15 10:39:41 <Luke-Jr> gmaxwell: meh, it's not really more work than indexing transaction ids :p
161 2014-08-15 10:41:00 <gmaxwell> Luke-Jr: txids is 44m records, txout indexing is about 5 times that, and double again to index txins to catch where txouts were spent. If you store just the txid for each of these, with no overhead it's 14 gigabytes of data.
162 2014-08-15 10:41:16 <wumpus> if currently something like insight needs huge redundant indexes, then maybe we should think how we can help them reduce that, and in general make it easier to do that kind of things
163 2014-08-15 10:42:46 <wumpus> (without having to add lots of code to bitcoind)
164 2014-08-15 10:43:08 <gmaxwell> wumpus: sipa's implementation uses a lossy index and then postprocesses the results to remove the false hits. So this requires you to have fast access to the blocks. Json serializing over several gigabytes of data to throw out most of it doesn't make for a fast query.
165 2014-08-15 10:43:32 <dsnrk> wumpus: for Insight thats ~42GB at the moment.
166 2014-08-15 10:43:32 <wumpus> gmaxwell: json serializing is just one option
167 2014-08-15 10:43:44 <gmaxwell> e.g. [push you're looking for] gets reduced to a 64 bit query key, and that tells you the list of blocks you need to scan.
168 2014-08-15 10:43:53 <gmaxwell> (IIRC)
169 2014-08-15 10:45:37 <gmaxwell> dsnrk: better than that postgres full node thing electrum used to be based on that took >350GBytes two years ago. :P
170 2014-08-15 10:45:53 <dsnrk> Bitcoin-ABE
171 2014-08-15 10:45:57 <wumpus> gmaxwell: but say that specific case is merged; people may come with compeltely reasonable arguments to merge yet another type of index, and another one, until bitcoind is insight :p
172 2014-08-15 10:46:11 <wumpus> I'd really like a more general solution here
173 2014-08-15 10:46:45 <gmaxwell> wumpus: well that particular one is pretty generic, in that it indexes every object pushed in a script pub key over some size threshold (needed to prevent index pollution).
174 2014-08-15 10:46:49 <Jouke> I am running bitcoin-abe, and it has grown to 125GB allready :o
175 2014-08-15 10:46:59 <gmaxwell> Jouke: it used to be much worse.
176 2014-08-15 10:47:34 <gmaxwell> wumpus: but yes, I agree with the general point you're making. Whatever is done there should be pretty modular.
177 2014-08-15 10:47:41 <wumpus> gmaxwell: yes it is completely reasonable, and sounds good, I give up :)
178 2014-08-15 10:48:01 <gmaxwell> But it also must have very high performance to be usable, and integrate with chain reorginzation and so on.
179 2014-08-15 10:48:21 <wumpus> yes it needs entry point for all of those
180 2014-08-15 10:48:46 <wumpus> that's my point really... we could define an API that gives all those events, and people could index in any way they think of
181 2014-08-15 10:49:25 <gmaxwell> the obvious thing to do is that the addrindex patch could just be refactored around to basically create that interface, internally at least. Source modularity is a good step even if its not some external plugin thing.
182 2014-08-15 10:49:30 <wumpus> could even send it in binary serialized format through shared memory, I believe ZMQ (but not the lossy pubsub) may come in handy there again
183 2014-08-15 10:59:17 <wumpus> not only for indexing, there would be tons of other use cases for a high-performance API that queries bitcoind's databases (while bitcoind is running) and can subscribe to certain events
184 2014-08-15 11:00:45 <wumpus> it would reduce the cliff between whatever is merged into bitcoind and what is not, so we have less difficult (or even policitical) decisions
185 2014-08-15 11:01:32 <wumpus> there is the enormous pressure to keep bitcoind small and maintainable on one hand and all kind of adventurous ideas people would like to do in the other hand
186 2014-08-15 11:14:09 <wumpus> another part that desparately needs to be modularized is mining/mempool policy
187 2014-08-15 11:28:25 <Luke-Jr> wumpus: the transaction selection code should be considered merely example code also
188 2014-08-15 11:31:55 <hearn> bitcoin works best when all nodes agree on everything. i don't think "policy" is something we should encourage: the desire to have it should normally reflect some bug in inadequacy somewhere else in the system
189 2014-08-15 11:33:00 <Luke-Jr> bitcoin doesn't work at all when all nodes agree on everything
190 2014-08-15 11:33:00 <wumpus> Luke-Jr: what would be the preferred way to do it, then? not use getblocktemplate?
191 2014-08-15 11:33:06 <Luke-Jr> then you just have a centralised digital currency
192 2014-08-15 11:33:14 <wumpus> hearn: nodes only have to agree on the consensus rules
193 2014-08-15 11:33:17 <Luke-Jr> wumpus: not use mainline bitcoind for mining
194 2014-08-15 11:33:25 <wumpus> hearn: the rest is 'policy'
195 2014-08-15 11:33:50 <wumpus> Luke-Jr: ok, but if bitcoind had the approriate API to be able to make those decisions externally they could just use mainline bitcoind
196 2014-08-15 11:34:05 <hearn> Luke-Jr: eh? no. all you need for decentralisation is that people *can* disagree, not that they actually must always do so
197 2014-08-15 11:34:08 <Luke-Jr> wumpus: sure, it's just a different way of applying a patch
198 2014-08-15 11:34:26 <wumpus> Luke-Jr: the only reason they have to use a forked bitcoind in the first place is because there isn't one, everything is done in one big monolithic process
199 2014-08-15 11:34:28 <Luke-Jr> hearn: if they don't, then the person making the decision for them is de facto in charge
200 2014-08-15 11:35:04 <hearn> wumpus: in practice, nothing stops miners with divergent policies fighting over the block chain as well and calling it policy. the mempool vs block chain difference is something of a distraction there. as there's no sane way to explain what blocks or confirmations are to non-expert users, in practice we must have as much consistency over "policy" as we possibly can.
201 2014-08-15 11:35:06 <wumpus> Luke-Jr: no, it's not a different way of applying a patch, a well-defined isolated API means much less porting work from version to version
202 2014-08-15 11:35:31 <hearn> Luke-Jr: no, not at all. there are lots of rules in bitcoin that everyone agrees to because ..... they feel it's worth agreeing. not because someone told them to.
203 2014-08-15 11:35:36 <hearn> Luke-Jr: consider the 21 million coin limit
204 2014-08-15 11:36:00 <wumpus> hearn: that's why there should be an API for decisions that don't involve consensus, it creates a clear barrier between what shouldn't be varied and what could be
205 2014-08-15 11:36:03 <hearn> that is "policy" and miners could choose to ignore it. but we find it better and more profitable to agree.
206 2014-08-15 11:36:27 <jgarzik> People agree on that stuff mostly because it is an arbitrary number to which you agree, to join the system.
207 2014-08-15 11:36:33 <wumpus> hearn: right now miners generally to run their own fork of bitcoind, which means they are slower to update, and also that it's easier to make a mistake that DOES affect consensus rules
208 2014-08-15 11:36:45 <jgarzik> 21M, 10 min, nobody gave deep significance to the actual number (versus 22M, 9 min, ...)
209 2014-08-15 11:36:56 <hearn> wumpus: seems like most miners don't have much in the way of policy today. certainly we should encourage them to have less, and fix any problems that result in people wanting divergence.
210 2014-08-15 11:37:19 <hearn> e.g. fees were being manually set, gavin is trying to move everyone towards floating fees
211 2014-08-15 11:37:23 <wumpus> hearn: 'we' don't need to fix everything. I want 'us' out as bottleneck. we could be hit by a bus every day
212 2014-08-15 11:37:33 <jgarzik> I don't think developers alone should be determining by fiat such existential questions.
213 2014-08-15 11:37:59 <hearn> wumpus: sure but for as long as the buses stay away, "we" (the greater "we") can fix things.
214 2014-08-15 11:38:13 <hearn> ideally for any given question around what bitcoin accepts, there should be one obviously best answer
215 2014-08-15 11:38:23 <hearn> it's true that we don't always have that today, but imho that should be the goal
216 2014-08-15 11:38:52 <wumpus> I don't share your idealism in that
217 2014-08-15 11:39:16 <wumpus> what is best for you doesn't need to be best for me, certainly not on a larger scale... of course it's great if everyone agrees, but you can't force that
218 2014-08-15 11:39:28 <hearn> otherwise you end up with (worst case) miners forking each others chains and (best case) inability to judge the value of an unconfirmed transaction. but as end users will struggle to handle this notion of some unconfirmed transactions being "riskier" than others depending on unknowable factors, that'd make bitcoin a lot harder to use
219 2014-08-15 11:39:53 <wumpus> anyhow, this drifts off point, I didn't mean to start a philosophical discussion
220 2014-08-15 11:40:45 <jgarzik> Resist planning too far ahead. Better to be informed by current problems, current events, current uses and users.
221 2014-08-15 11:41:19 <hearn> well, we do currently have problems caused by miner policy :( as was pointed out a bunch of times, people use eligius' divergent policy to double spend
222 2014-08-15 11:41:26 <Luke-Jr> hearn: FUD
223 2014-08-15 11:41:36 <hearn> same thing for the fee drop which in retrospect should have been done using some kind of coinbase flag to get consensus
224 2014-08-15 11:41:41 <hearn> (or whatever mechanism)
225 2014-08-15 11:42:08 <Luke-Jr> fee is policy, no consensus needed
226 2014-08-15 11:42:09 <hearn> Luke-Jr: i think we went over this before, a few months ago, no? there were people double spending against dice addresses exploiting the fact that eligius drops them from the mempool and other pools don't
227 2014-08-15 11:42:27 <Luke-Jr> hearn: irrelevant, they could have been double spending without that, with better performance
228 2014-08-15 11:42:36 <Luke-Jr> because they were double spending UNCONFIRMED transactions
229 2014-08-15 11:43:33 <hearn> yes, well, we've been around that one before. the notion of confirmed vs unconfirmed mattering has to be shrunk as much as possible for usability reasons. otherwise lots of basic tasks people want to do just become impractical.
230 2014-08-15 11:43:42 <hearn> so fixing them one by one is the way to go
231 2014-08-15 11:44:12 <Luke-Jr> spreading FUD that's been shown false does not help anything.
232 2014-08-15 11:44:25 <hearn> and yes people can/could double spend thanks to the fee drop and some miners not upgrading. floating fees should fix this in the long run, as long as people watch out for transactions on the edge
233 2014-08-15 11:44:42 <hearn> (still lots of open questions about how this will work in practice though)
234 2014-08-15 11:45:07 <hearn> (given that most people will want to pay the lowest fee possible so will always be on the edge of what actually "works" for some flexible definition of "work")
235 2014-08-15 11:48:25 <jgarzik> Having instant, secure transactions is possible without nutty schemes that attempt to synchronize mempools or miner policy.
236 2014-08-15 11:48:33 <jgarzik> And without centralized services.
237 2014-08-15 11:53:34 <hearn> bitcoin *is* an attempt to synchronise everything together. that's the core of what it does. it's hardly nutty.
238 2014-08-15 11:58:02 <jgarzik> "that attempt to synchronize mempools or miner policy."
239 2014-08-15 11:58:34 <jgarzik> thus in the content of the current discussion, and covering efforts over and above current blockchain consensus mechanism.
240 2014-08-15 11:58:59 <jgarzik> *context
241 2014-08-15 11:59:22 <Luke-Jr> indeed, it's because that's what bitcoin does that trying to do that is nutty ;)
242 2014-08-15 11:59:36 <Luke-Jr> if mempools could be synchronised, we wouldn't need a blockchain
243 2014-08-15 11:59:52 <wumpus> it's not an attempt to synchronize everything together, it's an attempt to agree on a very limited set of rules that are set in stone in the beginning, and use that to keep an agreement about a certain state of the world
244 2014-08-15 12:01:06 <wumpus> ie, the utxo set
245 2014-08-15 12:02:41 <sipa> and only after time passes
246 2014-08-15 12:03:06 <Luke-Jr> wumpus: btw, there *is* a tool for deterministic key generation already ;)
247 2014-08-15 12:03:08 <sipa> there is no consensus on what the 'current utxo set' is; there is exponentially increasing chance for consensus about what it was in the past
248 2014-08-15 12:03:23 <jgarzik> I was not speaking generally about bitcoin, as just noted.
249 2014-08-15 12:03:26 <wumpus> Luke-Jr: oh?
250 2014-08-15 12:03:29 <Luke-Jr> wumpus: pycoin
251 2014-08-15 12:03:47 <jgarzik> IMO hearn often pursues the path of synchronize mempools+miner policy -> zero conf transactions are now safe.
252 2014-08-15 12:03:56 <jgarzik> which is not bitcoin, IMNSHO
253 2014-08-15 12:04:08 <wumpus> Luke-Jr: oh, right, wouldn't be surprised if 'sx' could do it too
254 2014-08-15 12:04:30 <hearn> well, it's an optimisation problem. obviously the perfect p2p currency would have instant, guaranteed safe transactions without any of this mining malarky
255 2014-08-15 12:04:30 <sipa> i expect that there will be a strong incentive in the future for mempool consensus, as it can be used improve block forwarding
256 2014-08-15 12:04:41 <hearn> bitcoin is not that. but why not strive to get closer to perfection?
257 2014-08-15 12:04:42 <sipa> but as a goal on itself i do not think it should be enforced or even encouraged
258 2014-08-15 12:05:13 <hearn> wumpus: what tool do you need?
259 2014-08-15 12:05:18 <jgarzik> hearn, bitcoin == IP of TCP/IP. Don't do everything in the same layer.
260 2014-08-15 12:05:20 <wumpus> Luke-Jr: I was just mentioning deterministic key generation as an example of a stand-alone, as sipa has been working on that
261 2014-08-15 12:05:25 <Luke-Jr> sipa: optimising it as a special case *is* encouragement
262 2014-08-15 12:05:29 <wumpus> Luke-Jr: +utility
263 2014-08-15 12:05:45 <jgarzik> hearn, you can accomplish the goal without fundamentally altering bitcoin to some Platonic ideal
264 2014-08-15 12:05:49 <Luke-Jr> wumpus: sure, I just mean it already exists
265 2014-08-15 12:06:18 <jgarzik> IP (irreversible packets) required TCP and HTTP before "the web" became useful
266 2014-08-15 12:06:20 <hearn> it's hardly altering bitcoin. the original bitcoin has no notion of miner policy at all. that's all stuff that's been added later, after satoshi left.
267 2014-08-15 12:06:39 <jgarzik> malarky. miner policy has always existed in bitcoin.
268 2014-08-15 12:07:23 <sipa> i'd very much like to hear satoshi's view on the current system
269 2014-08-15 12:07:27 <hearn> yes? which flags/options in the original bitcoin let you customise mining policy?
270 2014-08-15 12:07:37 <sipa> but appealing to how he may have wanted to design things is pointless
271 2014-08-15 12:07:41 <wumpus> hearn: none, that's why a lot of miners run with ooooold bitcoind forks :(
272 2014-08-15 12:08:00 <wumpus> hearn: they made a few changes once and refuse to spend time to port it forward, because yeah it works
273 2014-08-15 12:08:02 <Luke-Jr> wumpus: well, I think "old bitcoind forks" died with 0.8
274 2014-08-15 12:08:07 <Luke-Jr> wumpus: that was a "must have" bump ;)
275 2014-08-15 12:08:13 <wumpus> Luke-Jr: okay :)
276 2014-08-15 12:08:42 <Luke-Jr> Deepbit probably still runs 0.3, but they're not relevant anymore :p
277 2014-08-15 12:08:54 <wumpus> hearn: you can't avoid miner policy, if you don't make it explicit, it's implicit
278 2014-08-15 12:09:52 <jgarzik> hearn, All of the early miners ran forks. The policy was easy to change, as it is encoded into BitcoinMiner() (grep for 'nMinFee' inside BitcoinMiner)
279 2014-08-15 12:10:36 <hearn> right, i know people can always introduce it. it's not like the absence of a flag will stop people making changes if they really want to. however, it's definitely something we should find ways to resolve, by having a "clearly right" answer for whatever the question is. e.g. fee estimation clearly gets us closer to this for fees.
280 2014-08-15 12:10:47 <jgarzik> hearn, early satoshi crypto posts (perhaps whitepaper itself) touched on miner policy vis a vis fees and transaction selection.
281 2014-08-15 12:11:40 <jgarzik> hearn, That's not a universally shared opinion by any stretch of the imagination
282 2014-08-15 12:11:54 <hearn> if it was, i would be arguing for it would i :-)
283 2014-08-15 12:11:58 <jgarzik> hehehe
284 2014-08-15 12:12:13 <Luke-Jr> hearn: let's mainline my spamfilter branch then, it's clearly an improvement
285 2014-08-15 12:13:10 <Luke-Jr> if all miners ran with it, it'd also close that one-of-many ways to double spend you mentioned earlier
286 2014-08-15 12:13:25 <hearn> i'd point out that has been a lot more controversial than fee estimation, except then i suspect you'd want to go and start trying to make that controversial as well :)
287 2014-08-15 12:14:00 <hearn> well if all miners ran with it, and all blocked dice addresses, those sites would stop reusing addresses and the code would become useless/inactive. so .....
288 2014-08-15 12:14:30 <Luke-Jr> that was sarcasm, to make a point that even when there is a clear improvement, it doesn't mean everyone ought to be pressured into it
289 2014-08-15 12:15:08 <hearn> i think we're vigorously agreeing without realising it (ironic given the topic). i don't think people should be pressured into consensus (how?) but that we should strive to obtain it via natural means, like finding solutions that everyone agrees on
290 2014-08-15 12:15:39 <hearn> e.g. everyone agrees that bitcoin does not have infinite capacity and some transactions are more useful than others, but is a spam filter the best solution? or is it better to just let the fee market sort things out (if the fee market worked....)
291 2014-08-15 12:16:10 <Luke-Jr> it's better to have empty blocks, than blocks full of spam
292 2014-08-15 12:17:16 <hearn> yeah but that's not the choice we face
293 2014-08-15 12:18:06 <jgarzik> ...and when 1MB block size changes, fee market implodes
294 2014-08-15 12:18:41 <Luke-Jr> jgarzik: depends on how and when? :p
295 2014-08-15 12:19:29 <jgarzik> current proposals for "constant sized blocks" (headers plus enough metadata to reconstruct full block) easily allow 1GB per block
296 2014-08-15 12:19:57 <gmaxwell> jgarzik: thats confused and incorrect.
297 2014-08-15 12:20:41 <gmaxwell> Relaying blocks more efficiently doesn't change the inherent tradeoffs at allâ (and p2pool has done more efficient realying for years, okay, it's not _quite_ constant size, but assuming consistent memory pools its a few bytes per transaction)
298 2014-08-15 12:21:20 <gmaxwell> Making relaying more efficent is important and good, but it doesn't change the resource required to keep up with the network.
299 2014-08-15 12:23:29 <jgarzik> gmaxwell, It self evidently does. You stop relaying transactions twice.
300 2014-08-15 12:23:53 <jgarzik> that resource usage is 0.5x
301 2014-08-15 12:24:01 <gmaxwell> jgarzik: I haven't been relaying things twice for years. :P
302 2014-08-15 12:24:34 <gmaxwell> You can be not relaying things twice yourself in a couple minutes, tooâ just go grab matt's jar.
303 2014-08-15 12:25:24 <gmaxwell> Sorry, I should have clarified thenâ perhaps for you it was a 0.5x change in scaling relative to your thinking. OKAY, perhaps so. I certantly wasn't assuming people would continue sending things twice.
304 2014-08-15 12:26:03 <gmaxwell> There is a maximum advantage in that of 0.5x however. Sorry, I'm a bit irritated by people running around saying that bitcoin can now support infinite transactions per second because zomg-constant.
305 2014-08-15 12:28:04 <jgarzik> Relative to what is going on in the real world on today's network, you are removing a major [dumb, needed-to-be-removed] impediment to simply running the network at the speed at which <number> transactions may relayed across the network, verified and remembered.
306 2014-08-15 12:28:24 <jgarzik> It is certainly not infinite, but it changes bitcoin in subtle ways.
307 2014-08-15 12:29:06 <jgarzik> The costs of IBD and blockchain size are typically not seen by in-sync nodes, except perhaps by blockchain download-to-peers traffic.
308 2014-08-15 12:29:46 <jgarzik> Oddly, that situation becomes more asymmetric with the removal of sending TXs twice, the second time via 'block'.
309 2014-08-15 12:29:51 <jgarzik> Block size itself becomes less visible.
310 2014-08-15 12:31:08 <jgarzik> That has implications on the block size debate, which in turn impacts how engineers code efficiently and bid for space.
311 2014-08-15 12:31:11 <gmaxwell> I dunno why no one seemed to care about the saving the resend six months ago.
312 2014-08-15 12:32:16 <jgarzik> Merge this useful change, and next will come the call to remove block size limit altogether, which will throw a nuclear bomb onto any nascent fee market.
313 2014-08-15 12:32:45 <hearn> there will always be *some* block size limit, obviously, as computers are finite machines bound by the laws of physics
314 2014-08-15 12:33:00 <jgarzik> All the VCs and execs want an infinite block size limit. It's a sad fixation.
315 2014-08-15 12:33:08 <hearn> the "block size limit" we talk about is really the one that satoshi put in place to stop troll blocks. if we had a better solution for that, it wouldn't be needed anymore.
316 2014-08-15 12:33:13 <hearn> e.g. a floating limit
317 2014-08-15 12:33:30 <hearn> but this goes back to the 2010 era debates about what the purpose of fees is/should be
318 2014-08-15 12:33:41 <gmaxwell> A floating limit is not magic though. (also actual in the wild implementations have failed)
319 2014-08-15 12:34:06 <gmaxwell> There is a split in interest between pool operators and the general users of bitcoin.
320 2014-08-15 12:35:07 <gmaxwell> If I was running a mining operation with 100k/month fee income then sure I could easily support transaction rates corresponding to 100GB blocks. But then there would pratically be only a dozen or so full nodes in the network, and under that assumption set ripple has a superior security model.
321 2014-08-15 12:35:23 <hearn> i used to worry about this
322 2014-08-15 12:35:29 <hearn> then i noticed that tx traffic for bitcoin has been flat for a year
323 2014-08-15 12:35:41 <hearn> these days i worry way more about how the hell we're going to increase usage to reach that magical 1mb limit
324 2014-08-15 12:36:01 <gmaxwell> hearn: you can happily generate traffic to drive everyone out of competition. (which is what floating fees seem to have permitted elsewhere :( )
325 2014-08-15 12:36:04 <hearn> i suspect one day we're going to look back on these discussions and say, wow, we were so optimistic :)
326 2014-08-15 12:37:03 <gmaxwell> and yea, I'd agree in general that increasing usage is a bigger concern today... also preventing the system from slipping into a state of centeralization which leaves it with no competative advantage in practice against alternatives like ripple (which are centeralized and make use of that capability)
327 2014-08-15 12:37:34 <hearn> i worry about visa more than ripple
328 2014-08-15 12:37:52 <hearn> contactless batteryless payment cards are pretty nice, if your merchant and tx size supports them
329 2014-08-15 12:37:59 <gmaxwell> just search and replace with your preferred massively more scalable because its centeralized cryptocurrency alternative.
330 2014-08-15 12:43:01 <hearn> lol. the dark wallet budget includes 300 EUR per month for "herbs"
331 2014-08-15 12:44:25 <gmaxwell> O_o
332 2014-08-15 12:44:40 <gmaxwell> And on that bit of perplexity, Goodnight everybody.
333 2014-08-15 12:44:43 <hearn> https://wiki.unsystem.net/en/index.php/DarkWallet/Budget
334 2014-08-15 12:44:46 <hearn> good night
335 2014-08-15 12:49:57 <wumpus> :') I wonder what TBF's budget for herbs is
336 2014-08-15 12:52:00 <hearn> oooh, we're getting close to the magical 1 tx/second mark :)
337 2014-08-15 12:52:01 <hearn> https://blockchain.info/charts/n-transactions
338 2014-08-15 12:52:11 <hearn> (again)
339 2014-08-15 12:55:03 <gmaxwell> BlueMatt: Cory: is pulltester suck? 4701 hasn't had a build yet.
340 2014-08-15 12:56:00 <factorialsquare> are there any good guides anywhere for best practice for building a bitcoin-accepting payment engine?
341 2014-08-15 12:57:22 <wumpus> pulltester indeed seems to be stuck
342 2014-08-15 12:58:16 <hearn> factorialsquare: there's the developer guide on bitcoin.org, but otherwise, honestly we lack a really great open source merchant engine of the caliber of what bitpay/coinbase have built. at least if one exists i've never seen or heard about it
343 2014-08-15 12:58:25 <wumpus> hopefully for the last time, we're very close to enabling the travis-based one
344 2014-08-15 12:59:07 <factorialsquare> hearn: ok, thanks
345 2014-08-15 12:59:26 <hearn> factorialsquare: if you'd like to build one i'm happy to give you free opinions on how to do it ;)
346 2014-08-15 13:00:07 <factorialsquare> hearn: go ahead
347 2014-08-15 13:00:17 <factorialsquare> hearn: :)
348 2014-08-15 13:00:45 <hearn> factorialsquare: have you made many payments via bitpay or coinbase?
349 2014-08-15 13:01:11 <factorialsquare> hearn: no, I havn't made any via them
350 2014-08-15 13:01:23 <factorialsquare> hearn: maybe I should start checking them out
351 2014-08-15 13:01:35 <factorialsquare> hearn: what's so special about them?
352 2014-08-15 13:01:44 <hearn> well, i'd suggest doing at least one so you get the idea of how they work. here's the one i use for testing: http://bitgivefoundation.org/donate-now/
353 2014-08-15 13:01:55 <hearn> nothing special exactly. but you should IMHO aim to be at least as slick
354 2014-08-15 13:02:14 <hearn> for example, they render a QR code and give you a clickable button, but also automatically refresh the page once the transaction is broadcast
355 2014-08-15 13:02:26 <hearn> so you see immediately that your payment was accepted by the merchant the moment you hit "Send" in your wallet.
356 2014-08-15 13:02:31 <hearn> no need to refresh the browser manually.
357 2014-08-15 13:02:47 <gmaxwell> https://github.com/slickage/baron might be worth checking out.
358 2014-08-15 13:02:56 <hearn> they use BIP70 so tx submission is even faster, you don't even have to wait for propagation across the network. it's very nice. they also show you the prices in local currency, etc
359 2014-08-15 13:03:06 <hearn> ah ha!
360 2014-08-15 13:03:52 <hearn> oh. except i tried their demo and it says "baron appears to be down"
361 2014-08-15 13:03:58 <hearn> also you have to pick min_confirmations of at least one.
362 2014-08-15 13:04:12 <factorialsquare> is it possible to build it without any custom bitcoin code or is the JSON-API via bitcoind enough?
363 2014-08-15 13:04:41 <hearn> you can use json-rpc with bitcoind, you could use bitcoinj (when a bitcoind is running locally it'll be used automatically and you get the same security level), there are other APIs you could use too
364 2014-08-15 13:04:50 <jgarzik> factorialsquare, you'll need at least some custom software, usually.
365 2014-08-15 13:04:58 <jgarzik> factorialsquare, bitcoind isn't built for direct website payments
366 2014-08-15 13:04:59 <hearn> the bitcoin interaction part isn't the difficult bit, imo. it's all the UI stuff.
367 2014-08-15 13:05:23 <gmaxwell> hearn: ah, I've tried it out before, and it's pretty good for what its intended to be.
368 2014-08-15 13:05:26 <hearn> baron sounds good
369 2014-08-15 13:05:33 <hearn> factorialsquare: yeah definitely check out baron
370 2014-08-15 13:05:35 <hearn> ACTION never heard of it before
371 2014-08-15 13:05:43 <gmaxwell> It's not a system for doing zeroconf, since that usually requires some more "application specific" risk management logic.
372 2014-08-15 13:05:58 <gmaxwell> maybe in the future though.
373 2014-08-15 13:07:01 <hearn> right. though people usually do want that. so that risk management is a very useful feature to have.
374 2014-08-15 13:07:59 <factorialsquare> jgarzik: what functions is not included in the json-rpc API with bitcoind that you would need for a payment engine?
375 2014-08-15 13:08:59 <hearn> well what do you mean by "payment engine"
376 2014-08-15 13:09:12 <factorialsquare> accepting payments
377 2014-08-15 13:09:14 <hearn> to me it means things like exchange rate conversion, rendering a web page, doing something useful when the payment is received
378 2014-08-15 13:09:31 <hearn> making a record of the payment etc
379 2014-08-15 13:09:39 <hearn> you can receive payments with just the json-rpc and -walletnotify flags. however that is the very basics of actually receiving money
380 2014-08-15 13:09:48 <hearn> you'd still want to implement e.g. BIP70 yourself
381 2014-08-15 13:29:14 <Luke-Jr> gmaxwell: fwiw, I just calculated 185 MB required to do a full index by sPK and spends; though I haven't determined what kind of lookups/sec it would be capable of
382 2014-08-15 13:29:31 <Luke-Jr> (also implemented part of the index)
383 2014-08-15 13:31:28 <Luke-Jr> 83 MB to lookup block by number (not necessarily height) to file number (11-bit) and word offset (29-bit); indexing sPKs by 20-bit hash160 prefix to 24-bit block number
384 2014-08-15 13:31:40 <Luke-Jr> (the latter being 6 MB + overflow)
385 2014-08-15 13:31:57 <Luke-Jr> ACTION ponders whether he should spend time completing this
386 2014-08-15 13:54:59 <ThomasV> what does it mean if "getblock <hash>" returns a result with "confirmations:-1", for an old block ?
387 2014-08-15 14:02:19 <helo> ThomasV: return -1; // Not in chain, not in mempool
388 2014-08-15 14:03:29 <jrick> block not in mempool? whut
389 2014-08-15 14:03:31 <ThomasV> helo: it occurs with block 000000000000000000fdb0eda63a91460fabc03ef4e648e2b672a649f88bfccd
390 2014-08-15 14:03:37 <ThomasV> which is in chain
391 2014-08-15 14:03:53 <ThomasV> so there's probably some corruption in my db
392 2014-08-15 14:03:59 <ThomasV> is this known about?
393 2014-08-15 14:04:04 <helo> i get "confirmations" : 16143,
394 2014-08-15 14:06:05 <wumpus> "confirmations" : 14624,
395 2014-08-15 14:06:17 <wumpus> ThomasV: do you get it for all blocks, or just that one?
396 2014-08-15 14:06:30 <ThomasV> wumpus: just that one so far
397 2014-08-15 14:06:51 <wumpus> can you pastebin the entire output of that getblock?
398 2014-08-15 14:06:57 <ThomasV> wumpus: compared with others, I have one txid that differs
399 2014-08-15 14:07:16 <ThomasV> wumpus: https://electrum.org/block299616.txt
400 2014-08-15 14:07:46 <wumpus> hm so the height is correct
401 2014-08-15 14:08:00 <ThomasV> someone suggested it's a flipped bit in a tx
402 2014-08-15 14:08:45 <helo> random - what kind of storage?
403 2014-08-15 14:09:54 <ThomasV> helo: http://www.hetzner.de/en/hosting/produkte_rootserver/ex40
404 2014-08-15 14:09:58 <ThomasV> hard drive
405 2014-08-15 14:10:07 <wumpus> yup - 96d3f7e192853e85488ca950f1d0cea07d076f23f0a6f923241734ed081a2fd9 for you versus d6328aedce03c9d94c48ebf320a4d7f4b0a8a08f58e2705c6a60e33528066d93 for me
406 2014-08-15 14:10:46 <gmaxwell> is it the coinbase txn that differs?
407 2014-08-15 14:10:57 <ThomasV> gmaxwell: no
408 2014-08-15 14:11:12 <wumpus> no, it has one input and two outputs
409 2014-08-15 14:11:34 <wumpus> can you get hold of the contents of that transaction?
410 2014-08-15 14:12:02 <ThomasV> note thar getrawtransaction fails for both hashes
411 2014-08-15 14:12:06 <mr_burdell> ThomasV: do getblock 000000000000000000fdb0eda63a91460fabc03ef4e648e2b672a649f88bfccd false
412 2014-08-15 14:12:10 <mr_burdell> and compare
413 2014-08-15 14:13:08 <gmaxwell> the reason for the -1 is probably because SetMerkleBranch is rehashing the block then it can't find it in the index.
414 2014-08-15 14:14:08 <wumpus> here's my version to compare https://download.visucore.com/tmp/000000000000000000fdb0eda63a91460fabc03ef4e648e2b672a649f88bfccd.txt
415 2014-08-15 14:15:34 <ThomasV> they differ
416 2014-08-15 14:15:37 <wumpus> anyhow, this indeed seems like a bitflip in your block files (not the block index or utxo database)
417 2014-08-15 14:16:33 <ThomasV> wumpus: how likely is this?
418 2014-08-15 14:16:52 <wumpus> that's completely dependent on your hardware
419 2014-08-15 14:17:04 <gmaxwell> ThomasV: can you post the actual data you got?
420 2014-08-15 14:17:26 <wumpus> I've had a pc with a malfunctioning CPU, which created all kinds of weird random corruptions at random times
421 2014-08-15 14:17:54 <wumpus> also it can also be caused by bugs in the kernel with dma'ing
422 2014-08-15 14:18:00 <gmaxwell> ThomasV: misplaced writes are not uncommon as are random bitflips on non-error corrected systems. sadly people are running servers on things without even ecc ram.
423 2014-08-15 14:18:21 <ThomasV> gmaxwell: https://electrum.org/block299616
424 2014-08-15 14:18:26 <ThomasV> that's the raw one
425 2014-08-15 14:18:36 <ThomasV> https://electrum.org/block299616.txt is the json
426 2014-08-15 14:18:46 <gmaxwell> how did you obtain the raw one?
427 2014-08-15 14:18:54 <gmaxwell> oh false, thats fine
428 2014-08-15 14:19:00 <ThomasV> getblock 000000000000000000fdb0eda63a91460fabc03ef4e648e2b672a649f88bfccd false
429 2014-08-15 14:19:14 <wumpus> a malfunction in the harddisk itself is unlikely to cause one bit flip
430 2014-08-15 14:20:16 <wumpus> as those do have error-correction logic
431 2014-08-15 14:21:49 <wumpus> a[841920] -> '3' versus '1'
432 2014-08-15 14:21:53 <gmaxwell> it's actually a 1 bit difference here.
433 2014-08-15 14:21:53 <wumpus> yep, a one-bit flip
434 2014-08-15 14:22:25 <ThomasV> ok
435 2014-08-15 14:22:30 <gmaxwell> wumpus: whats more common on disks is flying writes where a whole sector gets trashed with a write that should have been to another sector
436 2014-08-15 14:22:58 <gmaxwell> now for bonus points figure out what difference the 1 bit makes. :)
437 2014-08-15 14:23:01 <wumpus> gmaxwell: heh, a mistargeting
438 2014-08-15 14:23:41 <ThomasV> gmaxwell: is there a way to fix that block without redoing all the rest?
439 2014-08-15 14:24:02 <gmaxwell> shut down the node go hex edit the blockfile.
440 2014-08-15 14:24:05 <wumpus> yes, figure out at what offset that block is in the file
441 2014-08-15 14:24:09 <ThomasV> heh, ok
442 2014-08-15 14:24:13 <wumpus> (ie, search for the data)
443 2014-08-15 14:24:17 <wumpus> then patch in the right value :P
444 2014-08-15 14:24:25 <ThomasV> that's kinda raw
445 2014-08-15 14:24:57 <ThomasV> maybe I should scann all blocks for "confirmations: -1"
446 2014-08-15 14:24:59 <wumpus> bitcoind regards the block files as immutable, so it has no functions to 'edit' the block file
447 2014-08-15 14:25:09 <wumpus> ThomasV: would be better to just do a reindex then
448 2014-08-15 14:25:19 <ThomasV> yeah
449 2014-08-15 14:25:30 <ThomasV> well
450 2014-08-15 14:26:51 <ThomasV> gmaxwell: do you think it comes from the ram, or from the disk? ( http://www.hetzner.de/en/hosting/produkte_rootserver/ex40 )
451 2014-08-15 14:27:05 <ThomasV> it says "software raid 1"
452 2014-08-15 14:27:14 <wumpus> as discussed above, disks seldom fail in this manner
453 2014-08-15 14:27:52 <ThomasV> I will still do a scan, to see if it happened with other blocks
454 2014-08-15 14:28:15 <ThomasV> it might not be the only one
455 2014-08-15 14:28:52 <wumpus> but yes the ram is a likely suspect
456 2014-08-15 14:29:00 <gmaxwell> ThomasV: it's likely ram
457 2014-08-15 14:29:35 <gmaxwell> I think I computed the expected bitflip rate for my 64gb hosts at a couple per day.
458 2014-08-15 14:31:50 <ThomasV> gmaxwell: ok, they also offer servers with ecc ram, so that means mine does not have ecc
459 2014-08-15 14:32:02 <ThomasV> I guess I will upgrade :)
460 2014-08-15 14:32:38 <ThomasV> thank you guys
461 2014-08-15 14:32:47 <ThomasV> I learned something
462 2014-08-15 14:34:27 <hearn> one time in google we had a bitflip in a bigtable database master
463 2014-08-15 14:34:35 <hearn> it flipped a character in a string controlling replication datacenter paths
464 2014-08-15 14:35:15 <hearn> so /bigtable/yq-i/something became (handwave) /bigtable/ys-i/something
465 2014-08-15 14:35:29 <hearn> the entire web crawl ground to a halt as the system tried to replicate from a non-existent database :)
466 2014-08-15 14:38:42 <wumpus> hehe, ouch
467 2014-08-15 14:40:34 <gmaxwell> ThomasV: just be glad it wasn't a pubkey you were computing when that bitflip happened.
468 2014-08-15 14:43:40 <hearn_> grrr internets
469 2014-08-15 14:43:44 <hearn_> gmaxwell: i thought you were going to sleep?
470 2014-08-15 14:44:03 <gmaxwell> people were wrong on the internets.
471 2014-08-15 14:44:04 <hearn_> ACTION was surprised to learn that gmaxwell sleeps and is therefore unsurprised to learn that it's a lie
472 2014-08-15 14:44:19 <gmaxwell> hah
473 2014-08-15 14:46:37 <hearn> hmm. the problem with trying to recalculate pubkeys at the last moment before they're given to the app is encrypted wallets
474 2014-08-15 14:46:48 <hearn> though i guess you could rederive from the parent
475 2014-08-15 14:47:00 <hearn> that should be just as good w.r.t. bitflip checking
476 2014-08-15 14:49:08 <gmaxwell> hearn: you need to worry about bitflips in the ecdsa implementation ram too, which are fun because they'll result in stable errors.
477 2014-08-15 14:49:49 <hearn> ah yes
478 2014-08-15 14:49:53 <hearn> the precomputed tables?
479 2014-08-15 14:50:04 <hearn> well you can never really win at this sort of thing. just add a few tripwires here and there.
480 2014-08-15 14:50:05 <gmaxwell> But they'll also break the homorphic properties so you could compute G*(key+n)-G*n and G*k and compare.
481 2014-08-15 14:50:11 <helo> so you'd have to save the private key to disk, compute the pubkey, read from disk, recompute pubkey to ensure no bitflip?
482 2014-08-15 14:50:21 <hearn> helo: bitcoinj serialises pubkeys to disk
483 2014-08-15 14:50:33 <hearn> helo: so basically i'm just adding rederive from parent pubkey and check they match
484 2014-08-15 14:50:59 <helo> okay
485 2014-08-15 14:51:36 <gmaxwell> hearn: along the lines of minimizing these risks I went and implemented sha512 that works in the bit reversed (1=0 0=1) domain, then its also easy to implement the ecmult that takes bitreversed input.
486 2014-08-15 14:52:36 <gmaxwell> with the idea of you could compute the key in parallel, so even freaky faults like registers getting bits set to 1 wouldn't pass. dunno if its really work the complexity vs the check via homomorphism that I described above, which is simple easy.
487 2014-08-15 15:02:47 <hearn> gmaxwell: "key" in that formula is the pubkey?
488 2014-08-15 15:03:05 <hearn> gmaxwell: the problem is I don't have "k", just the parent pubkey
489 2014-08-15 15:03:12 <hearn> because the users wallet may be encrypted at the point a key is requested
490 2014-08-15 15:03:42 <hearn> i suppose i could check both at vending time, and at signing time for change addresses, at least for locally signed wallets.
491 2014-08-15 15:04:03 <hearn> no key must be the private key, i guess
492 2014-08-15 15:05:11 <gmaxwell> it's private in that example.. in that case of derriving from the pubkey you're doing a tweak operation, where generating from the master is MP + G*(tweak) which can still be checked e.g. by doing it twice, once with tweak and once with tweak+something and then adding the inverse of something *G to the result and comparing.
493 2014-08-15 15:07:34 <hearn> does it matter if something is dynamically random? i think not
494 2014-08-15 15:07:38 <hearn> but i guess making it random can't hurt either
495 2014-08-15 15:08:23 <gmaxwell> well it hurts in that the 1/something * G can be precomputed and just sit around if something is static.
496 2014-08-15 15:08:30 <hearn> so doing it the normal bip32 way when keys are first derived, and then doing tweak+something + modinverse(something*G) at vend time
497 2014-08-15 15:08:49 <hearn> seems reasonable
498 2014-08-15 15:25:12 <hearn> gmaxwell: to double check: (G*tweak)+MP == (G* (tweak+rnd_uint256).mod(n)) + MP + (G*(rnd_uint256.modInverse(n))) ?
499 2014-08-15 15:25:18 <hearn> ACTION is trying to implement this but the results don't match
500 2014-08-15 15:33:01 <gmaxwell> sorry, lack of sleep, you need the additive inverse. (rnd + n-1 mod n)
501 2014-08-15 15:46:18 <hearn> gmaxwell: sorry, more sucky internets. additive inverse is also just (-rnd_uint256 mod n) right? so G*(-rnd_uint256 mod n)
502 2014-08-15 15:47:59 <gmaxwell> yup.
503 2014-08-15 15:48:54 <hearn> hmm. i must still be doing something wrong.
504 2014-08-15 15:51:07 <gmaxwell> works here
505 2014-08-15 15:51:08 <gmaxwell> sage: G*tweak + P == G*(tweak+r) + G*int(N(-r)) + P
506 2014-08-15 15:51:09 <gmaxwell> True
507 2014-08-15 15:54:03 <hearn> yeah. wonder where my mistake is
508 2014-08-15 15:54:06 <hearn> my code is http://hastebin.com/arijukokod.java
509 2014-08-15 15:54:07 <hearn> the else branch
510 2014-08-15 15:56:01 <gmaxwell> is your N correct? it should be 115792089237316195423570985008687907852837564279074904382605163141518161494337
511 2014-08-15 15:56:43 <hearn> urgh
512 2014-08-15 15:56:55 <hearn> never mind. it does work now. for some reason intellij stopped rebuilding properly
513 2014-08-15 15:57:00 <hearn> i just forced a rebuild and now it does work.
514 2014-08-15 15:57:11 <hearn> the last update made things so flaky. it used to be solid as a rock.
515 2014-08-15 16:06:34 <hearn> there we go. https://github.com/bitcoinj/bitcoinj/commit/f00aef2048a2c3bce0b5d325bf4f3a16fe104cdc
516 2014-08-15 16:06:36 <hearn> gmaxwell: thanks
517 2014-08-15 16:06:51 <hearn> will try it out for a while and see if it causes any problems (shouldn't do)
518 2014-08-15 16:27:29 <dgenr8> visa
519 2014-08-15 16:28:37 <hearn> gmaxwell: is there a simple formula for encrypting a small integer using a secp256k1 private key?
520 2014-08-15 16:28:45 <hearn> i.e. a bip32 key path number
521 2014-08-15 16:29:06 <hearn> ecies looks complicated
522 2014-08-15 16:29:41 <dgenr8> hearn: i just saw sdlerner's nimblecoin. seems more relevant then either visa or ripple. 5s block time solves a big problem if it works
523 2014-08-15 16:29:49 <gmaxwell> It's very easy to be massively insecure. When you say encrypt using a private key, do you mean encrypt using a public key?
524 2014-08-15 16:31:52 <gmaxwell> The reason I ask if you mean public/private is that if you're encrypting just to yourself then you can skip the asymetric crypto entirely, e.g. https://bitcointalk.org/index.php?topic=726495.0
525 2014-08-15 16:36:41 <hearn> stupid internet.
526 2014-08-15 16:36:53 <hearn> gmaxwell: sorry, now i'm the sleep deprived one. i can just the the wallet seed as an AES key
527 2014-08-15 16:36:58 <hearn> (for this use case)
528 2014-08-15 16:37:20 <Luke-Jr> hearn: "just use the"?
529 2014-08-15 16:37:22 <gmaxwell> hearn: yea, thats the idea in that bitcointalk post.
530 2014-08-15 16:37:30 <hearn> yes sorry, just use the
531 2014-08-15 16:37:52 <gmaxwell> (not the seed but the first private key or whatever, same idea)
532 2014-08-15 16:37:53 <hearn> i want to encrypt some data so only the wallet owner can read it, and ended up thinking "i have EC keys, therefore I must encrypt with ECC"
533 2014-08-15 16:37:55 <hearn> which makes no sense
534 2014-08-15 16:37:58 <hearn> right
535 2014-08-15 16:38:28 <gmaxwell> it's a good idea, I also recommended the btcd people use that approach for their wallet metadata.
536 2014-08-15 16:38:48 <dgenr8> the recent focus on block propagation times. is there is a secret plan to reduce the block time someday?
537 2014-08-15 16:38:57 <dgenr8> cuz if you don't get down to 5s, why bother
538 2014-08-15 16:39:01 <Luke-Jr> dgenr8: â¦
539 2014-08-15 16:39:40 <Luke-Jr> dgenr8: there is no benefit to reducing the block interval; there is lots of benefit to reducing propagation time
540 2014-08-15 16:40:24 <Luke-Jr> (and 5s block interval is insane)
541 2014-08-15 16:40:35 <gmaxwell> Or at least no benefit to reducing the block interval that can't be had in some other way that doesn't massively increase overheads for SPV validators.
542 2014-08-15 16:40:47 <gmaxwell> (and create huge centeralization pressures)
543 2014-08-15 16:41:05 <dgenr8> well then we are stuck with finding other ways to address 0-conf reliability
544 2014-08-15 16:41:24 <Luke-Jr> dgenr8: no, it's not a problem.
545 2014-08-15 16:41:53 <Luke-Jr> dgenr8: and reducing block interval does not "address" it
546 2014-08-15 16:42:25 <dgenr8> sdlerners simulations are quite interesting
547 2014-08-15 16:43:30 <dgenr8> Luke-Jr: you are delirious to think that 10min +- 12 min confirmation time is not a problem
548 2014-08-15 16:43:58 <Apocalyptic> dgenr8, you are delirious to think it is
549 2014-08-15 16:44:02 <Luke-Jr> dgenr8: confirmation time is 1 hour. making the block interval shorter does not change that noticably.
550 2014-08-15 16:44:23 <Luke-Jr> confirmation time for credit cards is 6+ months
551 2014-08-15 16:44:50 <Luke-Jr> I think an hour is good enough considering the alternative
552 2014-08-15 16:45:19 <dgenr8> sigh, i do don't want to be the one to create the double-spending wallet app
553 2014-08-15 16:45:36 <dgenr8> s/do/so/
554 2014-08-15 16:45:49 <BigBitz> wait, what, Luke-Jr.
555 2014-08-15 16:45:56 <BigBitz> 6+months for a Credit Card confirmation?
556 2014-08-15 16:45:59 <gmaxwell> dgenr8: You can't be, Bc.i did so though they didn't maintain it so it stopped working about a year ago.
557 2014-08-15 16:46:16 <gmaxwell> BigBitz: it takes months for a credit card payment to become irreversable.
558 2014-08-15 16:46:16 <Luke-Jr> BigBitz: yes, that's how long until you know a credit card transaction is irreversible
559 2014-08-15 16:46:39 <BigBitz> You mean in regards to a fraudulant transaction?
560 2014-08-15 16:46:52 <dgenr8> this CC settlement talk is way off point. the guy in argentina selling sandwiches will get stolen from soon
561 2014-08-15 16:46:54 <BigBitz> They're covered by the payment proceesor. Isn't his point that a Bitcoin 'confirm' takes a long time.
562 2014-08-15 16:46:59 <gmaxwell> Thats also what Bitcoin is doing.
563 2014-08-15 16:47:04 <hearn> comparing CC and bitcoin reversals is a bit misleading
564 2014-08-15 16:47:10 <BigBitz> Yeah it's a bit... off, surely.
565 2014-08-15 16:47:16 <hearn> in practice card payments cannot be arbitrarily reversed. you have to file a dispute with the bank.
566 2014-08-15 16:47:23 <BigBitz> correct hearn.
567 2014-08-15 16:47:23 <gmaxwell> BigBitz: they are absolutely not "covered" by the payment processor. Merchant eats any reversals.
568 2014-08-15 16:47:24 <hearn> the bank sides with merchants about 40% of the time
569 2014-08-15 16:47:56 <hearn> if a merchant is careful to avoid obvious fraud, they don't typically have much to fear. for example outside the USA most countries are using EMV now and accepting those payments is basically risk free, instantly (in person)
570 2014-08-15 16:48:05 <gmaxwell> hearn: the dispute takes a five minute phone call, then they'll later mail you a filled out form to sign off on.
571 2014-08-15 16:48:08 <Luke-Jr> BigBitz: um, they're not covered by the payment processor. it gets takes back from the merchant.
572 2014-08-15 16:48:49 <Luke-Jr> gmaxwell: and in the meantime, the merchant is pressured to just refund you to avoid the chargeback cost
573 2014-08-15 16:48:57 <dgenr8> i'll feel bad stealing cupcakes ... i mean, i will reimburse them, but I will shake their faith in btc
574 2014-08-15 16:49:34 <Luke-Jr> dgenr8: why don't you go into Walmart and run off with some expensive item, with the understanding you'll reimburse them later?
575 2014-08-15 16:49:42 <Luke-Jr> because that's no different than what you're proposing
576 2014-08-15 16:49:43 <BigBitz> I meant the anti-fraud element is covered by the payment processor, but, you are correct there is no magic win/win situation for a Merchant taking stolen CC.
577 2014-08-15 16:49:56 <BigBitz> and infact what Luke-Jr says is right - the Merchant will likely refund before they get hit with costs.
578 2014-08-15 16:50:44 <Luke-Jr> I wouldn't be surprised if retail bitcoin acceptance ends up requiring a photo id, but oh well
579 2014-08-15 16:51:17 <dgenr8> Luke-Jr: lol the poor cupcake shop can't find out for 10 +- 12 minutes
580 2014-08-15 16:51:22 <gmaxwell> Luke-Jr: yea, it's something I've recommended to merchants. Moves the security model much closer to the cash and credit card ones.
581 2014-08-15 16:51:33 <Luke-Jr> (it might not, if people don't fraud more than their expected rate)
582 2014-08-15 16:52:35 <Luke-Jr> dgenr8: shorter block interval wouldn't change that. neither would they find out you used a stolen credit card until probably days later.
583 2014-08-15 16:52:41 <stevenleeg> Luke-Jr: why photo id?
584 2014-08-15 16:52:51 <Luke-Jr> stevenleeg: so if you fraud them, they know who to go after legally
585 2014-08-15 16:53:00 <Luke-Jr> stevenleeg: same as personal checks
586 2014-08-15 16:53:16 <gmaxwell> (or ban you from further business)
587 2014-08-15 16:53:25 <dgenr8> Luke-Jr: give it up ... the reason they take CC's is because *visa takes the risk*
588 2014-08-15 16:53:28 <Luke-Jr> gmaxwell: true, that might actually be the cheaper option XD
589 2014-08-15 16:53:31 <stevenleeg> Luke-Jr: hmm what sort of fraud though?
590 2014-08-15 16:53:36 <Luke-Jr> dgenr8: Visa does not.
591 2014-08-15 16:53:38 <dgenr8> or the bank rather
592 2014-08-15 16:53:41 <gmaxwell> dgenr8: visa does _not_ take the risk.
593 2014-08-15 16:53:45 <BigBitz> anyways I only poked in here to make you aware of some FUD --> <Xuthus> OMG new exploit found in bitcoin Protocol !!
594 2014-08-15 16:53:45 <BigBitz> conversations are above my intellect in here :)
595 2014-08-15 16:53:47 <Luke-Jr> dgenr8: Visa/banks put ALL the risk of the merchant
596 2014-08-15 16:53:48 <stevenleeg> like, is there any way for someone to revoke bitcoins after they've paid to a merchant?
597 2014-08-15 16:53:50 <gmaxwell> Nor does the bank. The merchant takes the risk.
598 2014-08-15 16:53:51 <hearn> dgenr8: it's complicated and varies a lot
599 2014-08-15 16:53:55 <hearn> the bank takes the risk in EMV zones
600 2014-08-15 16:54:06 <Luke-Jr> stevenleeg: yes
601 2014-08-15 16:54:08 <hearn> which is basically everywhere now except america. but even the USA is planning a liability shift in 2015
602 2014-08-15 16:54:17 <hearn> so soon Luke-Jr and gmaxwell will be wrong in the USA too, unless there's another delay (quite possible)
603 2014-08-15 16:54:20 <gmaxwell> Well I know squat about EMV zones, in the US the merchant takes 100% of the risk.
604 2014-08-15 16:54:35 <hearn> however this applies only to card-present transactions where EMV is used. if you accept magstripe, it's back to merchant liability.
605 2014-08-15 16:54:38 <BigBitz> US banking system is awful though.
606 2014-08-15 16:54:43 <hearn> basically the rule post-liability-shift is "whoever has the weakest technology pays"
607 2014-08-15 16:54:55 <hearn> if the bank and users card both support EMV but the merchant doesn't, merchant pays for fraud
608 2014-08-15 16:55:02 <dgenr8> well we're in the running for that prize