1 2012-06-19 00:15:25 <luke-jr> BlueMatt: ping
  2 2012-06-19 00:15:42 <luke-jr> BlueMatt: how do I call ConnectBlock directly after cblockstore-class ?
  3 2012-06-19 00:23:40 <l1l1ll11l11> Hellow all
  4 2012-06-19 00:28:20 <jimbit> lol,  rig1 has quit..  ll,  hello
  5 2012-06-19 00:39:44 <jimbit> has there been any talk of 512, vs 256 because of the asic news ?   or is this a stupid question?
  6 2012-06-19 00:40:43 <Diablo-D3> no, and maybe
  7 2012-06-19 00:40:47 <galambo> heh
  8 2012-06-19 00:41:10 <gmaxwell> jimbit: I'm not following your question.
  9 2012-06-19 00:41:38 <gmaxwell> Why would 512 vs 256 (what?) have to do with asic (what?) ?
 10 2012-06-19 00:41:49 <galambo> hes asking if bfls ASIC is going to be so powerful that we have to "upgrade" to sha-512
 11 2012-06-19 00:41:55 <jimbit> well, I have heard of talk of forking on purpose changing the proof of work from 256 to 512 because of the bfl asic news
 12 2012-06-19 00:42:18 <gmaxwell> jimbit: heard "talk" where?
 13 2012-06-19 00:42:31 <gmaxwell> And whats the justification given?
 14 2012-06-19 00:42:36 <galambo> the bfl cult secret forum
 15 2012-06-19 00:42:49 <jimbit> good question where....  the forums have become a park of trolling.
 16 2012-06-19 00:42:58 <copumpkin> jimbit: a 512-bit hash wouldn't change anything
 17 2012-06-19 00:43:08 <gmaxwell> IIRC the BFL announcement is regarding a product which will be available to the general public at sane sounding prices, and not otherwise compromising the bitcoin system.
 18 2012-06-19 00:43:46 <jimbit> so, then it is a question of just selling my gpus to gamers and buying the new stuff i guess.
 19 2012-06-19 00:44:11 <gmaxwell> jimbit: that already seemed inevitable.
 20 2012-06-19 00:45:12 <gmaxwell> after the reward halving no gpu operation would be profitable over power in any case... not unless a lot of gpus get shut off.
 21 2012-06-19 00:45:19 <jimbit> then it will be important to have good timing
 22 2012-06-19 00:45:24 <gmaxwell> or the bitcoin exchange rate doubles.
 23 2012-06-19 00:45:43 <l1l1ll11l11> or free power
 24 2012-06-19 00:45:50 <gmaxwell> l1l1ll11l11: no such thing.
 25 2012-06-19 00:45:54 <jimbit> which i expect it will go up to just above what it cost to 'make' a btc,  like any commodity will do
 26 2012-06-19 00:46:21 <jimbit> so it will cost me about 4 bucks to make a btc after the half
 27 2012-06-19 00:46:25 <gmaxwell> I mean, sure there are people here and there that don't pay for their power. But they aren't running large numbers of gpus and shouldn't care about asics or not.
 28 2012-06-19 00:46:57 <jimbit> I think i am at about 1.80 USD per btc now
 29 2012-06-19 00:46:57 <l1l1ll11l11> I'm running 45GH on zero (extra) cost power (included in my lease)
 30 2012-06-19 00:47:23 <jimbit> l1l1ll11l11,   shhh, dont tell anyone ;)
 31 2012-06-19 00:47:26 <gmaxwell> jimbit: supply side economics are usually BS. Especially for things without long term concrete demands.
 32 2012-06-19 00:47:38 <luke-jr> jimbit: also, were changing the hashing algorithm necessary for some reason, it wouldn't simply be to SHA512 most likely
 33 2012-06-19 00:48:01 <luke-jr> jimbit: so long as BFL is playing nice, most likely the community would want to ensure their ASICs continue to be usable
 34 2012-06-19 00:48:21 <Diablo-D3> bfl isnt playing nice
 35 2012-06-19 00:48:31 <luke-jr> Diablo-D3: yes they are
 36 2012-06-19 00:48:44 <galambo> anyone here have a bfl single?
 37 2012-06-19 00:48:48 <l1l1ll11l11> I have 5
 38 2012-06-19 00:48:51 <luke-jr> galambo: I do
 39 2012-06-19 00:48:52 <jimbit> well,  I cant say i understand alot about the algarithm.   I plug in quality psu's and gpus and that is about it....
 40 2012-06-19 00:48:54 <Diablo-D3> luke-jr: did you NOT see the thread I stickied?
 41 2012-06-19 00:49:02 <jimbit> I have 2 bfl singles
 42 2012-06-19 00:49:02 <luke-jr> Diablo-D3: nope
 43 2012-06-19 00:49:10 <Diablo-D3> https://bitcointalk.org/index.php?topic=88363.0
 44 2012-06-19 00:49:19 <galambo> luke-jr: okay i gyuess i can calm down about this then
 45 2012-06-19 00:49:19 <l1l1ll11l11> BFL got a little snippy
 46 2012-06-19 00:49:26 <gmaxwell> right, if we wanted to change something it would be a change to shut out a clear attacker, not someone who is behaving fairly and honestly and contributing to making the system secure against bad players that would use asics without cooperating.
 47 2012-06-19 00:49:48 <l1l1ll11l11> BFL singles are rock solid, I've never had one go down, I was among the first 10 to order
 48 2012-06-19 00:49:54 <jimbit> gmaxwell, sounds right!
 49 2012-06-19 00:50:12 <luke-jr> Diablo-D3: trolling isn't hostile
 50 2012-06-19 00:50:17 <gmaxwell> having good fast asics available to all miners will make it so that an attacker with asics isn't a risk unless he has a lot more asics than all the good users.
 51 2012-06-19 00:50:19 <galambo> well i wasnt quite sure if they actually existed or not
 52 2012-06-19 00:50:36 <jimbit> my bfls are good,  albiet they throttle themselvs with just a little over 58C
 53 2012-06-19 00:50:37 <galambo> everyone promoting the things seems kind of scammy (gigavps and inaba)
 54 2012-06-19 00:50:37 <l1l1ll11l11> just a very, very poor track record of timing
 55 2012-06-19 00:50:56 <l1l1ll11l11> 4-6 weeks really is 8-12 weeks
 56 2012-06-19 00:51:05 <gmaxwell> Once asics made on modern process (at least 45nm) at prices which aren't extortionary are available I'll buy a bunch.
 57 2012-06-19 00:51:13 <jimbit> yes, and OCT seeams a little quick too l1l1ll11l11
 58 2012-06-19 00:52:02 <gmaxwell> engineering is hard, everyone inexpirenced misses their deadlines. The only reason expirenced places get them right is that they learn how to sandbag.
 59 2012-06-19 00:52:06 <jimbit> with the prices they declared on the news release...well I will be at 1T myself
 60 2012-06-19 00:52:40 <jimbit> ftr:  http://news.yahoo.com/butterfly-labs-announces-next-generation-asic-lineup-054626776.html
 61 2012-06-19 00:52:57 <jimbit> 1 TH/s, priced at $29,899
 62 2012-06-19 00:53:08 <galambo> i dont have an advanced academic degree or defense industry job that would enable someone to pay me to figure out FPGA
 63 2012-06-19 00:53:10 <Diablo-D3> gmaxwell: Ill buy a bunch even before that
 64 2012-06-19 00:53:31 <jimbit> timing will be important
 65 2012-06-19 00:54:08 <jimbit> if you preorder too early, you tie up to much doe...  too late and your late to the party
 66 2012-06-19 00:55:21 <galambo> i know the basics and classmates did very simple things with fpga ( i guess sha-256 is simple too, without optimization )
 67 2012-06-19 00:56:11 <l1l1ll11l11> The only way to strike it big in the switcharoo is to be in line for first wave of delivery
 68 2012-06-19 00:56:18 <galambo> the spartan 6 guys are the most interesting to read about
 69 2012-06-19 00:56:21 <galambo> they know the most, but apparantly that doesn't mean anything
 70 2012-06-19 00:57:36 <l1l1ll11l11> Would be interesting if you coul dbe the fist to receive the 1TH unit, obviously all depends on how many are delivered the first wave, how much would 1TH make you that first day?
 71 2012-06-19 00:57:53 <jimbit> hmmmmm
 72 2012-06-19 00:58:36 <galambo> some parts of mining are interesting, but i think most of it is missing the point a bit
 73 2012-06-19 00:59:05 <l1l1ll11l11> point being network stability/security?
 74 2012-06-19 00:59:39 <jimbit> galambo, tru, but if it wasnt for the profit, most would not mine.
 75 2012-06-19 01:00:46 <l1l1ll11l11> which essentially means, long term, there must be a low cost way to mine, i.e. asics drawing tiny power
 76 2012-06-19 01:01:02 <gmaxwell> l1l1ll11l11: That doesn't follow in fact.
 77 2012-06-19 01:02:49 <gmaxwell> But what is important that the bulk of the honest miners can mine at least as cheap as some malicious miner so the malicious miner doesn't get a big advantage.
 78 2012-06-19 01:02:54 <l1l1ll11l11> wouldn't continued network security require a huge amount of hash power?
 79 2012-06-19 01:03:39 <l1l1ll11l11> It just seems that if the rewards were to cease today, hashing would just get tranaction fees, thus not many people would keep hashing, thus making it easier to overtake 51%
 80 2012-06-19 01:04:08 <gmaxwell> And thus the rewards aren't going to cease today.
 81 2012-06-19 01:05:29 <jimbit> and not for 20 years :)
 82 2012-06-19 01:05:55 <gmaxwell> 20??
 83 2012-06-19 01:06:14 <galambo> infinity?
 84 2012-06-19 01:06:14 <gmaxwell> The rewards don't go to _0_ until 2140 and thats if bitcoin isn't changed to have more precision before then.
 85 2012-06-19 01:06:47 <gmaxwell> Also, the correct word is subsidy, so I'll be using that in further comments. :)
 86 2012-06-19 01:06:50 <galambo> the question is when do rewards go below the market determined (it will be in the future) transaction fees
 87 2012-06-19 01:07:36 <gmaxwell> galambo: right, no way to tell until it becomes close I think.
 88 2012-06-19 01:11:00 <jimbit> i just put 20 years out there cause then it will be down in low single digits
 89 2012-06-19 01:20:35 <gmaxwell> jimbit: sure but it's hard to know what the economic significance of that would be.
 90 2012-06-19 01:23:32 <jimbit> yes,  people need to start 'using' btcs, not just hoarding them or cashing them in
 91 2012-06-19 01:23:47 <jimbit> I always give a transaction fee.
 92 2012-06-19 01:25:00 <galambo> that wont happen
 93 2012-06-19 01:25:48 <MysteryBanshee> whats wrong with hoarding?
 94 2012-06-19 01:25:49 <jimbit> galambo, then we are dead in the water when it goes below 6.25
 95 2012-06-19 01:26:04 <jimbit> unless they are 100bucks each
 96 2012-06-19 01:26:13 <jimbit> :)
 97 2012-06-19 01:26:33 <galambo> well they probably will increase in price
 98 2012-06-19 01:27:14 <galambo> i think bitcoin will be more useful in the future as a backing for inflationary sub-currencies
 99 2012-06-19 01:27:52 <jimbit> galambo, sub-currencies being fiat? or ?
100 2012-06-19 01:28:25 <galambo> issued by bitcoin banks basically
101 2012-06-19 01:28:29 <galambo> for lack of a better term
102 2012-06-19 01:29:05 <galambo> http://en.wikipedia.org/wiki/File:Delaware_Bridge_Company_Dollar.jpg
103 2012-06-19 01:29:09 <galambo> kind of like that
104 2012-06-19 01:29:32 <gmaxwell> galambo: they don't even have to be 'banks' they could themselves be distributed systems.
105 2012-06-19 01:29:41 <galambo> i agree
106 2012-06-19 01:32:29 <galambo> bitcoin can be a digital notary stamp for future subcurrencies, temporarily loaned out
107 2012-06-19 01:32:32 <jimbit> himmm   jimbit dollars :)
108 2012-06-19 01:32:59 <galambo> basically, but the proof of work keeps you almost honest :)
109 2012-06-19 01:47:07 <[Tycho]> How many new TXes per second do you see currently ?
110 2012-06-19 02:00:19 <devrandom> gmaxwell: Jenkins + gitian FTW
111 2012-06-19 02:00:27 <devrandom> gmaxwell: problems with lxc?
112 2012-06-19 03:18:06 <MysteryBanshee> Has anyone thought about making a plugin system for the official bitcoin client?
113 2012-06-19 03:18:17 <MysteryBanshee> I was thinking, there might be ideas that will make bitcoin appeal to a wider audience
114 2012-06-19 03:18:46 <MysteryBanshee> With features such as skinning, built-in charts, games, flexible backup system (to USB drive, CD, network, email)
115 2012-06-19 03:18:52 <MysteryBanshee> But also I realise people dont want feature creep
116 2012-06-19 03:19:12 <MysteryBanshee> So what if the dev team made a modular plugin system where features can be turned on and off at choice?
117 2012-06-19 03:19:59 <MysteryBanshee> Is this an idea?
118 2012-06-19 03:21:02 <MysteryBanshee> gmaxwell, [Tycho], sipa, anyone?
119 2012-06-19 03:21:07 <[Tycho]> Hello.
120 2012-06-19 03:21:12 <MysteryBanshee> Hi
121 2012-06-19 03:21:31 <[Tycho]> I'm not a client developer, sorry.
122 2012-06-19 03:21:51 <MysteryBanshee> oh
123 2012-06-19 03:24:12 <gmaxwell> MysteryBanshee: Perhaps, but perhaps not it would depend on a lot of things.
124 2012-06-19 03:24:25 <MysteryBanshee> oh
125 2012-06-19 03:24:32 <MysteryBanshee> such as?
126 2012-06-19 03:24:42 <gmaxwell> MysteryBanshee: puting a lot of potentially untrusted code in the process handling the user's private keys doesn't sound wise. And things like 'skinning' would be pretty invasive to implement.
127 2012-06-19 03:25:10 <MysteryBanshee> what about if each plugin had its own process and used IPC to communicate?
128 2012-06-19 03:25:17 <gmaxwell> What might be interesting is first creating a tightly sandboxed sidecar pluging system that runs along side bitcoin and uses the RPC.
129 2012-06-19 03:25:33 <MysteryBanshee> yeh something like that
130 2012-06-19 03:25:55 <gmaxwell> That doesn't let you get skinning does it does allow things like games. Which I do think is a neat idea.
131 2012-06-19 03:26:06 <MysteryBanshee> Im just thinking, because, when people download the bitcoin client theres not alot to do... if we had like a MtGox plugin, an intersango plugin, a bitcoin-otc plugin, it would be much more intuitive
132 2012-06-19 03:26:11 <Ukto> games? 0,o build SD right into clients? :P
133 2012-06-19 03:26:16 <Ukto> create even more txn fodder?
134 2012-06-19 03:26:21 <MysteryBanshee> yes Ukto
135 2012-06-19 03:26:24 <gmaxwell> Ukto: they don't have to be TXN inefficient.
136 2012-06-19 03:26:49 <MysteryBanshee> I remember reading about probabilistic TXNs
137 2012-06-19 03:26:53 <Ukto> i think the standard client should just be a standard client
138 2012-06-19 03:26:55 <MysteryBanshee> that may be used in SD in the future
139 2012-06-19 03:27:04 <gmaxwell> MysteryBanshee: 'SD in the client' is a very bad idea and is perhaps an argument against such things.
140 2012-06-19 03:27:08 <Ukto> if someone wants to make a special client with bells/whistles/plugins, thats another deal
141 2012-06-19 03:27:15 <Ukto> but i do like the idea
142 2012-06-19 03:27:16 <Ukto> :)
143 2012-06-19 03:27:24 <gmaxwell> if SD wanted to not be so bloaty it wouldn't have to.
144 2012-06-19 03:27:37 <MysteryBanshee> Ukto: technically the client would come with nothing, its up to the user to browse/install/download what plugin he/she wants
145 2012-06-19 03:27:38 <Ukto> couldnt they just use sendmany anyways?
146 2012-06-19 03:27:43 <MysteryBanshee> perhaps with a plugin browser
147 2012-06-19 03:27:57 <gmaxwell> In any case. Making a sandboxed lua enviroment which can talk to a filtered subset of RPC.. sounds grand to me.
148 2012-06-19 03:28:16 <Ukto> hmm, isnt there a browser plugin that RPC's to the client?
149 2012-06-19 03:28:20 <gmaxwell> Ukto: they could do lots of things. Sendmany, have accounts, etc.
150 2012-06-19 03:28:23 <MysteryBanshee> gmaxwell: could you propose the idea to gavinandressen for me anyway :)
151 2012-06-19 03:28:49 <gmaxwell> he'll probably read it in the logs. But it's not something that would be high on the priority list for some time.
152 2012-06-19 03:28:49 <Ukto> MysteryBanshee: im sure he would be interested, but as he has said for other things, it would be pretty low on the list :)
153 2012-06-19 03:29:05 <gmaxwell> (The idea of plugins isn't at all a new thing)
154 2012-06-19 03:29:15 <MysteryBanshee> oh, its been proposed before?
155 2012-06-19 03:29:21 <gmaxwell> Yes, many times.
156 2012-06-19 03:29:29 <gmaxwell> If you'd personally like to work on it, thats somewhat interesting. Otherwise, meh.
157 2012-06-19 03:30:03 <MysteryBanshee> I have thought about a fork but to be honist, its beyond my skills right now to get it done securely and effieciently
158 2012-06-19 03:30:41 <gmaxwell> Trying is the best way to learn, even if you fail.
159 2012-06-19 03:31:18 <MysteryBanshee> :/
160 2012-06-19 03:32:59 <MysteryBanshee> Perhaps, lol, but I think someone with better coding skills than me would have far better luck
161 2012-06-19 03:33:15 <MysteryBanshee> Theres alot of features that would be really useful in the client that wouldnt even be hard to do... like for example, a calculator plugin
162 2012-06-19 03:33:31 <gmaxwell> probably, but no one with better coding skills is going to work on it soon.
163 2012-06-19 03:33:31 <MysteryBanshee> It would be nice if someone could write a proof of concept plugin system with like 1 or 2 plugins
164 2012-06-19 03:33:53 <MysteryBanshee> :)
165 2012-06-19 03:38:52 <MysteryBanshee> I might consider putting a bounty for it :)
166 2012-06-19 03:39:10 <MysteryBanshee> I think it might really increase the number of users, just if it had a few simple plugins to begin with, like say a BTC Faucet plugin and a calculator plugin
167 2012-06-19 03:39:35 <MysteryBanshee> And perhaps a "Bitcoin Tutorial plugin"
168 2012-06-19 03:41:38 <gmaxwell> I'd probably recommend against that.
169 2012-06-19 03:41:46 <MysteryBanshee> oh, how come gmaxwell?
170 2012-06-19 03:42:41 <gmaxwell> You'll get someone creating some quickest possible will never be merged ever thing in order to collect your bounty, and then you'll be in the uncomfortable state of either paying someone for something that isn't actually useful, or having to tell someone you won't pay them when they already put in work.
171 2012-06-19 03:42:54 <gmaxwell> Scoping out that sort of thing is an unsolved problem.
172 2012-06-19 03:43:04 <MysteryBanshee> oh
173 2012-06-19 03:43:19 <gmaxwell> And if you instead make the payment requirement getting it merged the core devs don't want pressure from people trying to get stuff merged so they can get paid.
174 2012-06-19 03:43:52 <gmaxwell> I mean if you just want to offer a tiny bounty or something, I suppose thats no issue. But it probably won't motivate anyone to do it.
175 2012-06-19 03:43:58 <MysteryBanshee> what if one of the dev's escrowed the bounty payment and a condition of the payment was that it was acceptable to x number of devs
176 2012-06-19 03:44:10 <gmaxwell> MysteryBanshee: you get the pressure thing.
177 2012-06-19 03:44:43 <MysteryBanshee> oh
178 2012-06-19 03:44:46 <gmaxwell> We've got enough to do when not dealing with people who have income incentives.
179 2012-06-19 03:44:53 <gmaxwell> (Including a backlog of pull requests)
180 2012-06-19 03:45:46 <MysteryBanshee> is the bitcoin wiki free for any one to edit?
181 2012-06-19 03:46:08 <gmaxwell> Yes, you need to make an account.
182 2012-06-19 03:46:25 <MysteryBanshee> I wont try the bounty idea then... is it ok to just propose my idea there?
183 2012-06-19 03:47:38 <freewil> the forums are a better place for that unless you're going to carefully write up a proper BIP
184 2012-06-19 03:47:58 <freewil> ... bitcoin improvement proposal
185 2012-06-19 03:48:23 <freewil> and i think this might be too client-specific for a BIP
186 2012-06-19 03:49:38 <MysteryBanshee> it would be nice if the plugins were cross-client though
187 2012-06-19 03:49:53 <MysteryBanshee> ie, you could use a plugin in Electrum or the official client
188 2012-06-19 03:50:16 <gmaxwell> Well, personally I'd like a pony. A pony made of cake. :)
189 2012-06-19 03:50:31 <MysteryBanshee> lol
190 2012-06-19 03:51:04 <freewil> that sounds nice but probably a fantasy
191 2012-06-19 03:51:36 <gmaxwell> freewil: only .. half made of cake then?
192 2012-06-19 03:52:12 <freewil> im going for the full on unicorn cake
193 2012-06-19 03:53:43 <MysteryBanshee> freewil: why a fantasy?
194 2012-06-19 03:53:59 <MysteryBanshee> if the plugin system was documented and used a popular scripting system like Lua
195 2012-06-19 03:54:03 <MysteryBanshee> why would that not be possible?
196 2012-06-19 03:54:04 <gmaxwell> MysteryBanshee: the functionality available in different clients is very different.
197 2012-06-19 03:54:10 <freewil> what about all the different gui elements
198 2012-06-19 03:54:28 <gmaxwell> I mean if the plugin had no interaction at all with bitcoin, then perhaps but whats the point of a plugin then?
199 2012-06-19 03:54:36 <MysteryBanshee> the plugin system could use its own window, and simply use a standardised protocol to communicate with the client
200 2012-06-19 03:54:48 <MysteryBanshee> gmaxwell: its purely for ease of use
201 2012-06-19 03:55:00 <freewil> why not just use rpc then
202 2012-06-19 03:55:19 <gmaxwell> freewil: it's not like anything _else_ offers the rpc interface.
203 2012-06-19 03:55:49 <MysteryBanshee> freewil: that is possible
204 2012-06-19 03:56:47 <freewil> gmaxwell, not sure what you mean there? sarcasm?
205 2012-06-19 03:57:47 <gmaxwell> freewil: No. It's a statement of fact. If it uses the rpc interface it won't be portable to other clients because the other clients have nothing like that.
206 2012-06-19 03:57:59 <freewil> oh
207 2012-06-19 03:58:19 <freewil> well i think the different guis would probably be the biggest hurdle
208 2012-06-19 03:58:47 <freewil> it'd be like trying to make a modern web app work in ie6
209 2012-06-19 03:59:15 <gmaxwell> A process seperated plugin can't be part of the same gui interface anyways, not without horribleness.
210 2012-06-19 04:00:24 <freewil> actually what if you made the gui html-based
211 2012-06-19 04:00:59 <gmaxwell> Have fun with that!
212 2012-06-19 04:02:30 <freewil> i guess you'd just have to use some sort of common gui interface
213 2012-06-19 04:02:37 <freewil> that could be html, xul, xaml whatever
214 2012-06-19 04:02:49 <freewil> just not sure how easy that would be to implement across all the clients
215 2012-06-19 04:03:06 <MysteryBanshee> freewil: HTML-based is possible indeed, there are many easy solutions
216 2012-06-19 04:03:29 <MysteryBanshee> its the scripting system that worries me, lol, how to implement that...
217 2012-06-19 04:04:07 <freewil> building a plugin system on the current satoshi client sounds like a real waste of time
218 2012-06-19 04:04:12 <freewil> plenty of more important things
219 2012-06-19 04:05:13 <MysteryBanshee> the current satoshi client is too complex to use...
220 2012-06-19 04:05:27 <gmaxwell> MysteryBanshee: two complex?!
221 2012-06-19 04:05:29 <MysteryBanshee> alot of features that are already in it, would be better as a module
222 2012-06-19 04:05:44 <gmaxwell> if it were any simpler there would just be one button that said GIVE ALL MY COIN TO SOMEONE AT RANDOM.
223 2012-06-19 04:05:46 <MysteryBanshee> or perhaps unintuitive
224 2012-06-19 04:05:53 <gmaxwell> s/two/too/
225 2012-06-19 04:06:15 <luke-jr> testnet3 is totally broken
226 2012-06-19 04:06:16 <luke-jr> sigh
227 2012-06-19 04:06:23 <gmaxwell> luke-jr: huh?
228 2012-06-19 04:06:42 <gmaxwell> luke-jr: it was adversely impacted by that bug I fixed a day ago.. but otherwise?
229 2012-06-19 04:06:42 <luke-jr> gmaxwell: it thinks there's 40k blocks to download, but never makes any progress
230 2012-06-19 04:06:57 <MysteryBanshee> gmaxwell: ok for most users here its simple, but I was thinking, how could you introduce bitcoin to a 8 year old
231 2012-06-19 04:07:02 <gmaxwell> luke-jr: Wow, wouldn't have expected that to fool you.
232 2012-06-19 04:07:11 <luke-jr> &
233 2012-06-19 04:07:24 <freewil> MysteryBanshee, 8-year olds dont need to play SD /s
234 2012-06-19 04:07:26 <gmaxwell> luke-jr: it's fully synced up. The report is because of the wonderful GUI code beleving the median of the peers claims.
235 2012-06-19 04:08:05 <gmaxwell> luke-jr: you've got some old peers, and they're telling you there is 40k blocks. In reality there are...
236 2012-06-19 04:08:19 <luke-jr> gmaxwell: I thought the magic bytes were changed so testnet3 wouldn't even talk to them?
237 2012-06-19 04:08:22 <gmaxwell> 7198.
238 2012-06-19 04:08:37 <gmaxwell> Were they?
239 2012-06-19 04:08:39 <gmaxwell> hm.
240 2012-06-19 04:08:50 <luke-jr> I have 7187 blocks
241 2012-06-19 04:09:04 <luke-jr> is there a peer I can addnode?
242 2012-06-19 04:09:09 <gmaxwell> sure.
243 2012-06-19 04:10:34 <luke-jr> can you tell me what it is?
244 2012-06-19 04:11:06 <luke-jr> <.<
245 2012-06-19 04:11:09 <gmaxwell> luke-jr: see PM
246 2012-06-19 04:11:11 <gmaxwell> (I did!)
247 2012-06-19 04:11:19 <luke-jr> o ty
248 2012-06-19 04:12:28 <gmaxwell> In any case, I don't think the network magic was changed though perhaps it should have been. But doing so would break tools that speak the network protocol.
249 2012-06-19 04:12:49 <luke-jr> (for miners)
250 2012-06-19 04:15:15 <gmaxwell> that would be nice, I have a hack now that makes testnet cpu mine when the difficulty drops.
251 2012-06-19 04:15:30 <gmaxwell> (because I was too lazy to do what you've done)
252 2012-06-19 04:15:52 <gmaxwell> might be better to even be able to set a time.. so that it only starts after a half hour or something. ::shrugs::
253 2012-06-19 04:16:37 <luke-jr> haven't done
254 2012-06-19 04:16:38 <luke-jr> just thought of
255 2012-06-19 04:16:58 <luke-jr> hmm, Bitcoin-Qt is segfaulting a lot trying to mine on it directly
256 2012-06-19 04:28:08 <MysteryBanshee> gmaxwell/anyone else interested: I have created a draft of my idea... https://en.bitcoin.it/wiki/Plug-in_System any feedback?
257 2012-06-19 04:39:00 <luke-jr> MysteryBanshee: &
258 2012-06-19 04:39:16 <luke-jr> MysteryBanshee: that's already been invented a few times
259 2012-06-19 04:39:25 <luke-jr> it's called XUL, Chromium, and a few other names I think
260 2012-06-19 04:41:16 <MysteryBanshee> luke-jr: but its never been implemented in bitcoin
261 2012-06-19 04:41:21 <MysteryBanshee> in a bitcoin client :)
262 2012-06-19 04:41:32 <luke-jr> because it doesn't belong there.
263 2012-06-19 04:41:39 <MysteryBanshee> luke-jr: it doesnt?
264 2012-06-19 04:41:43 <luke-jr> no.
265 2012-06-19 04:41:52 <MysteryBanshee> why not?
266 2012-06-19 04:42:33 <luke-jr> because Bitcoin's goal is to be a currency, not a web browser
267 2012-06-19 04:43:08 <MysteryBanshee> im not proposing it becomes a web browser
268 2012-06-19 04:43:41 <MysteryBanshee> im just proposing that different services in bitcoin are brought under a single application
269 2012-06-19 04:43:59 <MysteryBanshee> so new users dont need to go around figuring out how to do everything
270 2012-06-19 04:45:07 <MysteryBanshee> it would be better to help people when they download teh software, than to have people flock to teh forums or irc all the time
271 2012-06-19 04:46:32 <weex> MysteryBanshee: i think the main concern with the wallet is security and that plugin system would have to offer some serious benefits to counter the potential security issues
272 2012-06-19 04:47:07 <wumpus> yeah, there's no way there will be html (except the limited qt rich text subset) anywhere near the client, just too much risk. This may be a good idea for one of the other (thin) wallets applications but not the satoshi client.
273 2012-06-19 04:47:12 <gmaxwell> I mean the very first comment I made: 22:24 < gmaxwell> MysteryBanshee: puting a lot of potentially untrusted code in the process handling the user's private keys doesn't sound wise. And things like 'skinning'  would be pretty invasive to implement.
274 2012-06-19 04:47:24 <gmaxwell> and now you're suggesting using a web browser in the client.
275 2012-06-19 04:47:55 <weex> you could see if one of the other clients wants to do that though
276 2012-06-19 04:49:28 <wumpus> and if you see how much trouble even merging URL support gives :-)
277 2012-06-19 04:49:54 <wumpus> anything that opens up an attack surface...
278 2012-06-19 04:51:16 <MysteryBanshee> gmaxwell: i thought we agreed the plugins would use a seperate process
279 2012-06-19 04:51:30 <MysteryBanshee> gmaxwell: it would not handle private keys, the client does that, the plugin simply sends requests via IPC
280 2012-06-19 04:51:45 <MysteryBanshee> gmaxwell: skinning im not sure how to do that safely, but im sure it can be done
281 2012-06-19 04:51:57 <wumpus> if it can send requests through IPC it can already do everything
282 2012-06-19 04:52:00 <gmaxwell> ... except your now predicating it on the client UI being in the same browser.
283 2012-06-19 04:52:14 <gmaxwell> Which breaks the process seperation.
284 2012-06-19 04:52:18 <wumpus> "skinning" you mean changing the theme? that's already possible with qt as-is
285 2012-06-19 04:52:19 <MysteryBanshee> gmaxwell: huh?
286 2012-06-19 04:52:37 <MysteryBanshee> ok, forget skinning for now
287 2012-06-19 04:52:50 <MysteryBanshee> the client UI is not the same as the plugin UI...
288 2012-06-19 04:52:52 <gmaxwell> wumpus: you could have a plugin executor that filters the IPC, so thats not a killer issue on its own.
289 2012-06-19 04:53:02 <weex> there are some clients to the rpc interface around right?
290 2012-06-19 04:54:05 <gribble> New news from bitcoinrss: xanatos opened pull request 1485 on bitcoin/bitcoin <https://github.com/bitcoin/bitcoin/pull/1485>
291 2012-06-19 04:55:39 <wumpus> no issue is a killer issue on its own, it's just the combination of things and the added complexity and attack surface
292 2012-06-19 04:56:39 <MysteryBanshee> wumpus: and there would be many things to protect users, such as a feedback rating system on the plugins... and the option to disable the entire plugin system if a user chooses
293 2012-06-19 04:56:40 <wumpus> of course you could try to filter things, but those things tend to have subtle bugs, it's very hard to keep a sandbox airtight. And it would be a prime target...
294 2012-06-19 04:57:30 <MysteryBanshee> wumpus: most of the modules would be trusted and open-source and scrutinised heavily
295 2012-06-19 04:57:58 <MysteryBanshee> they could even be cryptographically signed by trusted developers
296 2012-06-19 04:58:04 <MysteryBanshee> before even being executed
297 2012-06-19 04:58:23 <wumpus> anyway, if there were many people dying to add features to bitcoin-qt that we are somehow unable to accept, we'd have noticed in the pull request list...
298 2012-06-19 04:58:25 <MysteryBanshee> to proove they are compiled from source
299 2012-06-19 04:59:06 <wumpus> the only thing mildly controversial at the moment is the coin control stuff, which goes very deep into the internals so wouldn't really work as a plugin anyway
300 2012-06-19 04:59:23 <MysteryBanshee> wumpus: plugin systems are often underestimated, look at winamp for example... the plugin system pretty much made that as popular as it is today... people may not see it yet, but years down the line it could make bitcoin much more popular/usable
301 2012-06-19 04:59:38 <MysteryBanshee> wumpus: it wouldnt control coins though...
302 2012-06-19 04:59:45 <MysteryBanshee> wumpus: it would simply send requests via IPC
303 2012-06-19 04:59:55 <gmaxwell> We do have a little backlog. But I don't see how plugins would address any of this at all, because some of the backloged stuff is backlogged in part because its scarry part touching.
304 2012-06-19 05:00:14 <wumpus> the backlog is generally the dangerous core stuff, not gui fun
305 2012-06-19 05:00:43 <gmaxwell> yea, the gui stuff gets merged pretty quickly in general.
306 2012-06-19 05:01:39 <MysteryBanshee> gmaxwell: plugins would reduce the bitcoin client to its minimal, and allow developers to focus on that, while other people develop plugins
307 2012-06-19 05:01:49 <MysteryBanshee> to make it more usable
308 2012-06-19 05:02:10 <wumpus> it is already pretty minimal.. it's not like we have kitchen sink syndrome
309 2012-06-19 05:02:13 <gmaxwell> They would also cure war, bringing peace to the world and save endangered speces.
310 2012-06-19 05:02:20 <gmaxwell> Think of the pandas!
311 2012-06-19 05:02:24 <wumpus> hehe
312 2012-06-19 05:02:25 <gmaxwell> Sad sad pandas.
313 2012-06-19 05:02:32 <luke-jr> sigh
314 2012-06-19 05:02:37 <MysteryBanshee> it would also reduce people making too many different clients, because instead of someone developing a new client just to add a few features they like, they can simply develop a new module instead
315 2012-06-19 05:02:40 <luke-jr> 85 more blocks till I can spend my TN3 coins
316 2012-06-19 05:02:51 <gmaxwell> luke-jr: if you wanted I would have just sent you some. :)
317 2012-06-19 05:02:58 <gmaxwell> luke-jr: gimme an address.
318 2012-06-19 05:03:18 <wumpus> the people developing their own clients usually have deep technical reasons
319 2012-06-19 05:03:31 <wumpus> not "hey, I want to add this and this feature"
320 2012-06-19 05:03:52 <luke-jr> gmaxwell: I did ask yesterday:P
321 2012-06-19 05:03:59 <luke-jr> n42rRpKcTpxkqkLBxEoqT3ddDGyLJmGZJJ
322 2012-06-19 05:04:21 <gmaxwell> luke-jr: ?!
323 2012-06-19 05:04:33 <wumpus> also I don't think it's a problem that there are more clients
324 2012-06-19 05:04:38 <luke-jr> MysteryBanshee: more clients is a good thing; one problem with Bitcoin today is the software centralization
325 2012-06-19 05:04:43 <wumpus> yep...
326 2012-06-19 05:05:01 <MysteryBanshee> luke-jr: yes, but if there are 50 clients that will just spread the developers a bit thin wouldnt it?
327 2012-06-19 05:05:32 <luke-jr> MysteryBanshee: even 2 would be an improvement
328 2012-06-19 05:05:40 <wumpus> I think you see that wrong.. the developers starting their own client would usually never join another project
329 2012-06-19 05:05:57 <luke-jr> wumpus: to be fair, I kindof did
330 2012-06-19 05:06:01 <MysteryBanshee> wumpus: yes, but with a plugin system that could change...
331 2012-06-19 05:06:22 <luke-jr> wumpus: my main reason for working on Spesmilo last year, was because it was Qt instead of wx ;)
332 2012-06-19 05:07:01 <gmaxwell> luke-jr: and you shall never be testnet hungry again.
333 2012-06-19 05:07:20 <luke-jr> gmaxwell: wow, thanks
334 2012-06-19 05:07:31 <luke-jr> gmaxwell: but you mean, until Gavin resets testnet again ;)
335 2012-06-19 05:07:46 <gmaxwell> or until I rewrite this chunk of the chain.
336 2012-06-19 05:07:48 <gmaxwell> :)
337 2012-06-19 05:07:56 <luke-jr> <.<
338 2012-06-19 05:08:14 <gmaxwell> maybe I'm already rewriting it. dum dum dum. :)
339 2012-06-19 05:09:13 <wumpus> luke-jr: yes, spesmilo was ahead of its time :)
340 2012-06-19 05:09:44 <MysteryBanshee> gmaxwell: so is the main problem with my idea just security?
341 2012-06-19 05:09:48 <wumpus> only reason for not using that was that I didn't think embedding python and pyqt4 was a good idea
342 2012-06-19 05:12:06 <luke-jr> wumpus: hah, now we have Matt embedding Python just for automatic upgrades <.<
343 2012-06-19 05:12:18 <luke-jr> MysteryBanshee: and the kitchen sink too
344 2012-06-19 05:12:18 <wumpus> yes, the problem with your idea is security (not "just" security, security should be the main requirement)
345 2012-06-19 05:12:44 <wumpus> luke-jr: in hindsight... :p
346 2012-06-19 05:13:06 <luke-jr> but more seriously, I think it'd be viable to modify Bitcoin-Qt to be embedable ;)
347 2012-06-19 05:13:10 <gmaxwell> MysteryBanshee: no the main problem with your idea is that it's missing the _most_ important part.
348 2012-06-19 05:13:22 <luke-jr> and have another program embed it and other "plugins"
349 2012-06-19 05:14:04 <gmaxwell>  a person who will actually implement something.   But beyond that, yes, security should be a prime criteria in everything we do.
350 2012-06-19 05:14:24 <luke-jr> too true
351 2012-06-19 05:14:37 <luke-jr> nothing gets anywhere without a developer behind it :p
352 2012-06-19 05:15:37 <gmaxwell> and presumably someone who actually did something would throw out half your ideas, if not more, and replace them with whatever they think would be possible/useful to implement.
353 2012-06-19 05:15:48 <gmaxwell> Ideas are cheap. Show me the code.  (says I... oh well)
354 2012-06-19 05:15:57 <wumpus> yes, people actually implementing something are also rare ... especially the people with big ideas generally disappear very fast when you ask them to implement something
355 2012-06-19 05:17:26 <MysteryBanshee> gmaxwell: well, im not sure who will do it, but i think the first step is to figure out how to design it best first
356 2012-06-19 05:17:52 <gmaxwell> And what makes you think you're qualified to do that when you don't think you're qualified to actually do the work?
357 2012-06-19 05:18:06 <MysteryBanshee> im not tbh
358 2012-06-19 05:18:17 <MysteryBanshee> which is why im asking you gmaxwell, you seem to be one of the most insightful people here
359 2012-06-19 05:18:26 <gmaxwell> (I'm not saying you're not qualified to do it I did after all encourage you to try)
360 2012-06-19 05:18:51 <gmaxwell> In any case, I'm not trying to discourage you. I think you should try. If you really don't know how you'll fail, but you'll learn a bunch of stuff in the process.
361 2012-06-19 05:19:12 <MysteryBanshee> i might give it a try, even if its just a simple system with a Hello World module that can get the balance from the client
362 2012-06-19 05:19:23 <gmaxwell> You should.
363 2012-06-19 05:20:35 <gmaxwell> What you learn will also influence what you think about the design of the idea. Or in the process you may even find something else to do instead which is a little less ambitious and more within your reach and we'll all benefit.  There is nothing but upsides to trying.
364 2012-06-19 05:25:12 <luke-jr> wumpus: some of the people implement it, and get tired of rebasing for months ;)
365 2012-06-19 05:25:29 <wumpus> luke-jr: yes, I don't think it'd be too difficult technically, still thinking about a viable IPC protocol for the UI though... json-rpc just doesn't cut it if you want events/notifications
366 2012-06-19 05:26:01 <luke-jr> wumpus: I thought of that probably over a year ago now& it got hairy
367 2012-06-19 05:26:11 <ThomasV> MysteryBanshee: do you know about libbitcoin?
368 2012-06-19 05:26:14 <luke-jr> wumpus: https://en.bitcoin.it/wiki/Wallet_protocol
369 2012-06-19 05:26:22 <gmaxwell> wumpus: just limit it to simple buttons in order to bring up a UI actually provided by the plugin.
370 2012-06-19 05:26:47 <wumpus> luke-jr: yeah I've seen that page
371 2012-06-19 05:39:27 <luke-jr> hmm
372 2012-06-19 05:39:32 <luke-jr> all my transactions show 01/01/1970
373 2012-06-19 05:39:39 <luke-jr> I think maybe refactor_times has a bug <.<
374 2012-06-19 07:07:45 <da2ce7> New Open Transactions build for Windows:  https://bitcointalk.org/index.php?topic=77301  :)
375 2012-06-19 14:20:00 <gribble> 185337
376 2012-06-19 14:20:00 <sipa> ;;bc,blocks
377 2012-06-19 14:32:34 <luke-jr> wumpus: Diapolo wants me to troll you; is there a reason you use the integer 0 to initialize pointer types? :p
378 2012-06-19 14:34:04 <luke-jr> (as opposed to the pointer-type NULL)
379 2012-06-19 14:35:32 <gmaxwell> luke-jr: At least in C 0 is defined to be acceptable as a NULL pointer (even when the null pointer isn't zero).  I personally prefer to use NULL because its more clear what I mean, but styles differ.
380 2012-06-19 14:36:07 <Diablo-D3> no its not
381 2012-06-19 14:36:14 <drizztbsd> NULL is usually a (void*) 0
382 2012-06-19 14:36:30 <Diablo-D3> gmaxwell: C does not make NULL == an address of 0
383 2012-06-19 14:36:40 <gmaxwell> Diablo-D3: Read what I wrote again.
384 2012-06-19 14:36:47 <Diablo-D3> oh
385 2012-06-19 14:36:49 <Diablo-D3> the other way
386 2012-06-19 14:36:55 <drizztbsd> Diablo-D3: but if (NULL) should return false
387 2012-06-19 14:36:57 <Diablo-D3> yes, you can set your address to any bullshit value
388 2012-06-19 14:37:08 <Diablo-D3> as long as you're consistent
389 2012-06-19 14:37:20 <Diablo-D3> drizztbsd: should, yes
390 2012-06-19 14:37:29 <Diablo-D3> I usually specifically do == NULL
391 2012-06-19 14:37:54 <drizztbsd> if (!strcmp(blablabla))
392 2012-06-19 14:37:56 <gmaxwell> Null doesn't have to be an address of zero, and it isn't on some things (mostly microcontrollers with memmapped hardware registers at zero). But on those devices the compiler is required to treat 0 pointers in the code as NULL too.  E.g. NULL==0 will return true. Which is .. mildly psycho, but there you have it.
393 2012-06-19 14:38:06 <drizztbsd> oh sorry, wrong example
394 2012-06-19 14:38:13 <drizztbsd> strcmp does not return NULL :P
395 2012-06-19 14:38:22 <drizztbsd> I mean strstr
396 2012-06-19 14:38:30 <Diablo-D3> gmaxwell: oh god thats example
397 2012-06-19 14:38:31 <Diablo-D3> er
398 2012-06-19 14:38:35 <Diablo-D3> gmaxwell: oh god that example is bad
399 2012-06-19 14:39:01 <luke-jr> gmaxwell: _wtf_
400 2012-06-19 14:39:11 <drizztbsd> often you can use MMU
401 2012-06-19 14:39:34 <Diablo-D3> see, this is why I write my code one way
402 2012-06-19 14:39:39 <Diablo-D3> "sane 32 and 64 bit platforms only"
403 2012-06-19 14:39:39 <gribble> The bot responds when you start a line with the ! character. A good starting point for exploring the bot is the !facts command. You can also visit the bot's website for a list of help topics and documentation: http://gribble.sourceforge.net/
404 2012-06-19 14:39:39 <zeiris> ;;help
405 2012-06-19 14:39:45 <sipa> so, casting the integer 0 to a pointer always yields the NULL pointer defined for that system?
406 2012-06-19 14:40:07 <Diablo-D3> but like I said, I always compare addresses properly
407 2012-06-19 14:40:09 <Diablo-D3> == NULL, etc
408 2012-06-19 14:40:15 <Diablo-D3> so its obvious what the code does
409 2012-06-19 14:40:20 <sipa> yeah
410 2012-06-19 14:40:22 <Diablo-D3> and theres nothing wrong with overly obvious code
411 2012-06-19 14:40:36 <sipa> i wish pointers were not castable to pointers, and NULL was a separately defined constant
412 2012-06-19 14:40:43 <Diablo-D3> sipa: errr
413 2012-06-19 14:40:46 <Diablo-D3> malloc void *.
414 2012-06-19 14:40:47 <sipa> eh
415 2012-06-19 14:40:48 <luke-jr> sipa: O.o?
416 2012-06-19 14:40:53 <Diablo-D3> how do you make objects now
417 2012-06-19 14:40:54 <drizztbsd> sipa: use ADA :P
418 2012-06-19 14:40:58 <sipa> *integers* not castable to pointer
419 2012-06-19 14:41:02 <luke-jr> ah
420 2012-06-19 14:41:03 <Diablo-D3> ahh
421 2012-06-19 14:41:09 <Diablo-D3> sipa: yes, but thats wrong too
422 2012-06-19 14:41:13 <luke-jr> Diablo-D3: also, C++ *does* forbid implicit casting with void* ;)
423 2012-06-19 14:41:22 <Diablo-D3> integers that arent special pointer c99 types not castable back and forth
424 2012-06-19 14:41:22 <sipa> indeed
425 2012-06-19 14:41:32 <Diablo-D3> like uintptr_t and shit
426 2012-06-19 14:42:38 <jrmithdobbs> yes they are, it just spits warnings according to the spec, it's not an error
427 2012-06-19 14:43:24 <sipa> jrmithdobbs: ok
428 2012-06-19 14:43:27 <sipa> could be
429 2012-06-19 14:44:01 <TD> sipa: NULL is defined to be __null in c++
430 2012-06-19 14:44:47 <jrmithdobbs> it should be avoided, of course, in most circumstances unless you're 100% sure of the results and doing it any other way would add complexity, but we all know that
431 2012-06-19 14:44:50 <jrmithdobbs> heh
432 2012-06-19 14:45:36 <gribble> I do not know about 'thash', but I do know about these similar topics: 'trade'
433 2012-06-19 14:45:36 <zeiris> ;;thash
434 2012-06-19 14:45:39 <gribble> Error: "total" is not a valid command.
435 2012-06-19 14:45:39 <zeiris> ;;total
436 2012-06-19 14:45:43 <gmaxwell> jrmithdobbs: wasn't it you that had that capabilities patch for bitcoin a while back?
437 2012-06-19 14:45:46 <gribble> I do not know about 'blockchain', but I do know about these similar topics: 'blockchainsnapshot'
438 2012-06-19 14:45:46 <zeiris> ;;blockchain
439 2012-06-19 14:45:59 <gribble> Error: "difficulty" is not a valid command.
440 2012-06-19 14:45:59 <zeiris> ;;difficulty
441 2012-06-19 14:46:00 <jrmithdobbs> gmaxwell: ya i think i accidentally killed that repo
442 2012-06-19 14:46:20 <jrmithdobbs> gmaxwell: i just added a function in main before logs or anything were open to do it, why?
443 2012-06-19 14:46:41 <gmaxwell> jrmithdobbs: Have you looked at seccomp mode 2 yet? It's pretty awesome. You install a berkley packet filter to ACL every syscall the app makes. Other than writing the filter it's as easy to use as capabilities.
444 2012-06-19 14:46:55 <gmaxwell> ( http://outflux.net/teach-seccomp/ )
445 2012-06-19 14:47:22 <jrmithdobbs> gmaxwell: no I haven't because you can already do similar with pf and most of my personal machines are free/openbsd ;p
446 2012-06-19 14:48:00 <gmaxwell> jrmithdobbs: huh? PF filters syscalls? E.g. preventing apps from performing I/O outside of their dot directory?
447 2012-06-19 14:48:09 <jrmithdobbs> oh i misread
448 2012-06-19 14:48:26 <jrmithdobbs> gmaxwell: looking now
449 2012-06-19 14:50:02 <jrmithdobbs> gmaxwell: ewww
450 2012-06-19 14:50:05 <jrmithdobbs> gmaxwell: that shit is ugly
451 2012-06-19 14:50:14 <jrmithdobbs> gmaxwell: cool concept, ugly api
452 2012-06-19 14:51:03 <jrmithdobbs> gmaxwell: so you have to define it in the code? that's not like capabilities at all
453 2012-06-19 14:51:23 <gmaxwell> jrmithdobbs: you can have a launcher that does the sandboxing.
454 2012-06-19 14:51:33 <gmaxwell> They persist through exec, so...
455 2012-06-19 14:51:47 <gmaxwell> (and you can make them immutable)
456 2012-06-19 14:52:23 <jrmithdobbs> similar end result but completely app/code defined but not system/environment custmoizable without a wrapper at which point it almost might as well be an suid wrapper as far as the sys admin is concerned (there are more benefits to it, but the complexity of it is back to that of suid wrapper binaries instead of which capabilities how you can mark them in filesystem metadata)
457 2012-06-19 14:52:46 <jrmithdobbs> s/which capabilities/with capabilities/
458 2012-06-19 14:53:17 <jrmithdobbs> gmaxwell: caps persist through exec and can be made immutable as well so it's a different api for the same thing?
459 2012-06-19 14:54:23 <jrmithdobbs> it is much more fine grained though, which is cool
460 2012-06-19 14:54:34 <gmaxwell> Again, caps don't express anywhere near this level of flexiblity. You get full matching on all syscalls. So you can fully sandbox things. E.g. "append only for these files"
461 2012-06-19 14:54:37 <gmaxwell> right.
462 2012-06-19 14:55:07 <jrmithdobbs> well, a *lot* of the cool stuff you could do with it can be done in other long-accepted well defined ways
463 2012-06-19 14:55:44 <jrmithdobbs> but ya, i'll keep an eye on this, it could be very handy with a runit-like wrapper/helper app
464 2012-06-19 14:56:02 <jrmithdobbs> it's more the framework for cool stuff to come from instead of the resulting coll stuff ;p
465 2012-06-19 14:56:08 <jrmithdobbs> s/coll/cool/
466 2012-06-19 14:56:15 <gmaxwell> Yes, and in linux a lot of things this can do can be done with SELinux for example. But thats rather system level, while this can be dynamic. E.g. you could have he sandboxed app have a socket open to a helper which it asks to modify its filters.
467 2012-06-19 14:56:44 <jrmithdobbs> gmaxwell: ya, grsec and apparmor also provide a good chunk of this functionality
468 2012-06-19 14:56:49 <gmaxwell> so you could even have crazy stuff like the sandbox trapping on fault and prompting the user for permission to run the blocked syscall.
469 2012-06-19 14:57:00 <jrmithdobbs> gmaxwell: always interesting to see different ways of doing it, though
470 2012-06-19 14:57:01 <jrmithdobbs> for sure
471 2012-06-19 14:57:29 <gmaxwell> In any case, I thought you'd find it interesting. It was only recently added to the linux kernel, so the only distro with it yet is ubuntu 12.04.
472 2012-06-19 14:57:34 <jrmithdobbs> gmaxwell: ya, it does seem to provide a much nicer failure branch for elevated syscalls in user code
473 2012-06-19 14:58:08 <jrmithdobbs> gmaxwell: i'll keep an eye and watch for fun utilities for it because we're about to migrate an assload of stuff to 12.04
474 2012-06-19 14:58:45 <jrmithdobbs> gmaxwell: because we're sick of the shittastic xfs and nfs perf in <.32 ;p
475 2012-06-19 14:58:58 <jrmithdobbs> (what i mean is, we don't just upgrade to upgrade)
476 2012-06-19 15:15:38 <gribble> 185342
477 2012-06-19 15:15:38 <[Tycho]> ;;bc,blocks
478 2012-06-19 15:20:53 <gmaxwell> kinda odd, ran callgrind for an hour on a node not processing free txn... and basically all of the time it logged was time in loadblockindex on startup... which apparently makes about 533k calls to sha256.
479 2012-06-19 15:21:30 <gmaxwell> oh bleh. I bet it didn't follow the threads.
480 2012-06-19 15:24:53 <luke-jr> still, should LoadBlockIndex be doing SHA256sums?
481 2012-06-19 15:25:48 <gmaxwell> yea, this profile is still useful, just not what I wanted to measure:
482 2012-06-19 15:26:34 <sipa> luke-jr: i was positively surprised when i discovered that
483 2012-06-19 15:27:08 <sipa> luke-jr: it means someone can't give you a blockchain db, with a faked best chain in it
484 2012-06-19 15:27:15 <sipa> as that would require PoW
485 2012-06-19 15:27:19 <luke-jr> ah
486 2012-06-19 15:28:03 <phantomcircuit> PoW?
487 2012-06-19 15:28:14 <luke-jr> phantomcircuit: you shouldn't have to ask :P
488 2012-06-19 15:28:33 <sipa> phantomcircuit: proof of work
489 2012-06-19 15:28:35 <sipa> phantomcircuit: mining
490 2012-06-19 15:29:15 <gmaxwell> sipa: sure but they can give you a chain with a randomly corrupted index.
491 2012-06-19 15:29:20 <gmaxwell> So, "meh"
492 2012-06-19 15:29:38 <phantomcircuit> if you sent someone a fake chain they would only be fooled until they connected to the first peer that told them otherwise
493 2012-06-19 15:29:51 <phantomcircuit> and i think a reorg that large causes the client to simply exit
494 2012-06-19 15:29:55 <sipa> phantomcircuit: not if the faked chain has more work in it
495 2012-06-19 15:29:58 <gmaxwell> https://people.xiph.org/~greg/bitcoin-start.png < here is the time spent on startup.
496 2012-06-19 15:30:16 <phantomcircuit> sipa, shrug if you did more work you could just do it on the open network anyways
497 2012-06-19 15:30:53 <sipa> phantomcircuit: my point is that since bitcoin checks the PoW at startup, you cannot create fake work by giving a corrupt db to someone
498 2012-06-19 15:31:10 <phantomcircuit> sipa, yeah i was agreeing
499 2012-06-19 15:31:18 <sipa> phantomcircuit: sorry, i didn't notice you agreed :)
500 2012-06-19 15:31:40 <sipa> gmaxwell: then again, i don't want to claim we're safe when people start accepting untrusted block db's
501 2012-06-19 15:31:41 <phantomcircuit> gmaxwell, std::vector range_insert is weird
502 2012-06-19 15:32:08 <phantomcircuit> that should mostly be an append to a linked list which should be very cheap
503 2012-06-19 15:32:24 <sipa> std::vector is not a linked list
504 2012-06-19 15:32:32 <gmaxwell> It also spends about 20% of its time computing the difficulty of the blocks.
505 2012-06-19 15:33:09 <gmaxwell> phantomcircuit: well this is callgrind output.. it's estimates can be a little wonky. It has a very simplified view of instruction costs.
506 2012-06-19 15:33:10 <phantomcircuit> oh it's an array that's right
507 2012-06-19 15:33:27 <gmaxwell> It's mostly useful for looking at algorithmic complexity.
508 2012-06-19 15:33:53 <phantomcircuit> i wonder if there is anywhere that an std::vector is being used with insert
509 2012-06-19 15:34:05 <phantomcircuit> cause that can be super ridiculously expensive
510 2012-06-19 15:34:20 <sipa> in script it happens
511 2012-06-19 15:34:36 <phantomcircuit> shouldn't be running the scripts on startup though should it?
512 2012-06-19 15:34:47 <sipa> oh, no
513 2012-06-19 15:34:49 <sipa> certainly not
514 2012-06-19 15:34:58 <phantomcircuit> and the scripts are short enough that they'll fit in L2
515 2012-06-19 15:35:03 <phantomcircuit> so that should mostly not matter
516 2012-06-19 15:35:34 <phantomcircuit> but if there are inserts happening on anything that doesn't fit in L2 that could be a serious performance issue
517 2012-06-19 15:35:54 <Eliel> SHA-256 calculations take a bigger chunk than I'd have expected.
518 2012-06-19 15:36:05 <gmaxwell> That vector:range_insert is in CDatastream::write
519 2012-06-19 15:36:38 <sipa> oh, right
520 2012-06-19 15:36:48 <phantomcircuit> possibly the serializer should be changed to use a stack
521 2012-06-19 15:39:25 <phantomcircuit> also the sha256 ops are still naive
522 2012-06-19 15:39:37 <phantomcircuit> they should probably have runtime checks for cpu features
523 2012-06-19 15:39:45 <phantomcircuit> but that's getting into big risk of fuck ups
524 2012-06-19 15:40:03 <gmaxwell> I think we shouldn't be doing that sha256 at every startup. When someone gives you a chain you should loadblocks it. We should have a signature at the top of the blocks file to make it so that it knows if its foreign and then it does the check once or something like that.
525 2012-06-19 15:40:06 <Eliel> ah, they're unoptimized too?
526 2012-06-19 15:40:12 <gmaxwell> phantomcircuit: they wouldn't be subtle screwups.
527 2012-06-19 15:40:30 <gmaxwell> "bitcoin can't validate the genesis block on my computer!"  no big deal.
528 2012-06-19 15:40:43 <phantomcircuit> gmaxwell, well it depends
529 2012-06-19 15:40:56 <phantomcircuit> it could be something that happens to only some fraction of blocks
530 2012-06-19 15:41:09 <phantomcircuit> so like the first 45k blocks would be fine but 45001 wouldn't
531 2012-06-19 15:41:53 <Eliel> well, the good thing about hash-algo bugs is that it won't look like it's working if it's not :)
532 2012-06-19 15:42:04 <Eliel> it'll be obvious it's failing :)
533 2012-06-19 15:42:17 <gmaxwell> phantomcircuit: thats pretty unlikely. Also, testnet has a max size block in it now.
534 2012-06-19 15:42:21 <sipa> gmaxwell: the txout dataset needs one more element: for coinbase transaction, you need to keep the height as well, to know when they become spendable
535 2012-06-19 15:43:15 <sipa> gmaxwell: combined with a few extra optimizations for amounts and low-txout-count transactions, the size is still only 65 MB
536 2012-06-19 16:00:16 <PK> odd, after crosscompiling bitcoin 0.6.2 for arm it just segfaults when I try to run it. I even recompiled all dependencies from scratch.
537 2012-06-19 16:03:05 <jrmithdobbs> PK: is your arm flavor big, little, or dual endian? ;p
538 2012-06-19 16:03:11 <amiller> i've never looked at patricia trees before
539 2012-06-19 16:03:16 <jrmithdobbs> PK: actually, it doesn't matter, you're going to need code changes unless it's little endian. lots of them. (unless you can flip it at compile time like on some arm platforms)
540 2012-06-19 16:03:17 <amiller> does that really give you O(k) worst case?
541 2012-06-19 16:04:11 <amiller> if we're storing 256 bit hashes, then 256 operations is a maximum lookup cost regardless of the size, up to 2^256 possible entries?
542 2012-06-19 16:04:25 <amiller> i guess log N <= k in any case
543 2012-06-19 16:04:48 <MysteryBanshee> blockchain.info still havent replied back to my 2 emails!
544 2012-06-19 16:04:52 <MysteryBanshee> :(
545 2012-06-19 16:05:30 <PK> I wish I had gdb on the arm box :( or at least a hint at the error.
546 2012-06-19 16:05:36 <gmaxwell> amiller: yea, level compressed tries with high fanout and probably pretty good for this sort of thing. But meh. libjudy is 30,000 lines of dense code.
547 2012-06-19 16:06:29 <amiller> one of the points made in the General Model for Authenticated Data Strucutres paper is that the fanout doesn't need to be related to the logical tree
548 2012-06-19 16:06:40 <amiller> you can take a red black tree and store it as an equivalent 2-3 tree or b-tree
549 2012-06-19 16:07:30 <amiller> the wikipedia page for Radix tree doesn't seem too complicated, so i'm not sure what the 30,000 lines are for unless there's more to it in libjudy...
550 2012-06-19 16:08:24 <gmaxwell> amiller: it does a lot of fancy stuff to control the tree shape to maximize cache locality.
551 2012-06-19 16:09:13 <gmaxwell> Considering how precious icache is these days its impressive that it performs so well in spite of its size.
552 2012-06-19 16:12:13 <PK> now it runs... removing PIE=1 made a difference.
553 2012-06-19 16:13:09 <luke-jr> [17:42:21] <sipa> gmaxwell: the txout dataset needs one more element: for coinbase transaction, you need to keep the height as well, to know when they become spendable
554 2012-06-19 16:13:12 <amiller> so it looks like a merkle trie would also require worst-case O(log N) effort, but it would be much simpler to balance, and you'd get the (useless) property that etotheipi likes, where any two trees with the same leaves are identical
555 2012-06-19 16:13:26 <luke-jr> sipa: instead, it should be safe to just keep track of the last ~200 coinbase txids?
556 2012-06-19 16:14:08 <luke-jr> perhaps when connecting a block, just store its coinbase txid in a 0x100 entry table at the height modulo 0x100 position
557 2012-06-19 16:15:19 <luke-jr> even if you need to add it to the txout dataset for hash lookup purposes, you can at least cut it down to 1 octet this way
558 2012-06-19 16:30:34 <gmaxwell> amiller: I don't think that property is _useless_. I'd prefer it too if we could have it without giving up anything else.
559 2012-06-19 16:31:13 <amiller> how could you come into posession of a set of O(N) data that you _trust_, without also having an O(1) root hash that you trust?
560 2012-06-19 16:32:21 <sneak> does theymos irc?
561 2012-06-19 16:33:48 <gmaxwell> amiller: by following the chain from the start?  of course then you get the same structure anyways because you make the whole algorithim normative.
562 2012-06-19 16:34:17 <amiller> right, so if you follow the chain from the start, then you get the same structure
563 2012-06-19 16:34:30 <amiller> etotheipi keeps using the phrase 'construct the tree from scratch'
564 2012-06-19 16:34:35 <amiller> as in two people 'constructing the tree from scratch' will get identical trees
565 2012-06-19 16:34:46 <amiller> to construct a merkle tree from scratch, you must first invent the universe.
566 2012-06-19 16:34:47 <gmaxwell> amiller: the other advantage of the trie stuff is that its very space efficient. Though I still worry about creating local imbalances, though I guess the worst case isn't so bad.
567 2012-06-19 16:34:58 <amiller> space efficiency is entirely orthogonal
568 2012-06-19 16:35:02 <amiller> it doesn't matter how you store the trees
569 2012-06-19 16:35:47 <gmaxwell> amiller: I mean space efficiency for a fast lookup friendly structure, since you know we want people to actually perform queries.. and not just that perform queries for third parties hopefully for free.
570 2012-06-19 16:37:42 <amiller> my point is that doesn't need to be part of the normative protocol
571 2012-06-19 16:37:50 <amiller> there are all sorts of different ways to efficiently store the same logical tree
572 2012-06-19 16:38:09 <amiller> no need to all have the same format there
573 2012-06-19 16:38:56 <gmaxwell> No, it doesn't. But whatever is normative needs to be very friendly with at least one efficient representation.
574 2012-06-19 16:38:58 <amiller> :]
575 2012-06-19 16:39:11 <amiller> yo etotheipi_
576 2012-06-19 16:39:18 <etotheipi_> I figured I should just come here instead of spamming the mailing list with this
577 2012-06-19 16:39:40 <amiller> lets hash this out!
578 2012-06-19 16:39:43 <etotheipi_> it's just that when I get on IRC, I tend to lose a couple hours of my life
579 2012-06-19 16:39:44 <etotheipi_> :)
580 2012-06-19 16:39:56 <gavinandresen> not lost, recorded for posterity
581 2012-06-19 16:40:07 <amiller> irc time is deductible, you get reimbursed for it at the gates of heavne
582 2012-06-19 16:40:33 <etotheipi_> heh, is that how it works?
583 2012-06-19 16:40:44 <gmaxwell> Unfortunately IRC time is the opposite of time spent exercising, every hour on IRC subtracts two from your life.
584 2012-06-19 16:41:00 <gmaxwell> but oh what good hours they are.
585 2012-06-19 16:41:19 <etotheipi_> so am I the only one who doesn't like history-dependent tree structures?
586 2012-06-19 16:41:41 <BlueMatt> luke-jr: you dont, thats kinda the point...
587 2012-06-19 16:41:52 <etotheipi_> or rather, tree structures that have multiple valid representations for a given set of inputs
588 2012-06-19 16:42:01 <BlueMatt> sipa: no, jenkins uses the makefiles in git, and uses the default, is ipv6 not default?
589 2012-06-19 16:42:13 <gmaxwell> etotheipi_: Well they don't because the history of updates is the input.
590 2012-06-19 16:42:17 <amiller> there's only one representation for an ordered set of inputs
591 2012-06-19 16:44:19 <amiller> i think you are beginning your scenario with "suppose you are in possession of the unordered set of N currently-valid unspent outputs" and then you want to build the tree
592 2012-06-19 16:44:44 <amiller> my question is, under what kind of circumstance could you acquire that set of outputs?
593 2012-06-19 16:45:09 <etotheipi_> amiller: that's exactly right -- and not because I know of a particular situation that could lead to that (the future may hold lots of situations we don't know about yet)...
594 2012-06-19 16:45:24 <amiller> either you have processed the blockchain history yourself, from the genesis, or else you are trusting someone else to give you the _correct_ set of N unspent outputs
595 2012-06-19 16:45:50 <amiller> or something in the middle, where you trust someone's opinion for time T, but you process all the transactions yourself from time T up to time T+M
596 2012-06-19 16:46:07 <etotheipi_> it just seems like an unnecessary restriction to require not only all the tree nodes, but all the past tree nodes and the specific order of insertion and deletion
597 2012-06-19 16:46:13 <gmaxwell> My own belief that this was a requirement was removed by amiller pointing out to me that you can do provable not founds on queries without it.
598 2012-06-19 16:47:06 <etotheipi_> gmaxwell: I don't quite follow
599 2012-06-19 16:48:40 <gmaxwell> etotheipi_: I'd previously thought you could only be convinced that the tree with root X did not contain a particular leaf if you could know exactly where it was placed without having the whole tree. (e.g. if the ordering of the tree only depended on the direct data itself.
600 2012-06-19 16:49:18 <gmaxwell> amiller pointed out that a peer can trace a query through the tree and give you the hash nodes that you'd hit along the trace, which would prove the query was correct.
601 2012-06-19 16:50:30 <gmaxwell> as far as why the tree a function of inputs only is irrelevant: Either you're building the tree from scratch, so you can just get the same result by replaying the ordered updates. OR you trust someone to tell you the identity of the tree, so they can just give you the sate at some point in time.
602 2012-06-19 16:50:45 <jjjx> From an anonymity perspective, what are the benefits of routing Bitcoin through Tor? At what point is it possible to see what IP address a given Bitcoin address is 'receiving' from?
603 2012-06-19 16:51:00 <gmaxwell> Now, if we can have the tree be a pure function of the data in it, I agree thats better all things equal.
604 2012-06-19 16:51:21 <etotheipi_> well that's a feature of just about any of these structures, isn't it?  You can basically ask for nodes between a lower bound and upper bound of your desired leaf node and confirm the validity
605 2012-06-19 16:51:21 <gmaxwell> jjjx: http://blockchain.info/ip-address/206.71.179.116
606 2012-06-19 16:52:10 <jjjx> gmaxwell: I see. Is this data stored in the blockchain itself, or simply captured by services like blockchain.info?
607 2012-06-19 16:52:11 <gmaxwell> etotheipi_: yea I was stupid for a moment because I hadn't considered that the intermediate nodes which aren't a function of the data would be part of the committed structure.
608 2012-06-19 16:52:23 <gmaxwell> jjjx: The latter, and anyone else who wants to.
609 2012-06-19 16:52:42 <etotheipi_> gmaxwell: so you're saying that you originally felt the way I do, and mainly because you were concerned about being able to prove that a node also *wasn't* in the tree
610 2012-06-19 16:53:10 <gmaxwell> etotheipi_: Correct. But now I see that you don't have to have a data-determinstic structure for that.
611 2012-06-19 16:53:16 <etotheipi_> gotcha
612 2012-06-19 16:54:24 <gmaxwell> So now I'd still prefer what you want because it seems easier to test to me. But I suspect some alternative may give us better tradeoffs in some other dimension (e.g. query speed, space, resistance to worst cases in these).
613 2012-06-19 16:55:34 <amiller> etotheipi_, i think where you're at now, you still feel like there's an example that might be able to be constructed, but i don't think you agree yet with what gmaxwell and i are saying, which is that _any possible_ example will either involve an iteration from genesis, or a trusted starting point somewhere
614 2012-06-19 16:55:57 <gmaxwell> Though n-way tries are usually very space efficient.. so.. I dunno.
615 2012-06-19 16:56:08 <etotheipi_> amiller: I agree with that statement almost exactly
616 2012-06-19 16:56:12 <etotheipi_> which is why I wanted more input
617 2012-06-19 16:56:53 <etotheipi_> I believe they're both valid arguments -- but I feel that there's a great degree of inelegance in the history-based tree that I can't quantify... I figured gmaxwell would
618 2012-06-19 16:56:54 <etotheipi_> :)
619 2012-06-19 16:57:15 <amiller> okay well deferring that for now, the trie is really interesting
620 2012-06-19 16:57:21 <amiller> it's far simpler than red black balancing, which is kind of a bitch
621 2012-06-19 16:57:36 <gmaxwell> Yea, I can't come up with anything really compelling for amiller's question there.  But I'd also offer amiller the counter: What do we gain from a structure which doesn't allow determinstic generation from the 'unordered' set of inputs?
622 2012-06-19 16:57:39 <amiller> the worst case is worse than with the balanced tree
623 2012-06-19 16:57:47 <etotheipi_> right... but tries are *not* so space efficient
624 2012-06-19 16:58:02 <gmaxwell> Yes, but is it a worst case that requires colliding all of sha256 ?  I'm not too worried about that.
625 2012-06-19 16:58:15 <gmaxwell> In fact I _strongly_ prefer worst cases that reduce to "attack the hash function".
626 2012-06-19 16:58:27 <amiller> i'm just talking about worst case path length
627 2012-06-19 16:58:27 <etotheipi_> if you use a raw trie, branching on the next byte, each branch node requires storing 256 pointers, most of which will be null after the first few levels
628 2012-06-19 16:58:36 <amiller> with a trie the worst case path length is k > log N
629 2012-06-19 16:58:43 <etotheipi_> amiller: k is fixed
630 2012-06-19 16:58:52 <gmaxwell> amiller: right but to get that you need hashes which share a lot of common bytes.
631 2012-06-19 16:59:08 <amiller> and if you have fewer than that, then the balanced tree has a better worst case
632 2012-06-19 16:59:13 <etotheipi_> k = 256, or 32, depending on whether you branch on bytes or bits
633 2012-06-19 16:59:15 <amiller> etotheipi_, k may be fixed, but log N is bounded
634 2012-06-19 16:59:28 <amiller> log N <= k
635 2012-06-19 17:00:13 <etotheipi_> I don't understand:  we have 1e+73 nodes in the tree:  it still takes exactly 32 hops to find your data (or insert or delete)
636 2012-06-19 17:00:23 <etotheipi_> same if you only have 1 node
637 2012-06-19 17:01:32 <amiller> if we have more than 2^256 elements, then we have a big problem (guaranteed hash collision)
638 2012-06-19 17:01:40 <etotheipi_> yes yes...
639 2012-06-19 17:01:55 <etotheipi_> I'm just making the point that access/insert/delete time is exactly 32 under all circumstances
640 2012-06-19 17:02:04 <etotheipi_> (with a raw trie)
641 2012-06-19 17:02:15 <amiller> if you are branching by bytes rather than bits, then you also have buckets to worry about
642 2012-06-19 17:02:27 <gmaxwell> etotheipi_: as far as my libjudy comment goes it's a well known level-compressed (256-ary) trie implementation which has excellent performance due to a lot of care taken to preserve cache locality. ... and its implementation is 30kloc of C. 0_o
643 2012-06-19 17:03:05 <amiller> so you've got 32 + log N / 32
644 2012-06-19 17:03:44 <jgarzik> sorry to intrude...  has anyone successfully profiled "block propagation", i.e. traced the delay?
645 2012-06-19 17:03:45 <etotheipi_> gmaxwell: so it sounds like a further-optimized PATRICIA tree
646 2012-06-19 17:04:03 <etotheipi_> amiller: which buckets?
647 2012-06-19 17:04:25 <gmaxwell> (oh actually seems it isn't actually level-compressed)
648 2012-06-19 17:04:29 <gmaxwell> etotheipi_: correct.
649 2012-06-19 17:04:42 <jgarzik> I'm wondering if the network simply needs well-placed nodes running on SSD + signature cache?  or maybe we're just doing a whole lot of SHA256 per block?
650 2012-06-19 17:05:06 <amiller> etotheipi_, suppose you branch by 256 bits
651 2012-06-19 17:05:11 <amiller> then you only have to do one operation?
652 2012-06-19 17:05:50 <gmaxwell> jgarzik: well I have a node running in callgrind for a profile right now.  It's hard to say, the propagation times being observed in the wild don't seem justifyable with any obvious causes.
653 2012-06-19 17:06:24 <gmaxwell> jgarzik: e.g. block processing is _slow_ even on tmpfs+fast cpu. .. where slow means, seconds. But seconds per hop don't really explain multiminute propagations.
654 2012-06-19 17:06:46 <etotheipi_> gmaxwell: if we were to use a raw trie: then we'd be storing 256, 8-byte pointers at every level -- each node would take at least 2048 bytes -- and there are exactly 32-nodes on the way to any given leaf.  The top 4 levels will have a high degree of sharing with other branches, but the bottom 28 will be mostly used to support single leafs
655 2012-06-19 17:07:11 <etotheipi_> therefore, if we were to even consider a trie- structure, I think it would *have* to be level-compressed
656 2012-06-19 17:07:38 <etotheipi_> which then gets out of the realm of "straightforward" in terms of implementation
657 2012-06-19 17:07:48 <gmaxwell> etotheipi_: you can switch the order when the trie becomes too unsparse, I think thats what judy does.. been a long time.
658 2012-06-19 17:08:37 <jgarzik> gmaxwell: one of my background projects is a bitcoin backbone.  I've been wanting to set up hardware at key network points around the world, just running full peer nodes; making sure to have direct connections to popular pools and sites, etc.
659 2012-06-19 17:09:13 <etotheipi_> amiller: that would be a hell of a root node!
660 2012-06-19 17:09:21 <gmaxwell> jgarzik: sounds like a grand idea to me.
661 2012-06-19 17:10:03 <gmaxwell> jgarzik: did you see what luke was proposing wrt relaying prior to validation. That would really cut the delay down for with pretty much any cause.
662 2012-06-19 17:10:32 <etotheipi_> amiller: perhaps I just described your concern about tries in my previous comment. .. it's my concern too, which is why I wasn't taking them too seriously
663 2012-06-19 17:10:53 <amiller> it just ends up working out to log N
664 2012-06-19 17:11:34 <etotheipi_> what works out to log(N)?
665 2012-06-19 17:11:35 <jgarzik> if I had enough resources, I'd build a -nolisten bastion network to popular sites, then connect those to separate, public nodes for added security
666 2012-06-19 17:12:14 <jgarzik> gmaxwell: relay-before-validate seems very interesting, has anyone thought through the consequences?
667 2012-06-19 17:12:45 <jgarzik> internally, it should be straightforward to create a pre-validated holding pen for blocks
668 2012-06-19 17:12:55 <jgarzik> and then pipeline those through ProcessBlock
669 2012-06-19 17:12:59 <BlueMatt> jgarzik: done
670 2012-06-19 17:13:21 <jgarzik> BlueMatt: cool... part of CBlockStore?  link?
671 2012-06-19 17:13:35 <jgarzik> seems like something easily offload-able to a thread at that point
672 2012-06-19 17:13:41 <gmaxwell> jgarzik: it got discussed some. There is some minor attack potential, but it has the cost of producing valid block headers because we can at least do all the cheap validaion first.
673 2012-06-19 17:14:38 <amiller> if there is only a single root node (branch by 256 bits) then you might have to store up to N <= 2^256 elements in that node
674 2012-06-19 17:14:54 <BlueMatt> jgarzik: https://github.com/TheBlueMatt/bitcoin/commit/07672222debe0584a136babe43b5fc7803c452b3
675 2012-06-19 17:15:24 <amiller> if you only branch by one bit, then you only need to store on item per node, you only have to do worst case k work, but there can't be more than 2^k elements anyway
676 2012-06-19 17:15:31 <BlueMatt> jgarzik: (and, yes, its offloaded to another thread)
677 2012-06-19 17:15:55 <amiller> there's a tradeoff between the size of k (the branch size) and the number of elements that must be stored in each node
678 2012-06-19 17:16:22 <etotheipi_> amiller: I'm with you
679 2012-06-19 17:16:50 <amiller> no where in between can you get better than log N worst case, given that N <= 2^256
680 2012-06-19 17:16:54 <luke-jr> BlueMatt: I need to :p
681 2012-06-19 17:17:02 <BlueMatt> luke-jr: why?
682 2012-06-19 17:17:11 <luke-jr> BlueMatt: CheckNewBlock
683 2012-06-19 17:17:22 <BlueMatt> link?
684 2012-06-19 17:18:11 <luke-jr> https://github.com/bitcoin/bitcoin/pull/1246
685 2012-06-19 17:21:32 <BlueMatt> luke-jr: let me rephrase: why do you need to call ConnectBlock directly instead of using ProcessBlock/EmitBlock?
686 2012-06-19 17:22:16 <etotheipi_> amiller, give me a sec
687 2012-06-19 17:22:50 <amiller> take your time, i'm having a blast :D (love me some merkle trees)
688 2012-06-19 17:23:05 <Eliel> my feeling about a trie is that it sounds horribly space inefficient. Even with level compression.
689 2012-06-19 17:23:06 <luke-jr> BlueMatt: ProcessBlock does way too much
690 2012-06-19 17:23:08 <BlueMatt> luke-jr: if you are adding stuff to just check that inputs are connect-able in ConnectBlock, why not do that for ProcessBlock instead so that you get the additional checks in there?
691 2012-06-19 17:23:27 <etotheipi_> amiller: we can agree about that, then
692 2012-06-19 17:23:47 <amiller> :]
693 2012-06-19 17:23:47 <etotheipi_> I'm one of the rare few that actually enjoyed data structures in college
694 2012-06-19 17:24:26 <etotheipi_> so amiller, I just want to clarify:  there are two different aspects to consider: space efficiency and access efficiency
695 2012-06-19 17:24:37 <luke-jr> BlueMatt: ProcessBlock writes it to disk
696 2012-06-19 17:24:44 <jjjx> How much did the core development team grow since the craziness of the 'bubble' and all the media hype surrounding it? Did many new developers flock to the project? Did any of them stick around?
697 2012-06-19 17:24:47 <gmaxwell> Eliel: the actual node data can up in the intermediate nodes, this can make them very space efficient.
698 2012-06-19 17:24:55 <BlueMatt> luke-jr: ok, so add a fDontCheckWork that doesnt check work and doesnt write to disk?
699 2012-06-19 17:25:36 <amiller> there are also two modes to consider, the normative specification, and everything else that's optional
700 2012-06-19 17:25:55 <amiller> the normative spec needs to be about worst case bounds, but practical efficiency can be handled all sorts of ways
701 2012-06-19 17:26:02 <amiller> storage, O(N), access, O(log N), worst case
702 2012-06-19 17:26:05 <BlueMatt> luke-jr: anyway, the point of cblockstore is to encapsulate crap that no one should ever access directly...anyway, Im off
703 2012-06-19 17:26:21 <amiller> underneath the big O, there are all sorts of different ways to store and access the trees on a real disk
704 2012-06-19 17:26:27 <amiller> it's basically orthogonal to the balancing rules
705 2012-06-19 17:26:43 <gmaxwell> amiller: I don't believe you about orthogonal.
706 2012-06-19 17:26:48 <amiller> take a red black tree, store 32 nodes at a time
707 2012-06-19 17:26:56 <amiller> now if you have to update any of those nodes, you need to update all 32 in a block
708 2012-06-19 17:27:14 <amiller> or just cache however many of the top nodes you need, since you're going to be accessing those the most
709 2012-06-19 17:27:33 <amiller> all of the optimizations come from cache locality in some sense, chunking node data together
710 2012-06-19 17:29:56 <amiller> http://truthsayer.cs.ucdavis.edu/algorithmica.pdf this paper describes IO efficient schemes for storing the merkle search structures
711 2012-06-19 17:30:17 <amiller> it's a b-tree
712 2012-06-19 17:31:10 <amiller> figures 6 and 7 especially
713 2012-06-19 17:31:59 <amiller> "This also emphasizes the fact that the Search DAG is only a logical view of the data, and need not restrict the publisher???s physical implementation." is what i'm referring to by orthogonal
714 2012-06-19 17:32:52 <amiller> any time you see VO in one of these papers, they mean Verification Object, which is just a path of data through the tree
715 2012-06-19 17:33:08 <Eliel> how do you merkleize a trie?
716 2012-06-19 17:33:22 <amiller> everywhere you have a pointer, store a hash right next to it
717 2012-06-19 17:33:31 <Eliel> I don't think it'd work too well to need 256 sha-256 hashes to verify one level.
718 2012-06-19 17:34:25 <amiller> you only need to verify the one you follow
719 2012-06-19 17:35:12 <Eliel> you have the root hash, to verify the first level is correct, you'd need all 256 hashes it contains. Correct?
720 2012-06-19 17:35:35 <Eliel> then to verify the second level, another 256 hashes
721 2012-06-19 17:36:08 <Eliel> or am I missing something?
722 2012-06-19 17:36:37 <TuxBlackEdo> Fujitsu and its Japanese partners have taken 148 days to carry out a cryptanalysis that had been estimated to take hundreds of thousands of years http://www.techweekeurope.co.uk/news/fujitsu-cryptography-standard-83185
723 2012-06-19 17:36:40 <etotheipi_> Eliel: close, you only need hash a concatenation of all non-null pointers, I believe
724 2012-06-19 17:36:41 <Eliel> ... ah, you could still organize those 256 in a merkle-like hierarchial structure.
725 2012-06-19 17:36:46 <amiller> ah yeah you'd need to hash that much data 256*256
726 2012-06-19 17:37:00 <amiller> but you don't have to compute that many hashes
727 2012-06-19 17:37:14 <Eliel> so you'd need 16 to verify a level then.
728 2012-06-19 17:37:16 <amiller> i'm not being very clear about hash (the act of computation) vs hash (the resulting data)
729 2012-06-19 17:37:29 <sipa> luke-jr: good point, but it is 3 bytes per block now :)
730 2012-06-19 17:38:28 <Eliel> ... oh wait, not 16, 8
731 2012-06-19 17:38:59 <amiller> you need to compute one hash over a 256*256 bit chunk of data
732 2012-06-19 17:39:48 <Eliel> this needs to be space efficient enough that light clients can verify a leaf node without needing to download lots of data.
733 2012-06-19 17:40:11 <amiller> okay so the client verification requirements are separate from the server's storage requirements as well
734 2012-06-19 17:40:14 <PK> If I have a really mini-patch wish for bitcoind. Just a one liner. Who can I give the patch for that? I'd really like to have " obj.push_back(Pair("transactions",  (int) pwalletMain->mapWallet.size())); " added to getinfo in bitcoinrpc.cpp around line 348. It doesn't have to be there, that was just the easiest way for me to do it not knowing the code that well. I need a total of the records that listtransactions can return.
735 2012-06-19 17:40:19 <amiller> the client needs on the order of O(log N) data to verify a leaf node
736 2012-06-19 17:41:05 <etotheipi_> I'm gonna need ot break out my whiteboard in a sec
737 2012-06-19 17:41:10 <Eliel> yes, I think the amount of data client needs to verify a leaf is one of the primary purposes of this scheme.
738 2012-06-19 17:41:22 <Eliel> to make that small
739 2012-06-19 17:41:42 <amiller> well the verification data required is minimized when the fanout is minimized
740 2012-06-19 17:41:50 <sipa> etotheipi_: hey
741 2012-06-19 17:41:54 <etotheipi_> hey sipa
742 2012-06-19 17:42:12 <sipa> etotheipi_: just so you know, i am working on pruning
743 2012-06-19 17:42:31 <etotheipi_> sipa: an implementation?  theory?
744 2012-06-19 17:42:59 <sipa> currently just by keeping a txid -> coins map, with very compact representation
745 2012-06-19 17:43:10 <Eliel> also, the amount of data that needs hashing also makes a big difference for the updates. It makes up the constant multiplier for the O(log N)
746 2012-06-19 17:43:19 <sipa> the entire map is serialized to 65 MB now
747 2012-06-19 17:43:30 <sipa> without compression
748 2012-06-19 17:43:55 <etotheipi_> Eliel, amiller: it sounds like in order for any trie structure to be ideal, it would be best to have minimal branching, but most importantly, level-compression
749 2012-06-19 17:44:23 <sipa> etotheipi_: i have little time know, but i'll explain later if you like
750 2012-06-19 17:44:33 <etotheipi_> sipa: yeah, I'd like to hear about that
751 2012-06-19 17:44:40 <etotheipi_> unfortuantely I overwhelmed myself
752 2012-06-19 17:44:47 <etotheipi_> so now's not a good time for me, either
753 2012-06-19 17:45:14 <sipa> basically, i believe i can implement a fully validating node with 200 MB of storage
754 2012-06-19 17:45:17 <etotheipi_> I hardly can keep up with this conversation
755 2012-06-19 17:45:31 <etotheipi_> my debugger has been sitting at the same place for the last hour
756 2012-06-19 17:46:59 <gmaxwell> luke-jr: so interesting, getmemorypool appears to be spending almost all its time doing signature validations.
757 2012-06-19 17:47:11 <etotheipi_> sipa: I'd also like to hear your input on the two core components of my proposal, or how they could be modified:  address/script-based aggregation in some tree structure, and using alternate chain to track the new data
758 2012-06-19 17:48:01 <sipa> etotheipi_: i like it, because it can be done as an arbitrary chain
759 2012-06-19 17:48:13 <Eliel> etotheipi_: I think index by script hash would work great. You can always calculate the script hash for regular transactions and P2SH transactions have it right there.
760 2012-06-19 17:48:31 <luke-jr> gmaxwell: with or before CheckNewBlock?
761 2012-06-19 17:48:48 <amiller> etotheipi_, i like that part too, gmaxwell suggested that in irc a few months ago
762 2012-06-19 17:49:02 <sipa> etotheipi_: i don't think address->tx lookups should burden the core protocol, but as an extension, i don't care
763 2012-06-19 17:49:03 <luke-jr> sipa: what happens if I reuse the same coinbase from an orphan? ;)
764 2012-06-19 17:49:18 <sipa> luke-jr: ?
765 2012-06-19 17:49:19 <gmaxwell> luke-jr: I'm profiling 0.6.2.2 right now.
766 2012-06-19 17:49:31 <etotheipi_> I don't think I can take credit for any single piece of the proposal
767 2012-06-19 17:49:36 <amiller> oh, wait, no, gmaxwell just said that etotheipi_ wanted it http://bitcoinstats.com/irc/bitcoin-dev/logs/2012/05/07/3#l3477735
768 2012-06-19 17:49:38 <etotheipi_> I just was inspired and put it together
769 2012-06-19 17:50:17 <luke-jr> sipa: just thinking it should be tested in the scenario where coinbase is at height X, orphaned, then again at height Y
770 2012-06-19 17:50:19 <Eliel> etotheipi_: so you can take credit for the overall structure :)
771 2012-06-19 17:50:24 <gmaxwell> amiller: I'd proposed that kind of indexing for namecoin litenode lookups, but I thought maintaining a balanced tree would be a really hard problem.
772 2012-06-19 17:50:54 <sipa> luke-jr: the txout db only contains active open txouts in the currently connected branch
773 2012-06-19 17:51:08 <Eliel> etotheipi_: that's pretty much what satoshi did when he put bitcoin together. Took already existing parts and put them together. :)
774 2012-06-19 17:51:20 <luke-jr> sipa: i c; still would make a test case with that :P
775 2012-06-19 17:52:42 <etotheipi_> it sounds like there's a lot of important people who like the overall concept
776 2012-06-19 17:52:51 <etotheipi_> the question is, where is it lacking?
777 2012-06-19 17:53:06 <etotheipi_> we are debating the specific tree structure to use, but I think we have too many options there, not too few
778 2012-06-19 17:53:26 <etotheipi_> what is it that I have missed that I will find out, halfway through implementation, that this doesn't actually work?
779 2012-06-19 17:53:45 <etotheipi_> or two months into people using, find some vulnerability
780 2012-06-19 17:54:04 <amiller> etotheipi_, if you're inclined, you could look at my reference implementation and give some feedback on how to clarify it
781 2012-06-19 17:54:05 <Eliel> etotheipi_: if that happens, then it's something that got past all of us.
782 2012-06-19 17:54:14 <amiller> https://github.com/amiller/redblackmerkle/blob/master/redblack.py
783 2012-06-19 17:54:21 <amiller> my goal in implementing it was to make it as clear as possible that it is secure
784 2012-06-19 17:54:24 <etotheipi_> amiller: I will read that redblack paper
785 2012-06-19 17:54:49 <amiller> they way that i did that is by having a single function definition for each operation, insert delete and so on
786 2012-06-19 17:54:49 <etotheipi_> err.. code
787 2012-06-19 17:54:56 <amiller> and then there are two modes, Record and Replay
788 2012-06-19 17:55:14 <amiller> record simply collects every node it passes through, and replay only looks at input data after verifying it against the hash it already ahs
789 2012-06-19 17:55:37 <amiller> ideally i would implement this in Coq and show that there is no way for two queries to return different results without constructing a hash collision
790 2012-06-19 17:55:40 <etotheipi_> maybe I just prefer tries and patricia trees because I spent 17 hours programming the insert-function for a patricia treein college
791 2012-06-19 17:56:03 <etotheipi_> and while I understood red-black trees, the implementation was my last assignment that I didn't do :)
792 2012-06-19 17:56:09 <amiller> we would need to agree on a set of balancing rules, since that will be a normative part of the protocol
793 2012-06-19 17:56:24 <amiller> although for a given set of balancing rules, there are still potentially many ways of serializing and storing the data, that doesn't need to be normative
794 2012-06-19 17:56:54 <amiller> you could store it as a btree or a linked list for all it matters to the spec
795 2012-06-19 17:57:28 <gmaxwell> amiller: (broken record) you do need to come up with at least one reasonable one just to show that your rules don't create problems for it.
796 2012-06-19 17:57:50 <amiller> mm and i suppose there has to be an agreed-upon serialization format since thats what you would take the hash of
797 2012-06-19 17:58:03 <amiller> gmaxwell, what would be the target for a reasonable one?
798 2012-06-19 17:58:53 <luke-jr> gavinandresen: please don't make a 0.6.3 branch, there is already a 0.6.x
799 2012-06-19 17:59:08 <gmaxwell> One consideration should be working things out so that distributed nodes work somewhat better. e.g. having reasonably high branching factor right at the root lets you easily load balance work.
800 2012-06-19 17:59:48 <gmaxwell> amiller: You've got me. There should be some representation which is fast for the normal operations an which doesn't have a large space inflation. E.g. what would the reference implementation use?
801 2012-06-19 17:59:50 <gavinandresen> luke-jr: ?
802 2012-06-19 18:00:06 <luke-jr> gavinandresen: 0.6.3 is already a work-in-progress
803 2012-06-19 18:00:17 <gavinandresen> says who?
804 2012-06-19 18:00:27 <luke-jr> me
805 2012-06-19 18:00:39 <luke-jr> I maintain the stable branches, in case you've not noticed
806 2012-06-19 18:00:45 <amiller> gmaxwell, you could store every possible branch (N log N) and then do constant time lookups
807 2012-06-19 18:00:47 <amiller> that would distribute well
808 2012-06-19 18:01:10 <gavinandresen> I've repeatedly told you that I disagree with the notion of "stable" branches that nobody is testing.
809 2012-06-19 18:01:27 <luke-jr> gavinandresen: so stop trying to reinvent them, and discourage people from testing them?
810 2012-06-19 18:01:43 <luke-jr> if you disagree with 0.6.3, then why are you trying to reinvent it?
811 2012-06-19 18:02:29 <amiller> gmaxwell, the difference between fast and slow will pretty much come down to cache locality, won't it?
812 2012-06-19 18:02:38 <gavinandresen> If we were closer to a stable 0.7rc1 I wouldn't, but we aren't, and there are a couple of pressing issues that I think requires a 0.6.3
813 2012-06-19 18:02:45 <amiller> maybe the target should be a parameterized storage model where you indicate the size of cache you want
814 2012-06-19 18:02:46 <gmaxwell> gavinandresen: I take back my earlier comments, as far as I can tell all the interesting cpu time for both miners and regular nodes is in the ecdsa validation. The caches should help a lot.
815 2012-06-19 18:02:55 <luke-jr> gavinandresen: then I'll roll a 0.6.3rc1
816 2012-06-19 18:03:18 <gavinandresen> luke-jr: where's your 0.6.3 tree?
817 2012-06-19 18:03:48 <luke-jr> gavinandresen: git://gitorious.org/+bitcoin-stable-developers/bitcoin/bitcoind-stable.git 0.6.x
818 2012-06-19 18:04:08 <gmaxwell> amiller: or disk locality. Bu also the branching factor. E.g. if throughput >> latency you want less branching.
819 2012-06-19 18:05:33 <Eliel> etotheipi_: also, one thing that piques my interest with this approach is that my intuition keeps telling me this could perhaps allow for full validating nodes that function on headers only basis for storage.
820 2012-06-19 18:05:51 <amiller> Eliel, most definitely
821 2012-06-19 18:06:01 <gmaxwell> Eliel: ...
822 2012-06-19 18:06:05 <gmaxwell> Eliel: thats the point.
823 2012-06-19 18:06:09 <Eliel> etotheipi_: of course, for that, every transaction would need to be packaged with it's inputs
824 2012-06-19 18:06:29 <gmaxwell> Eliel: or at least available from the peer that gave you the transaction.
825 2012-06-19 18:07:51 <Eliel> gmaxwell: yes. It'd increase the bandwidth requirements though.
826 2012-06-19 18:08:18 <gmaxwell> Eliel: but you can choose that trade-off at runtime rather than in the protocol.
827 2012-06-19 18:09:22 <etotheipi_> heh... actually the point was to allow new nodes to get on the network (validating or not) and be ready to go with their wallet in a few minutes
828 2012-06-19 18:09:31 <etotheipi_> regardless of how old that wallet is
829 2012-06-19 18:10:16 <etotheipi_> err... the point was blockchain compression -- that just happened to have some nice by-products :)
830 2012-06-19 18:10:21 <Eliel> I like that this approach would even allow mining without a need to store anything but blockchain headers :)
831 2012-06-19 18:10:41 <Eliel> would push storage to each user's wallet.
832 2012-06-19 18:11:16 <amiller> my original goal was to replace the proof of work system with a proof-of-throughput to the paths through the merkle tree
833 2012-06-19 18:11:26 <amiller> this would unify the two main tasks in bitcoin, mining, and transaction validation
834 2012-06-19 18:11:30 <etotheipi_> does merged mining provide the benefits I think it does, in this case?
835 2012-06-19 18:12:00 <Eliel> etotheipi_: I think this would function best as it's own chain, but merged mining doesn't add any significant drawbacks. Just some extra data.
836 2012-06-19 18:12:03 <etotheipi_> mainly that it's "free" for someone to participate, and once you have a significant proportion of miners doing it, it's got a comparable level of mining-majority-required-to-break security
837 2012-06-19 18:12:53 <etotheipi_> Eliel: I think merged-mining is critical -- it means the miners supporting it don't have to divert mining power to provide the proof-of-work to keep the second chain secure
838 2012-06-19 18:12:57 <Eliel> as a merged mining chain it lacks miner incentives though.
839 2012-06-19 18:13:24 <etotheipi_> that is a minor point I wasn't worried about... but others might be
840 2012-06-19 18:13:29 <etotheipi_> I don't think the second chain needs incentive
841 2012-06-19 18:13:46 <etotheipi_> if the computations are efficient and the code is already written... it basically costs nothing for miners to support the second chain through merged mining
842 2012-06-19 18:14:17 <amiller> etotheipi_, i have another suggestion as well. Since the point is that lite clients could maintain the tree without having to store anything themselves, you could also just implement it as a centralized untrusted service
843 2012-06-19 18:14:20 <Eliel> if this gets included with bitcoind someday, that removes the issue :)
844 2012-06-19 18:14:53 <amiller> the way it would work is that a client has a root hash for block B. Then they look at the transactions for block B+1, add it to the tree, and they don't even need to query the service
845 2012-06-19 18:14:55 <luke-jr> gavinandresen: if it helps, http://codepad.org/do2dlohf is a list of commits backported
846 2012-06-19 18:15:03 <etotheipi_> Eliel: but the question is: will it make it off the ground without incentive?
847 2012-06-19 18:15:25 <etotheipi_> I believe so --- and once it does, it can then, in the future, be integrated safely into the main chain, or just left where it is
848 2012-06-19 18:16:22 <amiller> i think what i just said was contradictory so nevermind, etotheipi_ i like your alternate blockchain idea as well
849 2012-06-19 18:16:39 <etotheipi_> amiller: hah
850 2012-06-19 18:16:41 <amiller> especially as a staging area for future main chain features
851 2012-06-19 18:17:33 <etotheipi_> I didn't think of it until galambo mentioned it in the thread: but it does seem "staging area" is a great analogy for it
852 2012-06-19 18:17:41 <etotheipi_> stage-chain
853 2012-06-19 18:17:49 <Eliel> etotheipi_: if not, there's always the option of forking bitcoin ;)
854 2012-06-19 18:18:01 <Eliel> this is big enough improvement that it could take off.