1 2012-06-20 00:03:17 <gmaxwell> hehe bun5vdg2agmaxwell.onion   vanity onions are a bit easier than vanity bitcoin addresses.
  2 2012-06-20 00:17:49 <luke-jr> O.O
  3 2012-06-20 00:17:58 <luke-jr> gmaxwell: you got half? how long to get 100%?
  4 2012-06-20 00:19:02 <gmaxwell> Well, it's exponential of course. The onion addresses are 80 bits.
  5 2012-06-20 00:20:03 <dk5> where in the official code are non-standard scripts parsed? also - I like your template matching algorithm, are there opcodes that would be safe to use to expand it?
  6 2012-06-20 00:20:58 <dk5> i.e. any hex values you guys don't plan on using later on that would create a conflict
  7 2012-06-20 00:21:45 <gmaxwell> You can't just 'extend' it simply every node in the network must validate scripts. If you write a script that one node accepts and another doesn't then the world ends.
  8 2012-06-20 00:22:10 <gmaxwell> There are opcodes set aside for additions, the NOPs.
  9 2012-06-20 00:22:39 <gmaxwell> Because then you can construct it so that old nodes manage to ignore the new stuff.
 10 2012-06-20 00:23:24 <gmaxwell> Of course, the scripts have to be written so they always evaluate to true under the NOP interpertation, and this must also be enforced by the new implementation or someone can split the network.
 11 2012-06-20 00:25:18 <dk5> i'm more interested in just parsing the existing scripts, not sending strange ones out into the wild. i'd want to try to parse input scripts using templates as well.
 12 2012-06-20 00:25:41 <jgarzik> BlueMatt: well if you wanna do all the lock auditing immediately... I'm all for it :)
 13 2012-06-20 00:26:35 <jgarzik> BlueMatt: I was thinking even more incrementally
 14 2012-06-20 00:27:13 <dk5> gmaxwell: so, for instance, a template for signatures, or one for scripts with messages followed by an OP_DROP
 15 2012-06-20 00:29:25 <dk5> gmaxwell: you set aside some for OP_PUBKEY and OP_PUBKEYS and you do something special when those are hit in the script. i'd want to put one that can indicate to skip the next chunk of data (like a message) and go on to the next opcode. is there a safe one to use for that purpose?
 16 2012-06-20 00:30:50 <gmaxwell> dk5: you would just push the data onto the stack and then pop it off again, you don't need to add an opcode for this, though you shouldn't be contemplating using the blockchain as a message transport/store. It's not good for that.
 17 2012-06-20 00:31:59 <dk5> gmaxwell: lol, no that's not my plan. I'm just trying to parse the data. kind of like a toString() function for scripts but able to identify where the key portion is.
 18 2012-06-20 00:32:21 <BlueMatt> jgarzik: I really didnt, I just did the really obvious ones
 19 2012-06-20 00:33:24 <jgarzik> BlueMatt: it's mainly a matter of taste.  while auditing, does one (a) poop some boilerplate in each function, or (b) update vRPCCommands[] with a new 'unlocked' column?
 20 2012-06-20 00:33:37 <gmaxwell> oh, well we have script pretty printing code in 0.7 (git master)
 21 2012-06-20 00:33:44 <gmaxwell> 'pretty' at least.
 22 2012-06-20 00:33:59 <BlueMatt> jgarzik: if it were binary, I would say (b), but its not
 23 2012-06-20 00:34:44 <BlueMatt> and then you end up going back and switching to (a) after doing (b), which is just needless work (imho)
 24 2012-06-20 00:34:45 <gmaxwell> (e.g. it looks like             "scriptPubKey" : "OP_DUP OP_HASH160 06f1b66ffe49df7fce684df16c62f59dc9adbd3f OP_EQUALVERIFY OP_CHECKSIG"
 25 2012-06-20 00:34:48 <gmaxwell> )
 26 2012-06-20 00:35:22 <jgarzik> BlueMatt: no, you update VRPCCommands[] each time you audit and properly lock a function
 27 2012-06-20 00:35:27 <dk5> gmaxwell: it's in the 0.6.2 version too, no? i saw something like that
 28 2012-06-20 00:36:00 <jgarzik> BlueMatt: not all of them need LOCK2(cs_main, pwalletMain->cs_wallet), as I'm sure you're discovering
 29 2012-06-20 00:36:20 <BlueMatt> jgarzik: meh, seems like needless steps...
 30 2012-06-20 00:36:20 <jgarzik> BlueMatt: (a) is not an intermediate step
 31 2012-06-20 00:36:43 <dk5> gmaxwell: it's actually the toString() function for the CScript class
 32 2012-06-20 00:36:56 <jgarzik> both (a) and (b) are easy and provable.  as I said, a matter of taste.
 33 2012-06-20 00:37:05 <gmaxwell> dk5: I don't reacall where we exposed it in 0.6.2 except the debugging stuff.
 34 2012-06-20 00:37:11 <BlueMatt> jgarzik: it took me like 20 minutes to do sanity auditing to get to a pretty decent a, if it took more effort, Id agree, but it doesnt
 35 2012-06-20 00:37:24 <gmaxwell> dk5: in 0.7 you can just run gettransaction <hash>
 36 2012-06-20 00:37:28 <dk5> gmaxwell: is the one in 0.7 different?
 37 2012-06-20 00:37:31 <dk5> i see
 38 2012-06-20 00:37:32 <jgarzik> BlueMatt: first step should be an equivalent transformation
 39 2012-06-20 00:37:42 <jgarzik> BlueMatt: subsequent steps update the locking
 40 2012-06-20 00:38:41 <dk5> gmaxwell: alright, thanks. i'll take a look
 41 2012-06-20 00:38:46 <jgarzik> BlueMatt: first step for (a) would be to create a patch that does nothing but move the lock from CRPCTable::execute() to each function this code, "LOCK2(cs_main, pwalletMain->cs_wallet)"
 42 2012-06-20 00:39:07 <jgarzik> then separate commits would change 'LOCK2(cs_main, pwalletMain->cs_wallet)' to the improved locking
 43 2012-06-20 00:40:08 <BlueMatt> jgarzik: feel free to do it in a cleaner way, I wasnt planning on doing it for real until I had deleted cs_main, added a cs to CBlockStore and moved mapBlockIndex/etc to it.  and since I dont plan on doing that until I get some response, and its not like thats coming soon
 44 2012-06-20 00:44:47 <jgarzik> BlueMatt: the main driver (or lack thereof) is user interest.  if gmaxwell or luke-jr or miners think it's worthwhile to unlock GW/GMP, it can be done
 45 2012-06-20 00:46:11 <luke-jr> jgarzik: gmp_bip and gmp_longpoll have been waiting for a while
 46 2012-06-20 00:50:58 <BlueMatt> jgarzik: never said it should or shouldnt be considered important, I was just saying I wasnt planning on doing it for real for a while, if at all
 47 2012-06-20 00:53:08 <BlueMatt> (though shared lock support would mean getmemorypool would be trivially multithreaded...
 48 2012-06-20 00:54:57 <gmaxwell> getmemorypool miners do internal work generation more important is that other RPCs like getblock / getinfo don't obstruct it.
 49 2012-06-20 00:55:07 <gmaxwell> (also that it be fast in general)
 50 2012-06-20 00:55:16 <BlueMatt> (which shared lock support would provide ;) )
 51 2012-06-20 02:07:23 <gmaxwell> hm, http://people.xiph.org/~greg/main.cpp.ann.txt  the profile on the O0 compile seems to say that it's spending all the time in getmemorypool at main.cpp:3388
 52 2012-06-20 02:17:42 <luke-jr> gmaxwell: Mitre is having a hard time figuring out whether our block propagation mis-incentive should be considered a vuln or not >_<
 53 2012-06-20 02:19:25 <luke-jr> I'm kindof leaning toward the interpretation that "The Bitcoin node p2p protocol suffers from a bitcoin-loss bug affecting miners", and not a vulnerability
 54 2012-06-20 02:39:55 <jgarzik> GetTransaction() is an odd, odd function
 55 2012-06-20 02:40:18 <jgarzik> one of those "sometimes we give you a tx, sometimes we give you a block" type of deals
 56 2012-06-20 04:43:31 <amiller> etotheipi_, what if we compromise
 57 2012-06-20 04:43:45 <amiller> suppose we balance a merkle tree the way i'd like to, with a red-black balancing bit
 58 2012-06-20 04:44:06 <amiller> but actually store the data in an implementation of tires using libjudy
 59 2012-06-20 04:45:08 <amiller> ah i don't know that this actually has the ideal io efficiency
 60 2012-06-20 04:48:02 <amiller> as a server, what i would really like to do is store every possible 'path' (verification object) for the merkle tree, that the clients could ask me to fetch
 61 2012-06-20 04:48:20 <amiller> the naive solution is N log N - it's trivially parallel! but it's a ridiculous solution because it duplicates so much data.
 62 2012-06-20 04:50:00 <amiller> (the naive solution is literally to store each possible path, there are only N, each one is N log N. but for example you'd have N duplicates of the root node)
 63 2012-06-20 04:50:24 <amiller> ah nvm i don't think this is going where i want it to, since every update would change _all_ the possible paths, it's not useful to think of it that way
 64 2012-06-20 04:53:20 <amiller> the next best thing though, still using tries + red black merkle, is that you just store each logical merkle-tree node as an ordinary element in the trie
 65 2012-06-20 04:53:43 <amiller> ick, but that discards any potential use of the locality
 66 2012-06-20 04:54:49 <amiller> if you access a root node you definitely have to access one of your children, so that's the value of higher fanout
 67 2012-06-20 04:56:28 <amiller> my goal will still be to have a logical binary tree as far as the normative protocol is concerned, so that parameters fanout can be left up to the different implementations
 68 2012-06-20 05:03:21 <amiller> "Binary search trees (BSTs) and skip lists are good for main memory implementations, but for large data sets requiring secondary storage, they have poor I/O performance. A classic way to get good I/O performance is to use a B-tree." is from one of the papers i like http://truthsayer.cs.ucdavis.edu/algorithmica.pdf
 69 2012-06-20 05:22:31 <amiller> THEOREM 5. For a one-dimensional range query with T answer points we get a VO of size (log N + T ) which can be constructed with (logB N + T/B) I/O operations using a multi-way tree of size (N).
 70 2012-06-20 05:23:00 <amiller> range query there, especially means "give me all the spendable coins at this address"
 71 2012-06-20 05:24:27 <amiller> B is the size of the block, the main parameter, accessing it costs a single operation in this expression
 72 2012-06-20 05:29:49 <amiller> so i'll go away for a bit and come back when i've made a shitty b-tree simulation using python tuples i guess, it'll still be a 'look at my code' sort of thing
 73 2012-06-20 05:30:16 <amiller> i suppose i could profile it too
 74 2012-06-20 05:32:05 <amiller> since i have a 'reference' implementation of the logical tree, that is intended to be fairly simple to reason about, i shouldn't have to worry about any of the hashes or introduce any new security claims for this exercise
 75 2012-06-20 08:03:38 <grondilu> I don't understand how IP are encoded on IRC.  For instance, on bitcoinTEST I see an address as u71xQ2UmhBG22nq, but this gives me at most 11 bytes.  Clearly not enough to encode an IPv6.  Apart from the C++ source code which is tough to grasp, is there a documentation about that somewhere?
 76 2012-06-20 08:08:19 <wumpus> does irc bootstrapping support ipv6 at all?
 77 2012-06-20 08:11:01 <grondilu> wumpus: it's kind of the default since v0.6, I think.
 78 2012-06-20 08:13:43 <wumpus> looking at irc.cpp it seems not, it uses an "u"+a base58-encoded (ipv4 address + port) structure
 79 2012-06-20 08:14:14 <wumpus> see EncodeAddress and the ircaddr structure right at the top
 80 2012-06-20 08:16:12 <genjix> grondilu: irc isnt used anymore
 81 2012-06-20 08:16:20 <genjix> it's off by default
 82 2012-06-20 08:17:12 <wumpus> yeah only for testnet
 83 2012-06-20 08:18:32 <grondilu> ok so what's the method for bootstapping now?
 84 2012-06-20 08:21:20 <genjix> grondilu: bootstrap nodes + peerlist (which is far more sensible)
 85 2012-06-20 08:22:51 <grondilu> is it about this pnSeed big array I see in net.cpp?
 86 2012-06-20 08:23:05 <grondilu> Some IP are hard-coded?
 87 2012-06-20 08:24:31 <wumpus> and the dns mechanism
 88 2012-06-20 08:25:13 <grondilu> indeed, I see the strDNSSeed array.  Ok, why not.
 89 2012-06-20 08:26:59 <grondilu> One thing I don't like with C++ is that it is tough to find out where something is defined.  I use etags and it can not find the source of "in_addr" in irc.cpp, for instance :(
 90 2012-06-20 08:27:39 <[Tycho]> How is this related to C++ ?
 91 2012-06-20 08:27:46 <[Tycho]> It's just a flaw in your IDE
 92 2012-06-20 08:27:56 <grondilu> yeah, probably.
 93 2012-06-20 08:29:08 <wumpus> no problem in qt creator, and many other ides which have a working go-to-definition
 94 2012-06-20 08:29:37 <grondilu> Maybe I should try one.
 95 2012-06-20 08:30:32 <[Tycho]> The best IDE ever is MSVC 6
 96 2012-06-20 08:31:20 <grondilu> [Tycho]: I can't use any MS software.
 97 2012-06-20 08:31:39 <grondilu> The only one I know is Eclypse.  What do you think about it?
 98 2012-06-20 08:31:49 <wumpus> you need a fast machine
 99 2012-06-20 08:31:54 <wumpus> for eclipse
100 2012-06-20 08:32:08 <[Tycho]> I tried eclipse. Don't know what can be worse.
101 2012-06-20 08:32:25 <wumpus> I really like qt creator
102 2012-06-20 08:32:44 <grondilu> wumpus: is it ok with a small, netbook like, machine?
103 2012-06-20 08:32:46 <wumpus> (and yes, you can use it for non-qt projects)
104 2012-06-20 08:32:48 <wumpus> yes
105 2012-06-20 10:36:33 <gmaxwell> < luke-jr> gmaxwell: Mitre is having a hard time figuring out
106 2012-06-20 10:36:53 <gmaxwell> I think it's not,  if it were than almost every performance bug could be argued to be one.
107 2012-06-20 10:37:15 <gmaxwell> "this bad performance might cause people to use something other than bitcoin!"
108 2012-06-20 10:37:39 <sipa> mitre?
109 2012-06-20 10:39:45 <sipa> wumpus: lfnet nicknames are not long enough to encode 128 bits
110 2012-06-20 10:40:50 <sipa> maybe you could put them in realnames, and have nodes do a whois on eachother
111 2012-06-20 10:41:11 <sipa> but i don't think it was important enough to bother
112 2012-06-20 10:42:11 <kinlo> the best chain is determined by only looking at the sum of the difficulties of the chain of blocks right?
113 2012-06-20 10:42:36 <kinlo> there are no other reasons why bitcoin would choose a different chain - given that all blocks are valid?
114 2012-06-20 10:42:54 <gmaxwell> luke-jr_: It sort of goes along with this very famous linus post: http://kerneltrap.org/mailarchive/linux-kernel/2008/7/15/2506154   sometimes we forget that other bugs can be very important too, and want to start calling every serious issue a security bug.
115 2012-06-20 10:43:20 <gmaxwell> kinlo: well, they can be tied, so you'll use whatever you had first in that case.
116 2012-06-20 10:43:33 <kinlo> except for ties
117 2012-06-20 10:43:49 <kinlo> would bitcoind download blocks from their peers if they are tied?
118 2012-06-20 10:44:17 <kinlo> ie, if I have a chain, and you're offering a chain with last block different, would I even attempt to download the block from you?
119 2012-06-20 10:44:25 <sipa> kinlo: yes
120 2012-06-20 10:44:32 <kinlo> so orphans propagate?
121 2012-06-20 10:44:37 <sipa> no
122 2012-06-20 10:44:52 <sipa> blocks are only announced when they extend the main chain
123 2012-06-20 10:44:58 <sipa> but you can always download them
124 2012-06-20 10:45:02 <kinlo> ic
125 2012-06-20 10:45:07 <gmaxwell> Your peer will only offer it if your peer has accepted it as best.
126 2012-06-20 10:45:14 <sipa> i don't think you can download an orphan even
127 2012-06-20 10:45:33 <gmaxwell> If you don't accept it as best you won't pass it on... so the propagation stops.
128 2012-06-20 10:45:46 <kinlo> and you will download everything that it presented?
129 2012-06-20 10:46:09 <kinlo> given that it is valid
130 2012-06-20 10:46:11 <sipa> unless you already have it
131 2012-06-20 10:46:19 <sipa> and you don't know it's valid before downloading it
132 2012-06-20 10:46:37 <kinlo> yeah, but you know that the header matches difficulty
133 2012-06-20 10:46:47 <gmaxwell> No you don't.
134 2012-06-20 10:46:50 <sipa> no, only hashes are announced, not headers
135 2012-06-20 10:47:08 <kinlo> oh, so I can just send fake hashes to my peer, and my peer will download the block from me?
136 2012-06-20 10:47:19 <sipa> no, only hashes are announced, not headersyes
137 2012-06-20 10:47:23 <sipa> yes
138 2012-06-20 10:47:25 <kinlo> ofcourse, it will then subsequently invalidate the block and block me
139 2012-06-20 10:47:30 <sipa> indeed
140 2012-06-20 10:47:43 <kinlo> but I could trigger a huge download very easily
141 2012-06-20 10:47:49 <kinlo> well huge, up to blocklenght
142 2012-06-20 10:47:56 <kinlo> maxblocklength*
143 2012-06-20 10:48:17 <gmaxwell> kinlo: of course you could also DOS your peer with unwanted pings. And?
144 2012-06-20 10:48:37 <kinlo> gmaxwell: I'm just trying to grasp the mechanism, not trying to point out flaws :)
145 2012-06-20 10:48:50 <sipa> upon announce: 1) check if we already have it, if not, request data 2) when data arrives, check superficial validity (PoW, difficulty) 3) check whether its parent is known 4a) if not, store as orphan in memory and request parents 4b) is parent is known, store on disk 5) try to connect it to make it part of the the new best chain
146 2012-06-20 10:51:26 <kinlo> mmmz, how does it know the difficulty if it doesn't know where the block comes from?
147 2012-06-20 10:51:51 <kinlo> in theory, the difficulty could drop back to 1 if everybody stopped mining, so the client should accept diff=1 blocks too, right?
148 2012-06-20 10:52:05 <sipa> you're right; difficulty is only checked after finding the parent, of course
149 2012-06-20 10:52:10 <sipa> but PoW is checked before that
150 2012-06-20 10:52:17 <kinlo> PoW could be of diff=1
151 2012-06-20 10:52:25 <sipa> yes
152 2012-06-20 10:52:38 <kinlo> mmz, can I then upload 1000 blocks of 1 meg if I just generate 1000 with valid diff=1?
153 2012-06-20 10:52:57 <kinlo> as you first need to download parent to do proper checking...
154 2012-06-20 10:53:38 <gmaxwell> kinlo: there is a check for plausable decline from the prior checkpoint.
155 2012-06-20 10:53:55 <sipa> oh, righty
156 2012-06-20 10:54:54 <kinlo> :)
157 2012-06-20 10:54:56 <gmaxwell> e.g. it takes log4(difficulty)*2016 blocks to get to diff 1.
158 2012-06-20 10:54:56 <kinlo> thanks
159 2012-06-20 10:55:28 <kinlo> and in that case the timestamp must be changed accordingly too
160 2012-06-20 10:55:43 <kinlo> and the client refuses blocks more then a few hours in the future right?
161 2012-06-20 10:56:08 <gmaxwell> Not more than two hours, right.
162 2012-06-20 10:56:23 <kinlo> makes sense
163 2012-06-20 10:56:48 <gmaxwell> There are ways to improve this process to make it better but they haven't been done yet.
164 2012-06-20 10:57:12 <kinlo> I dunno wether improving that part is important
165 2012-06-20 10:57:22 <gmaxwell> A little.
166 2012-06-20 10:57:26 <kinlo> the main problem still looks to be io
167 2012-06-20 10:57:34 <gmaxwell> Main problem of what?
168 2012-06-20 10:57:40 <kinlo> block propagation
169 2012-06-20 10:57:46 <kinlo> chain downloading
170 2012-06-20 10:58:33 <gmaxwell> I would normally take those to be two very different issues.
171 2012-06-20 10:59:14 <gmaxwell> It appears that absent undiscovered bugs the latency for a block to reach most of the network is due to just the time it takes to validate blocks right now. Though it doesn't have to be.
172 2012-06-20 10:59:44 <gmaxwell> Downloading the whole chain is just a question of raw data but there isn't any particular reason that downloading the whole chain needs to get in the user's way.
173 2012-06-20 10:59:52 <kinlo> yeah, but chain downloading means block validating right?
174 2012-06-20 11:00:34 <kinlo> during initial download, one does verify the entire chain, or does it assume certain blocks are already verified and valid?
175 2012-06-20 11:00:37 <gmaxwell> kinlo: In one case, the propagation of a new block, the latency is important  in the other the throughput is important.
176 2012-06-20 11:00:53 <kinlo> true
177 2012-06-20 11:01:10 <sipa> never underestimate the bandwidth of a truck filled with DVD's
178 2012-06-20 11:01:18 <gmaxwell> kinlo: it verifies the whole chain. But as I mentioned, there isn't any reason that this should matter to the user. It should happen in the background.
179 2012-06-20 11:01:44 <kinlo> but in both cases validation is a big reason of it being slow
180 2012-06-20 11:01:58 <gmaxwell> kinlo: A node that started up and pulled headers first in reverse time order could pick the probably best chain and act as a SPV node until the chain sync caught up.
181 2012-06-20 11:02:20 <kinlo> sipa: ack on that, we often use the bandwith of the e19 to get our stuff uploaded, much faster then the average provider here :)
182 2012-06-20 11:02:31 <gribble> New news from bitcoinrss: zootreeves opened pull request 1487 on bitcoin/bitcoin <https://github.com/bitcoin/bitcoin/pull/1487>
183 2012-06-20 11:02:37 <gmaxwell> kinlo: depends on the hardware. We've fixed enough bugs that for fast machines network bandwidth and unfortunate peer selection limit the block download.
184 2012-06-20 11:17:36 <gribble> New news from bitcoinrss: sipa opened pull request 1488 on bitcoin/bitcoin <https://github.com/bitcoin/bitcoin/pull/1488>
185 2012-06-20 11:19:48 <genjix> sipa: might want to remove that subscribe stuff in net.cpp too
186 2012-06-20 11:19:58 <sipa> wasn't that removed already?
187 2012-06-20 11:20:08 <sipa> yup
188 2012-06-20 11:20:16 <genjix> ah ok then
189 2012-06-20 11:33:17 <gavinandresen> I just tagged v0.6.3rc1, and am gitian-building it now.  Any other gitian-capable people here and able to build?
190 2012-06-20 11:52:45 <Graet> any info on the proposed "backbone" available yet?
191 2012-06-20 11:52:47 <sipa> gavinandresen: building (assuming my gitian still works)
192 2012-06-20 11:52:54 <sipa> Graet: backbone?
193 2012-06-20 11:53:15 <gavinandresen> jgarzik says a miner backbone is in the works....
194 2012-06-20 11:53:24 <sipa> ah, jeff's project
195 2012-06-20 11:53:51 <jgarzik> suddenly accelerated.  ETA 7 days to a 3-node, network dispersed backbone of bitcoin-optimized machines.
196 2012-06-20 11:54:08 <jgarzik> people will be able to sponsor nodes too, if they wish to contribute with money
197 2012-06-20 11:54:19 <jgarzik> (or simply donate a dedicated server)
198 2012-06-20 11:54:20 <Eliel> the wonders of additional publicity :)
199 2012-06-20 11:54:49 <jgarzik> gavinandresen: not necessarily a miner backbone...  a backbone connecting "major" sites of all sorts
200 2012-06-20 11:55:05 <jgarzik> TX producers as well as miners
201 2012-06-20 11:55:07 <gavinandresen> right, sorry, I mean to say "major backbone"
202 2012-06-20 11:55:10 <Graet> dedicated server ? or a vps on a dedi server?
203 2012-06-20 11:55:24 <gavinandresen> (ooh, new nickname for Jeff,  "Major Backbone")
204 2012-06-20 11:55:29 <Graet> hehe
205 2012-06-20 11:55:55 <jgarzik> rofl
206 2012-06-20 11:56:02 <Eliel> Graet: I'd think a VPS works about as well as a dedicated.
207 2012-06-20 11:56:13 <jgarzik> Eliel: VPS is not so great for security
208 2012-06-20 11:56:17 <Eliel> well, depending on the specs though :)
209 2012-06-20 11:56:18 <Graet> what resources/bandwidth do you expect a node to need jgarzik ?
210 2012-06-20 11:56:28 <jgarzik> Eliel: latencies are too variable on a VPS, also
211 2012-06-20 11:56:54 <sipa> hmm, so i won't be able to connect my VPS that runs the DNS seed :(
212 2012-06-20 11:56:57 <Graet> wel i have dedi servewr is several countries, a vps on one of my servers is as secure as another dedi :) its not linode :P
213 2012-06-20 11:56:57 <jgarzik> Graet: standard bitcoin node, plus a little bit of extra bandwidth for statistics gathering
214 2012-06-20 11:57:26 <Graet> ok, i may be able to provide 1 or 2 nodes
215 2012-06-20 11:57:27 <jgarzik> sipa: you may connect to backbone, sure.  I'm saying I will not build backbone nodes using a VPS.
216 2012-06-20 11:57:58 <jgarzik> Graet: that would be helpful...
217 2012-06-20 11:58:09 <Graet> jgarzik, i have several powerful machinhes i split into vms, how is onhe less secure than a dedi p[lease ?
218 2012-06-20 11:58:38 <sipa> jgarzik: i see
219 2012-06-20 11:59:22 <jgarzik> Graet: it is more difficult to hack into, or clone, a dedicated server unobserved.
220 2012-06-20 11:59:43 <SomeoneWeird> err
221 2012-06-20 11:59:46 <jgarzik> Graet: VMs introduce a new level of security issues, as we've seen with LInode and RackSpace hacks
222 2012-06-20 12:00:00 <Graet> err dont they have to hack into my dedi b4 they can do those thiongs tho?
223 2012-06-20 12:00:11 <jgarzik> Graet: yes
224 2012-06-20 12:00:31 <Graet> so how is a vm on my dedi less secure
225 2012-06-20 12:00:38 <Graet> sorry, i'm confudsed :)
226 2012-06-20 12:00:40 <sipa> it is not
227 2012-06-20 12:00:49 <sipa> Graet: the question is just whether you operate the actual hardware it is running on
228 2012-06-20 12:01:10 <Graet> well i do , but <jgarzik> sipa: you may connect to backbone, sure.  I'm saying I will not build backbone nodes using a VPS.
229 2012-06-20 12:01:21 <jgarzik> Graet: from the perspective of the outside party, it requires trusting the machine operator's security, in addition to your own.  one additional level of trust.
230 2012-06-20 12:01:22 <Graet> so i was trying to get clarification :)
231 2012-06-20 12:02:04 <jgarzik> but for backbone, latency is also a big issue.  VPS's tend to "stutter" (have super-high latencies at unpredictable times)
232 2012-06-20 12:02:30 <Graet> i'm the machine operator? or do you require dedivated servers only you can access similar to freenodes way?
233 2012-06-20 12:02:35 <gmaxwell> 06:56 <@jgarzik> Eliel: VPS is not so great for security
234 2012-06-20 12:03:01 <gmaxwell> with our current on the wire protocol it hardly matters, anyone with access to the network from $here to $there can just intercept all the connections.
235 2012-06-20 12:03:28 <jgarzik> nod
236 2012-06-20 12:03:37 <gmaxwell> Though perhaps someday we'll add authenticated addnode or something.
237 2012-06-20 12:03:39 <jgarzik> if you are the machine op, you may of course trust your own VPS
238 2012-06-20 12:03:51 <jgarzik> I'll need to heavily audit this machines, if I'm building peering agreements with major sites
239 2012-06-20 12:04:04 <jgarzik> VPS adds an additional audit hurdle
240 2012-06-20 12:04:54 <gmaxwell> jgarzik: yea, I'm not arguing at all with the decision, just pointing out that without authenticated connections its not the biggest hurdle.
241 2012-06-20 12:05:03 <luke-jr> imo, if it's just another node without a wallet, it doesn't need to be secure
242 2012-06-20 12:05:15 <jgarzik> luke-jr: hah, that's your first mistake ;-)
243 2012-06-20 12:05:27 <sipa> peering with major sites = with their inner nodes?
244 2012-06-20 12:05:40 <luke-jr> in fact, the point of such a hub would be to NOT be secure - ie, not do validity checks on the blocks
245 2012-06-20 12:05:41 <jgarzik> gmaxwell: well the trusted backbone will only connect to known parties
246 2012-06-20 12:05:54 <luke-jr> &
247 2012-06-20 12:05:55 <jgarzik> that will, in turn, connect to a public backbone
248 2012-06-20 12:06:04 <luke-jr> sounds like needless centralization
249 2012-06-20 12:06:16 <gmaxwell> luke-jr: it's DOS resistance.
250 2012-06-20 12:06:33 <luke-jr> the point of a miner hub is to relay blocks faster and be available to any miner without a formal arrangement
251 2012-06-20 12:06:39 <jgarzik> luke-jr: I have plenty of interest from pools and exchanges.  It's no big deal if you aren't interested.
252 2012-06-20 12:06:43 <BlueMatt> and /very/ helpful for eg p2pool miners who want quicker connections to big pools
253 2012-06-20 12:06:54 <jgarzik> Eligius can lag behind other pools ;p
254 2012-06-20 12:07:37 <gmaxwell> jgarzik: right, but I'm mr. badguy and I operate or have just hacked any router on the path between the backbone and a known party. (or really any BGP speaking router in the default free zone, considering how promiscious internet backbone routing is). I can intercept these connections, and you won't be able to tell.
255 2012-06-20 12:08:16 <SomeoneWeird> oh noes
256 2012-06-20 12:08:31 <luke-jr> jgarzik: Eligius already peers directly with other big pools. as described, your node would just be an extra delay.
257 2012-06-20 12:09:06 <gmaxwell> luke-jr: where you've been able to track down what you guess is their IP addresses.
258 2012-06-20 12:09:39 <luke-jr> gmaxwell: as long as jgarzik's node requires special peering agreements, it's not much different
259 2012-06-20 12:10:38 <luke-jr> the only difference is, the pools participating outsource that job to him personally
260 2012-06-20 12:10:50 <BlueMatt> letting anyone connect to the main nodes would kinda ruin the whole point of being more dos-resistant.  but I do agree its important for there to be public nodes connected to the backbone
261 2012-06-20 12:10:53 <gmaxwell> luke-jr: but he'll perhaps have better luck in getting them, because he's not a competitor.
262 2012-06-20 12:11:47 <gmaxwell> (and because it's one hub to negotiate with instead of N)
263 2012-06-20 12:12:06 <luke-jr> BlueMatt: the problem isn't DoS, it's propagation time
264 2012-06-20 12:12:30 <BlueMatt> adding one layer of fast-nodes with tmpfs backing wont hurt propagation much
265 2012-06-20 12:12:59 <BlueMatt> and if you let anyone connect, it may
266 2012-06-20 12:13:13 <BlueMatt> (dos, generally more nodes to deal with, ...)
267 2012-06-20 12:14:55 <Eliel> definitely helpful for tmpfs use I think :)
268 2012-06-20 12:15:06 <BlueMatt> would be easier to just prune blkindex and would provide most of the optimization, but sipa's working on in-memory stuff that would be even better
269 2012-06-20 12:15:40 <Eliel> some will want to keep the whole history too... of course, you can run another node for that though.
270 2012-06-20 12:16:12 <BlueMatt> ofc
271 2012-06-20 12:17:32 <Eliel> just, if there was a file that contains only unspent txouts, disk caches would get a bigger hit rate. I have no idea how big effect that would have though.
272 2012-06-20 12:18:03 <jgarzik> will of course be running latest code w/ signature caching, plus some ideas akin to luke-jr's delayed block validation
273 2012-06-20 12:26:51 <luke-jr> jgarzik: so, how much are you charging for peering with this "backbone" that seems to have no other peers yet?
274 2012-06-20 12:28:14 <jgarzik> luke-jr: just like in the internet world, big sites peer for free
275 2012-06-20 12:29:09 <kinlo> in the internet world, one can measure the traffic in megabits
276 2012-06-20 12:29:09 <luke-jr> jgarzik: that doesn't really answer the question on cost
277 2012-06-20 12:29:21 <kinlo> how are you going to measure the size of a peer?
278 2012-06-20 12:29:28 <kinlo> in found blocks per week?
279 2012-06-20 12:30:10 <jgarzik> with subjective judgements.  "big" includes exchanges that produce no blocks, for example.
280 2012-06-20 12:31:36 <kinlo> I don't really see why exchanges benefit that much from fast block propagation, depositing coins takes like 5 confirmations, so the minute the last block propagation takes...
281 2012-06-20 12:32:42 <gavinandresen> exchanges benefit from double-spend detection/prevention
282 2012-06-20 12:33:03 <gavinandresen> ... and if they're peered with a good percentage of hashing power then that's good for them
283 2012-06-20 12:35:14 <kinlo> true
284 2012-06-20 12:35:26 <jgarzik> it's not just fast block propagation, it's fast TX propagation too.  The hardware and software will be optimized for quick database accesses (using well known techniques like SSDs, increasing bdb cache, etc.)
285 2012-06-20 12:36:08 <BlueMatt> s/SSDs/tmpfs/ ?
286 2012-06-20 12:36:12 <gavinandresen> somebody should write a version of bitcoind that stores the blockchain and index in memcache.
287 2012-06-20 12:36:16 <jgarzik> Old news to most, I imagine, but I just found a SatoshiDice clone :/  http://www.bitcoin-dice.com/
288 2012-06-20 12:36:27 <BlueMatt> gavinandresen: sipa's working on it
289 2012-06-20 12:36:55 <gavinandresen> cool.  Then you could write "slave" bitcoind's that read-only memcache...
290 2012-06-20 12:37:10 <sipa> not directly, but a 100 MB database would trivially fit in memory anyway
291 2012-06-20 12:37:33 <gavinandresen> I want the entire N gigabyte blockchain in memcached memory, so it can be served up quickly
292 2012-06-20 12:37:35 <jgarzik> it's doable... but IMO RAM is so cheap I can just build a node that does not need to worry about hitting disk (SSD) except in rare cases of spending ancient coins
293 2012-06-20 12:38:16 <jgarzik> or serving old blocks (an unlikely event on a trusted backbone node)
294 2012-06-20 12:38:20 <sipa> if ultraprune works out, even ancient coins won't need to hit disk
295 2012-06-20 12:38:27 <jgarzik> sipa: indeed
296 2012-06-20 12:38:48 <jgarzik> though I would still prefer to store the entire block chain
297 2012-06-20 12:38:55 <jgarzik> at least for this backbone project
298 2012-06-20 12:39:12 <sipa> sure, i think it should
299 2012-06-20 12:39:22 <sipa> even if just as an example
300 2012-06-20 12:40:42 <gmaxwell> jgarzik: the author of dice has been selling his code to anyone who wants to pony up 1100 btc.
301 2012-06-20 12:40:54 <gmaxwell> jgarzik: I expect we'll see a few more.
302 2012-06-20 12:40:59 <jgarzik> lovely
303 2012-06-20 12:41:15 <gmaxwell> though I don't think more will increase the traffic.
304 2012-06-20 12:41:29 <gmaxwell> Pretty sure these things are bounded by the supply of suckers, not by the supply of sites.
305 2012-06-20 12:41:46 <kinlo> the url you just gave might exist, but it probably won't get much traffic
306 2012-06-20 12:41:58 <jgarzik> heh
307 2012-06-20 12:42:09 <Graet> gmaxwell, yes
308 2012-06-20 12:42:26 <kinlo> I think the power of satoshidice is that one can verify that they are not cheating
309 2012-06-20 12:42:37 <gmaxwell> I also expect that the dice traffic will mostly fall off once the people running bots against it lose all their money.
310 2012-06-20 12:42:45 <kinlo> I'd rather gamble there then somewhere else
311 2012-06-20 12:42:57 <gmaxwell> kinlo: you can validate that on many gambling sites.
312 2012-06-20 12:43:21 <kinlo> gmaxwell: true, but not on the bitcoin-dice.com site
313 2012-06-20 12:43:29 <gmaxwell> kinlo: for example there is the blackjack site that uses the cryptographic proof I suggested, or even joric's mtgox code slot machine.
314 2012-06-20 12:43:49 <sipa> gavinandresen: windows build matches; linux one doesn't
315 2012-06-20 12:43:55 <gmaxwell> kinlo: also, dice has failed the proof test and no one cares.
316 2012-06-20 12:44:11 <gavinandresen> sipa: that's... different....
317 2012-06-20 12:44:23 <sipa> gavinandresen: not even bitcoind
318 2012-06-20 12:44:27 <gmaxwell> kinlo: if someone double spends them then other customers get screwed... and this has happened, and its ~indistinguishable from the site ripping them off.
319 2012-06-20 12:44:45 <gavinandresen> sipa:  d'oh.   Is one of the inputs different?
320 2012-06-20 12:45:00 <sipa> just pushed my sigs
321 2012-06-20 12:45:26 <sipa> i'll upload my binaries
322 2012-06-20 12:49:18 <gavinandresen> Release notes and my gitian-built binaries up at:  https://sourceforge.net/projects/bitcoin/files/Bitcoin/bitcoin-0.6.3/test/
323 2012-06-20 12:50:07 <sipa> http://bitcoin.sipa.be/builds/0.6.3rc1
324 2012-06-20 12:50:57 <gavinandresen> sipa:  your sigs aren't in   /gitian.sigs/0.6.3rc1
325 2012-06-20 12:51:22 <gavinandresen> (they are in the rc1-win32/ directory)
326 2012-06-20 12:51:52 <sipa> gavinandresen: oh right; now they are
327 2012-06-20 12:56:19 <sipa> gavinandresen: the linux binaries seem to have things in a different order
328 2012-06-20 12:56:58 <luke-jr> my win32 matches
329 2012-06-20 12:57:17 <luke-jr> sigs pushed
330 2012-06-20 12:57:30 <luke-jr> we have 3 - who wants to update .org?
331 2012-06-20 12:58:02 <gavinandresen> hold your horses, "we" need to sanity test to make sure they actually run
332 2012-06-20 12:58:10 <luke-jr> oh right, this is rc1 :P
333 2012-06-20 12:59:07 <gavinandresen> sipa:  there wasn't a 'randomize segments in the binary' flag added to makefile.unix between 0.6.2 and 0.6.3, was there ?
334 2012-06-20 12:59:28 <sipa> not that i know of
335 2012-06-20 13:00:15 <gavinandresen> Looking at the build-asserts from 0.6.2 versus 0.6.3rc1, I don't see any reason they'd differ.  I'll reboot and rebuild and see if I get the same bits.
336 2012-06-20 13:00:52 <sipa> only changes to the makefile are BOOST_SPLITIT_THREADSAFE and sh before genbuild
337 2012-06-20 13:30:40 <luke-jr> sipa: gavinandresen
338 2012-06-20 13:30:46 <luke-jr> my linux build matches gavin's
339 2012-06-20 13:33:20 <sipa> fhmm
340 2012-06-20 13:34:06 <sipa> i'll rebuild
341 2012-06-20 13:35:41 <gavinandresen> out doing errands for a while, back later (I'm rebuilding too, will see if I match myself....)
342 2012-06-20 13:40:17 <gavinandresen> rebuild finished, identical binaries for me
343 2012-06-20 14:00:10 <gribble> New news from bitcoinrss: jgarzik opened pull request 1489 on bitcoin/bitcoin <https://github.com/bitcoin/bitcoin/pull/1489>
344 2012-06-20 14:03:35 <gmaxwell> jgarzik: did you see that callgrind annotated trace from createnewblock I posted? It /claims/ to be spending a signficant fraction of the total time twiddling with heap allocations for mapTestPoolTmp.
345 2012-06-20 14:03:39 <gmaxwell> But I don't quite see why.
346 2012-06-20 14:04:14 <luke-jr> gmaxwell: did you run it for txn_prio?
347 2012-06-20 14:04:27 <BlueMatt> sipa: your pgp key expired
348 2012-06-20 14:04:54 <gmaxwell> luke-jr: No. did it touch mapTestPoolTmp?
349 2012-06-20 14:06:37 <luke-jr> gmaxwell: it significantly changes CreateNewBlock
350 2012-06-20 14:07:18 <sipa> BlueMatt: what? again? :(
351 2012-06-20 14:07:32 <BlueMatt> sipa: it expired in 2011, or you need to update gitian-downloader/sipa.pgp
352 2012-06-20 14:07:55 <BlueMatt> (and push to key servers, because --update-keys isnt finding an unexpired one)
353 2012-06-20 14:08:23 <BlueMatt> eg --refresh-keys
354 2012-06-20 14:08:27 <BlueMatt> s/eg/eh/
355 2012-06-20 14:10:27 <BlueMatt> hm, no it finally found the key, still needs updated in contrib/gitian-downloader/sipa.pgp
356 2012-06-20 14:10:40 <sipa> sub  3072R/C87992E0  created: 2011-09-23  expires: 2014-09-22  usage: S
357 2012-06-20 14:11:54 <luke-jr> oh right, we store the actual keys in git
358 2012-06-20 14:12:15 <BlueMatt> sipa: please dont push sigs to gitian.sigs if they dont match, it freaks gverify out
359 2012-06-20 14:12:24 <sipa> BlueMatt: ok
360 2012-06-20 14:12:46 <BlueMatt> ;;later tell devrandom any chance we could get gverify to figure out the number of matching sigs and not freak out if it finds one sig that doesnt match?
361 2012-06-20 14:12:46 <gribble> The operation succeeded.
362 2012-06-20 14:13:59 <sipa> BlueMatt: i think that'd be nice; for example when you are the first two pushers, and you don't know yet which one seems to be the 'faulty' one
363 2012-06-20 14:14:36 <BlueMatt> agreed, and btw pushed my sigs for 0.6.3rc1, which matched gavin's
364 2012-06-20 14:14:51 <sipa> updated my GPG key
365 2012-06-20 14:15:21 <gmaxwell> I think it's good to push non-matching ones out because it lets people watching know that there is a problem. When the downloader is something people use it should absolutely hold off doing anything in that case.
366 2012-06-20 14:15:55 <sipa> same binaries here after rebuild
367 2012-06-20 14:16:07 <BlueMatt> gmaxwell: downloader doesnt read gitian.sigs, it expects sigs to be in the zip that you download with binaries
368 2012-06-20 14:16:44 <TD> good evening
369 2012-06-20 14:16:56 <BlueMatt> hello TD
370 2012-06-20 14:17:15 <luke-jr> BlueMatt: how does it work with the Windows installer then?
371 2012-06-20 14:17:27 <BlueMatt> luke-jr: ???
372 2012-06-20 14:19:20 <BlueMatt> installer doesnt do anything with gitian, it just installs bitcoin.  updater downloads a zip with pgp keys, sigs, and binaries, then checks the pgp keys against its list of signers, checks the sigs, and installs
373 2012-06-20 14:20:26 <luke-jr> BlueMatt: but updater should never bypass the system package management
374 2012-06-20 14:20:27 <gribble> New news from bitcoinrss: luke-jr opened pull request 1490 on bitcoin/bitcoin <https://github.com/bitcoin/bitcoin/pull/1490>
375 2012-06-20 14:20:48 <BlueMatt> luke-jr: windows doesnt have package management, thats why gitian-updater in bitcoin-qt isnt implemented for non-windows
376 2012-06-20 14:21:25 <luke-jr> BlueMatt: it does, it just sucks
377 2012-06-20 14:21:51 <BlueMatt> you mean the uninstall dialog crap? meh, it may have an incorrect version, but oh well
378 2012-06-20 14:21:56 <luke-jr> &
379 2012-06-20 14:22:19 <BlueMatt> would be nice to update the version in registry, but it doesnt currently
380 2012-06-20 14:23:04 <luke-jr> imo, it should download the installer, verify that, and use it :p
381 2012-06-20 14:23:22 <luke-jr> (when it was installed from installer only& sigh)
382 2012-06-20 14:24:01 <BlueMatt> does nsis have an unattended install option?
383 2012-06-20 14:24:19 <sipa> why does it need that?
384 2012-06-20 14:24:46 <BlueMatt> updating and forcing users through the install dialog again would be kinda...ugly
385 2012-06-20 14:24:59 <sipa> that's only reasonable way, imho
386 2012-06-20 14:25:13 <BlueMatt> sipa: I know no other app that does that
387 2012-06-20 14:25:16 <sipa> ?
388 2012-06-20 14:25:24 <sipa> when i used windows, they all did :)
389 2012-06-20 14:25:27 <luke-jr> ^
390 2012-06-20 14:25:33 <BlueMatt> sipa: afaik the standard on win32 is to show an upgrade dialog, if the user hits yes, show a progress bar and restart the app
391 2012-06-20 14:25:40 <BlueMatt> no next, next, accept, next crap
392 2012-06-20 14:26:01 <luke-jr> anyhow, I think NSIS does have an autoinstall mode of some form
393 2012-06-20 14:26:03 <sipa> well then make an installer that doesn't require a next next accept next
394 2012-06-20 14:26:16 <BlueMatt> sipa: thats my point (unattended install option)
395 2012-06-20 14:26:17 <sipa> maybe optionally
396 2012-06-20 14:26:40 <BlueMatt> sipa: would have to be
397 2012-06-20 14:27:38 <BlueMatt> anyway, Ill look at tweaking gitian-updater to have a post-install command, make the only required file in gitian-win32 the installer and look into it when I get around to redoing auto-update
398 2012-06-20 14:45:19 <luke-jr> wb gavinandresen
399 2012-06-20 14:53:07 <jgarzik> gmaxwell: interesting
400 2012-06-20 14:53:26 <gavinandresen> luke-jr : why is https://github.com/bitcoin/bitcoin/compare/0.6.2...0.6.3    such a mess?
401 2012-06-20 14:53:41 <jgarzik> gmaxwell: I wonder if it's this line: swap(mapTestPool, mapTestPoolTmp)
402 2012-06-20 14:54:17 <jgarzik> gmaxwell: it does that swap on each iteration (for each tx in priority map)
403 2012-06-20 14:55:30 <BlueMatt> gavinandresen: because 0.6.3 has all of the stable commits that luke did over the years, whereas 0.6.2 doesnt?
404 2012-06-20 14:56:01 <gavinandresen> is there a way to make it less messy in the future?  I want to give people an easy way to see the diffs between 0.6.2 and 0.6.3
405 2012-06-20 14:57:43 <luke-jr> gavinandresen: I think part of it is the splitoff branches for 0.6.2 and 0.6.3
406 2012-06-20 14:58:08 <BlueMatt> luke-jr: there are a ton of like "Bump version to 0.4.5" in there
407 2012-06-20 14:58:25 <luke-jr> BlueMatt: yes, but those can mostly be filtered out easily otherwise
408 2012-06-20 14:58:32 <gavinandresen> If anybody has better git-fu than me, suggestions welcome for improving the process.
409 2012-06-20 14:59:20 <BlueMatt> gavinandresen: there is a command, but I dont think you can get to it from github, and I dont remember the proper invocation...
410 2012-06-20 14:59:36 <BlueMatt> luke-jr: will they are part of the "diff" in any case
411 2012-06-20 14:59:37 <luke-jr> git log --no-merges 65077e1..origin/0.6.3 will do changes between 0.6.1 and 0.6.3
412 2012-06-20 14:59:52 <luke-jr> 0.6.2 had fairly few, so those can be removed easily
413 2012-06-20 15:00:10 <luke-jr> BlueMatt: not really
414 2012-06-20 15:00:27 <BlueMatt> they arent actually different but they are a part of the list of commits different
415 2012-06-20 15:01:38 <BlueMatt> gavinandresen: git diff-tree -u v0.6.2 v0.6.3rc1 does the actual diff, though not the commit list
416 2012-06-20 15:02:12 <gavinandresen> yes, git diff does the right thing, I'd just like to give people a github URL that clearly shows what changed
417 2012-06-20 15:03:31 <gavinandresen> I posted a "help sanity test" request to https://bitcointalk.org/index.php?topic=88734.0
418 2012-06-20 15:08:45 <gavinandresen> sipa: I want to pick your brain about the signrawtx API when you have a few minutes
419 2012-06-20 15:21:27 <luke-jr> sipa: What is this doing? 839c4e7 Fix initialization of inaddr_any
420 2012-06-20 15:40:38 <sipa> luke-jr: iirc that's a bugfix, but not at a computer right now
421 2012-06-20 15:41:01 <sipa> gavinandresen: i'll be back in 2-3 hours
422 2012-06-20 15:41:31 <gavinandresen> sipa: ack
423 2012-06-20 16:17:04 <gribble> New news from bitcoinrss: darsie opened issue 1491 on bitcoin/bitcoin <https://github.com/bitcoin/bitcoin/issues/1491>
424 2012-06-20 16:48:48 <luke-jr> gavinandresen: 0.6.3rc1 for Mac?
425 2012-06-20 16:49:00 <gavinandresen> luke-jr: right!  thanks for reminding me
426 2012-06-20 16:51:53 <gavinandresen> d'oh, macdeployqtplus is unhappy
427 2012-06-20 16:52:05 <luke-jr> >_<
428 2012-06-20 16:57:37 <gribble> New news from bitcoinrss: gavinandresen opened issue 1492 on bitcoin/bitcoin <https://github.com/bitcoin/bitcoin/issues/1492>
429 2012-06-20 17:39:13 <sipa> gavinandresen: back
430 2012-06-20 17:40:41 <gavinandresen> sipa: cool.  So, signrawtx has a third param in case you want to create chains of transactions that aren't yet in the blockchain/memorypool
431 2012-06-20 17:40:53 <sipa> gavinandresen: i noticed
432 2012-06-20 17:40:56 <gavinandresen> As currently implemented, that third param is just an array of serialized parent transactions
433 2012-06-20 17:41:13 <gavinandresen> I'm wondering if it would be better to make it an array of [txid,nout,scriptPubKey]
434 2012-06-20 17:41:13 <sipa> oh wait; no, i saw you adding a list of secret keys
435 2012-06-20 17:41:33 <gavinandresen> (I may be misremembering third param versus second)(
436 2012-06-20 17:42:08 <sipa> well, txid/nout/scriptPubKey is all you need, the full transaction is overkill
437 2012-06-20 17:42:53 <gavinandresen> yup.  If I make that change, it ripples through a couple of other RPC calls-- e.g. listunspent should return [txid, nout, scriptPubKey, nConfirmations]  (I think)
438 2012-06-20 17:43:47 <Eliel> no amounts?
439 2012-06-20 17:44:06 <sipa> technically, you don't need those
440 2012-06-20 17:44:11 <gavinandresen> inputs are always completely spent....
441 2012-06-20 17:44:28 <sipa> but the transaction may be invalid if you fail to provide enough worth of inputs
442 2012-06-20 17:46:01 <gavinandresen> I think I've convinced myself, [txid,n,hex_script_pubkey] is the right thing to do.
443 2012-06-20 17:46:46 <sipa> Yes, I like that better as well.
444 2012-06-20 17:47:02 <sipa> more transparent, at least
445 2012-06-20 17:47:09 <gavinandresen> To me that implies the default for reporting txOuts for gettransaction should be hex_script_pubkey.  Which brings up the whole 'decompositions' feature, which I don't like
446 2012-06-20 17:49:18 <gavinandresen> luke-jr : how upset will you be if I rip out the "decompositions" code ?   What is the use case for gettransaction with "please decompile the scripts into human-readable output" ?
447 2012-06-20 17:49:58 <gavinandresen> (the RPC interface is meant to be great for machines, not humans)
448 2012-06-20 17:55:18 <gmaxwell> gavinandresen: because humans use it to and we do actually want power users learning the system better and becoming more powerful power users.
449 2012-06-20 17:56:03 <jgarzik> gavinandresen: historically speaking, people liked my getblock patch (dumped all fields and all tx's, including scripts) because it was readable -and- computer parseable back into useable form
450 2012-06-20 17:56:05 <gavinandresen> gmaxwell: meh.  blockchain.info/blockexplorer.com ....
451 2012-06-20 17:56:12 <gmaxwell> Its a feature that I now personally use while diagnosing transactions and such.
452 2012-06-20 17:56:23 <jgarzik> gavinandresen: but it was unconditionally decoded.  none of this "choose your own adventure" 12 types of decomposition
453 2012-06-20 17:56:25 <gmaxwell> Which are often unreliable/slow/unavailable.
454 2012-06-20 17:57:03 <gmaxwell> yea, I don't really care about the choose your own adventure crap, I only found it tolerable because both raw and human readable had usecases.
455 2012-06-20 17:57:09 <sipa> jgarzik: i have no idea what you're trying to say
456 2012-06-20 17:57:14 <gavinandresen> A decodetransaction / decodescript RPC call that takes hex-encoded crap and spits out human-readable JSON would be fine with me.
457 2012-06-20 17:58:08 <gavinandresen> Those would work better with the 'raw transaction' rpc calls, because they deal entirely with hex-encoded serialized strings.
458 2012-06-20 17:59:04 <sipa> and they would be a nice example of an RPC that doesn't need cs_main or cs_wallet :)
459 2012-06-20 17:59:14 <gmaxwell> That sounds okay to me. Being able to decode a hex encoded transaction is important before you sign the thing.
460 2012-06-20 18:00:08 <gmaxwell> It's a little less elegant from the CLI usage perspective than getting it directly through the decomp, but I think thats OKAY as you say the primary target audience of the rpc is machines.
461 2012-06-20 18:00:41 <sipa> It does extend the usefulness of decomposition to all transactions, not just those that can be seen through getblock or gettransaction
462 2012-06-20 18:00:51 <jgarzik> true
463 2012-06-20 18:01:28 <gavinandresen> OK, my TODO is:  replace the 'decompositions' stuff with decode RPC call or calls.  And modify the rawtransaction API to deal with prior inputs as [txid, n, scriptPubKey]  (might mean some refactoring of the transaction creation code, too....)
464 2012-06-20 18:01:39 <sipa> i'm sure luke's response will be that a getblock that can decompose the transactions can save hundreds of rpc calls
465 2012-06-20 18:01:41 <jgarzik> gavinandresen: ACK
466 2012-06-20 18:02:03 <jgarzik> sipa: RPC is helpfully multi-threaded now... :)
467 2012-06-20 18:02:29 <gavinandresen> ... then we should teach the RPC code to speak JSON-RPC-2.0 and take an array of commands to execute.... (I actually wrote that code a while ago, it's not hard)
468 2012-06-20 18:02:44 <gmaxwell> sipa: no reason that a decodetransaction couldn't take 100 transactions as arugments, though there would be more (de)serialization overhead.
469 2012-06-20 18:03:07 <jgarzik> sipa: original getblock dumped -everything- in a single call, which was useful for debugging.  Mind you, I'm NOT arguing for that now...  just noting.
470 2012-06-20 18:03:13 <jgarzik> I think theymos still uses my getblock patch
471 2012-06-20 18:03:15 <sipa> i think that
472 2012-06-20 18:03:22 <gmaxwell> but realistically if you're asking for 100 things, you're able to do the darn decoding yourself.
473 2012-06-20 18:03:53 <gmaxwell> helo: 12 simultaneous outputs in one transaction confirmation!
474 2012-06-20 18:04:30 <gavinandresen> While I've got your attention:  "Reworking Bitcoin Transaction Fees"  :  https://gist.github.com/2961409
475 2012-06-20 18:09:44 <gmaxwell> gavinandresen: "meh". I full expect anyone to look at that and set the first option as high as they can. Because they'll mentally visualize missing out on a mysterious 100 BTC fee transaction otherwise.
476 2012-06-20 18:10:34 <gavinandresen> gmaxwell: hmm?   100 BTC fee would get sorted to the very top of the to-be-included list
477 2012-06-20 18:11:08 <gmaxwell> "But there might be 4000 100 BTC transactions!"  The key is that you don't know how big the excluded transaction will be.
478 2012-06-20 18:11:30 <gmaxwell> This is something that stresses senders out about fee rules. They worry that the fees might be enormous on some transaction and so they'd rather send none at all.
479 2012-06-20 18:12:10 <gmaxwell> Perhaps instead "BTC amount of lost fees which you're willing to tolerate in order to support free txn" ?
480 2012-06-20 18:12:40 <helo> 'receiver pays fees model' -- neat
481 2012-06-20 18:12:47 <gavinandresen> Huh?  Paid would always overrule free.  So if you set the "paid" space to 1MB and the free space to 100K and there WERE 1MB of paid transactions there wouldn't be room for any free ones.
482 2012-06-20 18:12:48 <gmaxwell> e.g. sum up the list of txn over the minimum fee from the _bottom_.
483 2012-06-20 18:13:20 <gmaxwell> gavinandresen: thats what I'm saying. I'm saying your proposal strongly encourages people to set that to 1MB because they can't reason about how much they'll be losing except retrospectively.
484 2012-06-20 18:13:40 <gavinandresen> I don't have a problem with that.
485 2012-06-20 18:14:18 <gavinandresen> I expect SOME people will decide that a little priority-based space is a good idea, so free transactions will eventually be confirmed
486 2012-06-20 18:14:29 <gmaxwell> I'm suggesting you can solve this by doing the test another way and have much more use of it.
487 2012-06-20 18:14:59 <gavinandresen> I still don't understand.  Either miners will want to be greedy and take as many fee-paying transactions as possible, or they won't.
488 2012-06-20 18:15:32 <gmaxwell> Instead, 'Minimum transaction fee-per-kilobyte to be considered "paid"'  'Maximum amount sum fee you're willing to miss out on to process free txn'
489 2012-06-20 18:15:48 <gavinandresen> why wouldn't the greedy miners just set that to zero?
490 2012-06-20 18:16:38 <gmaxwell> They will, the issue isn't greed its that your proposal says you can either be generous and take potentially unbounded losses, or you can be a scrooge.
491 2012-06-20 18:17:21 <helo> might as well make the default to include all transactions with fees that will fit (sorted increasing), with an option to include free transactions if there is room?
492 2012-06-20 18:17:22 <gavinandresen> minimum-fee-per-kb times kb-of-free-transactions gives that bound
493 2012-06-20 18:17:34 <helo> *decreasing
494 2012-06-20 18:18:24 <gmaxwell> gavinandresen: No it doesn't, Say the mempool has 1MB of 100 BTC fee transactions. .. every free txn you take costs you 100 BTC.
495 2012-06-20 18:19:32 <gmaxwell> I've seen this same reasoning play out with people on the sending side of fees. People running webservices being VERY anxious about paying any fees at all because they're worried they'll come back and find out they are in the red.  Even though its quite unlikely that the automatic fees will be large.
496 2012-06-20 18:19:34 <helo> default behavior of "make money please", so every wiki guide to setting up a miner doesn't start with "turn off free transactions"
497 2012-06-20 18:19:38 <gavinandresen> Ok, I propose that greedy self-interested miners set the params as:  1MB of paid transactions.  100K-max of free transactions.  "free" if paying less than 0.001 BTC/kb.
498 2012-06-20 18:20:11 <gavinandresen> They'll never lose more than 100K*0.001 BTC of potentially fee-paying transactions, but will include "free" transactions if there is room.
499 2012-06-20 18:20:46 <Diablo-D3> gavinandresen: what, enforced globally?
500 2012-06-20 18:20:48 <gmaxwell> hm. I must be misunderstanding you. Not sure where.
501 2012-06-20 18:20:53 <Diablo-D3> or can individual nodes still do whatever?
502 2012-06-20 18:21:08 <gmaxwell> oh yes, in that case they would.
503 2012-06-20 18:21:12 <gavinandresen> I'm talking default, out-of-the-box rules.  (and pulling numbers outta my ass)
504 2012-06-20 18:21:23 <Diablo-D3> gavinandresen: I'd do it this way
505 2012-06-20 18:21:40 <sipa> Diablo-D3: read https://gist.github.com/2961409 first
506 2012-06-20 18:21:42 <Diablo-D3> 4mb max, paid or unpaid, but DONT do it by fee per kb
507 2012-06-20 18:21:54 <Diablo-D3> do it fee per kb already used in the blocj
508 2012-06-20 18:21:56 <gmaxwell> Well the out of box rules should be reasonably altruistic ones.
509 2012-06-20 18:21:57 <sipa> we can't change the max block size without hardfork
510 2012-06-20 18:22:05 <Diablo-D3> oh, max is still 1?
511 2012-06-20 18:22:09 <sipa> yes
512 2012-06-20 18:22:15 <Diablo-D3> okay, then pretend I said one
513 2012-06-20 18:22:24 <gavinandresen> Increasing the block size (or not) is a separate issue
514 2012-06-20 18:22:34 <gavinandresen> (that I don't want to talk about right now)
515 2012-06-20 18:22:40 <Diablo-D3> sort all possible tx by fee per kb, and just start with the cheapest ones and ramp up the fee as you fill the block
516 2012-06-20 18:23:06 <Diablo-D3> so, if you have, like, room for 10 tx in our fictional micro-block example chain
517 2012-06-20 18:23:07 <gmaxwell> gavinandresen: I don't expect anyone to set that 1MB number to anything smaller.  Which then also means you'll get weird outcomes like processing 0.0001 transaction spam over high priority free txn.
518 2012-06-20 18:23:38 <gmaxwell> Which I don't think even fairly greedy miners want.
519 2012-06-20 18:23:39 <Diablo-D3> 0% fee, 10% fee, 20% fee, 30% fee, etc
520 2012-06-20 18:23:43 <gribble> New news from bitcoinrss: jgarzik opened pull request 1493 on bitcoin/bitcoin <https://github.com/bitcoin/bitcoin/pull/1493>
521 2012-06-20 18:23:59 <Diablo-D3> so only large blocks require extra high fees
522 2012-06-20 18:23:59 <gavinandresen> gmaxwell: set the "free" threshold to something reasonable and The Right THing happens
523 2012-06-20 18:24:04 <gmaxwell> or basically, the minimum-which-is-considered-a-fee doesn't actually encapsulate the priority vs fee tradeoff.
524 2012-06-20 18:24:36 <Diablo-D3> my idea would end up making priority queues
525 2012-06-20 18:24:56 <Diablo-D3> low priority transactions would get, say, 10% of a block (assuming we have 10 priority queues)
526 2012-06-20 18:25:05 <gavinandresen> gmaxwell: I'm not following.  If a miner set:  1MB paid.  100K free.  "free" threshold 0.001 BTC then it seems to me all the right things happen.
527 2012-06-20 18:25:20 <Diablo-D3> high priority transactions would pay ten times more fees per kb but almost always get into a block
528 2012-06-20 18:25:42 <Diablo-D3> yes/no?
529 2012-06-20 18:26:43 <gavinandresen> Diablo-D3: I don't see how a user or their software would figure out what fee to pay to know whether or not they have a good chance of getting into the next block or two.
530 2012-06-20 18:26:55 <Diablo-D3> gavinandresen: make a gui that lists priority
531 2012-06-20 18:27:11 <Diablo-D3> gavinandresen: low medium high guaranteed
532 2012-06-20 18:27:26 <Diablo-D3> you pay the fee no matter what even if you would have gotten into a free slot because of low contention
533 2012-06-20 18:27:29 <gmaxwell> gavinandresen: at 0.001/block the miner potentially walks on 4BTC in fees. So I think people would set it lower than that. With it set lower than that, its cheap to crowd all the free txn out of a block just for burn-the-world purposes.
534 2012-06-20 18:28:06 <Diablo-D3> default rules could split the blocks into 4, 250kb free, 250kb low, 250kb medium, 250kb high
535 2012-06-20 18:28:29 <gavinandresen> Huh?  0.001 BTC/k times 100K free-transaction-space would be losing out on 0.1 BTC
536 2012-06-20 18:28:45 <Diablo-D3> high can spill over into medium, medium into low, low into free, but not the reverse
537 2012-06-20 18:29:31 <gavinandresen> Diablo-D3: I think you're overcomplicating it.  All most users care about is "fast" or "free"
538 2012-06-20 18:30:10 <gmaxwell> gavinandresen: say there are 4000 very very low priority transactions in the queue with 0.001 fees. And enough high priority zero fee transactions above them to completely keep the fees out of the block. You'd lose out on 4BTC. (assuming a maximum 4k txn in a block, which is vaguely right)
539 2012-06-20 18:30:49 <Diablo-D3> gavinandresen: well
540 2012-06-20 18:30:59 <Diablo-D3> its currently wrong to have a value in the prefs like it does now
541 2012-06-20 18:31:20 <Diablo-D3> gmaxwell: thats vs my example? doesnt work
542 2012-06-20 18:31:21 <gavinandresen> gmaxwell: ah, right, I sit corrected.
543 2012-06-20 18:31:24 <Diablo-D3> because priority is fees.
544 2012-06-20 18:31:32 <Diablo-D3> the more fee you pay, the higher priority you are
545 2012-06-20 18:32:24 <sipa> sure, i'd rather include a 1 MB transaction that pays 0.1 BTC than 1000 1 kB transactions that pay 0.01 each
546 2012-06-20 18:32:47 <gavinandresen> gmaxwell: that scenario seems pretty far-fetched, though.
547 2012-06-20 18:32:51 <gmaxwell> gavinandresen: thats why I was suggesting an alternative of "how much are you willing to lose per block to support free txn"  even if you set that to zero you'd still process some because there still would be some fee floor under which you would consider feed transactions free.
548 2012-06-20 18:33:01 <Diablo-D3> sipa: its per kb though
549 2012-06-20 18:33:18 <gmaxwell> gavinandresen: it absolutely is, I'm not suggesting that its likely. I'm saying that the possibility will encourage irrational settings because people are anxious about 'losing money'.
550 2012-06-20 18:34:04 <Diablo-D3> so 1mb that pays 0.1 btc is a hundred times less than 1000kb of 0.01 each
551 2012-06-20 18:34:39 <Diablo-D3> so I dunno
552 2012-06-20 18:34:43 <Diablo-D3> maybe Im overcomplicating it
553 2012-06-20 18:34:50 <Diablo-D3> just sort all tx by fee/kb
554 2012-06-20 18:35:06 <Diablo-D3> and allow 100% free blocks by default
555 2012-06-20 18:35:08 <gavinandresen> I really like the simplicity of "this chunk of the block is reserved for fee-paying transactions.  I'll take them in order of fees paid."   And "if there's any space leftover, I'll take this many other transactions, sorted by priority."
556 2012-06-20 18:35:17 <Diablo-D3> gavinandresen: I dont
557 2012-06-20 18:35:27 <Diablo-D3> because it doesnt lead to people trying to outbid people for the space
558 2012-06-20 18:35:27 <gavinandresen> Adding fees-paid to the priority calculation might be a good idea.
559 2012-06-20 18:35:42 <Diablo-D3> so having a priority that is ONLY fee/kb seems to be the best bet
560 2012-06-20 18:35:44 <Diablo-D3> its what we want
561 2012-06-20 18:35:47 <gavinandresen> Diablo-D3: huh?  sure it does, the fee-paying space is up for bid
562 2012-06-20 18:35:58 <Diablo-D3> gavinandresen: yes, but why isnt ALL the space up for bid?
563 2012-06-20 18:36:03 <Diablo-D3> why is any of it avail for free only?
564 2012-06-20 18:36:11 <Diablo-D3> the market itself should set the price
565 2012-06-20 18:36:22 <Diablo-D3> if everyone doesnt want to pay, then we have zero fee blocks
566 2012-06-20 18:36:26 <gavinandresen> Diablo-D3: could be, set max kilobytes of fee-paying transactions to 1Meg
567 2012-06-20 18:36:27 <gmaxwell> gavinandresen: I'm not suggesting anything much more complicated: "The block is filled wiht fee paying transactions in order of fees, but I'll replace the X BTC lowest fee transactions with transactions ordered by priority instead"
568 2012-06-20 18:36:30 <Diablo-D3> if everyone wants to pay exorbant costs, then we have high fee blocks
569 2012-06-20 18:36:35 <gmaxwell> s/wiht/with/
570 2012-06-20 18:36:42 <Diablo-D3> gavinandresen: yeah but we're arguing defaults
571 2012-06-20 18:36:57 <gavinandresen> gmaxwell: ok, got it.  I like that.
572 2012-06-20 18:37:01 <Diablo-D3> I personally can do whatever I want
573 2012-06-20 18:37:10 <Diablo-D3> such as ripping out all the priority code and sort by fee/kb.
574 2012-06-20 18:37:13 <gmaxwell> (and that with same augmentation that fees under some value are taken to be zero for the calculation)
575 2012-06-20 18:37:29 <Diablo-D3> we shouldnt sort by things like age of inputs or total amount or anything
576 2012-06-20 18:37:33 <gavinandresen> Diablo-D3: the "knobs" i'm proposing would allow you to do exactly that
577 2012-06-20 18:37:42 <Diablo-D3> make everyone pay per kb since thats what our most expensive thing
578 2012-06-20 18:37:46 <Diablo-D3> nothing costs more than the kb.
579 2012-06-20 18:38:09 <Diablo-D3> even computation time means nothing now
580 2012-06-20 18:38:12 <gmaxwell> gavinandresen: WRT dust spam, we need fixes in the coin selection code if you want to remove that. Right now we have a dos vulnerability  litecoin has a crazy kludge to fix it.
581 2012-06-20 18:38:20 <gavinandresen> Diablo-D3: ... you still need a minimum fee to be considered 'paying' to avoid an asshat filling up blocks with spam
582 2012-06-20 18:38:21 <Diablo-D3> its growing a thousand times faster than our harddrive space is
583 2012-06-20 18:38:26 <Diablo-D3> gavinandresen: no
584 2012-06-20 18:38:40 <Diablo-D3> gavinandresen: because if I sort by fee/kb, most blocks will have NO free tx in them eventually
585 2012-06-20 18:39:05 <gavinandresen> sigh.  I attach a 1-satoshi fee to my spam and your sort now happily includes them....
586 2012-06-20 18:39:15 <Diablo-D3> thats not what I meant.
587 2012-06-20 18:39:21 <Diablo-D3> ifs not free vs not free
588 2012-06-20 18:39:28 <Diablo-D3> its a linear scale of how much you want to pay
589 2012-06-20 18:39:32 <Diablo-D3> with no minimum or maximum
590 2012-06-20 18:39:39 <gmaxwell> gavinandresen: is there a way we can express the "minimum fee to be considered non zero" which won't result in fair numbers of people ignorantly setting it to 0?
591 2012-06-20 18:39:58 <Diablo-D3> so if everyone pays 0.01btc per kb except the spammer, and we max the blocks out every single time with 0.01btc per kb tx, then the spammer cant spam
592 2012-06-20 18:40:24 <Diablo-D3> or at least, if he wants to spam, then he pays 0,01 btc per kb like everyone else
593 2012-06-20 18:40:30 <Diablo-D3> which then the market goes to 0.02btc per kb
594 2012-06-20 18:40:30 <gavinandresen> gmaxwell: good question-- that's the downside of letting people twiddle the knobs, they might be idiots
595 2012-06-20 18:40:45 <gmaxwell> gavinandresen: maybe express it as a fraction of the fees in the prior block and not let them set it under 1?
596 2012-06-20 18:40:58 <Diablo-D3> gmaxwell: fraction of fees over the past 100.
597 2012-06-20 18:41:01 <gmaxwell> Not just idiots. It's not obvious that "0" is really bad.
598 2012-06-20 18:41:33 <gmaxwell> gavinandresen: E.g. "I'm super generous, so I consider any fee to be a fee!"  or "I'm really poor, every cent counts!"  without really thinking through the implications.
599 2012-06-20 18:41:44 <Diablo-D3> "send with no fee" "send with low (half of average)" "send with medium (average)" "send with high (twice average)"
600 2012-06-20 18:42:09 <Diablo-D3> and then the reverse "allow % maximum of each"
601 2012-06-20 18:42:24 <Diablo-D3> so a guy really could allow 100% free
602 2012-06-20 18:42:33 <Diablo-D3> or set free, low, and medium to 0
603 2012-06-20 18:42:49 <Diablo-D3> or hell
604 2012-06-20 18:42:51 <Diablo-D3> set all four to 0
605 2012-06-20 18:42:53 <Diablo-D3> and mine with no tx
606 2012-06-20 18:46:15 <Diablo-D3> gavinandresen: seriously, if someone wants to do that, whats wrong with that
607 2012-06-20 18:48:11 <gavinandresen> it's anti-social behavior that I think should be discouraged.
608 2012-06-20 18:48:31 <gavinandresen> ... other miners should be encouraged to discourage those blocks.
609 2012-06-20 18:48:35 <Diablo-D3> gavinandresen: okay lets try this differently
610 2012-06-20 18:48:38 <gavinandresen> in my humble opinion.
611 2012-06-20 18:48:39 <Diablo-D3> 1mb is the max, right?
612 2012-06-20 18:48:42 <gavinandresen> yup
613 2012-06-20 18:48:50 <Diablo-D3> take the average usage over the past 100 blocks
614 2012-06-20 18:49:00 <gavinandresen> usage?
615 2012-06-20 18:49:05 <Diablo-D3> usage of the 1mb
616 2012-06-20 18:49:05 <gavinandresen> you mean block size?
617 2012-06-20 18:49:26 <gavinandresen> ok
618 2012-06-20 18:49:34 <Diablo-D3> then let 100% average usage be 1btc in fees per 1mb (or 0.001 per kb)
619 2012-06-20 18:49:50 <Diablo-D3> and then make that the fee that is both sent and required by default nodes
620 2012-06-20 18:49:54 <gavinandresen> why is 1btc in fees the right number?
621 2012-06-20 18:50:01 <Diablo-D3> its nice and even.
622 2012-06-20 18:50:04 <gavinandresen> nack
623 2012-06-20 18:50:28 <sipa> you just said you wanted block space price to be determined by the market
624 2012-06-20 18:50:30 <Diablo-D3> lets say the past 100 blocks were 10%, the average fee would be 0.1 BTC per 1mb
625 2012-06-20 18:50:39 <Diablo-D3> sipa: yes but gavin said it was wrong
626 2012-06-20 18:50:43 <Diablo-D3> so Im trying a new idea
627 2012-06-20 18:50:54 <sipa> you don't sound very consistent
628 2012-06-20 18:51:00 <Diablo-D3> sipa: Im trying out ideas here
629 2012-06-20 18:51:21 <Diablo-D3> gavinandresen: it gets rid of the knobs and it scales the fees with usage
630 2012-06-20 18:51:25 <sipa> yes, you're just coming up with things before thinking them through, while other people here are discussion some well-formed plan
631 2012-06-20 18:51:37 <Diablo-D3> sipa: shush, Ive had this on a backburner for awhile
632 2012-06-20 18:51:49 <sipa> i'm going to shut up
633 2012-06-20 18:52:02 <Diablo-D3> gavinandresen: spammers cant spam for very long before it starts dragging the average usage up
634 2012-06-20 18:52:28 <Diablo-D3> and people dont have to set fee because the network sets it dynamically
635 2012-06-20 18:52:53 <gavinandresen> Diablo-D3: I'm with sipa.  Write up a counter-proposal so I can see it all at once.
636 2012-06-20 18:54:38 <gavinandresen> gmaxwell: so if we go with "maximum BTC of transactions to replace with priority-sorted-free-transactions", the knobs would be....
637 2012-06-20 18:56:12 <gavinandresen> uh....    sort all transactions by fee, chop the list when you get to 1MB...  still need to avoid relaying/considering obvious spam (close-to-no-fee/low-priority)
638 2012-06-20 18:58:25 <gmaxwell> gavinandresen: I just commented on the gist.
639 2012-06-20 18:58:39 <gmaxwell> Instead I suggest "Maximum amount of BTC to give up in order to process transactions in order of priority: M". The algorithm would be:
640 2012-06-20 18:58:42 <gmaxwell> Set (for the purpose of selection) all fees below the amount/kb to be considered a fee to zero.
641 2012-06-20 18:58:45 <gmaxwell> Order transactions by fee per kb. Stop when you hit maximum size.
642 2012-06-20 18:58:47 <gmaxwell> Go backwards and remove M BTC in fees worth of transactions.
643 2012-06-20 18:58:50 <gmaxwell> Add transactions in order of priority, stopping when the block is full or when the maximum amount of 0 fee transaction data is hit.
644 2012-06-20 19:02:38 <gavinandresen> gmaxwell: I like it.
645 2012-06-20 19:06:07 <gmaxwell> Note that you might just add back the same txn you just removed.. if they happened to have high priority. So it's only a maximum lost. This means it's also reasonably safe to set the option high.
646 2012-06-20 19:07:13 <gavinandresen> I wonder if jgarzik has a comment, he was looking at benchmarking/optimizing the block creation code today, yes?
647 2012-06-20 19:09:59 <Diablo-D3> gavinandresen: well, whatever you guys go with, it has to be simple to use
648 2012-06-20 19:21:42 <BlueMatt> gmaxwell: I was under the impression that was always kinda the plan...anyway, Ive always thought it would work
649 2012-06-20 19:22:49 <gmaxwell> BlueMatt: hm?   I'm just clarifying that the 'remove' in my algorithim didn't mean don't mine them at all.
650 2012-06-20 19:26:56 <BlueMatt> gmaxwell: no, the idea that we will always keep around a free tx area and let miners chose how big (or in this case how much) it is
651 2012-06-20 19:27:11 <BlueMatt> and other than that just sort by fee
652 2012-06-20 19:27:41 <BlueMatt> aside from getting really complicated, I kinda always assumed that was the idea
653 2012-06-20 19:28:48 <gmaxwell> BlueMatt: The problem with 'free txn area' as a 'minimum' is that it makes it hard to reason about how much fees you're missing out on.  E.g. you could potentially lose a billonity bitcoins. That encourages people to set it to zero if you make it easy to set.
654 2012-06-20 19:29:24 <BlueMatt> gmaxwell: agreed, size in btc is much better
655 2012-06-20 19:34:45 <gmaxwell> If I think a bit harder perhaps I can figure out how to make the amount closer to the amount you will lose (instead of just a maximum) without making the algorithim terribly complicated.
656 2012-06-20 19:35:54 <gmaxwell> ah. I've got it.   "sort by fee/kb, then replace txn one at a time from he bottom using txn from the top of the priority sorted list until your lost fee =M or you've hit the maximum amount of free txn"
657 2012-06-20 19:38:41 <BlueMatt> as long as its not individual txn, but txn groups, yep
658 2012-06-20 19:38:41 <gavinandresen> Do we need the 'max amount of free txn' knob?
659 2012-06-20 19:39:07 <BlueMatt> I dont think so, just "max Im willing to lose" should cover it imho
660 2012-06-20 19:39:23 <gavinandresen> I think so, too.
661 2012-06-20 19:40:03 <gmaxwell> I only left it in my description because I thought some people who want to fight chain bloat might want to turn it down.
662 2012-06-20 19:40:14 <gmaxwell> I don't feel strongly that its needed however.
663 2012-06-20 19:42:14 <BlueMatt> ok...lets see, how do spv nodes with the new filter* stuff get average fee/kb over last n blocks?
664 2012-06-20 19:42:31 <sipa> they can't
665 2012-06-20 19:42:38 <gavinandresen> block fees are easy, they're in the coinbase txout
666 2012-06-20 19:42:41 <sipa> unless they want to trust some full node to tell them
667 2012-06-20 19:43:04 <gavinandresen> e.g. if the last block was 100K and got a 50.01 BTC reward, you know there was 0.1 fees for 100KB
668 2012-06-20 19:43:08 <gavinandresen> 0.01
669 2012-06-20 19:43:24 <BlueMatt> gavinandresen: ignore free tx stuff then?
670 2012-06-20 19:43:26 <sipa> yes, but you cannot do any measurement of how long it takes for a tx to be include
671 2012-06-20 19:43:30 <sipa> d
672 2012-06-20 19:43:55 <BlueMatt> sipa: if you pay avg, its a pretty safe bet you will be in the next block?
673 2012-06-20 19:43:56 <gavinandresen> sipa: right.  SPV nodes need to rely on somebody else to tell them, I think
674 2012-06-20 19:44:21 <gmaxwell> Client behavior I feel pretty meh about. We need to get replacement working. Nothing automatic is going to work great.
675 2012-06-20 19:44:24 <sipa> BlueMatt: i think that may be serious overkill someday
676 2012-06-20 19:45:25 <BlueMatt> Id think avg would be serious overkill, yea.  But if you do the lowest, or low percentile, it just encourages a downward fee spiral
677 2012-06-20 19:45:40 <gmaxwell> If people start automatically paying the mean,  MinerEconomicus will start mining 100 BTC fee txn in his own blocks. (and hoping he doesn't get orphaned. ;) )
678 2012-06-20 19:45:41 <BlueMatt> plus then you'd need to trust a full node
679 2012-06-20 19:46:30 <yellowhat> does anyone know which protocol electrum currently uses for communication with the server?
680 2012-06-20 19:46:32 <yellowhat> is it the "stratum" protocol already?
681 2012-06-20 19:46:37 <gavinandresen> miners trying to influence the system is why I think watching the memory pool is the only reliable way to figure it out
682 2012-06-20 19:46:48 <BlueMatt> and for spv nodes?
683 2012-06-20 19:47:05 <gavinandresen> spv nodes will subscribe to a fee-reporting service that they trust, I think
684 2012-06-20 19:47:09 <gmaxwell> miners can influence the memory pool too..
685 2012-06-20 19:47:56 <gmaxwell> I suppose looking at txn which _aren't_ getting included which you consider valid is probably pretty good.
686 2012-06-20 19:48:01 <gavinandresen> but influencing the memory pool seems to me to be a lot costlier-- you'd have to submit large-fee transactions and then hope you mine them
687 2012-06-20 19:49:23 <gmaxwell> e.g. sort the memory pool by fee. When a block comes, whats the highest fee that didn't make it in?  ... now median that over the last N blocks.
688 2012-06-20 19:50:00 <gmaxwell> (or fit some logistic model to it so you get a probability out)
689 2012-06-20 19:50:08 <gavinandresen> gmaxwell: experimentation/data is needed, and I think it might take a couple of iterations to get it close to right
690 2012-06-20 19:51:29 <gmaxwell> In general I'm cautious about using transactions which get _in_ to blocks right away because that gets influenced by invisible agreements. But I think stays out is better, because invisible agreements to keep txn out are not above board at all and hopefully won't exist in any quantity.
691 2012-06-20 19:51:40 <gmaxwell> (and they would be futile because to keep something out everyone must agree)
692 2012-06-20 19:52:17 <gavinandresen> gmaxwell: excellent insight
693 2012-06-20 19:52:31 <gavinandresen> ... that's why we pay you the big bucks
694 2012-06-20 19:52:39 <gmaxwell> I was just about to say that. HAHA
695 2012-06-20 19:53:44 <gmaxwell> How to convert stays out data into an amount is an open question. But I can crunch some numbers on that once I figure out how to mesaure it.
696 2012-06-20 20:03:38 <jgarzik> drat, just missed gavin
697 2012-06-20 20:05:22 <CluckCreek> I'm hoping to release a software project soon and I want some legal advice, I was wondering if anyone here had consulted an attorney they would recommend for bitcoin-related things?
698 2012-06-20 20:16:18 <gmaxwell> sipa: were you thinking of hardcoding some onionseeds in the tor support eventually?
699 2012-06-20 20:16:31 <sipa> gmaxwell: yes
700 2012-06-20 20:16:43 <sipa> though not before there are few decent ones
701 2012-06-20 20:22:48 <sipa> except getaddrinfo and predecessors
702 2012-06-20 20:22:54 <Eliel> CluckCreek: you'll have better luck on bitcointalk.org I think.
703 2012-06-20 20:27:42 <CluckCreek> Would I still be restricted to the newbie forum when I register?
704 2012-06-20 20:27:53 <sipa> i believe so
705 2012-06-20 20:28:56 <galambo> bitcoin lawyer :) heh
706 2012-06-20 20:29:36 <CluckCreek> I know, I thought I'd at least give it a shot. :)
707 2012-06-20 20:52:25 <jgarzik> gmaxwell: one wonders if the tx.ConnectInputs() call in the CreateNewBlock() callchain isn't redundant?  we are iterating through the mempool, whose members IIUC must have passed ConnectInputs checks
708 2012-06-20 20:56:29 <sipa> jgarzik: there are some cases where transactions can end up in the mempool that conflict with the blockchain or with eachother, iirc
709 2012-06-20 20:56:34 <sipa> TD discovered that some time ago
710 2012-06-20 20:57:16 <jgarzik> sipa: that sounds... odd
711 2012-06-20 20:57:32 <sipa> maybe the solution is to more strictly enforce that invariant on the mempool better
712 2012-06-20 20:57:43 <sipa> jgarzik: it's not very likely, it require some reorganisations
713 2012-06-20 20:58:30 <jgarzik> sipa: hrm.  reorg does not sort into mapOrphans vs. mempool, but just tosses everything into mempool?
714 2012-06-20 20:59:24 <sipa> i can't remember the details
715 2012-06-20 20:59:36 <jgarzik> I suppose it must toss them into mempool, otherwise mapOrphans code could throw away transactions
716 2012-06-20 21:00:27 <jgarzik> thus, mempool requires ConnectInputs upon entry from outside world, but reorg hack might pollute mempool with orphans
717 2012-06-20 21:00:41 <jgarzik> I guess ConnectInputs is not redundant in CreateNewBlock, then, alas
718 2012-06-20 21:01:15 <maaku> well, sounds more like the reorg code needs to be fixed
719 2012-06-20 21:01:51 <sipa> iirc it required two reorganisations and a double spend in between to cause the issue
720 2012-06-20 21:02:57 <jgarzik> "fixing" reorg to not toss orphans into mempool would require special orphan-from-reorg handling inside mapOrphans, because mapOrphans code randomly deletes orphans if it exceeds a certain size, for anti-DoS reasons.  The ugly hack is understandable... you don't want a big reorg to kick in anti-DoS code
721 2012-06-20 21:03:23 <jgarzik> thus, orphans in mempool
722 2012-06-20 21:03:28 <maaku> i see
723 2012-06-20 21:07:47 <cande> which URI scheme does bitcoin use?
724 2012-06-20 21:08:22 <sipa> ;;google BIP 0022
725 2012-06-20 21:08:23 <gribble> BIP 0022 - Bitcoin: <https://en.bitcoin.it/wiki/BIP_0022>; Talk:BIP 0022 - Bitcoin: <https://en.bitcoin.it/wiki/Talk:BIP_0022>; Category:BIP - Bitcoin: <https://en.bitcoin.it/wiki/Category:BIP>
726 2012-06-20 21:08:28 <sipa> cande: ^
727 2012-06-20 21:08:41 <sipa> ow
728 2012-06-20 21:08:44 <cande> cool, thnx
729 2012-06-20 21:08:44 <sipa> ;;google BIP 0021
730 2012-06-20 21:08:45 <gribble> BIP 0021 - Bitcoin: <https://en.bitcoin.it/wiki/BIP_0021>; Category:BIP - Bitcoin: <https://en.bitcoin.it/wiki/Category:BIP>; Category:Developer - Bitcoin: <https://en.bitcoin.it/wiki/Category:Developer>
731 2012-06-20 21:08:47 <sipa> that ^
732 2012-06-20 21:08:54 <cande> oh.,
733 2012-06-20 21:08:57 <cande> :)
734 2012-06-20 21:09:15 <cande> since when did bitcoin change to bip 0021 =
735 2012-06-20 21:09:16 <cande> ?
736 2012-06-20 21:11:42 <sipa> "change" ?
737 2012-06-20 21:12:14 <sipa> BIP's are just proposed improvements; people agree with them or they don't; some implementations choose to follow some if they like
738 2012-06-20 21:12:38 <cande> ah
739 2012-06-20 21:13:07 <cande> does bitcoin-qt apply to Bip 0021?
740 2012-06-20 21:13:40 <sipa> on linux it does
741 2012-06-20 21:13:53 <sipa> there were issus with the uri handling on osx and windows
742 2012-06-20 21:14:16 <sipa> though windows will most likely be supported by 0.7.0 when it is release
743 2012-06-20 21:14:36 <sipa> d
744 2012-06-20 21:15:26 <cande> :)
745 2012-06-20 21:15:38 <cande> mm so nice!
746 2012-06-20 21:15:52 <BlueMatt> q
747 2012-06-20 21:16:13 <BlueMatt> oops, sorry
748 2012-06-20 21:18:05 <gmaxwell> sipa: I have my new hidden service bitcoin node up: bun5vdg2agmaxwell.onion:8333 it should be working now.
749 2012-06-20 21:20:02 <cande> gnight
750 2012-06-20 21:20:09 <sipa> gmaxwell: there is a small chance my crawler will notice it
751 2012-06-20 21:21:22 <doublec> does onlynet=tor cause the node to only distribute the tor node informationvia addr and not pass on non-tor addresses to peers?
752 2012-06-20 21:22:03 <sipa> doublec: not related; -onlynet is about which networks it connects to
753 2012-06-20 21:22:43 <doublec> sipa: ok thanks
754 2012-06-20 21:23:19 <sipa> doublec: if you pass -externalip=, it will not attempt to detect other local addresses
755 2012-06-20 21:23:27 <sipa> doublec: so it certainly won't pass any on
756 2012-06-20 21:23:40 <gmaxwell> I thought it passed on foreign ones but not many?
757 2012-06-20 21:24:02 <sipa> gmaxwell: i mean *local* addresses
758 2012-06-20 21:24:12 <sipa> it does relay other addresses it hears about
759 2012-06-20 21:28:44 <doublec> how does the a node on tor bridge to the real network? Does some node on it not use onlynet=tor so it can receive connections to the non-tor world and other onlynet=tor nodes rely on these existing?
760 2012-06-20 21:30:17 <gmaxwell> doublec: if you're using -proxy  without onlynettor you'll connect to both hidden services and IPv4 nodes via exits, and thus bridge.
761 2012-06-20 21:30:44 <sipa> and you can be reachable on both networks too
762 2012-06-20 21:30:46 <gmaxwell> also, there are nodes connected to both directly. E.g. mine is like that.
763 2012-06-20 21:31:17 <doublec> ok thanks. I've been using onlynet=cjdns modelled after the tor/i2p stuff and wanted to make sure I understood things correctly
764 2012-06-20 21:36:02 <jrmithdobbs> gmaxwell: i should really setup my public node to bridge tor
765 2012-06-20 21:41:33 <gmaxwell> jrmithdobbs: it'll be somewhat easier when we pull the hidden services patch.
766 2012-06-20 21:45:47 <gmaxwell> sipa: er.. my onion address is bun5vdg2agmaxwell.onion it's configured correctly... but
767 2012-06-20 21:45:50 <gmaxwell> 06/20/12 23:11:50 AddLocal(bun5vdg2agmaxwel.onion:8333,6)
768 2012-06-20 21:45:53 <gmaxwell> (one l instead of two)
769 2012-06-20 21:46:35 <sipa> heh
770 2012-06-20 21:48:12 <jrmithdobbs> gmaxwell: ah, i thought you were saying that was pulled, that's what i've been waiting for
771 2012-06-20 21:48:29 <sipa> gmaxwell: sure? it's supposed to be 16 characters
772 2012-06-20 21:48:34 <sipa> gmaxwell: you claim yours is 17
773 2012-06-20 21:48:43 <gmaxwell> !
774 2012-06-20 21:48:45 <gmaxwell> lemme check
775 2012-06-20 21:49:37 <gmaxwell> hm. well shallot gave me bun5vdg2agmaxwell.onion
776 2012-06-20 21:49:46 <gmaxwell> lemme see if I can get tor to write it out.
777 2012-06-20 21:49:59 <sipa> gmaxwell: .../service_dir/hostname
778 2012-06-20 21:50:09 <sipa> same place the secret key is in
779 2012-06-20 21:51:07 <gmaxwell> right but I'd written that with the shallot output.
780 2012-06-20 21:51:16 <gmaxwell> well crap some crazy shallot bug!
781 2012-06-20 21:51:22 <gavinandresen> sipa: running as a hidden service worked nicely once I set the obscure Vidalia option not to randomize the SOCKS port
782 2012-06-20 21:51:49 <sipa> gavinandresen: gotten any incoming connections?
783 2012-06-20 21:52:07 <gavinandresen> I'll fire it up again and check....
784 2012-06-20 21:52:20 <gmaxwell> sipa: well nevermind then!
785 2012-06-20 21:52:37 <gmaxwell> apparently my hidden service address is b4ymclq6qeh4um3q.onion ... /me goes to fix shallot
786 2012-06-20 21:53:04 <sipa> gavinandresen: my crawler didn't ever see any onion node except my own, so i doubt any onion addresses are circulating on the p2p net
787 2012-06-20 21:53:19 <jrmithdobbs> i've never seen one
788 2012-06-20 21:53:45 <jrmithdobbs> i've not been looking for them, though, either
789 2012-06-20 21:55:07 <sipa> i've been hunting a bug for hours now... a txindex entry is written to blkindex.dat, it is never deleted, but it can't be found afterwards :(
790 2012-06-20 21:55:37 <sipa> hmmm, maybe the db transaction where it gets written is aborted
791 2012-06-20 21:56:18 <gmaxwell> okay.. shallot consistently writes onion addresses that are too long.
792 2012-06-20 21:56:54 <gavinandresen> programming would be a lot more fun if time-to-fix-a-bug was a constant instead of an exponential function
793 2012-06-20 21:57:30 <gmaxwell> depends on the constant. ;)
794 2012-06-20 21:57:54 <gavinandresen> just about any constant would be better than "might take 2 minutes, might take 2 weeks..."
795 2012-06-20 21:58:10 <sipa> let's define this constant to be log(-1)
796 2012-06-20 21:58:19 <sipa> = an imaginary pie
797 2012-06-20 21:58:22 <BlueMatt> programming would be a lot more fun if bugs didnt exist
798 2012-06-20 21:58:48 <gavinandresen> fixing easy bugs is fun and rewarding.
799 2012-06-20 21:59:24 <sipa> fixing hard bugs is exhausting
800 2012-06-20 21:59:35 <sipa> (though also rewarding...)