1 2012-07-11 00:06:33 <copumpkin> that scriptSig field in http://snapplr.com/0cs1 isn't actually a script, right?
  2 2012-07-11 00:07:00 <copumpkin> in TxIn, that is
  3 2012-07-11 00:07:06 <sipa> it is
  4 2012-07-11 00:07:21 <sipa> but in every meaningful case, it only contains pushes
  5 2012-07-11 00:07:25 <copumpkin> oh okay
  6 2012-07-11 00:07:30 <copumpkin> cause I was only ever seeing pushes in it
  7 2012-07-11 00:07:40 <copumpkin> which doesn't seem very useful
  8 2012-07-11 00:08:08 <copumpkin> I'm seeing stuff like [[PushData "255255NULGS",PushData "223NUL"]]
  9 2012-07-11 00:08:09 <TigrBot> http://en.wikipedia.org/wiki/PushData "255255NULGS",PushData "223NUL"
 10 2012-07-11 00:08:21 <copumpkin> heh
 11 2012-07-11 00:09:27 <sipa> yes?
 12 2012-07-11 00:09:36 <copumpkin> does that seem legit?
 13 2012-07-11 00:09:42 <copumpkin> ignoring my fake opcode there
 14 2012-07-11 00:10:09 <sipa> not really
 15 2012-07-11 00:10:11 <luke-jr> copumpkin: also see BIP 18
 16 2012-07-11 00:10:13 <sipa> but it's not illegal
 17 2012-07-11 00:10:24 <BlueMatt> if ts coinbase it is
 18 2012-07-11 00:10:25 <copumpkin> sipa: these are the first blocks that I'm parsing
 19 2012-07-11 00:10:33 <BlueMatt> (seems legit)
 20 2012-07-11 00:10:45 <copumpkin> for the very first block, I have
 21 2012-07-11 00:10:46 <copumpkin> [[PushData "255255NULGS",PushData "EOT",PushData "The Times 03/Jan/2009 Chancellor on brink of second bailout for banks"]]
 22 2012-07-11 00:10:47 <TigrBot> http://en.wikipedia.org/wiki/PushData "255255NULGS",PushData "EOT",PushData "The Times 03/Jan/2009 Chancellor on brink of second bailout for banks"
 23 2012-07-11 00:10:56 <BlueMatt> wtf is TigrBot?
 24 2012-07-11 00:11:12 <nimdAHK> a failed automatic arbitrage system
 25 2012-07-11 00:11:18 <nimdAHK> but this is the wrong channel for that
 26 2012-07-11 00:11:19 <sipa> copumpkin: oh, those are coinbases
 27 2012-07-11 00:11:29 <sipa> copumpkin: they just contain arbitrary data being pushed really
 28 2012-07-11 00:11:48 <copumpkin> okay, so the script isn't meant to be run in isolation, I assume?
 29 2012-07-11 00:11:51 <sipa> originally the difficulty bits and the extranonce
 30 2012-07-11 00:11:52 <copumpkin> or there's some implied other operation afterwards
 31 2012-07-11 00:12:02 <sipa> coinbase inputs aren't every executed
 32 2012-07-11 00:12:04 <sipa> *ever
 33 2012-07-11 00:12:06 <copumpkin> okay
 34 2012-07-11 00:12:34 <copumpkin> so, if I understand correctly, the inputs to a given transaction aren't directly available in the transaction structure, right? I need to be stateful and look the inputs up in a previous transaction by reference?
 35 2012-07-11 00:12:45 <sipa> yes
 36 2012-07-11 00:12:48 <copumpkin> :(
 37 2012-07-11 00:12:58 <copumpkin> oh well, makes sense I guess
 38 2012-07-11 00:13:06 <copumpkin> I was hoping to run in near-constant memory :)
 39 2012-07-11 00:13:18 <BlueMatt> prune
 40 2012-07-11 00:13:44 <copumpkin> how do I know that no future transaction will refer to a given transaction output?
 41 2012-07-11 00:13:57 <BlueMatt> if its spent, you can
 42 2012-07-11 00:13:57 <sipa> it cannot be spent twice
 43 2012-07-11 00:14:02 <copumpkin> oh, fair enough
 44 2012-07-11 00:14:12 <sipa> i've been working on my "ultraprune" branch that only retains (for not-fully spent tx's): height, whether it was a coinbase or not, and its unspent outputs
 45 2012-07-11 00:14:28 <sipa> with some clever encodings, that takes around 70 MB
 46 2012-07-11 00:14:55 <luke-jr> sipa: any reason not to collapse the first 2 by using height=-1 for non-coinbases txns?
 47 2012-07-11 00:15:42 <sipa> luke-jr: i originally did it that way, but that makes it impossible to keep the current mining algorithm (because you need to know the confirmation count of inputs)
 48 2012-07-11 00:15:55 <luke-jr> ah
 49 2012-07-11 00:15:57 <sipa> technically not needed to verify, but quite cheap, and adds functionality
 50 2012-07-11 00:18:31 <sipa> almost everything is stored as base128 varints, so height is only 3 bytes for now (and will be for the coming 40 years)
 51 2012-07-11 00:19:45 <TigrBot> jgarzik ... that was not very nice!
 52 2012-07-11 00:20:37 <copumpkin> MrTiggr :O
 53 2012-07-11 00:23:03 <jgarzik> "so today i was completely wiped out by [SatoshiDice ...] after 75 consecutive losses i was wiped out, because i was placing relatively large bets. i have decided not to play this game any more.  "
 54 2012-07-11 00:23:18 <nimdAHK> lol
 55 2012-07-11 00:23:38 <[Tycho]> Smart decision.
 56 2012-07-11 00:25:21 <luke-jr> what was he expecting? :P
 57 2012-07-11 00:27:01 <walruscode> nimdAHK, I've run into a bit of a problem. You see, I'm writing the script to work with the Litecoin blockchain, but the litecoin daemon has no getblock method, it seems. /facepalm
 58 2012-07-11 00:27:12 <[Tycho]> A miracle.
 59 2012-07-11 00:27:17 <nimdAHK> :/
 60 2012-07-11 00:32:42 <gmaxwell> jgarzik: it still makes me sad!
 61 2012-07-11 00:42:03 <luke-jr> doublec: cjdns needs rebasing
 62 2012-07-11 00:46:49 <doublec> luke-jr: yeah, I'll do that and address the pull comments soon
 63 2012-07-11 00:47:37 <doublec> for some reason I sometimes see two connections from my node running on cjdns
 64 2012-07-11 00:47:44 <doublec> but both are too one peer
 65 2012-07-11 00:47:49 <doublec> s/too/to
 66 2012-07-11 00:48:03 <doublec> I don't know why it's connecting to it twice
 67 2012-07-11 00:48:22 <doublec> or that peers connecting back to me maybe
 68 2012-07-11 01:10:21 <nimdAHK> hodamn what a spam
 69 2012-07-11 01:33:44 <Karmaon> nimdAHK spam?
 70 2012-07-11 01:34:06 <Karmaon> You mean the connect/disconnect messages?
 71 2012-07-11 02:54:30 <amiller> hell yeah, i just worked out an O(N^2 * D * k) solution for bitcoin parameters
 72 2012-07-11 02:54:53 <amiller> i think that's what i thought i could get before, but i finally made it from start to finish with math
 73 2012-07-11 02:56:30 <amiller> the N means that the advantage of the network is 1/2 (1 + 1/N), so for example 51% is N=50
 74 2012-07-11 02:57:55 <amiller> the D is the message delay, like a communications radius, where a block gets propagated to everyone within D seconds or w/e
 75 2012-07-11 03:00:05 <luke-jr> amiller: try minutes
 76 2012-07-11 03:00:21 <amiller> w/evs
 77 2012-07-11 03:01:53 <gmaxwell> luke-jr: the units don't matter of course.
 78 2012-07-11 03:02:13 <luke-jr> maybe. I wasn't following what amiller was doing.
 79 2012-07-11 03:02:29 <gmaxwell> making a proof under what conditions bitcoin will converge to the majority consensus.
 80 2012-07-11 03:03:32 <gmaxwell> (in the face of some badguys)
 81 2012-07-11 03:31:09 <midnightmagic> hey I wonder if the magic word still works..
 82 2012-07-11 03:31:34 <midnightmagic> amiller: do you have something we can read up somewhere?
 83 2012-07-11 03:31:44 <midnightmagic> maybe it's time to try using the magic word again..
 84 2012-07-11 03:32:23 <amiller> not yet, gotta run through it a few more times
 85 2012-07-11 03:32:41 <midnightmagic> raouaw
 86 2012-07-11 03:32:48 <midnightmagic> lol
 87 2012-07-11 03:32:49 <amiller> gonna submit this shit to a conference, if i don't break down in tears in the next few days if it's broken
 88 2012-07-11 03:34:30 <midnightmagic> in case anybody's curious, for about 6 months from Nov '10 onwards, anytime someone would mention ra oul's nickname he'd suddenly join the channel and answer questions. it was like rubbing a magic lamp, it was great.
 89 2012-07-11 03:34:37 <amiller> i'm providing a lower bound for the longest chain miners know about, even accounting for the delay
 90 2012-07-11 03:34:54 <amiller> that lower bound is more than half the upper bound for the total number of blocks anyone could have found
 91 2012-07-11 03:35:09 <amiller> hence, there is a unique longest chain that every miner knows about
 92 2012-07-11 03:35:52 <amiller> i'm modeling this lower bound as a simple random process, it's alternating phases of waiting for a block to arrive, and then just skipping over the message delay
 93 2012-07-11 03:36:06 <amiller> the upper bound for the total number of blocks is just a normal poisson process
 94 2012-07-11 03:40:30 <midnightmagic> very awesome. :)
 95 2012-07-11 06:48:15 <gribble> New news from bitcoinrss: luke-jr opened issue 1579 on bitcoin/bitcoin <https://github.com/bitcoin/bitcoin/issues/1579>
 96 2012-07-11 09:36:10 <yellowhat> did anyone of the devs get a notice from Jacob Appelbaum? http://twitter.com/ioerror/status/78480502641803264
 97 2012-07-11 09:36:43 <yellowhat> Jacob Appelbaum - @ioerror - Bitcoin prediction: Major bugs in the near future will mess with the "market"
 98 2012-07-11 09:37:17 <yellowhat> @ioerror Bitcoin prediction seem to become en vogue. Have any particular bug in mind?
 99 2012-07-11 09:37:19 <jeremias> ounds like FUD to me, until further information is given
100 2012-07-11 09:37:19 <yellowhat> @ioerrorandreasdotorg Yes. :-)
101 2012-07-11 09:37:37 <yellowhat> jacob appelbaum is a serious guy i would not rule it out
102 2012-07-11 09:37:48 <yellowhat> he is a good guy (TM)
103 2012-07-11 09:38:27 <jeremias> well, I don't care if he is a good guy or not, just spreading messages like that without any details is annoying
104 2012-07-11 09:39:32 <yellowhat> maybe he is waiting for gavin to contact him privately?
105 2012-07-11 09:40:37 <cccp> isnt that tweet 13 months old?
106 2012-07-11 09:41:04 <yellowhat> he is the lead TOR developer, so he shoud know what he is talking about
107 2012-07-11 09:41:08 <yellowhat> ahh
108 2012-07-11 09:41:18 <yellowhat> you are right 11 months old ...
109 2012-07-11 09:41:24 <sipa> iirc gavin contacted him about back then
110 2012-07-11 09:41:31 <yellowhat> 13 to be correct like your
111 2012-07-11 09:41:51 <yellowhat> just forget everything i wrote then
112 2012-07-11 09:44:02 <gribble> New news from bitcoinrss: laanwj opened pull request 1580 on bitcoin/bitcoin <https://github.com/bitcoin/bitcoin/pull/1580>
113 2012-07-11 10:14:17 <gribble> New news from bitcoinrss: laanwj opened pull request 1582 on bitcoin/bitcoin <https://github.com/bitcoin/bitcoin/pull/1582> || Diapolo opened pull request 1581 on bitcoin/bitcoin <https://github.com/bitcoin/bitcoin/pull/1581>
114 2012-07-11 11:07:12 <helo> 'mess with the "market"' sounds like a prediction of problems with exchanges
115 2012-07-11 11:56:16 <jgarzik> whee, got a blk0002.dat here
116 2012-07-11 11:56:30 <kinlo> everybody does :)
117 2012-07-11 11:56:37 <jgarzik> time to start distributing a canonical blk0001.dat via torrent
118 2012-07-11 11:56:58 <jgarzik> rather than burdening the DNS seed-listed nodes with a heavy block download burden
119 2012-07-11 11:58:48 <kinlo> aren't the dns seeded nodes just random nodes with high uptime, hence always different ones?
120 2012-07-11 12:00:02 <jgarzik> kinlo: no
121 2012-07-11 12:00:33 <kinlo> dns seeding doesn't allow different ports either I assume
122 2012-07-11 12:00:38 <jgarzik> kinlo: that's only 100% true for sipa's seed.  others like bitseed.xf2.org are taken from the wiki list of fallback nodes, and are largely static.
123 2012-07-11 12:00:51 <kinlo> ic
124 2012-07-11 12:01:30 <jgarzik> kinlo: but even so, BitcoinJ exclusively connects to DNS seed-returned nodes, and no other.  And Bitcoin's Initial Block Download (IBD) logic winds up defaulting to download from the first well connected remote P2P node.
125 2012-07-11 12:01:47 <kinlo> ic
126 2012-07-11 12:01:49 <jgarzik> thus heaping more work on seed-listed nodes, rather than other listening nodes on the P2P network
127 2012-07-11 12:02:19 <jgarzik> the initial block download, in particular, always downloads the same data, making it a prime torrent candidate IMO
128 2012-07-11 12:02:44 <kinlo> in any case, we might should enforce bitcoin not being able to start without an existing blockchain, forcing people to download a bootstrap chain
129 2012-07-11 12:03:30 <jgarzik> ...as long as you validate the download somehow, sure
130 2012-07-11 12:04:42 <sipa> you mean download via torrent, to be fed to loadblock or similar?
131 2012-07-11 12:04:45 <kinlo> should it be?
132 2012-07-11 12:04:48 <gavinandresen> I nominate kinlo to pay for hosting blk0001.dat
133 2012-07-11 12:05:20 <kinlo> jgarzik: imagine that someone gives you an invalid blk0001.dat, your client will still verify it, no?
134 2012-07-11 12:05:35 <kinlo> jgarzik: given that you do not just load it as is, but fully verify it
135 2012-07-11 12:05:56 <kinlo> gavinandresen: depending on the bandwith, I might be able to arrange something, I work at an isp...
136 2012-07-11 12:06:01 <jgarzik> kinlo: -loadblock verifies, yes
137 2012-07-11 12:06:09 <jgarzik> sipa: yes
138 2012-07-11 12:06:19 <sipa> btw, separating the block info and coin info has another nice advantage: remove the coindb, and all that happens is a reconnect
139 2012-07-11 12:06:24 <kinlo> jgarzik: the bootstrap should do somethng like loadblock...
140 2012-07-11 12:06:35 <jgarzik> kinlo: it already does (over the network)
141 2012-07-11 12:06:48 <sipa> no new block files created, or new indexes to be created
142 2012-07-11 12:07:00 <jgarzik> the point is to do it over torrent or other instead, while still verifying
143 2012-07-11 12:07:13 <gavinandresen> last time we talked about this, the conclusion was there was no real advantage to using torrent versus just asking peers to download
144 2012-07-11 12:07:22 <kinlo> gavinandresen: in the end, the current dnsseeds seem to be doing the hosting of those files already, so I don't see it to be that expensive in bandwith
145 2012-07-11 12:07:23 <gavinandresen> if we can do a better job of asking peers, then cool
146 2012-07-11 12:08:06 <gavinandresen> on a meta level, I'm very "meh" -- since in the medium-term-run only solo miners and pools will need the full chain
147 2012-07-11 12:08:09 <kinlo> gavinandresen: what about the idea of forcing a client to request the user of a blockchain if the current datadir is empty?
148 2012-07-11 12:08:26 <gavinandresen> kinlo: meh
149 2012-07-11 12:08:51 <gavinandresen> what problem are y'all trying to solve?
150 2012-07-11 12:09:33 <kinlo> just answering jgarzik's question about distributing the blockchain
151 2012-07-11 12:09:46 <BlueMatt> jgarzik: I was under the impression that was 100% for every one /but/ yours
152 2012-07-11 12:10:13 <jgarzik> bitcoin P2P protocol is a poor substitute for a real file transfer protocol.  Sure, we can import all the useful file transfer heuristics into bitcoin, for better peer selection
153 2012-07-11 12:10:33 <BlueMatt> do we get a blk0002.dat when -loadblock'ing, or just with a few orphans in blk0001.dat?
154 2012-07-11 12:10:37 <jgarzik> the bigger the blockchain, the more the protocol use segments into (a) bulk chain downloads and (b) everything else
155 2012-07-11 12:11:16 <jgarzik> it's a mess for server profiling and long term planning
156 2012-07-11 12:11:40 <gavinandresen> jgarzik: the problem is difficulty in predicting server bandwidth?
157 2012-07-11 12:11:41 <jgarzik> BlueMatt: natural chain spills over into blk0002.dat now
158 2012-07-11 12:11:43 <BlueMatt> jgarzik: anyway, I still disagree that just distributing blk0001.dat is the way to go...obvious optimizations to the way bitcoin does ibd could get the same effect without having to
159 2012-07-11 12:12:03 <sipa> i figure there will be verification nodes and archival servers in the long term
160 2012-07-11 12:12:06 <jgarzik> BlueMatt: ...thus reinventing bittorrent, but poorly
161 2012-07-11 12:12:26 <jgarzik> BlueMatt: once you have all its IP ToS optimizations, peer selection optimizations, etc.
162 2012-07-11 12:13:04 <BlueMatt> who cares? its not the the bottleneck, and using it over bittorrent blk0001.dat would have only a minor bandwidth cost over bittorrent
163 2012-07-11 12:13:07 <jgarzik> "good peers for blockchain download " will be a shrinking set of total nodes, and selection becomes ever more important
164 2012-07-11 12:13:21 <BlueMatt> and just the really obvious and simple optimizations would give it the same result
165 2012-07-11 12:13:31 <BlueMatt> no need for IP ToS stuff, or getting nearly as fancy as bittorrent
166 2012-07-11 12:13:46 <jgarzik> BlueMatt: it matters if you observe the impact DNS seeds inflict upon public nodes
167 2012-07-11 12:13:51 <gmaxwell> The blockchain download should be in the background in any case, so it shouldn't be time critical.
168 2012-07-11 12:14:24 <gmaxwell> It can't be done in the background if you're depending on the user torrenting some multigigabyte file before they start.
169 2012-07-11 12:14:26 <BlueMatt> jgarzik: thats my point, just downloading blocks from all 8 peers during ibd would get rid of most of that impact
170 2012-07-11 12:14:26 <jgarzik> gmaxwell: not true from user's PoV...  they are waiting to use the problem.
171 2012-07-11 12:15:20 <gavinandresen> again, I think we need to figure out what problem(s) we want to solve.  User experience won't be helped by switching to a torrent download
172 2012-07-11 12:15:20 <jgarzik> gmaxwell: *program
173 2012-07-11 12:15:22 <gmaxwell> jgarzik: You don't have to have the whole chain to use the software. The node could bootstrap as SPV, fetching headers first.
174 2012-07-11 12:15:27 <BlueMatt> simple, obvious optimizations that we should do anyway that will get the same result without forcing people to go download a blk0001.dat file to get started (which is horrible for users, btw): download blocks from all peers, thread/buffer blocks, done
175 2012-07-11 12:15:45 <gavinandresen> I'm not sure server usage would, either, unless we ship a full bittorrent client
176 2012-07-11 12:16:00 <gmaxwell> (sorry I'm just tuning into this discussion, forgive me if I repeated something already said)
177 2012-07-11 12:16:20 <jgarzik> Idelaly users would see a rapid download (hopefully with progress bar), then a rapid import (hopefully with progress bar), then immediate live network access.
178 2012-07-11 12:16:24 <jgarzik> *Ideally
179 2012-07-11 12:16:36 <BlueMatt> jgarzik: yep, and bittorrent wont speed any of those bars up
180 2012-07-11 12:16:39 <gavinandresen> sure, that should be pretty easy, headers first then backfill....
181 2012-07-11 12:16:52 <jgarzik> BlueMatt: over today's client, yes it will
182 2012-07-11 12:17:10 <gmaxwell> jgarzik: "rapid download" .. how rapidly will you download 50 GBytes?
183 2012-07-11 12:17:16 <gavinandresen> if we get bloom filters (hint hint) then do SPV stuff and fetch full blocks in the background....
184 2012-07-11 12:17:17 <BlueMatt> jgarzik: again, implementing bittorrent would be more effort than just solving the underlying issues
185 2012-07-11 12:17:48 <gmaxwell> gavinandresen: the bitcoinj stuff make a useful optimization which is that a new wallet knows there will be no transactions for it before the height at which it was created.
186 2012-07-11 12:18:03 <BlueMatt> jgarzik: I just feel like you are working around an issue that has low-hanging fruit optimizations that would solve the issue itself
187 2012-07-11 12:18:04 <jgarzik> gmaxwell: bittorrent peer selection hunts for fast peers.  we don't.  the obvious result occurs.  :)
188 2012-07-11 12:18:09 <gmaxwell> gavinandresen: so it never even bothers downloading the blocks before that heigh... For a fullnode + wallet it means those could come last.
189 2012-07-11 12:18:17 <gavinandresen> gmaxwell: yup
190 2012-07-11 12:18:41 <BlueMatt> jgarzik: bandwidth is very far from the problem, just one peer with 100k upload would be just fine for block download
191 2012-07-11 12:18:48 <gavinandresen> jgarzik: but unless you're a very impatient solo miner, you shouldn't need the full blockchain at all
192 2012-07-11 12:19:38 <jgarzik> gavinandresen: today's bitcoin-qt users need the full blockchain...  and they are impatient enough to continue complaining on the forums.
193 2012-07-11 12:19:41 <gmaxwell> jgarzik: Yea, you're not following my point. :)  Even if you have _perfect_ transfer, 50 gigs is going to take 1.2 hours over a 100mbit/s link at line rate.
194 2012-07-11 12:19:46 <jgarzik> BlueMatt: running several public nodes, I disagree
195 2012-07-11 12:19:59 <sipa> i think it woulfmd not be hard to have a shipped blockchain with ultraprune, which is imported at first startup
196 2012-07-11 12:20:08 <jgarzik> BlueMatt: if you judge from log lines, they spend the majority of their time shoveling blocks
197 2012-07-11 12:20:14 <jgarzik> and bandwidth usage
198 2012-07-11 12:20:20 <BlueMatt> jgarzik: yep, because we dont thread/buffer blocks
199 2012-07-11 12:20:31 <gavinandresen> jgarzik: right... but I think if we switch to IBD via bittorrent then we'd just swap those complaints with a whole 'nother set of complaints
200 2012-07-11 12:20:37 <BlueMatt> jgarzik: if you do that, you can spend most of your time in block checking on tmpfs with just 10m down
201 2012-07-11 12:20:39 <jgarzik> gmaxwell: yes, agreed
202 2012-07-11 12:21:22 <gmaxwell> jgarzik: which means for user expirence just changing download mechnism isn't sufficient. We need to get that download of historical data into the background. And once it is simple factors of two probably don't matter much.
203 2012-07-11 12:21:36 <jgarzik> gavinandresen: perhaps, but x<y.  it eliminates slow downloads, bad peer choice, and public node punishment.
204 2012-07-11 12:21:56 <gavinandresen> Shipping a "bitcoin starter" that is a 3GB download, blockchain snapshot included, seems like the easiest/quickest short-term fix.  I'm ok with that, as long as somebody else pays the bandwidth costs.
205 2012-07-11 12:22:19 <BlueMatt> gavinandresen: shipping a bitcoin starter that is 100m would be better ;)
206 2012-07-11 12:22:29 <gavinandresen> BlueMatt: agreed
207 2012-07-11 12:22:42 <gmaxwell> jgarzik: As mentioned before, I'm seeing no evidence of public node punishment here, and my nodes are returned by sipa's seeder at least. Perhaps you should make your dnsseed list non-static.
208 2012-07-11 12:23:00 <gavinandresen> BlueMatt: the work would be all installer/packaging, which should be reusable for whatever starting state you want to ship
209 2012-07-11 12:23:37 <BlueMatt> gavinandresen: yep...
210 2012-07-11 12:24:02 <jgarzik> gmaxwell: that just makes the problems more difficult to measure
211 2012-07-11 12:24:07 <jgarzik> e.g. burying head in sand
212 2012-07-11 12:24:32 <gavinandresen> I agree with gmaxwell: if the problem is some seeds getting hammered because of a static DNS list....
213 2012-07-11 12:25:03 <gmaxwell> And also because nodes pull only from the first node they connect to.
214 2012-07-11 12:25:11 <gavinandresen> that one would be easy to fix
215 2012-07-11 12:25:11 <jgarzik> which is a major problem
216 2012-07-11 12:25:24 <gavinandresen> (famous last words)
217 2012-07-11 12:25:57 <jgarzik> gavinandresen: yes, I had thought about that as a possible solution:  separate program which torrents the block chain, indexes, then executes bitcoin-qt.
218 2012-07-11 12:26:06 <BlueMatt> jgarzik: one other thing, as of 0.7, we give users an easy way to import blk0001.dat, and tcatm already (graciously) hosts chain snaps, so maybe we could link to his site to download snaps if people want the import to go faster
219 2012-07-11 12:26:34 <gmaxwell> The pulling behavior has other negative consequences.. if you first connect to a broken node you won't start pulling for real until the next block (perhaps an half hour later!)
220 2012-07-11 12:27:50 <gmaxwell> BlueMatt: yea, the fact that we have import is good. Though... perhaps loadblock needs a rpc call or even a file->open_blockfile. 0_o because asking people to change commandline parameters isn't exactly user friendly.
221 2012-07-11 12:28:10 <BlueMatt> yea, that would be nice
222 2012-07-11 12:28:58 <BlueMatt> (also, loadblock that downloads from a url and imports as it downloads would be cool, but thats potentially another lib...
223 2012-07-11 12:28:59 <gavinandresen> how about:  if there are blk????.dat files in the executable's directory, automagically -loadblock them
224 2012-07-11 12:29:09 <gavinandresen> (command line switch to disable for us geeks)
225 2012-07-11 12:29:26 <BlueMatt> gavinandresen: I was looking at that for a checkpoints file, and it gets pretty ugly
226 2012-07-11 12:29:44 <jgarzik> well
227 2012-07-11 12:29:57 <jgarzik> do we mean -loadblock or -justreindex ?
228 2012-07-11 12:30:04 <BlueMatt> (have to have specific code per os to figure out what dir your exe is in to begin with, and Ive never seen any version for bsd/other not very standard oses)
229 2012-07-11 12:30:13 <jgarzik> -loadblock creates a duplicate blk0001.dat essentially
230 2012-07-11 12:30:30 <gavinandresen> BlueMatt: I thought boost::filesystem had a call to figure that out, might be misremembering though
231 2012-07-11 12:30:44 <BlueMatt> I didnt see one, but I may have missed it
232 2012-07-11 12:30:47 <BlueMatt> I only know qt has one
233 2012-07-11 12:30:48 <gmaxwell> jgarzik: yes, and if you're so short on space that this is a major problem.. then you're going to be hosed in N months in any case.
234 2012-07-11 12:31:16 <gmaxwell> With a file->load blocks there could be a checked by default [delete when done].
235 2012-07-11 12:31:19 <gavinandresen> Users won't know where to put a downloaded blk*.dat, won't know how to create the -datadir, etc
236 2012-07-11 12:31:44 <BlueMatt> gavinandresen: yea file->load block file url would be nice
237 2012-07-11 12:31:48 <jgarzik> gmaxwell: just pointing it out a fact, in case it wasn't obvious.  previously, some devs occasionally assumed -loadblock only indexed, if $datadir/blk0001.dat was being imported.
238 2012-07-11 12:31:55 <jgarzik> which is not true
239 2012-07-11 12:32:10 <jgarzik> (that came up in previous -loadblock discussions)
240 2012-07-11 12:32:21 <jgarzik> users must also avoid putting the -import- file in the bitcoin datadir
241 2012-07-11 12:32:27 <gmaxwell> ah, I assume that now everyone here uses it daily. :)
242 2012-07-11 12:33:12 <gmaxwell> well the import file should be called not blk0001.dat  but something like bitcoin-bootstrap.dat
243 2012-07-11 12:33:28 <gmaxwell> so even if they threw it in there it would be harmless.
244 2012-07-11 12:33:43 <jgarzik> ack
245 2012-07-11 12:35:03 <gmaxwell> also, I suppose it wouldn't just be a blk0001.dat .. it would be a concatination of blk000*.dat.  (I don't think it being over 2GB would be an enormous problem for anything anymore, esp since we don't seek in it, though that should be tested)
246 2012-07-11 12:37:06 <BlueMatt> auto-importing any found bitcoin-bootstrap.dats that are in datadir/exe dir/maybe one dir up from exe would be cool, as long as it more efficiently figured out if it didnt need to import a given set of blocks instead of ProcessBlock'ing them anyway
247 2012-07-11 12:37:58 <gmaxwell> We could perhaps make loadblock a bit smarter about blocks it already has...
248 2012-07-11 12:38:20 <gmaxwell> but, hm, you don't want to rescan that file every startup.
249 2012-07-11 12:38:32 <jgarzik> the first thing ProcessBlock() does is check the hash for dups... not much more efficiency to be gained
250 2012-07-11 12:38:58 <BlueMatt> jgarzik: except not importing 180k blocks and checking if we already have them to begin with ;)
251 2012-07-11 12:39:09 <jgarzik> nod
252 2012-07-11 12:39:19 <jgarzik> if height > 180k, ignore bootstrap.dat
253 2012-07-11 12:39:24 <gavinandresen> ack
254 2012-07-11 12:39:47 <gavinandresen> we really only care about the brand-new-user use case, right?
255 2012-07-11 12:40:04 <jgarzik> pretty much
256 2012-07-11 12:40:22 <BlueMatt> or, really simply, if our blk*.dat >> bitcoin-bootstrap.dat, dont import unless -forceimport
257 2012-07-11 12:40:41 <gmaxwell> oh ^ thats a sane sounding metric.
258 2012-07-11 12:40:54 <BlueMatt> heh, not really, but for new users it would work
259 2012-07-11 12:41:25 <jgarzik> gmaxwell BlueMatt: didn't one of you do peer selection work?
260 2012-07-11 12:41:32 <jgarzik> patches lying around anywhere?
261 2012-07-11 12:41:37 <BlueMatt> not me
262 2012-07-11 12:42:07 <BlueMatt> rebroad had (has?) a pull entitled block download from multiple peers simultaneously, but it doesnt actually do that :(
263 2012-07-11 12:42:36 <gmaxwell> I have a bunch of random peer selection hacks.. but nothing currently applicable. I think our selection isn't really problematic now. The download blocks from multiple peers thing is what we need.
264 2012-07-11 12:42:43 <jgarzik> even something that avoids the first few peer addresses found would be useful
265 2012-07-11 12:42:49 <jgarzik> to avoid the hammer-public-node problem
266 2012-07-11 12:42:51 <gmaxwell> Biggest gap in our peer handling is we have no peer rotation at all.
267 2012-07-11 12:42:55 <jgarzik> nod
268 2012-07-11 12:43:57 <jgarzik> for starters (a) don't IBD from the first peer we find, and (b) if requests receive no response within 10 seconds, try another connected peer
269 2012-07-11 12:44:55 <jgarzik> both of those should go a long way towards spreading the IBD load
270 2012-07-11 12:45:05 <jgarzik> and being responsive, if a remote peer burps
271 2012-07-11 12:45:15 <jgarzik> IBD pauses of ~30 minutes are still common
272 2012-07-11 12:45:31 <BlueMatt> jgarzik: we are going to have to download from public nodes no matter what, if dnsseeds return a random subset of what public nodes are out there, it is no different to download from a set of those nodes than just using them to get addresses
273 2012-07-11 12:45:41 <BlueMatt> jgarzik: your nodes being static is a separate issue...
274 2012-07-11 12:46:05 <BlueMatt> downloading from multiple peers to avoid burps/spread the load better is still something that should be done
275 2012-07-11 12:46:50 <gmaxwell> so.. the really obvious thing to do ...
276 2012-07-11 12:46:55 <gmaxwell> is to list     {"xf2.org", "bitseed.xf2.org"},
277 2012-07-11 12:46:58 <gmaxwell> _LAST_
278 2012-07-11 12:47:00 <gmaxwell> not first.
279 2012-07-11 12:47:16 <gmaxwell> since his list is static, I bet his first output node is very very frequently getting the IBD.
280 2012-07-11 12:47:32 <gmaxwell> so there is probably almost no load balance at all.
281 2012-07-11 12:49:01 <gmaxwell> delaying the ibd 10 seconds and picking a random peer would help.
282 2012-07-11 12:52:07 <sipa> ha
283 2012-07-11 12:53:40 <jgarzik> no objections to the list change
284 2012-07-11 12:54:11 <jgarzik> but the way addrman works, plus A record randomization at the DNS server level, I don't think one particular node is getting hit hard
285 2012-07-11 12:56:02 <BlueMatt> fun test: after my addnode pull, -addnode=all the dnsseeds then use getaddednodeinfo to see to which dnsseed you are connected (have to do it quick before the dnsseeds rotate and you lose that info)
286 2012-07-11 12:59:54 <BlueMatt> (in the few tests I did, while mine was still up, it would get mostly mine, and a connection or two to jgarzik's)
287 2012-07-11 13:01:27 <gmaxwell> I just went ahead and reordered it, it's an obviously safe change, and I did a quick test, and indeed the first node I connected to in two restarts was on of the ones returned by bitseed.xf2.org.
288 2012-07-11 13:01:54 <gmaxwell> jgarzik: addrman strongly prefers the recent data for the first peers IIRC.
289 2012-07-11 13:02:15 <sipa> less strongly than the old code
290 2012-07-11 13:03:26 <jgarzik> DNS seeds are added to addrman with a random age
291 2012-07-11 13:03:52 <gmaxwell> You mean my two sample test is not conclusive= :)
292 2012-07-11 13:04:08 <gmaxwell> Also matters for proxydns too.
293 2012-07-11 13:04:28 <sipa> still, the first ever entry to be added to addrman is quite likely to be the first connected to
294 2012-07-11 13:05:20 <sipa> whatever age it gets assigned
295 2012-07-11 13:05:28 <jgarzik> it adds to addrman in batches
296 2012-07-11 13:05:32 <jgarzik> one batch per DNS server
297 2012-07-11 13:05:51 <jgarzik> DNS servers, dnspark's and other BIND at least, return A's in random order
298 2012-07-11 13:06:01 <gmaxwell> even the static dns is a RR, so you'll get at least load balancing there.
299 2012-07-11 13:06:04 <gmaxwell> Right.
300 2012-07-11 13:06:21 <gmaxwell> but that still gives the dozen or so nodes on your list a disproportionate load.
301 2012-07-11 13:07:04 <jgarzik> yep
302 2012-07-11 13:07:32 <jgarzik> waiting, before choosing a peer, would do a lot I think (as mentioned earlier)
303 2012-07-11 13:07:44 <jgarzik> *[first | IBD] peer
304 2012-07-11 13:08:14 <jgarzik> might also shut down some attacks, by giving the user client better diversity?
305 2012-07-11 13:08:28 <jgarzik> well, s/shut down/make more difficult/
306 2012-07-11 13:08:55 <BlueMatt> does dnsseed wait for each dnsseed request to return before trying the next one?
307 2012-07-11 13:09:08 <sipa> yes
308 2012-07-11 13:09:09 <gmaxwell> it doesn't actually compare them... I mean you could get fancy like fetch the best block from each and choose among the ones with the highest pow.. which doesn't stop attacks but would avoid broken peers.
309 2012-07-11 13:09:21 <BlueMatt> (if it is, its waiting for mine to time out, which means you would pretty much only get jgarzik's nodes atm...)
310 2012-07-11 13:09:58 <gavinandresen> always fetching the best block from every new peer seems like a good idea in any case
311 2012-07-11 13:22:56 <gavinandresen> gmaxwell: I updated https://gist.github.com/2961409  with my latest transaction fee thoughts.  Away for a while, but will be back in an hour or so
312 2012-07-11 13:25:32 <BlueMatt> gavinandresen: did you see my fork of that gist that made some other recommendations? https://gist.github.com/3041216
313 2012-07-11 13:26:29 <gavinandresen> BlueMatt: yep, if I recall correctly you went off on some tangents I didn't want to follow
314 2012-07-11 13:26:59 <BlueMatt> gavinandresen: eg specify whats spv clients would do, note that a better prio algorithm should be implemented, etc
315 2012-07-11 13:29:19 <gavinandresen> BlueMatt: I decided to focus on what we could/should do right now (and right now I really am going away for a bit)
316 2012-07-11 13:33:39 <gmaxwell> gavinandresen: as far as the sendmany vs output reduction goes a size metric that is linear on size of outputs + size of inputs can never be cheaper/higher priority by spliting it up vs combining, and would still usually be cheaper merged.  (this doesn't work if you make the costs non-linear though)
317 2012-07-11 14:02:52 <jgarzik> gavinandresen: hmmm, does the review of current code accurately capture current policy?
318 2012-07-11 14:03:30 <jgarzik> gavinandresen: CreateNewBlock() passes progressively larger nBlockSize values to GetMinFee(), thereby creating different policy zones (and price points) with one block
319 2012-07-11 14:03:53 <jgarzik> as the block gets larger during CreateNewBlock(), block space is ever more expensive
320 2012-07-11 14:04:19 <jgarzik> gavinandresen: (reading your gist)
321 2012-07-11 14:05:29 <BlueMatt> gavinandresen: posted a comment that essentially diffs your gist from my fork and explains the motivations
322 2012-07-11 14:05:53 <jgarzik> maybe it is how I'm reading it, but under "Block creation rules" it is not as simple as "if block size > 27k, only include transactions with X fee"
323 2012-07-11 14:06:19 <jgarzik> read alone, that statement does not match the code
324 2012-07-11 14:20:25 <starsoccer> could someone help me with setting up dark exchange?
325 2012-07-11 14:48:24 <jgarzik> starsoccer: no, but if you have fibre, I could set up a light exchange
326 2012-07-11 14:48:29 <gavinandresen> jgarzik: "policy zones" don't start until you hit MAX_BLOCK_SIZE_GEN/2, which is 250K.  I'll fix the gist (I said it was 500K)
327 2012-07-11 14:49:03 <jgarzik> gavinandresen: they start immediately, by creating a free transaction area
328 2012-07-11 14:49:32 <jgarzik> gavinandresen: when CreateNewBlock() begins, nBlockSize < 27k for a while
329 2012-07-11 14:49:40 <gavinandresen> right.....
330 2012-07-11 14:50:27 <gavinandresen> I care mostly about whether or not you can pay a fee and get faster confirmation.  And right now, you can't.
331 2012-07-11 14:50:36 <jgarzik> gavinandresen: thus "if block size > 27k, only include transactions with X fee" is not true, because blocks >27k do include some free tx's
332 2012-07-11 14:50:59 <gavinandresen> ah, I see.  I mean "as the block is filling up"
333 2012-07-11 14:51:04 <gavinandresen> I'll fix the gist to make that clear
334 2012-07-11 14:51:10 <jgarzik> gavinandresen: thanks
335 2012-07-11 14:54:16 <gavinandresen> fixed, using the wording " if less than 27kb of transactions have been added" ... etc
336 2012-07-11 14:55:33 <gavinandresen> BlueMatt: RE: why priority fraction and not "fees to forgo" : because I agree with jgarzik, "fees to forgo" will be a hard sell to miners, and it is way too easy for a not-thinking miner to set fees-to-forgo to zero, not realizing that opens the door to massive spam attacks.
337 2012-07-11 14:56:40 <gmaxwell> I think we instead have something worse right now.
338 2012-07-11 14:56:46 <gmaxwell> (I mean in the gist)
339 2012-07-11 14:57:02 <gavinandresen> gmaxwell: ok, what did I screw up now?
340 2012-07-11 14:57:49 <gmaxwell> :) With what you have described miners have an unbounded lost fees liability unless they set the target to maximum and the free portion to 0.
341 2012-07-11 14:58:42 <gavinandresen> They always have a theoretical unbounded lost fees liability-- there may be a million 1-bitcoin transactions in the memory pool that won't fit into a 1MB block
342 2012-07-11 14:59:00 <gmaxwell> Because you could always have A full megabyte of of 1000 BTC fee transactions, or what have you. Not that this is likely, but we've seen how people obsess about fees on the sending side.
343 2012-07-11 14:59:12 <gavinandresen> If they're worried about losing fees, then they SHOULD set the free portion to zero, and the maximum (not target) size to 1MB
344 2012-07-11 14:59:32 <gmaxwell> And so they all will. I think that makes the proposal a failure. :(
345 2012-07-11 15:00:00 <gavinandresen> I disagree that they all will.  If they do, then so be it, that's the market speaking
346 2012-07-11 15:00:16 <gmaxwell> I don't think anyone cares about losing a tiny amount of fees especially for a good cause. It's the fact that the amount is unbounded. In theory it could be omg-millions-of-bitcoins lost.
347 2012-07-11 15:00:46 <BlueMatt> gavinandresen: I think prioriy fraction will be a harder sell than fees to forgo, actually
348 2012-07-11 15:00:58 <gmaxwell> No, its not the market speaking, it's paranoia speaking and a result of exposing knobs that don't allow people to express their real market preferences.
349 2012-07-11 15:01:55 <jgarzik> free or "high priority"  transactions currently exist solely due to momentum from the existing Satoshi client hardcoding several behaviors.  Simply fiddling with that status quo will be messy.  If you give miners a "fees to forgo" knob, it will never be non-zero.
350 2012-07-11 15:02:25 <jgarzik> And I don't think miners have the best knowledge of the dev's idea (community's idea?) of a "good" or high priority transaction
351 2012-07-11 15:03:03 <gmaxwell> Also with the median rule default plus a 50% default priority fraction people will probably observe non-trivial fees being lost. So that will encourage them to tweak the knobs. (I agree that default are powerful and people are lazy, but they'll notice missed fees)
352 2012-07-11 15:04:01 <gavinandresen> Tweaking the knobs is good, and if we move towards a purely "you must pay a fee to get into a block" model that's OK with me
353 2012-07-11 15:04:17 <jgarzik> the main market sensitive knob, tx-fee-too-low-and-is-considered-free is easy for miners to change
354 2012-07-11 15:04:25 <gmaxwell> jgarzik: So make it impossible to set it below the minfee unless they go hack it out?  Perhaps thats a dead idea too.. I'm not arguing for it.
355 2012-07-11 15:04:35 <gavinandresen> I just don't want to give them a knob that they can tweak and open up a huge DoS problem
356 2012-07-11 15:04:54 <gmaxwell> jgarzik: funny, previously I'd argued that it should be the only cli inaccessible one! because it's quite subtle due to dos issues.
357 2012-07-11 15:05:10 <jgarzik> anti-spam is a technical question and should remain in the dev realm. if miners want to #if 0 the spam code, fine :)
358 2012-07-11 15:05:33 <jgarzik> spam price is a market issue
359 2012-07-11 15:05:58 <jgarzik> if KimDotCom switches to bitcoin, miners will change the antispam fee limit :)
360 2012-07-11 15:06:01 <gmaxwell> I would happily take a bet that a random selection of miners (or even pool ops) would set that to zero ("I want to support low cost txn!", "I want all fees I can get!")  and the result is that a simple while loop takes out the network. :)
361 2012-07-11 15:06:10 <BlueMatt> I guess I havent been following close enough: how does setting fees to forgo to 0 open the door to massive spam?
362 2012-07-11 15:06:24 <gmaxwell> jgarzik: the fee to be treated as zero is mostly a spam control not a market control.
363 2012-07-11 15:06:27 <gavinandresen> I spam a bunch of 1-satoshi-fee transactions
364 2012-07-11 15:06:27 <jgarzik> gmaxwell: the knob exists today
365 2012-07-11 15:06:39 <gmaxwell> The market control is space availablity in blocks.
366 2012-07-11 15:07:08 <jgarzik> gmaxwell: that logically leads to all blocks being 1MB, perhaps padded with cheap spam if nothing else
367 2012-07-11 15:07:16 <BlueMatt> gavinandresen: thats why there are priority limits/fee limits in mempool?
368 2012-07-11 15:07:40 <gavinandresen> BlueMatt: that's why any transaction with less than 0.0001BTC/kilobyte fee is the same as a 0-fee
369 2012-07-11 15:08:06 <BlueMatt> yes, and what does that setting have to do with 'fees to forgo"
370 2012-07-11 15:08:21 <gmaxwell> jgarzik: that doesn't create any pressure to increase fees though.  The thing (perhaps the only thing? :( ) I like about gavin's latest gist is the use of the median operation to set the target size.
371 2012-07-11 15:08:24 <gavinandresen> If you set fees to forgo to zero, then a 1-satoshi transaction won't be forgone.
372 2012-07-11 15:08:44 <gavinandresen> 1-satoshi-fee tranactions won't be forgone....
373 2012-07-11 15:08:48 <BlueMatt> but if they are low priority, they wont get into memory pool...
374 2012-07-11 15:08:58 <gmaxwell> Because then you have a mechenism to coordinate on that behavior. But it fails in the GIST because there is such a motivation to change it.
375 2012-07-11 15:09:51 <gavinandresen> BlueMatt: good point, you'd have to twiddle two knobs
376 2012-07-11 15:09:57 <gmaxwell> I think instead we should have something where the median size sets a target but you'll gladly go over it for 'enough' income. I don't know how to parameterize enough so that people don't need to twiddle it all the time. Or so that it doesn't have a "zero setting" that would have bad outcomes.
377 2012-07-11 15:10:09 <BlueMatt> gavinandresen: no, I believe the mempool limits would be static
378 2012-07-11 15:10:43 <gavinandresen> gmaxwell: current version of the gist can go over the target size, up to the max size, if there are fee-paying transactions waiting.
379 2012-07-11 15:11:08 <gmaxwell> gavinandresen: er, perhaps I was reading old text then!
380 2012-07-11 15:11:22 <gavinandresen> "include as many as will fit until the block is maximum block size bytes big, not including "free" transactions (transactions with fee-per-kb of less than the default spam threshold of 0.0001 BTC/kilobyte)."
381 2012-07-11 15:12:55 <gmaxwell> Okay, thats better, though it's still kind of lame.. as you'll literally end up producing a block which is 10x larger just to earn a penny or two.
382 2012-07-11 15:13:04 <gavinandresen> RE: how to have a reasonable default that doesn't require constant twiddling:  I punted on that problem; I would like to do something straightforward right now so we get some miners on the network sorting transaction by fee.
383 2012-07-11 15:13:09 <gmaxwell> (and you lose that penny or two in expectation because the larger block takes more time to relay)
384 2012-07-11 15:13:38 <gavinandresen> Next step would then be to modify the client(s) to estimate fees to get a transaction mined quickly, and maybe later implement fancy algorithms to auto-adjust all the params.
385 2012-07-11 15:14:50 <gmaxwell> hm. I wonder if "adjust spam threshold based on the block size an value given a simple model of how size relates to orphaning risk" would give useful values.
386 2012-07-11 15:15:28 <gmaxwell> s/an value/and value/
387 2012-07-11 15:16:05 <gavinandresen> gmaxwell: probably. I think we should keep it simple to start, though, and keep track of metrics (see end of the gist) to make sure the network is behaving like we think it should
388 2012-07-11 15:16:45 <gmaxwell> I'm actually losing track of what we'd be gaining by making the changes beyond order by fee?
389 2012-07-11 15:17:01 <gmaxwell> (I mean the simple changes that you're talking about)
390 2012-07-11 15:19:49 <gmaxwell> ISTM that the median stuff is dangerous (both for market effects and anti-spam) unless we are reasonable sure most people won't just set it to max. Simply because a habbit of maxing out that value established now will moot smarter mechenisms which would let miners coordinate their fee pressure later.
391 2012-07-11 15:20:06 <gavinandresen> gmaxwell: good question.  I suppose the simplest possible thing that might actually work (on the mining side) is just change the transaction selection code to "After 27K, include transactions in fee-per-kb order."
392 2012-07-11 15:20:28 <gmaxwell> I think I'd be more comfortable about that as a step one change.
393 2012-07-11 15:20:45 <gmaxwell> fee-per-kb order is something no one has any concern with, I think.
394 2012-07-11 15:21:09 <helo> what's the easiest path to calculating percent-spent-outputs for each block?
395 2012-07-11 15:21:58 <gmaxwell> gavinandresen: I do really want to get the ball rolling on tx-out pressure in priority, because it will take time for network penetration.
396 2012-07-11 15:22:44 <gavinandresen> Can we do it in a way that is pretty compatible with existing code?  What do you think of the proposed modified priority algorithm in the gist
397 2012-07-11 15:23:03 <gavinandresen> priority = (sum(value*age)/size) * (10 + number of inputs) / (10 + number of outputs)
398 2012-07-11 15:23:29 <gavinandresen> (10 is arbitrary... thinking more about it, I think 11 would actually be better)
399 2012-07-11 15:24:17 <gavinandresen> Still linear, just de-emphasizes the effect of inputs/outputs on priority, for normal transactions with 1-in 2-out modifies the priority a tiny bit
400 2012-07-11 15:24:38 <gmaxwell> I need to run some numbers on actual data.  What I'd suggsted before is replacing size with  2*size_of_txn - size_of_old_txouts_consumed, with a minimum of size/2.  It could be scaled to make it mostly compatible.
401 2012-07-11 15:25:46 <gmaxwell> (basically "you get half the size you remove from the chain for free")
402 2012-07-11 15:26:07 <gavinandresen> mmm.   size of old txouts consumed isn't a number that's easy to get if you didn't save your inputs
403 2012-07-11 15:26:19 <gmaxwell> You have to have the inputs to write the transaction.
404 2012-07-11 15:26:37 <gmaxwell> (or validate it)
405 2012-07-11 15:27:04 <gavinandresen> I'm thinking of an implementation that validates then shoves in the memory pool... I suppose it could cache the size of previous txouts
406 2012-07-11 15:28:16 <gmaxwell> A downside of using the number is that you don't get bonuses for removing bigger or smaller tx outs.. so you can get crazy fun like making a bunch of zero value outputs with empty scripts to use to insert and increase your priority. :)  Though perhaps the incentive is large enough that it won't matter.
407 2012-07-11 15:28:39 <BlueMatt> can someone explain why "priority fraction" is better than "fees to forgo"? if you add the same 0.0001 btc/kb limit to counting fees to "fees to forgo" it ends up with the same result?
408 2012-07-11 15:28:51 <gmaxwell> Mostly my motivation is that later we'll add changes to coin selection to intentionally try to sweep up extra inputs when you already have change... and I don't want people to feel that doing so is against their interest. (omg! more fees!)
409 2012-07-11 15:28:53 <gavinandresen> huh? zero-value outputs would decrease your priority, right?
410 2012-07-11 15:29:24 <gavinandresen> more inputs == higher priority
411 2012-07-11 15:29:24 <gmaxwell> gavinandresen: I make one txn with 1000 zero value outputs... then I use them as inputs in my future txn to boost their priority.
412 2012-07-11 15:29:42 <gavinandresen> gmaxwell: so you pay fees then or now
413 2012-07-11 15:29:51 <gmaxwell> basically more inputs is higher priority is fine, but it should be proportional to the 'cost' of those inputs.
414 2012-07-11 15:30:05 <gmaxwell> Otherwise I can game the system by making a bunch of useless inputs that get more boost.
415 2012-07-11 15:31:03 <gavinandresen> ok.  That's why I wanted inputs/outputs to have only a mild effect on priority
416 2012-07-11 15:31:49 <gavinandresen> (and wouldn't it work the other way? If I want to "pre-pay priority" then I could make one txn with 1000 really big txouts and burn them later to increase future txn priorities)
417 2012-07-11 15:32:44 <BlueMatt> the goal would be that you would pay more in fees to get priority later, when you could just pay less fees later
418 2012-07-11 15:33:07 <gmaxwell> gavinandresen: sure but you paid a lot to create that. I'm not concerned with prepay. I'm concerned with pay-less-by-gaming. E.g. if we'll give 1 unit for sane normal inputs and 1 unit for cheaply created inputs.. but :::shrugs::: this is a tangent, I bougt your argument that the incentive is small.
419 2012-07-11 15:34:26 <gavinandresen> gmaxwell: I don't like the step function in "MAX(size*2 - size_consumed, size/2)"  -- seems like it will make it hard to reason about
420 2012-07-11 15:35:44 <gmaxwell> gavinandresen: yea, the step function sucks.  There needs to be a max to prevent it from going zero. Lest you get infinite priority.  The reason I suggested size/2 is for compatiblity with the existing behavior.
421 2012-07-11 15:35:49 <amiller> why not make it proportional to size and call that all?
422 2012-07-11 15:36:01 <gmaxwell> amiller: it is currently but not all size is equal.
423 2012-07-11 15:36:49 <gmaxwell> amiller: we're getting closer to producing a pruned node, and size in the txout set is _far_ more costly than network bandwidth.
424 2012-07-11 15:36:57 <gmaxwell> (under a model with lots of pruning nodes)
425 2012-07-11 15:37:00 <amiller> yup, it's the key resource
426 2012-07-11 15:37:03 <gavinandresen> BlueMatt: I don't like the step function in min(value,A)....
427 2012-07-11 15:37:10 <amiller> a bitcoin is basically a storage locker token in that tree
428 2012-07-11 15:37:32 <BlueMatt> gavinandresen: why? its no harder for any of the related algorithms (most coin selection)
429 2012-07-11 15:37:35 <gmaxwell> BlueMatt: sorry, I still don't like it though I admit you softened me some.
430 2012-07-11 15:37:51 <BlueMatt> s/most/mostly/
431 2012-07-11 15:37:57 <gmaxwell> BlueMatt: because non-linearies are always gaming fun.
432 2012-07-11 15:38:12 <gavinandresen> yep, find the edge and exploit it
433 2012-07-11 15:38:25 <gmaxwell> BlueMatt: for example, under that scheme I can _enormously_ boost my priority by never writing myself change greater than A in value.
434 2012-07-11 15:38:30 <BlueMatt> if, say, you set A to 1 BTC, that just means 1 BTC inputs are the same value as 100 BTC inputs...so?
435 2012-07-11 15:39:07 <BlueMatt> gmaxwell: hence why you strongly anti-prioritize many outputs?
436 2012-07-11 15:39:21 <amiller> i still think 1 satoshi (or something proportional to the size) should be deducted from each txoutput during every block that it isn't spent, like collecting rent
437 2012-07-11 15:39:22 <BlueMatt> (which should happen anyway?)
438 2012-07-11 15:39:26 <gmaxwell> BlueMatt: but thats a one time operation, and would discourage sendmany.
439 2012-07-11 15:39:31 <amiller> if you store all your bitcoins in one big txout, then that rent deduction is very small
440 2012-07-11 15:39:42 <gavinandresen> gmaxwell: priority = (sum(value*age)/size) * (CONSTANT + prev_txout_size) / (CONSTANT  + txout_size)
441 2012-07-11 15:39:43 <amiller> if you spread your change very thin, then it attritions faster
442 2012-07-11 15:40:01 <gmaxwell> amiller: there are arguments for it, but the cost of reasoning about it multiplied by all the current and future users of the system just isn't worth it.
443 2012-07-11 15:40:23 <amiller> understandble. well, simple proportional to size should be the next best thing
444 2012-07-11 15:40:31 <gmaxwell> amiller: it's also a totally non-starter because of course if we changed something like that, why not have a maximum of
445 2012-07-11 15:40:44 <gmaxwell> amiller: but what size? There are several sizes that matter?
446 2012-07-11 15:40:59 <amiller> txout size is all?
447 2012-07-11 15:41:02 <gmaxwell> gavinandresen: I like that too.
448 2012-07-11 15:41:35 <gmaxwell> amiller: network bandwidth is never irrelevant.  And the historial size (including the inputs) matters too.
449 2012-07-11 15:41:54 <amiller> isn't that just paying twice, and it would be simpler to only pay in one direction?
450 2012-07-11 15:42:15 <gmaxwell> amiller: I could write an enormous input to redeem a boring regular output.
451 2012-07-11 15:42:31 <gmaxwell> push push push push pop pop pop pop
452 2012-07-11 15:42:39 <amiller> hmmm
453 2012-07-11 15:42:43 <BlueMatt> gmaxwell: yes, the fact that its a one-time operation is a bit ugly, but I still think the positives of not strongly prioritizing people with 1 mill btc to spend over people with 1 outweighs that...and in terms of discouraging sendmany, it really doest have to, not discouraging sendmany with something like that is easy
454 2012-07-11 15:42:50 <amiller> interestingly, that input would never need to be stored in the merkle tree
455 2012-07-11 15:42:57 <amiller> but i agree it's not free
456 2012-07-11 15:43:21 <gmaxwell> BlueMatt: but when that person with a million btc uses their priority its gone. It's not in their rational interest to burn more priority than they have to.
457 2012-07-11 15:43:49 <BlueMatt> gmaxwell: because, again, at that point you defeat the purpose of free txn, and then we might as well throw away priority and /just/ use fee/kb
458 2012-07-11 15:44:41 <gmaxwell> BlueMatt: I haven't really seen any reason yet to worry that bitcoin millionaries are crowding out newbies for free space.
459 2012-07-11 15:44:43 <helo> do you folks have your bitcoin savings broken up into increments for multiple aged outputs?
460 2012-07-11 15:44:45 <amiller> how about if txoutputs indicate a worst-case size for an input that redeems it?
461 2012-07-11 15:45:12 <BlueMatt> gmaxwell: absolutely, but if you have a high priority cost to get into the free txn area, someone with a larger total btc could get into it easier than someone with less, older btc
462 2012-07-11 15:45:36 <BlueMatt> gmaxwell: the point of the fee restructuring is to make it work 10 years down the road, not work tomorrow
463 2012-07-11 15:45:37 <gmaxwell> BlueMatt: they can regardless of what you do though! you can't solve that through non-linearity.
464 2012-07-11 15:45:49 <BlueMatt> gmaxwell: the current stuff (arguably) works and will work fine for the next year+
465 2012-07-11 15:46:18 <gmaxwell> If I have a million btc and you cap at 1 BTC, then I just make an effort to split up my million whenever I transact.  Plus you've gained a magic number which will be invalidated if the exchange rate changes.
466 2012-07-11 15:46:47 <BlueMatt> gmaxwell: magic number: the number can change, which makes it harder to game, too
467 2012-07-11 15:47:01 <amiller> eventually we are going to need worst-case reasoning when we extend the scripting language anyway, so this is an appropriate place to begin that sort of thing
468 2012-07-11 15:47:12 <gmaxwell> worse it's encouraged people to write software that bloats the txout set. :( and ... I just haven't seen evidence of the problem. Right now free txn are croweded out by 0.0005 fee txn, not high priority ones.
469 2012-07-11 15:47:25 <amiller> if push push push pop pop pop is a plausible input, then potentially miners will have to chug through all of that many times just to get to realize the transaction is invalid
470 2012-07-11 15:47:41 <gmaxwell> amiller: you missed my point.
471 2012-07-11 15:48:16 <gmaxwell> amiller: I was arguing against your argument to ignore the input size. If you don't ignore the input size it's a non-issue. Because computation = size (times come constant), so they pay for that.
472 2012-07-11 15:48:49 <amiller> sure, but it's the worst case plausible input that's really expensive
473 2012-07-11 15:48:54 <BlueMatt> gmaxwell: even if it doesnt change, say someone has 100 BTC split into inputs of size A, and want to send A/2 to someone...they could take in 10 A inputs and send back change to themselves at 9.5A, but now they've lost a ton of their priority*age inputs, or they could send themselves back a bunch of A, in which case its no different than just 1 input, 2 outputs, which anyone can create
474 2012-07-11 15:49:04 <amiller> since the inputs aren't inserted in a tree, the only cost associated with an input is checking it
475 2012-07-11 15:49:25 <amiller> but a worst case plausible input might have to be checked many times
476 2012-07-11 15:49:46 <BlueMatt> gmaxwell: and the "I dont see the problem happening today" isnt a valid argument here..."I dont think the problem will ever occur", sure, but we arent trying to set up a fee system for use for the next 6 months, we are trying to set up something that works further out
477 2012-07-11 15:49:55 <amiller> so it would be good to limit that at the txoutput, and that's something to pay proportionally for as well
478 2012-07-11 15:50:18 <amiller> so i'm saying a txout should be prepaid with a fee covering its storage in the tree, and also the worst-case validation cost for a txinput that spends it
479 2012-07-11 15:50:28 <amiller> those are the two costs
480 2012-07-11 15:50:42 <gmaxwell> "input is checking it" and network bandwidth. And archival storage.
481 2012-07-11 15:51:16 <gavinandresen> the fee for an expensive-to-redeem-txout is paid when it is redeemed; I don't see an issue with that.
482 2012-07-11 15:51:42 <BlueMatt> its better that way, then the merchant can pay the fee, and its them who really want to pay the fee (or they do in the current system, and that works well)
483 2012-07-11 15:51:46 <amiller> it's not how much it costs when it's actually redeemed, but how much it might cost just to _validate_ whether a transaction succesfully redeems it
484 2012-07-11 15:52:04 <gmaxwell> amiller: you could imagine that it _is_ paid in the output, just not market. And in fact if the output is really small the amount paid might be negative. :) it's isomorphic.
485 2012-07-11 15:52:14 <BlueMatt> amiller: yes, and the tx that is redeeming should pay that cost
486 2012-07-11 15:52:21 <amiller> if the tx is invalid, there's no pay
487 2012-07-11 15:52:37 <gavinandresen> you pay by getting disconnected from the network
488 2012-07-11 15:52:48 <gavinandresen> (the DoS code kicks in if you send invalid transactions)
489 2012-07-11 15:52:51 <amiller> suppose there's a txoutput that i could redeem with a 10 byte message, but it also could be redeemed by a 10000 byte message
490 2012-07-11 15:53:30 <amiller> validating a signature has a worst-case cost, that's an important part
491 2012-07-11 15:54:13 <gmaxwell> amiller: yea good luck writing the solver for figuring that out, and for our language the worst case cost is always
492 2012-07-11 15:54:38 <gmaxwell> (well, except for p2psh, but you can't know the cost there)
493 2012-07-11 15:54:43 <amiller> gmaxwell, simple, the txoutput indicates a bound for the worst size cost and pays a fee proportional to that
494 2012-07-11 15:54:53 <amiller> anything larger is simply invalid
495 2012-07-11 15:54:57 <gmaxwell> amiller: this feels like navel gazing again.
496 2012-07-11 15:55:13 <amiller> i don't see what you meant by solver
497 2012-07-11 15:55:23 <gavinandresen> amiller: I still don't get it, there's an incentive to use the 10 byte message to redeem because you'll pay lower fees
498 2012-07-11 15:55:29 <BlueMatt> gavinandresen: anyway, the idea behind my suggestion is to actually make the free tx area useful in the case of high volume and competition for that area...in the original proposal, anyone who has a large btc-net-worth can get into the free txn area easily, whereas others cannot...this has several issues: those with high btc-worth could just pay a fee, so getting into the free txn area doesn't matter for them (though they would do so, becau
499 2012-07-11 15:55:31 <gavinandresen> again, when you redeem
500 2012-07-11 15:56:12 <gmaxwell> BlueMatt: I'm totally symapathitc to the concern you're expressing, but you have not solved it and I'm vaguely sure you can't solve it.
501 2012-07-11 15:56:19 <gavinandresen> BlueMatt: newbies with a tiny number of coins are really indistinguishable from spammers
502 2012-07-11 15:56:35 <amiller> gavinandresen, sure the incentive is to redeem with a 10 byte message, but if a 10MB message is plausible, then that could be the amount of effort a miner spends just checking even the invalid transaction
503 2012-07-11 15:57:00 <gmaxwell> amiller: okay, I see that point.
504 2012-07-11 15:57:12 <BlueMatt> gmaxwell: I dont see how it was not largely solved, I have yet to see an example of where it gets easier to create high-prio txn with large net-worth with a hard limit?
505 2012-07-11 15:57:24 <gmaxwell> E.g. the 100 BTC fee a transaction promises you means _nothing_ if the txn is invalid.
506 2012-07-11 15:57:34 <BlueMatt> gavinandresen: tiny number of coins, Im talking about low coins eg 1BTC or such, not 10 satoshi, which are clearly spammers
507 2012-07-11 15:58:20 <gmaxwell> BlueMatt: but I gave you one. I adjust my code that when I have change over 1 BTC it keeps adding more 1 BTC change outputs until my txn hits the minimum free priority.
508 2012-07-11 15:58:46 <BlueMatt> <BlueMatt> gmaxwell: even if it doesnt change, say someone has 100 BTC split into inputs of size A, and want to send A/2 to someone...they could take in 10 A inputs and send back change to themselves at 9.5A, but now they've lost a ton of their priority*age inputs, or they could send themselves back a bunch of A, in which case its no different than just 1 input, 2 outputs, which anyone can create
509 2012-07-11 15:58:52 <gmaxwell> BlueMatt: then over time I bloat up the txout set with enormous numbers of 1BTC inputs which I can use for maximum priority effect.
510 2012-07-11 15:59:24 <BlueMatt> even with 100 1btc outputs, you still have to spend a finite resource to get the high prio, and its more expensive per txn for you than it is for someone who only has 1 btc
511 2012-07-11 15:59:47 <gmaxwell> BlueMatt: yes, though you can avoid that loss by making better use of sendmany. You're taking a small incentive to break change into typical transaction sizes into a large incentive to split to your target value.
512 2012-07-11 15:59:52 <gavinandresen> amiller: again, I'm not seeing your point. If a miner gets an expensive-to-check transaction free-floating on the network, and it turns out to be invalid, then they disconnect and ban that peer.
513 2012-07-11 16:00:20 <gavinandresen> amiller: if they get it as part of a block, then whoever generated that block must have done proof-of-work....
514 2012-07-11 16:00:31 <gmaxwell> amiller: you have a point but gavin's answer is basically conclusive. ... we can't really get txn so costly that being banned is worth it.
515 2012-07-11 16:00:34 <gavinandresen> ... either case, there's a disincentive to broadcasting invalid transactions
516 2012-07-11 16:00:51 <gavinandresen> And if the transaction is valid, then fees will be paid
517 2012-07-11 16:01:36 <BlueMatt> gmaxwell: huh?
518 2012-07-11 16:02:54 <amiller> if that's the case then you might as well only pay for the txout storage
519 2012-07-11 16:03:18 <amiller> consider that all the other costs are bounded and there's no incentive for anyone to exceed them
520 2012-07-11 16:04:10 <BlueMatt> the way I see it, if high btc-worth people can more easily get into free txn area (as it is in the proposal) the free txn area should just be scrapped...maybe Im wrong and you cant solve it in priority, but then when the competition for the free txn area increases (ie when it starts to actually become useful) its use disappears because those who would otherwise have gladly paid a fee will be the only ones in it
521 2012-07-11 16:04:15 <gmaxwell> amiller: I'm not following you again. :(  I could make a random txin have 100kb data. Something needs to discourage me from doing that.
522 2012-07-11 16:04:39 <BlueMatt> and the whole fee structure would be way simpler if we dropped the free txn area altogether, which, afaict, we might as well do
523 2012-07-11 16:05:25 <gmaxwell> BlueMatt: I wish you'd stop repeating the motivation. I believe it and love it and cuddle it. And I now wish I never suggested that it might not really matter.
524 2012-07-11 16:06:04 <gmaxwell> also, why do you say 'as it is in the proposal' instead of 'as it is on bitcoin today' ?
525 2012-07-11 16:06:21 <gavinandresen> uff. I knew modifying how fees would work would be a great big tarbaby
526 2012-07-11 16:06:43 <BlueMatt> because Im largely ignoring as it is in bitcoin today, we all agree its largely broken, so...meh
527 2012-07-11 16:07:05 <gavinandresen> we can't ignore how it is today, whatever we do we need bitcoin to keep working while we do it
528 2012-07-11 16:07:39 <BlueMatt> gavinandresen: Id rather try to work through the fees issues than do nothing, I may very well be entirely wrong, so we can just focus on other parts of the fee stuff, but we might as well work through everything
529 2012-07-11 16:07:54 <gmaxwell> What we have isn't broken. Being a naysayer is cool but it doesn't make you wise. (advice I should take more often). The most obvious current dysfunction is the flood of 0.0005 fee dice txn flooding out transactions where people want fast confirms... and the free policy stuff is irrelevant for that.
530 2012-07-11 16:07:57 <gavinandresen> ... that's why I've retreated from "rework everything" to what I'm thinking now, which is just sort-by-fee-per-kb after 27K of the block has been filled.
531 2012-07-11 16:08:32 <gavinandresen> gmaxwell: +1  exactly
532 2012-07-11 16:09:06 <gavinandresen> maybe satoshi dice is the current best use of block space, but we need to move forward with a market so people can vote on that using transaction fees
533 2012-07-11 16:09:52 <BlueMatt> gmaxwell: ok, s/we all agree its largely broken/we all agree it has some big limitations, and should be upgraded/
534 2012-07-11 16:09:59 <gmaxwell> BlueMatt: perhaps we can address your newbie concern differently. How about this.... setup something so new users can solve some annoying captchas to get miners to except their transactions from the fee rules like luke does for commercial partners.
535 2012-07-11 16:10:19 <luke-jr> ^ interesting
536 2012-07-11 16:10:28 <BlueMatt> gmaxwell: captcha farms?
537 2012-07-11 16:10:46 <gavinandresen> good idea.  A RPC call to keep a list of artificially-high-priority transactions would be easy
538 2012-07-11 16:10:50 <gmaxwell> Sure, you rate limit it. I don't think it would expose a spam risk. It's just another way to pay for your txn.
539 2012-07-11 16:11:04 <BlueMatt> thats fine with me
540 2012-07-11 16:11:06 <luke-jr> gavinandresen: I actually have that already, I could clean it up if it's wanted
541 2012-07-11 16:11:43 <gavinandresen> luke-jr: settransactionpriority <txid> <priority>   ?
542 2012-07-11 16:11:47 <BlueMatt> gavinandresen: I still think the proposed priority changes that take into account pruning are better than nothing, Im just saying I think they could be better
543 2012-07-11 16:12:04 <amiller> i still think putting in a worst-case bound at the txout is ideal and simple, especially since in the future it can reflect a cycle/instruction count as well as a storage, and otherwise you're relying too much on the dos defense / banning, which i don't know much about but i'll take your word it's effective
544 2012-07-11 16:12:06 <luke-jr> gavinandresen: right now it's a boolean, but I could certainly make it variable
545 2012-07-11 16:12:27 <luke-jr> gavinandresen: IMO it makes more sense to specify a difference/boost tho
546 2012-07-11 16:12:33 <gavinandresen> would be good to prioritize OR de-prioritize certain txns.  I know some people would like to de-prio all 1dice.....
547 2012-07-11 16:13:16 <gavinandresen> I think   prioritizetransaction <txid> <delta> would be OK, too
548 2012-07-11 16:14:38 <gavinandresen> amiller: so what happens if somebody lies about the worst case bound?
549 2012-07-11 16:15:13 <amiller> the worst case bound is normative
550 2012-07-11 16:15:18 <amiller> it's not something you solve for or calculate
551 2012-07-11 16:15:36 <amiller> if a tx exceeds it, it's necessarily invalid
552 2012-07-11 16:15:42 <gavinandresen> we've already got hard-coded bounds on script execution
553 2012-07-11 16:16:34 <BlueMatt> gmaxwell: one issue with the captcha-with-pools-for-free-prio is its another odd thing to make new users go through, and their experience should generally be as simple as possible...but it is optional so its only a minor gripe
554 2012-07-11 16:17:24 <amiller> gavinandresen, sure but it's a bit inflexible now, the idea is to allow a txout to prepay for a larger bound
555 2012-07-11 16:17:48 <gavinandresen> I see, you want the bounds to be expanded if enough fees are paid?
556 2012-07-11 16:17:54 <amiller> right
557 2012-07-11 16:18:04 <gavinandresen> meh
558 2012-07-11 16:18:21 <amiller> heh
559 2012-07-11 16:18:23 <gavinandresen> on the list of priorities, bumping up against the hard-coded script limits is... well, it's not on it
560 2012-07-11 16:19:02 <amiller> well then for now it could just be a storage bound, it would prevent the 100kb inputs
561 2012-07-11 16:19:52 <gmaxwell> BlueMatt: yea, I'm mostly interested in the infrastructure, which would be the RPC for priority boosts. People can invent clever stuff.
562 2012-07-11 16:20:27 <luke-jr> gavinandresen: is it fair to assume if priority delta > 0, the minimum fee rule should be ignored?
563 2012-07-11 16:20:49 <gavinandresen> amiller: scripts (inputs) bigger than 10K are invalid
564 2012-07-11 16:21:06 <BlueMatt> gmaxwell: I think a priority rpc would be nice in any case, though Im not convinced that its not possible to solve with priority, and I do think that would be a much better way to address the issue
565 2012-07-11 16:21:11 <gmaxwell> (ah, I was hazy on it being 10k or 100k :) )
566 2012-07-11 16:21:30 <amiller> well 10k is still larger than necessary for a standard transaction, so you should have to pay more for a txout that can be redeemed with a 10k
567 2012-07-11 16:21:45 <gmaxwell> amiller: Yes, and we already do!
568 2012-07-11 16:21:53 <gmaxwell> amiller: because you pay based on the size of the transaction.
569 2012-07-11 16:22:01 <amiller> only for a valid transaction
570 2012-07-11 16:22:01 <gmaxwell> when you redeem it!
571 2012-07-11 16:22:05 <amiller> that 10k is the validation cost
572 2012-07-11 16:22:12 <gmaxwell> amiller: if the transaction is invalid you will be blacklisted!
573 2012-07-11 16:22:17 <gavinandresen> luke-jr: no, I'd keep it simple and just add the number to the priority and let it go through all the rest of the transaction selection rules
574 2012-07-11 16:22:43 <gavinandresen> e.g. if map[txid].count() > 0  then priority += map[txid]
575 2012-07-11 16:22:43 <gmaxwell> and there is no concevable way that a 10k (or 100k) txn would be so costly to validate that blacklisting wouldn't be a worse cost.
576 2012-07-11 16:22:56 <amiller> how sound is this blacklisting thing
577 2012-07-11 16:23:13 <amiller> can't i submit transactions from anywhere?
578 2012-07-11 16:23:19 <amiller> only a full node is capable of rejecting it
579 2012-07-11 16:23:26 <amiller> but lots of nodes will circulate it
580 2012-07-11 16:23:30 <gmaxwell> amiller: non-full nodes do not relay.
581 2012-07-11 16:23:48 <amiller> they're not allowed to? or they just typically don't?
582 2012-07-11 16:23:55 <gavinandresen> yes, rule is "if you can't validate it, you shouldn't be relaying it."
583 2012-07-11 16:23:58 <amiller> i see
584 2012-07-11 16:24:23 <gavinandresen> ... otherwise you can easily get yourself DoS'ed off the network as an accessory to the crime
585 2012-07-11 16:24:28 <gmaxwell> amiller: And it's as sound as the IP addresses you can get access to. Which is your cost.. perhaps you can pay a botnet operator for 10k IPs.. but once they're blacklisted.. you're spent.
586 2012-07-11 16:24:40 <gmaxwell> Acting as an attack force multiplier.
587 2012-07-11 16:25:11 <amiller> or connect through tor
588 2012-07-11 16:25:16 <gmaxwell> Morover, if we didn't have that even with your maximum txin size scheme, the attacker would just flood infinite amounts of maximum allowed cost. The cost is bounded but they'd just send more txn.
589 2012-07-11 16:25:18 <gavinandresen> That reminds me, Something probably Needs to be Done with IpV6 and the DoS code
590 2012-07-11 16:25:47 <BlueMatt> gavinandresen: easiest: just ban by /64
591 2012-07-11 16:25:57 <BlueMatt> easy and sane
592 2012-07-11 16:25:59 <gavinandresen> BlueMatt: that's "Something"
593 2012-07-11 16:26:03 <gmaxwell> eee.
594 2012-07-11 16:26:12 <gmaxwell> well start by looking at how the group handling works.
595 2012-07-11 16:26:21 <gmaxwell> E.g. we convert 6to4 to the IPv4 addresses.
596 2012-07-11 16:26:46 <gmaxwell> BlueMatt: "I can take deepbit off the network by getting a host in their colo"  fantastic!
597 2012-07-11 16:26:52 <BlueMatt> gmaxwell: ok s|ban by /64|ban by /64 for regular ipvs's|
598 2012-07-11 16:27:03 <BlueMatt> ipv6s
599 2012-07-11 16:27:46 <gmaxwell> BlueMatt: try again? I didn't parse.
600 2012-07-11 16:28:03 <BlueMatt> gmaxwell: just ban by /64 for regular ipv6s, not for anything else
601 2012-07-11 16:28:29 <BlueMatt> yea, if you are in a colo where they give you only one ipv6, you may get banned, but you still have ipv4 to fall back on, which should be safe
602 2012-07-11 16:29:06 <gmaxwell> BlueMatt: Thats true but it seems really inelegant to continue to depend on good v4 connectivity. I suppose it's an okay short term thing.
603 2012-07-11 16:29:45 <gmaxwell> 6to4 and such should still be treated as their v4 for this purpose too. The same normalization function can handle it.
604 2012-07-11 16:29:46 <BlueMatt> yea, its not really that pretty, sure, but for the forseeable future it will work just fine
605 2012-07-11 16:33:50 <amiller> gmaxwell, yeah good point, my input bound thing isn't at all an adequate dos-defense
606 2012-07-11 16:35:05 <amiller> it still might be a way of prepaying for the 10k inputs so that the fees are simple and one-way only
607 2012-07-11 16:35:44 <TD> or don't ban
608 2012-07-11 16:35:46 <TD> just reprioritize
609 2012-07-11 16:37:18 <BlueMatt> recommendations for profiling java?
610 2012-07-11 16:38:19 <TD> there's a simple tool included called jvisualvm
611 2012-07-11 16:38:28 <TD> it's not amazing but works well enough if you take the snapshots
612 2012-07-11 16:39:00 <BlueMatt> yea, thats what Im trying, but I just hit "Compute retained sizes" and it looks like its gonna take a day...
613 2012-07-11 16:39:09 <BlueMatt> (and is neatly using 1% cpu right now...wtf?)
614 2012-07-11 16:39:17 <TD> that's odd. i've used it many times before and had no trouble
615 2012-07-11 16:39:47 <TD> otherwise the android tools are pretty good, but obv you'd have to write a little android wrapper app to use them
616 2012-07-11 16:39:58 <TD> there are a bunch of commercial profiling tools too
617 2012-07-11 16:40:07 <TD> are you trying to do cpu or heap profiles?
618 2012-07-11 16:40:09 <gmaxwell> amiller: so I'm hoping an expecting that ultraprune (+ txtree commitments later) will make the cost of running a full validation cheap enough that continuing to limit relaying to nodes which can validate isn't an issue at all. I think anything less than that makes any public node an attack multipler, so it better be enough. :)
619 2012-07-11 16:40:15 <BlueMatt> TD: both...
620 2012-07-11 16:41:06 <TD> ok
621 2012-07-11 16:41:19 <TD> i've been ok with jvisualvm or jhat for heap walks
622 2012-07-11 16:41:26 <TD> but if you find a better one that's free, let me know
623 2012-07-11 16:41:53 <TD> BlueMatt: whereabouts in germany are you again?
624 2012-07-11 16:41:58 <BlueMatt> it might just be the fact that the heap peaks at ~500M thats killing it...
625 2012-07-11 16:42:02 <BlueMatt> frankfurt area
626 2012-07-11 16:42:51 <TD> ah ok
627 2012-07-11 16:43:04 <TD> BlueMatt: if you have an enormous heap then analyzing it will be difficult, yes. jhat may be more appropriate
628 2012-07-11 16:43:11 <TD> or reducing the amount of stuff on the heap :)
629 2012-07-11 16:43:33 <BlueMatt> reducing the amount of stuff in heap is difficult if you cant analyze whats in heap ;)
630 2012-07-11 16:44:16 <amiller> gmaxwell, agreed, though i'm also really hoping that the scripting language gets totally beefed up
631 2012-07-11 16:46:54 <TD> amiller: what would you want in it?
632 2012-07-11 16:49:24 <amiller> well, there are tons of cool things you could do with it, but the most important one (i don't expect anyone to agree with me) is Ripple
633 2012-07-11 16:50:03 <amiller> what i would want is to be able to restrict not just what kind of input is needed to spend a tx, but also how the next txoutputs could be used
634 2012-07-11 16:53:11 <TD> amiller: see here https://bitcointalk.org/index.php?topic=92421.0 for the latter
635 2012-07-11 16:53:51 <TD> amiller: and here https://en.bitcoin.it/wiki/Ripple_currency_exchange for the former
636 2012-07-11 16:54:35 <TD> amiller: yes, being able to [sig]hash the connecting transaction and match it would be neat. and not a difficult extension
637 2012-07-11 16:56:06 <TD> amiller: for now, there is plenty of mileage in the features we already have
638 2012-07-11 16:56:09 <amiller> mm thanks didn't know about those pages, but yeah that's exactly right
639 2012-07-11 16:56:28 <gribble> New news from bitcoinrss: luke-jr opened pull request 1583 on bitcoin/bitcoin <https://github.com/bitcoin/bitcoin/pull/1583>
640 2012-07-11 16:56:46 <TD> amiller: the community still hasn't done the basics yet. let alone the more complex stuff. and TBH i haven't found any really compelling use cases for checking the form of spending transactions yet. i keep coming up with places i want to use it, then realizing it isn't really necessary
641 2012-07-11 16:56:49 <jgarzik> gavinandresen: (returns, scrolling back)  Classify TX as either fee or free (fee < X).  Sort TXs by fee-per-kb.  Fill block from this sorted list.  If there is space left over, add up to X kb free TX's.
642 2012-07-11 16:56:52 <jgarzik> gmaxwell: ^
643 2012-07-11 17:02:50 <gmaxwell> As an aside,  I'd like to start thinking in terms of "Fee paid" transactions and "Priority paid" transactions. They're both paid for, but with different resources.
644 2012-07-11 17:03:44 <jgarzik> reasonable
645 2012-07-11 17:04:45 <amiller> TD, i'll know its working when i can sponsor a chess game
646 2012-07-11 17:04:49 <gmaxwell> jgarzik: instead of X kb, in that perhaps apply gavin's median thing.  The target is (x?)*median(y blocks). If you're under target, add up to Z KB of priority paid until you're at target up to target.
647 2012-07-11 17:05:08 <amiller> i post the bounty price, designated two players by their public key, they alternate making moves by making transactions, miners validate each move, and the winner gets the bitcoins
648 2012-07-11 17:05:33 <TD> that sounds like an inefficient way to implement it :-)
649 2012-07-11 17:05:45 <amiller> yeah, i also don't have any compelling use case, outside ripple i suppose
650 2012-07-11 17:05:45 <jgarzik> gmaxwell: reasonable -- but I don't understand why target != maximum in all cases
651 2012-07-11 17:05:56 <amiller> oh, also if you can do ripple, you can also build WoT into bitcoin, which will be cool
652 2012-07-11 17:06:03 <jgarzik> as a miner, why not build all the way up to 1MB of fee-paying tx's?
653 2012-07-11 17:06:10 <gmaxwell> jgarzik: because the target would only matter for non-fee txn.
654 2012-07-11 17:06:14 <TD> the way i've been going with my designs, is to move almost all logic out of bitcoin and just tie it to the block chain where necessary
655 2012-07-11 17:06:23 <jgarzik> gmaxwell: ok
656 2012-07-11 17:06:25 <amiller> TD, you mean with merge mining?
657 2012-07-11 17:06:30 <TD> if you wanted some kind of crypto-chess system, it's best to create an independent network for that
658 2012-07-11 17:06:38 <TD> well, not just for that
659 2012-07-11 17:06:48 <jgarzik> gmaxwell: I could ack that
660 2012-07-11 17:06:52 <gmaxwell> jgarzik: I'm saying you add up to the maximum of fee-paid, then if you're still under target, you add free ones. This allows miners to coordinate to keep the free pressure.
661 2012-07-11 17:06:58 <TD> look at the bonds implementation. the block chain contains pointers to data structures held elsewhere, that are dereferenced by software that knows how to speak the bitcoin protocol
662 2012-07-11 17:07:02 <jgarzik> gmaxwell: yes, agreed
663 2012-07-11 17:07:02 <TD> but to miners it's just dropped data
664 2012-07-11 17:07:50 <amiller> TD, i don't think that's satisfactory but let me read this more carefully first, the basic problem is the same as wtih merge mining, if it's just dropped data to the bitcoin miners, you get timestamp sure but you don't get well-orderedness
665 2012-07-11 17:08:06 <TD> amiller: for example. for your chess example, you could create a p2p network of nodes that take part in the creation of threshold signatures. when you start a game, you select 50 peers that have been online for >7 days, and they generate a 30 of 50 threshold key
666 2012-07-11 17:08:07 <jgarzik> gmaxwell: I'm just not a fan of the target system.  Seems easier to ignore block history, and just work on the current block with a simple rule:  block size + X kb, as long as that does not overflow 1MB
667 2012-07-11 17:08:35 <TD> amiller: the nodes all watch the moves and when a winner is found, group-sign the transaction
668 2012-07-11 17:08:40 <jgarzik> gmaxwell: but if consensus likes the complex target system, that's fine too
669 2012-07-11 17:08:56 <jgarzik> simple rules are easier for miners
670 2012-07-11 17:09:06 <TD> amiller: you can get it when you need it, if you use transactions carefully. trying to use miners as a general purpose logic engine isn't going to work though
671 2012-07-11 17:09:34 <jgarzik> the smaller the patch, the more likely miners with forked codebases pick it up
672 2012-07-11 17:09:57 <amiller> but now i have to select 50 peers rather than just using the bitcoin miners? why shouldn't bitcoin miners be a general purpose logic engine, if that's what i want to pay them for?
673 2012-07-11 17:11:18 <TD> amiller: because they weren't designed for it, as you have observed - script is not a general purpose programming language
674 2012-07-11 17:11:37 <TD> amiller: if people are interested in crypto-chess matches, it's not hard for them to run a node for your network
675 2012-07-11 17:11:38 <gmaxwell> jgarzik: So I do think we should have some mechenism so that there is always pressure to provide fees for faster processing even when the load is low.  By using the median it means that the free txn only slip in when the local load is low.. regardless of what the absolute load is.
676 2012-07-11 17:11:56 <TD> amiller: given that people interested in that would want/need an integrated game playing UI, it's not a big leap to combine them together
677 2012-07-11 17:11:58 <gmaxwell> If we had that all along free txns would always have been a bit slow and we'd be better adapted to where we are now.
678 2012-07-11 17:12:04 <gmaxwell> But I don't have super strong opinions on this.
679 2012-07-11 17:12:07 <TD> amiller: also, a confirm per move? sounds kinda dull.
680 2012-07-11 17:12:32 <jgarzik> gmaxwell: would having a run of same-sized blocks, with moderate-to-heavy transaction load, work to starve off free TXs even with block size < 1MB?
681 2012-07-11 17:12:39 <gmaxwell> amiller: BECAUSE THE NETWORK IS MORE THAN MINERS.
682 2012-07-11 17:12:44 <amiller> TD, the chess game is just an example, but enhancing the scripting language so that bitcoin becomes a giant distributed general-purpose-state machine is exactly what i'm after
683 2012-07-11 17:13:11 <amiller> gmaxwell, lol, i didn't mean miners, i meant verifiers of all shapes and sizes ;p
684 2012-07-11 17:13:17 <TD> amiller: i think that is very likely to result in a network that sucks as a logic engine, and also sucks as a financial platform. have you looked at MPC? building an independent yet linked network that does general MPC might be an interesting project
685 2012-07-11 17:13:24 <jgarzik> I think bitcoin users benefit from free TX's
686 2012-07-11 17:13:28 <TD> it'd have very different scaling and security properties to bitcoin though
687 2012-07-11 17:13:29 <gmaxwell> amiller: you're not paying them They validate for the 'payment' of having their currency be inflation proof.
688 2012-07-11 17:13:31 <jgarzik> and I don't want to squeeze them off just yet
689 2012-07-11 17:13:36 <jgarzik> not when there's still <1MB blocks
690 2012-07-11 17:13:43 <gmaxwell> amiller: but checking your chess games do nothing to make the currency inflation proof.
691 2012-07-11 17:13:54 <jgarzik> a target system sounds like it will slow existing free TX's
692 2012-07-11 17:14:03 <gmaxwell> jgarzik: absolutely, I don't want to stop free txn.
693 2012-07-11 17:14:03 <jgarzik> when the network bursts
694 2012-07-11 17:14:31 <gmaxwell> jgarzik: it would, it would make them slip into gaps and during load growth they'd take a long time.. which is the expected long term behavior.
695 2012-07-11 17:14:35 <jgarzik> a simple "+ Z Kb priority tx's" rule handles bursts with useful behavior
696 2012-07-11 17:14:56 <jgarzik> as we approach 1MB, the system slowly converts to fee only
697 2012-07-11 17:15:23 <amiller> if i pay a fair proportional fee to make a transaction with some logic in the script, i don't see why that makes any difference
698 2012-07-11 17:15:32 <jgarzik> gmaxwell: a target system hastens the point at which free tx's see poor performance, versus a simple +Z rule
699 2012-07-11 17:15:40 <TD> amiller: there are quite a few differences.
700 2012-07-11 17:15:50 <gmaxwell> jgarzik: it does, but also makes the transaction smoother instead of almost instant.
701 2012-07-11 17:16:19 <TD> amiller: firstly, people who run bitcoin aren't necessarily interested in providing a general purpose computation and storage system. they are definitely interested in a p2p financial system though, that is something we can safely assume. it's rude to assume everyone who takes part in bitcoin cares about your chess games
702 2012-07-11 17:16:30 <jgarzik> gmaxwell: I disagree with "almost instant" characterization
703 2012-07-11 17:16:35 <gmaxwell> jgarzik: you could do target + Z ... but meh.
704 2012-07-11 17:16:36 <TD> amiller: secondly, script is - by design - a pure function over the inputs. it does not support loops or anything else that could lead to non termination.
705 2012-07-11 17:16:44 <TD> amiller: that limits you a _lot_. really, what you want is MPC
706 2012-07-11 17:16:50 <jgarzik> gmaxwell: yes, I like target+Z a lot better
707 2012-07-11 17:17:08 <gmaxwell> jgarzik: I think it's a reasonable guess that the transition between 1m-27k to 1m shouldn't take long, perhaps I'm crazy.
708 2012-07-11 17:17:13 <amiller> TD, my chess game is easily a pure function that provably terminates, i'll show you my coq if you want ;p
709 2012-07-11 17:17:24 <TD> amiller: this argument that, well, i paid for it so i can do what i want with the network has come up before. it does not impress any of the main developers, largely because there are many people in the network who have to deal with the block chain but who do not mine and therefore, do not get any of your fees
710 2012-07-11 17:17:42 <TD> eg, people tried to argue for stuffing DNS data into fake transactions, for storing files in the block chain, etc
711 2012-07-11 17:17:48 <gmaxwell> jgarzik: yea, target+Z is fine to me. I just prefer something that is a bit adaptive to even out the load, and hopefully make the transation points a little more gradual instead of ALL happening at the 1MB boundary.
712 2012-07-11 17:17:52 <TD> DNS turned out to be more elegantly done with a merged mined separate network
713 2012-07-11 17:17:57 <amiller> TD, that's because we don't have a proportional fee system yet
714 2012-07-11 17:17:57 <TD> file storage was just a bad idea to begin with
715 2012-07-11 17:17:58 <amiller> bitcoin is like a notarized digital postage system, but all we do right now is send each other blank postcards covered with stamps
716 2012-07-11 17:18:21 <amiller> the stamps will become even more valuable when we can do various neat things with the envelopes
717 2012-07-11 17:18:25 <gmaxwell> amiller: 'proportional fee system' doesn't address this at all. :(
718 2012-07-11 17:18:35 <TD> i'm all for building powerful systems on top of the bitcoin protocol
719 2012-07-11 17:18:39 <TD> keyword: on top of. not built in.
720 2012-07-11 17:18:43 <jgarzik> gmaxwell: at the point in the future that block sizes start touching 1MB, it will not be an "almost instant" brick wall that sticks at 1MB, but standard organic transaction flow that bursts, ebbs, flows.  Weaning off free TX's would not be almost instant, and everybody would be long prepared.
721 2012-07-11 17:18:45 <gmaxwell> TD <3
722 2012-07-11 17:18:59 <amiller> meh, time will tell
723 2012-07-11 17:19:04 <TD> amiller: and i think when you drill into the details you'll almost always find there are better ways to do things. at least that's what i found
724 2012-07-11 17:19:18 <TD> amiller: a p2p MPC network would be a very interesting endevour, btw. i'd probably run a node
725 2012-07-11 17:19:20 <jgarzik> 1MB boundary is probably soft anyway
726 2012-07-11 17:19:21 <TD> just for the heck of it
727 2012-07-11 17:19:29 <gmaxwell> The ability to have hashed locked txn and oracle driven multisigs is super powerful.
728 2012-07-11 17:19:29 <jgarzik> when we consistently hit 1MB, changes happen
729 2012-07-11 17:19:32 <TD> though i'm not sure how well it'd work
730 2012-07-11 17:19:39 <jgarzik> thus simple rule will still be effective
731 2012-07-11 17:19:42 <jgarzik> +Z Kb
732 2012-07-11 17:20:01 <TD> amiller: btw, did you see about oracles?
733 2012-07-11 17:20:03 <amiller> TD, MPC is one of the things you can do in the general bitcoin-as-state-machine approach
734 2012-07-11 17:20:05 <gmaxwell> e.g. someone proposed a really complicated way of doing cross chain payments .. and I was like.. dur. two hashes + key. Done.  And TD pointed out that this was already an old scheme.
735 2012-07-11 17:20:20 <gmaxwell> jgarzik: I think gavin is actually off implementing just that in any case.
736 2012-07-11 17:20:46 <TD> amiller: i suspect MPC with transactions would be one of the least efficient implementations imaginable.
737 2012-07-11 17:20:58 <TD> if it was possible at all. huge extensions to the script language don't count, btw :)
738 2012-07-11 17:21:17 <amiller> mpc crypto isn't exactly fast yet so yeah
739 2012-07-11 17:21:50 <TD> amiller: you can probably make simple programs "fast enough" with some tricks, like having nodes associate with each other if they're on the same continent/within <30msec ping times
740 2012-07-11 17:21:54 <gmaxwell> It's actually really important to build these external things now. Because if we do need to get something into bitcoin to make the interconnect then now is a good time to do it.
741 2012-07-11 17:22:10 <TD> gmaxwell: yeah. but everyone is already busy :(
742 2012-07-11 17:22:24 <gmaxwell> Yea. :(
743 2012-07-11 17:22:36 <TD> lately i've been thinking about maybe doing my own little company, that'd publish design docs and then raise assurance contracts for implementation
744 2012-07-11 17:22:54 <TD> if the design docs were precise enough, the work modules small enough, it might be able to get to the point where the community is regularly funding work
745 2012-07-11 17:23:12 <TD> the question is, how big / rich / interested in exotic "apps" is the bitcoin userbase
746 2012-07-11 17:24:21 <TD> amiller: i once proposed a network of oracles, that'd run arbitrary imperative programs that yield an address, and which would then sign transactions if they matched the yielded template
747 2012-07-11 17:24:35 <TD> amiller: as a way to link arbitrary external state to the chain
748 2012-07-11 17:24:57 <TD> amiller: CHECKMULTISIG is kind of expensive, but there are ways to do threshold signatures that result in a single ECDSA signature
749 2012-07-11 17:25:14 <TD> amiller: so you could potentially have a lot of oracles all read the same list of moves, compute the winner and then sign correctly.
750 2012-07-11 17:25:18 <gmaxwell> Oracles are the obvious easy thing to start with.
751 2012-07-11 17:25:20 <jgarzik> TD: how difficult would 'addr' support in BitcoinJ be?  if I were to look into the task?
752 2012-07-11 17:25:23 <TD> the advantage being you can do loops and other stuff.
753 2012-07-11 17:25:32 <gmaxwell> The nice thing about them is that they're fully general.
754 2012-07-11 17:25:39 <jgarzik> TD: lack of 'addr' bugs me :)  seems like a minor security worry, too.
755 2012-07-11 17:26:12 <TD> jgarzik: it bugs me too. i am guessing - not so trivial. i'd start by having a crawler app (or adapting sipas code) to calculate fresh seed lists and having them be loaded from an external file.
756 2012-07-11 17:26:39 <TD> jgarzik: we could then refresh the seed list regularly offline. at least, the android wallet, is auto-updated pretty often. or the list could just be downloaded and signature checked.
757 2012-07-11 17:27:01 <TD> jgarzik: then the next step is to make that list be updated locally. not sure if it's best to do it all in RAM or use a real database. so far bitcoinj does not depend on a bdb-like lib anywhere
758 2012-07-11 17:27:15 <TD> jgarzik: probably you can say, max 5000 addresses or whatever, use a file as a ring buffer or something suitably bare-metal
759 2012-07-11 17:27:29 <jgarzik> yeah
760 2012-07-11 17:27:30 <TD> jgarzik: problem is how do you keep it fresh when the apps may remain dormant for some time.
761 2012-07-11 17:27:49 <TD> you don't want a huge delay if you don't run the mobile wallet for a month, and then it spends 2 mins connecting to vanished nodes
762 2012-07-11 17:28:33 <TD> jgarzik: a good implementation might require new protocol support, so you can ask nodes "tell me addresses you've seen reliably be announced for >2 weeks". except that nodes don't necessarily re-announce themselves. so it's an issue.
763 2012-07-11 17:28:54 <TD> i guess you'd build it up in layers
764 2012-07-11 17:29:26 <gmaxwell> You might be over thinking thing. Fast connect is only an issue at startup. Use DNSseed for an initial connection while you try ones from your database.
765 2012-07-11 17:34:29 <TD> gmaxwell: yeah, but i think the most common use case for these mobile wallets is fast start/fast end. that's how i use mine anyway
766 2012-07-11 17:34:51 <TD> as nodes migrate more to being run by "professionals" it'll be less of an issue. dead addresses won't be a big concern anymore
767 2012-07-11 17:35:08 <TD> and until then opening up 100 attempted connections in parallel and then dropping the ones that work until you have 5 is ugly, but doable
768 2012-07-11 17:35:25 <TD> we recently switched bcj to threaded async io, from one-thread-per-peer. so that approach is more feasible now
769 2012-07-11 17:35:51 <TD> gmaxwell: for most of the proposed schemes, btw, the only changes we need from bitcoin now are (a) reactivating tx replacement (b) whitelisting a few trivial script forms
770 2012-07-11 17:36:06 <TD> gmaxwell: it'd probably be pretty safe. those two changes open up a lot of possibilities
771 2012-07-11 17:37:43 <TD> amiller: for your example of a chess game, you don't need miners to order btw. in the unlikely and unsporting event of two plays by the same player on the same turn being created simultaneously, the one to build off can be chosen by his opponent. because turns are naturally ordered anyway it's a simple and fast way to get a real ordering
772 2012-07-11 17:38:02 <TD> amiller: then the final set of plays can be fed to a generic script that runs inside a bunch of oracles, who then threshold sign. now your coins are unlocked by the game.
773 2012-07-11 17:38:15 <TD> amiller: some of those oracles can prove what they're doing to you using a TPM, for additional low-trustyness
774 2012-07-11 17:38:32 <amiller> the oracle thing is useful but it's also an unnecessary additional trust vector if i trust the bitcoin miners but don't want to have to trust anyone else
775 2012-07-11 17:38:36 <amiller> you're right that the chess example sucks though
776 2012-07-11 17:39:30 <TD> well, the problem with miners (and bitcoin nodes in general) is that every additional bit of functionality is an attack vector
777 2012-07-11 17:39:38 <TD> so extending their power is a very hard problem
778 2012-07-11 17:39:54 <TD> if you get it wrong the entire system could collapse, or people could have their money stolen, or be double-spent into bankruptcy.
779 2012-07-11 17:39:58 <gmaxwell> amiller: the _miners_ don't trust the oracles. The oracles are _your_ problem.
780 2012-07-11 17:40:11 <TD> in contrast it's pretty safe to run arbitrary code inside an oracle. it can't do anything beyond sign your tx or not sign it.
781 2012-07-11 17:40:31 <gmaxwell> You release a transaction locked to the consent of some ensemble of oracles. If the oracles cheat, too bad for you.. but bitcoin doesn't care.
782 2012-07-11 17:40:33 <TD> the worst case scenario is, the transaction they're mediating goes the wrong way. but it only affects you. and nothing stops you having a big p2p network of oracles
783 2012-07-11 17:40:50 <gmaxwell> And you should make better decisions about your oracles in the future. :)
784 2012-07-11 17:40:53 <amiller> gmaxwell, i know that, i didn't mean to imply otherwise
785 2012-07-11 17:41:08 <amiller> also for things like measuring my grandfather's death, i nkow there's no hope of relying _only_ on bitcoin miners
786 2012-07-11 17:41:17 <gmaxwell> amiller: good I was about to say that.
787 2012-07-11 17:41:19 <amiller> but for things that are just about simple state machine transitions, it could be unnecessary to have to choose oracles
788 2012-07-11 17:41:56 <gmaxwell> amiller: I think thats wrong in practice unless you can make the validation as ~cheap as a regular finical transaction.
789 2012-07-11 17:42:08 <gmaxwell> Otherwise the usage is paracitic on everyone else who just wants inflation proof money.
790 2012-07-11 17:42:19 <gmaxwell> parasitic*
791 2012-07-11 17:43:14 <TD> amiller: i think it's easier to debate specific use cases than the general case. so the chess example isn't that great.
792 2012-07-11 17:43:17 <amiller> yeah, i'm assuming it's possible to pay a fair proportional fees for whatever costs i burden the network with
793 2012-07-11 17:43:19 <TD> there are other ways to do it
794 2012-07-11 17:43:23 <gavinandresen> speaking of "every additional bit of functionality is a potential attack vector" ...  I've been thinking about double spends and transaction replacement and fees
795 2012-07-11 17:43:32 <TD> can you find an example of a compelling app that would interest the community, that needs a much more powerful script?
796 2012-07-11 17:44:25 <amiller> TD, i suppose i should try. at least now i can purge my head of the random processes bullshit i've been struggling with for months.
797 2012-07-11 17:44:33 <amiller> i bet i can find one
798 2012-07-11 17:44:52 <TD> amiller: the issue is, you burden any user who runs a node, and they can't avoid processing your data. so if there are 25k nodes on todays tiny system, how do you recompense them for the storage and computation? you don't even know their costs
799 2012-07-11 17:45:09 <TD> amiller: miners can choose to ignore your transactions if they don't want to process them
800 2012-07-11 17:45:14 <TD> amiller: at least until they get included
801 2012-07-11 17:45:36 <TD> amiller: the oracle protocol + micropayment channels do allow you to recompense the oracle nodes, giving people a genuine incentive to run them
802 2012-07-11 17:46:05 <TD> amiller: and because you talk to each one directly, you can pick ones that are both trustworthy and cheap. whereas you can't really connect to every bitcoin node to give them some money for their troubles.
803 2012-07-11 17:46:14 <TD> so it seems more elegant, from my perspective
804 2012-07-11 17:46:17 <TD> gavinandresen: how so?
805 2012-07-11 17:46:22 <gmaxwell> Redeeming an oracle locked transactions will require a transaction regardless... so you can output to a multisig of the same oracles to pay them.
806 2012-07-11 17:47:57 <gavinandresen> TD: spamming a continuous stream of nSequence++ transaction variations into the network just because you can seems like the main risk to just re-enabling transaction replacement as Satoshi wrote it.
807 2012-07-11 17:48:29 <TD> i suppose you could have a cooldown period. but if you see nodes as work queues with prioritization rules, it shouldn't be an issue
808 2012-07-11 17:48:39 <TD> besides
809 2012-07-11 17:48:42 <gavinandresen> TD: we disagreed about using alternative replacement rules (I like the "every replacement needs to pay more in fees", where "more" follows the free-transaction rules)
810 2012-07-11 17:48:43 <TD> replacing a tx is cheap
811 2012-07-11 17:48:46 <TD> the inputs have to be the same
812 2012-07-11 17:48:53 <TD> so the signatures are already checked, i think. they should be cached.
813 2012-07-11 17:49:37 <TD> given that for replacement to occur the zeroth input must be connected to the same output, and there's no point changing the signatures in the input as then it'd simply fail to verify at commit time ..... it's unclear to me you can DoS nodes this way
814 2012-07-11 17:50:08 <TD> you could just do a trivial memcmp on the signature and if it's not the same as last time, ignore the update. i'd have to think carefully about if there were any issues with that
815 2012-07-11 17:50:35 <TD> hmm, actually, never mind. ignore me.
816 2012-07-11 17:50:40 <TD> i'm talking rubbish
817 2012-07-11 17:50:44 <TD> of course the signatures have to be differen
818 2012-07-11 17:50:48 <gavinandresen> good, I was about to dig out the code, I thought the sig covered nSequence
819 2012-07-11 17:51:25 <gavinandresen> So there's an unrelated thought that I think might actually be related
820 2012-07-11 17:52:00 <gavinandresen> The unrelated thought is the idea of broadcasting double-spend attempts instead of just dropping them... which also needs spam prevention
821 2012-07-11 17:52:00 <TD> ok
822 2012-07-11 17:52:09 <TD> right. like the paper suggested.
823 2012-07-11 17:52:15 <gavinandresen> which paper?
824 2012-07-11 17:52:16 <TD> btw the authors of that paper came to the last bitcoin switzerland meetup
825 2012-07-11 17:52:25 <TD> the ETH paper on double spending attacks. the one that suggested double spend alerts.
826 2012-07-11 17:52:28 <gmaxwell> gavinandresen: yea, I described how to do that on the mailing list enos ago.
827 2012-07-11 17:52:38 <TD> ok, for anti-spam
828 2012-07-11 17:52:58 <TD> let's say we multi-thread signature checking/tx verification. that needs to be done to buy more scaling time sooner or later anyway
829 2012-07-11 17:53:07 <TD> the worker threads pop signatures/txns off a shared queue.
830 2012-07-11 17:53:09 <gavinandresen> So if an input is spent twice, you relay a new 'doublespendinv' or something message
831 2012-07-11 17:53:10 <gmaxwell> gavinandresen: the spamming goes away if each spendable input can only get one alert. So your ability to cause alerts is rate limited by your ability to create valid txn spending inputs.
832 2012-07-11 17:53:26 <gavinandresen> (if spent a gazillion times, you just broadcast one though)
833 2012-07-11 17:53:31 <gavinandresen> right
834 2012-07-11 17:53:32 <gmaxwell> And the alert contains the two signatures, so it _proves_ there was a double spend.
835 2012-07-11 17:53:35 <TD> now whenever a new piece of work comes in, the queue is locked and we figure out where it should go. if it's a replacement of a transaction that's already in the mempool, clear any pending work items related to the previous one and add it to the bottom
836 2012-07-11 17:54:00 <TD> if it's from a newly broadcast, validly solved block, add the signatures at the top
837 2012-07-11 17:54:12 <TD> or have multiple queues, that are drained in priority (that's probably better)
838 2012-07-11 17:54:30 <TD> so you can spam tons of updates but the worst you can do is max out peoples CPUs. latency on valuable work won't be affected
839 2012-07-11 17:55:27 <gmaxwell> gavinandresen: also you only relay the alert when you've also heard one of the txn.  So the alert identifies the input, then gives you the two signatures for the input.  If you haven't seen txn for that input you just hold the alert in the queue for a bit. When you do see a txn for that input you forward the alert.
840 2012-07-11 17:55:49 <gavinandresen> gmaxwell: ACK.  why is there not a pull request for that yet :)
841 2012-07-11 17:56:26 <TD> gmaxwell: don't you need both whole transactions?
842 2012-07-11 17:56:42 <gavinandresen> TD:  I'm more worried about somebody trying to saturate network bandwidth... I suppose if you don't relay until after you validate... but if I spam 8 different versions of a transaction to 8 different peers
843 2012-07-11 17:56:42 <TD> gmaxwell: nodes don't know which tx they saw, and you can't verify a signature with only a partial transaction
844 2012-07-11 17:56:57 <gmaxwell> TD: you need everything the signatures cover, not the whole txn.
845 2012-07-11 17:57:25 <gmaxwell> gavinandresen: because the devil is in the details! :)
846 2012-07-11 17:57:54 <gavinandresen> doublespend message can just have the two txids, a node that doesn't already have both will request using regular inv message....
847 2012-07-11 17:58:08 <TD> gavinandresen: if the tx isn't a replacement (two different txns with the same sequence numbers?) then it's a double spend and can be treated as such. if it's not IsNewerThan(), you don't have to verify the signatures
848 2012-07-11 17:58:19 <gmaxwell> gavinandresen: ah, thats simpler indeed!
849 2012-07-11 17:58:27 <gmaxwell> anyone who gives you the alert already has the txn.
850 2012-07-11 18:00:26 <gavinandresen> TD: ok, I'll have to think more... I still don't think the incentives are right for miners, though.  If I want to break the rules I just spend an input involved in some complicated contract with a higher fee and economically rational miners will ignore all the previous non-final transactions and take the bird-in-the-hand fee
851 2012-07-11 18:00:34 <TD> gavinandresen: but i think network bandwidth can be seen the same way as cpu time/disk seeks. it's a limited resource. prioritize stuff that looks abusive below the stuff that doesn't
852 2012-07-11 18:00:46 <TD> but there's no need for hard cutoffs that could break in weird ways, or changes to fees
853 2012-07-11 18:01:41 <gavinandresen> I'm all for moving away from hard cutoffs.  But I'm all for movement, too
854 2012-07-11 18:02:49 <midnightmagic> TD/amiller: re chess game tied to the blockchain, there's a program called chronobit which is a special-case (timestamping) of tying basically arbitrary information to the main block chain via p2pool and merged-mining, so a large chunk of work for that is already done by mr Uukgoblin, here: https://github.com/goblin/chronobit/
855 2012-07-11 18:03:57 <UukGoblin> chess game?
856 2012-07-11 18:04:02 <UukGoblin> I was thinking about poker, but chess?
857 2012-07-11 18:04:04 <UukGoblin> like, for money?
858 2012-07-11 18:04:32 <midnightmagic> UukGoblin: it's a ways back in scrollback, I didn't see what the end purpose was, but it's amiller, so it's bound to be something interesting.
859 2012-07-11 18:06:07 <TD> gavinandresen: i think that strategy is self defeating. if miners apply such a patch in a widespread manner, it just means nobody would rely on the feature because it'd not work as intended. that in turn means there'd be no income to be had from doing it.
860 2012-07-11 18:06:27 <TD> gavinandresen: i mean, miners are not entirely self interested. if they were, we'd see empty blocks be the majority case.