1 2012-06-03 00:59:51 <luke-jr> TD[gone]: how can I contact YouTube people?
  2 2012-06-03 03:00:35 <amiller> etotheipi_, first of all i have a general purpose implementation of balanced merkle-trees here https://github.com/amiller/redblackmerkle
  3 2012-06-03 03:02:11 <amiller> you don't need to mess with the stuff like bucket-sort and uniform key space
  4 2012-06-03 03:02:27 <amiller> you simply balance the tree (a 2-3 tree or a redblack tree) each time you insert or delete something
  5 2012-06-03 03:02:39 <amiller> it still takes O(log N) to do either of those
  6 2012-06-03 03:03:47 <amiller> i think that aspect is the only thing missing from yours and gmaxwell's and dthdt's posts about using more merkle trees
  7 2012-06-03 03:05:10 <amiller> i'm still meaning to write a forum post or a wiki or a note to bitcoin-dev summarizing all this but i'm not too sure where to dig in, so if you give me any feedback i'll probably get a bit further along
  8 2012-06-03 03:05:47 <amiller> the point of balancing is that you maintain a strict worst-case bound for how long it takes to verify a tree.
  9 2012-06-03 03:06:37 <amiller> so there's no way for someone to 'congest a portion of tree' the way you described
 10 2012-06-03 03:07:12 <amiller> so there are at least two kinds of useful ways to index this tree
 11 2012-06-03 03:08:19 <amiller> 1) is by txhash/index - if you index it that way, a lite client (a verifier with O(1) of state, i.e. the root hash) can tell if any inputs in a new transaction have already been spent or not.
 12 2012-06-03 03:08:43 <amiller> 2) is by address - if you store them this way, then a lite client can verify its balance at any given time.
 13 2012-06-03 03:09:45 <amiller> i see no reason not to maintain at least two trees, one for each of those useful indexes.
 14 2012-06-03 03:11:29 <amiller> This stuff is pretty well described in an academic research using the phrase "Authenticated Data Structures" but I don't know of any other implementations
 15 2012-06-03 05:00:53 <forrestv> amiller, you're talking about the "flipping the chain" idea?
 16 2012-06-03 05:01:54 <amiller> i don't recognize that phrase?
 17 2012-06-03 05:02:17 <Diablo-D3> oh thats easy
 18 2012-06-03 05:02:24 <Diablo-D3> how about I give you this
 19 2012-06-03 05:02:31 <Diablo-D3> and you give me my phone call
 20 2012-06-03 05:03:04 <Diablo-D3> also, I feel old
 21 2012-06-03 05:03:15 <Diablo-D3> I just realized there are kids out there who were born AFTER the matrix.
 22 2012-06-03 05:03:28 <Diablo-D3> and they might be mining bitcoins.
 23 2012-06-03 05:03:41 <Diablo-D3> kids born AFTER the matrx... mining bitcoins.
 24 2012-06-03 05:03:51 <Diablo-D3> after the matrix, mining bitcoins.
 25 2012-06-03 05:04:08 <Diablo-D3> seriously. what the fuck.
 26 2012-06-03 05:05:44 <amiller> forrestv, i wonder if by 'flipping' you mean making the 'unspent coins database' the primary datastructure in the protocol, instead of the 'blockchain' history?
 27 2012-06-03 05:10:01 <bd_> not really possible, that. You can't trust another node to compute the transition between states (unspend DBs) correctly, so you need to see the state transitions (blockchain history) to check their work
 28 2012-06-03 05:10:23 <bd_> clients could of course start pruning old txns and just maintain the unspent coins. But the protocol itself needs to remain in terms of txns
 29 2012-06-03 05:13:51 <amiller> ah a great question, bd_
 30 2012-06-03 05:14:02 <amiller> you _can_ trust another node to compute the transition in the unspend DB
 31 2012-06-03 05:14:16 <bd_> not if you never see said txn :)
 32 2012-06-03 05:14:28 <amiller> you need to see the said txn, but you do not need the history
 33 2012-06-03 05:14:40 <amiller> to see why, consider that the unspend DB is a balanced binary search tree
 34 2012-06-03 05:14:48 <amiller> the transitions are always insertions or deletions in the tree
 35 2012-06-03 05:15:04 <amiller> insertions and deletions each only involve touching O(log N) nodes in the worst-case
 36 2012-06-03 05:15:05 <bd_> the representation of the unspent DB isn't really the interesting part, to be honest
 37 2012-06-03 05:15:24 <bd_> bitcoin is a state machine, where state transitions are determined by a distributed consensus protocol
 38 2012-06-03 05:15:41 <bd_> the unspent coin DB is the state, and blocks are state transitions
 39 2012-06-03 05:16:08 <bd_> the distributed consensus protocol must be resistant to byzantine failures, and this makes the whole thing more difficult :)
 40 2012-06-03 05:16:31 <bd_> as I said, nodes can certainly discard their transition log and just keep the current state, after they finish checking the work themselves
 41 2012-06-03 05:16:47 <bd_> however there are limits to this - someone needs a full log to help new nodes bootstrap themselves
 42 2012-06-03 05:17:00 <bd_> and you need to keep enough transactions to be able to back out and replay when the blockchain forks
 43 2012-06-03 05:17:18 <bd_> and, of course, what goes over the wire are blocks, not unspent-DBs
 44 2012-06-03 05:17:39 <bd_> of course, when bootstrapping, if you have a trusted node, you can inherit its current-state database and replay from there too
 45 2012-06-03 05:19:35 <bd_> anyway, all this follows from the security requirements of the protocol
 46 2012-06-03 05:19:48 <bd_> the representation of the unspent DB is a secondary concern
 47 2012-06-03 05:20:09 <amiller> i agree with everything you wrote
 48 2012-06-03 05:20:10 <bd_> (and a hash table might be a better choice than a binary tree, if you have a SSD :)
 49 2012-06-03 05:20:22 <amiller> i'm not sure what it is you objected to originally
 50 2012-06-03 05:20:31 <bd_> 03:05 < amiller> forrestv, i wonder if by 'flipping' you mean making the 'unspent coins database' the primary datastructure in the protocol, instead of the
 51 2012-06-03 05:20:33 <amiller> i think about which is the primary structure
 52 2012-06-03 05:20:35 <bd_> 'blockchain' history?
 53 2012-06-03 05:20:39 <bd_> 03:14 < amiller> you _can_ trust another node to compute the transition in the unspend DB
 54 2012-06-03 05:20:53 <bd_> You can't trust another node to compute the transition
 55 2012-06-03 05:21:04 <bd_> as they may be a malicious node (ie, byzantine failure)
 56 2012-06-03 05:21:11 <bd_> you need to check their work
 57 2012-06-03 05:21:25 <amiller> let me rephrase that then, you _can_ double check the transition in the unspend DB
 58 2012-06-03 05:21:30 <amiller> even if you only have the current root hash
 59 2012-06-03 05:21:33 <amiller> that isn't possible with a hash table
 60 2012-06-03 05:21:57 <amiller> you'd have to have the whole hash table to verify the transition
 61 2012-06-03 05:22:18 <bd_> sure, but why do you need the root hash of the unspent db then?
 62 2012-06-03 05:22:27 <bd_> oh, I see what you're saying now
 63 2012-06-03 05:23:36 <bd_> essentially representing the state of the system with the root of a hash tree, then transmitting all mutated nodes with the state transition (ie, you send prior values of the nodes and the transition function in the block)
 64 2012-06-03 05:24:00 <bd_> and presumably only store the nodes of interest to you locally? (ie, nodes where you have coins)
 65 2012-06-03 05:24:33 <amiller> i don't think that a block needs to contain the prior values of the nodes, or anything other than it does currently
 66 2012-06-03 05:24:53 <amiller> but in order to verify the transactions in a block, a client would need some untrusted auxiliary data called a Verification Object
 67 2012-06-03 05:24:57 <bd_> if it has nothing more than it has now, you haven't changed the protocol :)
 68 2012-06-03 05:25:30 <bd_> so
 69 2012-06-03 05:25:39 <bd_> the root hash of the unspent db does need to be trusted
 70 2012-06-03 05:25:57 <amiller> partially.
 71 2012-06-03 05:26:02 <bd_> although it can be trusted because it's locally computed in the bootstrapping process
 72 2012-06-03 05:26:07 <amiller> it needs to be trusted in the sense that the byzantine fault tolerant network decides on its ordering
 73 2012-06-03 05:26:21 <bd_> well, we already have transaction ordering
 74 2012-06-03 05:27:01 <bd_> anyway, what this boils down to is storing your _own_ unspent db in an untrusted third party, and just remembering the root hash so you can check that the untrusted third party hasn't done anything to it
 75 2012-06-03 05:27:08 <amiller> someone who has a full node now, would gain absolutely nothing from this
 76 2012-06-03 05:27:17 <amiller> the point is to make verification viable for people who only store O(1) root hashes of state
 77 2012-06-03 05:27:28 <amiller> i think that's compatible with what you just wrote
 78 2012-06-03 05:27:35 <bd_> as a further refinement, you can arrange to use the same procedure for storing your unspent db, and share storage with other users
 79 2012-06-03 05:27:41 <bd_> (mutually untrusted users, of course)
 80 2012-06-03 05:27:45 <bd_> seems sensible
 81 2012-06-03 05:28:16 <amiller> even better than that, i want to have an incentive system for the people who store these untrusted unspend dbs.
 82 2012-06-03 05:28:20 <bd_> you might also want to push tree nodes needed to verify a block along with the block itself, to speed block acceptance
 83 2012-06-03 05:29:04 <bd_> amiller: seems like it would be simple enough - just have a normal pay-to-get-a-login system?
 84 2012-06-03 05:29:13 <bd_> I mean, any untrusted third-party will do
 85 2012-06-03 05:29:16 <bd_> heck, store it in S3
 86 2012-06-03 05:29:17 <amiller> the incentive system i'd like is to replace mining proof-of-work with one based on demonstrating random access to this unspend db.
 87 2012-06-03 05:29:54 <bd_> amiller: all nodes require random access to the unspent DB to verify transactions... that's not much of a proof of work
 88 2012-06-03 05:30:19 <bd_> I mean, the system is based on the assumption that random access to the unspent DB is much cheaper than the proof of work
 89 2012-06-03 05:30:35 <bd_> I don't think this needs to be bolted onto the core protocol
 90 2012-06-03 05:30:56 <bd_> just send some BTC to $third-party to get them to issue you a password (or enable access from your public key, etc)
 91 2012-06-03 05:30:57 <amiller> well the proof of work is a single hash right now
 92 2012-06-03 05:31:06 <amiller> but mining requires millions and millions of hashes
 93 2012-06-03 05:31:33 <bd_> sure
 94 2012-06-03 05:31:34 <amiller> likewise the proof-of-work would be a constant number of entries from the unspent-coins db, but the work itself would require tons of throughput
 95 2012-06-03 05:31:49 <bd_> but the thing about the POW is that you need to be able to _verify_ the proof-of-work in a _much_ shorter time than it was created
 96 2012-06-03 05:31:55 <amiller> right
 97 2012-06-03 05:32:05 <bd_> if you start creating a random-access-based POW, how do you verify that without doing the work again?
 98 2012-06-03 05:32:20 <amiller> i'll describe it, i also have a link to a paper by ron rivest that gave me the idea
 99 2012-06-03 05:32:33 <amiller> you start with a nonce, and you hash it and use that to select a random element from the database
100 2012-06-03 05:33:02 <amiller> then you also hash that data when you look it up, and combine it with the previous hash. now you use that hash to select the next element
101 2012-06-03 05:33:17 <amiller> you keep going for k times, at the end, you see if your hash has a lot of zeroes in the front, if so you announce your winning block
102 2012-06-03 05:33:56 <bd_> hmm, okay. So you need fast access to the DB to compute this. But what's your incentive for providing it to others?
103 2012-06-03 05:34:39 <amiller> http://people.csail.mit.edu/rivest/BowersVanDijkJuelsOpreaRivest-HowToTellIfYourCloudFilesAreVulnerableToDriveCrashes.pdf this is the paper i mentioned by the way that includes the above description, basically.
104 2012-06-03 05:36:05 <amiller> it doesn't provide an incentive to share it, just an incentive to have it.
105 2012-06-03 05:36:14 <amiller> i think that's sufficient and optimal but i'm not too sure of this part.
106 2012-06-03 05:36:25 <amiller> the way i think of it is that right now, mining requires an unspent coin database and also a gpu
107 2012-06-03 05:36:33 <amiller> you get reimbursed in some sense by adding more gpus
108 2012-06-03 05:36:42 <bd_> mining nodes don't really need the unspent db
109 2012-06-03 05:36:47 <bd_> they just need a control node that has said db
110 2012-06-03 05:36:54 <bd_> heck, not even that
111 2012-06-03 05:36:57 <bd_> you just need the previous block
112 2012-06-03 05:37:04 <bd_> and you can issue new blocks that only have a coin-minting txn :D
113 2012-06-03 05:37:17 <amiller> right
114 2012-06-03 05:37:19 <bd_> I mean, currently, that is.
115 2012-06-03 05:37:20 <amiller> and in fact, it's cheaper that way
116 2012-06-03 05:37:21 <bd_> anyway
117 2012-06-03 05:37:29 <amiller> if the mining and the double checking were unified
118 2012-06-03 05:37:31 <bd_> the thing you want to incentivize is not keeping the unspent db around
119 2012-06-03 05:37:36 <bd_> it's making it available to _others_
120 2012-06-03 05:37:42 <bd_> the whole point is not everyone needs to maintain this db
121 2012-06-03 05:37:50 <amiller> everyone needs access to it though
122 2012-06-03 05:38:02 <bd_> so you don't want to create an incentive for everyone to maintain their own db in RAM because otherwise they can't mine
123 2012-06-03 05:38:20 <bd_> you want to create an incentive for a number of providers to provide access to their DBs
124 2012-06-03 05:38:38 <bd_> which is not what your hashing thing would do. Indeed, you'd want to monopolize access to your DB so others don't use it for their own mining
125 2012-06-03 05:38:47 <bd_> The solution is really very simple
126 2012-06-03 05:38:51 <bd_> just pay for access to a db provider
127 2012-06-03 05:39:11 <bd_> no need for fancy cryptography, just an ordinary contract
128 2012-06-03 05:39:33 <bd_> pay with micropayments so a scam provider doesn't get very much money, if you want to be fancy about it
129 2012-06-03 05:40:14 <amiller> the point of the technique in that rivest paper is to have a way of checking is a service provider is really doing the job they say they are
130 2012-06-03 05:40:15 <bd_> hmm
131 2012-06-03 05:40:19 <bd_> I find some of the flawed assumptions in that rivest paper quite interesting ;)
132 2012-06-03 05:40:43 <amiller> er, yeah if you're talking about the assumptions about hard drive performance characteristics then yeah
133 2012-06-03 05:40:51 <bd_> there are ways to get really good redundancy without improvements in seek times
134 2012-06-03 05:40:59 <bd_> can't say more about that though
135 2012-06-03 05:41:39 <amiller> you can purchase a subscription to a db provider and make them perform proofs-of-work for you
136 2012-06-03 05:42:57 <amiller> i think i'm still missing some kind of explanation here about why it works out that it's so useful to align the proof-of-work with the validation task.
137 2012-06-03 05:43:19 <amiller> mining has the effect of subsidizing investment in gpus and other mining hardware
138 2012-06-03 05:43:30 <bd_> well, no, I think you're asking people to prove the wrong thing
139 2012-06-03 05:43:31 <amiller> the work itself necessarily has to be burnt cycles
140 2012-06-03 05:43:34 <bd_> it's all about incentives
141 2012-06-03 05:43:42 <amiller> but the subsidized investment doesn't need to be wasted
142 2012-06-03 05:43:49 <bd_> what do we want to encourage people to do?
143 2012-06-03 05:44:06 <bd_> subsidize DRAM manufacturers? :)
144 2012-06-03 05:44:09 <amiller> you want to reduce the costs needed to participate in the network, to encourage decentralization
145 2012-06-03 05:44:29 <amiller> it's not about the storage hardware in this case, it's about what the storage hardware is _storing_
146 2012-06-03 05:44:41 <bd_> sure. but making the POW demonstrate possession of the unspent DB doesn't achieve that goal
147 2012-06-03 05:44:55 <bd_> you proved that you have the db, you haven't proved that you're making it available to others
148 2012-06-03 05:45:08 <amiller> but if you make it available to others, you can prove that to them
149 2012-06-03 05:45:09 <bd_> and like I said, there's a disincentive to making this db available
150 2012-06-03 05:45:56 <bd_> amiller: They need access to the db anyway, to participate in bitcoin at all. If I were mining, why would _I_ personally want to sacrifice memory bandwidth to making my db available?
151 2012-06-03 05:46:21 <bd_> A rational miner would just put it onto a dedicated box and have it only serve queries from the mining program.
152 2012-06-03 05:46:41 <amiller> think of the miner as separate from the db provider
153 2012-06-03 05:46:41 <bd_> They might _also_ operate a public db, potentially for a fee, but this is orthogonal to the proof-of-work
154 2012-06-03 05:46:46 <bd_> indeed
155 2012-06-03 05:46:48 <amiller> providing the db is an untrusted service
156 2012-06-03 05:47:04 <amiller> miners then are nodes that hire the untrusted db provider
157 2012-06-03 05:47:05 <bd_> sure. my point is making the POW be DB access does not impact whether there are db providers
158 2012-06-03 05:47:16 <bd_> no, that won't be the case
159 2012-06-03 05:47:22 <bd_> if your POW is based on latency to the DB
160 2012-06-03 05:47:24 <amiller> you mine when you pay an untrusted db provider to send you back a proof of work that includes a block winning bonus to yourself
161 2012-06-03 05:47:27 <bd_> you can't use an external DB
162 2012-06-03 05:47:50 <bd_> amiller: why wouldn't the db provider just find a proof-of-work that pays themselves then?
163 2012-06-03 05:48:22 <bd_> the db provider has no reason to implement proof-of-work computation only to pay part of it to a middleman
164 2012-06-03 05:48:32 <bd_> they can do the tiny amount of additional work to prepare the block themselves
165 2012-06-03 05:48:54 <bd_> hence why I say it'd be subsidizing DRAM manufacturers - here the best strategy is to store the entire database in memory, so lookups are extremely fast
166 2012-06-03 05:49:40 <amiller> we're not being very precise about what the different roles are here, especially between verifiers, unspent db providers, and miners
167 2012-06-03 05:49:51 <amiller> but the point is to subsidize the thing that's expensive and helps verifiers
168 2012-06-03 05:50:02 <bd_> and my point is that this incentive does not have that effect
169 2012-06-03 05:50:09 <bd_> miners have no incentive to make their db available to others
170 2012-06-03 05:50:21 <amiller> miners have an incentive to hire unspent db providers
171 2012-06-03 05:50:24 <bd_> db providers have no incentive to assist in mining
172 2012-06-03 05:50:26 <amiller> unspent db providers can sell subscriptions to verifiers
173 2012-06-03 05:50:40 <bd_> but db providers can indeed sell subscriptions to verifiers
174 2012-06-03 05:50:52 <amiller> right and those subscriptions are cheaper if they're subsidized by mining as well
175 2012-06-03 05:51:19 <bd_> why would they be subsidized? like I said, the db provider has no reason to let the miners have a cut of any proof-of-work computation said provider may or may not do
176 2012-06-03 05:51:31 <bd_> If the DB provider does the POW computation, that makes the DB provider a miner.
177 2012-06-03 05:51:35 <amiller> that's fine
178 2012-06-03 05:51:39 <amiller> that's not the part that needs subsidizing
179 2012-06-03 05:51:47 <bd_> Which is indeed fine, but my point is miners need to have a copy of the DB locally
180 2012-06-03 05:51:57 <bd_> so they have no reason to subsidize db providers
181 2012-06-03 05:51:58 <amiller> not if they hire unspent db providers
182 2012-06-03 05:52:12 <amiller> if the have the copy of the db locally then they've also invested in an unspend db
183 2012-06-03 05:52:14 <amiller> and they can sell a subscription
184 2012-06-03 05:52:29 <bd_> amiller: the network latency involved in connecting to a db provider makes it impractical to mine-at-a-distance. A local copy will produce much better results
185 2012-06-03 05:52:37 <amiller> you don't need to do round trips
186 2012-06-03 05:52:49 <bd_> and offloading the entire POW computation to the db is not something that a rational db provider would allow
187 2012-06-03 05:52:54 <amiller> you just tell the db provider to mine for you and you pay them for the best proof of work they can provide
188 2012-06-03 05:53:10 <bd_> because if they're going to do that computation, they should take the entire reward for themselves
189 2012-06-03 05:53:52 <bd_> and if you're paying for non-winning blocks
190 2012-06-03 05:54:00 <bd_> then you've just reinvented the mining pool :)
191 2012-06-03 05:54:17 <bd_> my point though is that having a db does not imply you want to provide public access
192 2012-06-03 05:54:42 <bd_> for a miner, memory bandwidth is precious. To give some of it up to provide public access would require a suitable payment to offset the loss in mining efficiency
193 2012-06-03 05:54:48 <amiller> my goal is to lower the marginal cost, if you're mining, of also providing public db access
194 2012-06-03 05:54:50 <bd_> and many miners might not bother doing so
195 2012-06-03 05:56:00 <bd_> amiller: the other thing is, a DB meant for mining might not necessarily scale.
196 2012-06-03 05:56:20 <bd_> consider one possible design for a public unspent DB: Shove all the blocks into S3 and have a server gating access to it
197 2012-06-03 05:56:25 <bd_> Easy.
198 2012-06-03 05:56:34 <bd_> Doesn't require the whole thing be in memory
199 2012-06-03 05:56:45 <bd_> And, vitally, easily gives access to historical blocks
200 2012-06-03 05:56:53 <bd_> The miner, on the other hand, only needs the _current_ blocks
201 2012-06-03 05:56:54 <bd_> er
202 2012-06-03 05:56:56 <bd_> current DB
203 2012-06-03 05:57:11 <bd_> they can immediately discard the old DB when a new block comes along
204 2012-06-03 05:57:18 <bd_> which is exactly when the old DB is needed by verifiers
205 2012-06-03 05:57:37 <bd_> A rational miner will discard the data needed by verifiers exactly when the verifiers need it
206 2012-06-03 05:57:51 <bd_> as they want to make room for the current DB to start mining from it immediately
207 2012-06-03 05:58:35 <amiller> the old DB and the current DB mostly overlap
208 2012-06-03 05:58:49 <bd_> mostly. but why bother storing the old one?
209 2012-06-03 05:59:07 <amiller> if you want to make old ones covered by the incentive plan
210 2012-06-03 05:59:10 <bd_> much easier to mutate it in-place
211 2012-06-03 05:59:33 <amiller> it's easy to include that as well, when you select random entries you either select them from the current db, or select them from any of the previous dbs
212 2012-06-03 05:59:47 <amiller> you could also have a sliding window that goes back to some finite amount of time
213 2012-06-03 06:00:03 <amiller> this is described in one paper as a Persistent Authenticated Datastructure and it has one figure that's really cool and simple
214 2012-06-03 06:00:26 <bd_> I suppose. It still doesn't incent people to make it accessible, though.
215 2012-06-03 06:00:47 <amiller> http://citeseerx.ist.psu.edu/viewdoc/summary?doi=10.1.1.23.9438
216 2012-06-03 06:00:50 <bd_> the CPU and memory bandwidth cost of providing access would have to be more lucrative than mining
217 2012-06-03 06:02:19 <amiller> you can lead a horse to water but you can't make it drink
218 2012-06-03 06:02:25 <bd_> perhaps :)
219 2012-06-03 06:02:46 <bd_> anyway, I don't know if it's necessary
220 2012-06-03 06:03:00 <bd_> If you were to introduce this, initially we have the current world, where everyone has their own unspent db
221 2012-06-03 06:03:09 <bd_> for people to discard these dbs, unspent db providers must appear first
222 2012-06-03 06:03:21 <bd_> these could be, e.g., pool master servers
223 2012-06-03 06:03:30 <bd_> or just a random node somewhere
224 2012-06-03 06:03:38 <sipa> 
225 2012-06-03 06:03:55 <bd_> there's no point in incentivizing people to have a full db when they _already do_
226 2012-06-03 06:04:00 <amiller> i didn't mean to get so distracted talking about the proof of work part
227 2012-06-03 06:04:04 <bd_> :)
228 2012-06-03 06:04:05 <amiller> it's not nearly as important
229 2012-06-03 06:04:09 <bd_> indeed
230 2012-06-03 06:04:20 <bd_> I do think having a way to offload the unspent db is a nice idea though
231 2012-06-03 06:04:49 <amiller> yeah what i am hoping to do is start solving that part, which doesn't even require any network changes
232 2012-06-03 06:05:11 <bd_> as I said before, it would be very helpful, though, to push any nodes needed to verify a block along with said block
233 2012-06-03 06:05:25 <bd_> otherwise you'll have a thundering herd scenario whenever a block is found
234 2012-06-03 06:05:51 <amiller> but you just look it up in your database
235 2012-06-03 06:06:09 <bd_> the verifiers don't have their db, right?
236 2012-06-03 06:06:25 <amiller> not if they have access to an untrusted db provider
237 2012-06-03 06:06:25 <bd_> when a new block is published, all verifiers need to fetch whatever tree nodes are needed to verify said block
238 2012-06-03 06:06:28 <bd_> at approximately the same time
239 2012-06-03 06:06:36 <bd_> thus overwhelming the untrusted db providers
240 2012-06-03 06:06:58 <amiller> the verifiers CAN have their own db, and they probably will continue to just as they do now
241 2012-06-03 06:07:07 <amiller> the point is that it's not requred because you CAN offload it if you choose to as well
242 2012-06-03 06:07:14 <bd_> offloaded clients, then
243 2012-06-03 06:07:21 <bd_> which also participate as verifiers
244 2012-06-03 06:07:28 <amiller> SPVs require helpers
245 2012-06-03 06:07:35 <amiller> that's still true
246 2012-06-03 06:07:36 <bd_> point is, when a block comes out, all offloaded clients start fetching the same data at once
247 2012-06-03 06:07:55 <bd_> far better to gossip the blocks around to avoid this kind of load spike
248 2012-06-03 06:08:01 <bd_> er, gossip the tree nodes around rather
249 2012-06-03 06:11:03 <amiller> i don't see why they need to be included in the block vs not included in the block
250 2012-06-03 06:11:15 <bd_> it's not necessary for the protocol, and I'
251 2012-06-03 06:11:21 <bd_> m not suggesting they be _part_ of the block
252 2012-06-03 06:11:35 <bd_> just gossiped around with the block, to reduce the impact of load spikes on these DB providers
253 2012-06-03 06:11:41 <amiller> okay, then i agree totally with that
254 2012-06-03 06:11:59 <bd_> you could argue that peers have no incentive to gossip the nodes
255 2012-06-03 06:12:08 <bd_> but they don't really have an incentive to gossip the blocks either
256 2012-06-03 06:12:17 <amiller> i would _not_ have made that argument :p
257 2012-06-03 06:12:22 <bd_> :)
258 2012-06-03 06:12:37 <amiller> so you mentioned that bitcoin is a byzantine consensus protocol
259 2012-06-03 06:12:49 <amiller> i've been hoping to talk to someone about that a bit
260 2012-06-03 06:13:06 <amiller> all the byzantine consensus protocols include 'message delays', that turns out to be like half the challenge of the problem
261 2012-06-03 06:13:14 <bd_> It would be so much simpler if it didn't need to be resilient to byzantine failures :)
262 2012-06-03 06:13:21 <bd_> well, that's not even a byzantine failure
263 2012-06-03 06:13:27 <bd_> eg even paxos has to deal with message delays
264 2012-06-03 06:13:34 <amiller> the challenges are 1) just shy of half the nodes might be faulty,  and 2) the messages between nodes, even the non-faulty ones, can be delayed by arbitrary amounts
265 2012-06-03 06:14:18 <bd_> which basically means it might take a very long time to converge in the presence of message delays
266 2012-06-03 06:14:36 <bd_> all nodes thus have a very strong incentive to _receive_ messages from other nodes quickly
267 2012-06-03 06:14:49 <amiller> regarding different kinds of message delays, a) in the synchronous case, messages are delayed by up to some maximum bound, but that bound is known ahead of time (e.g. paxos)  b) in the partially synchronous case, the message delay bound exists but is not known  and c) if there is no message bound, it's 'asynchronous' and its' impossible
268 2012-06-03 06:14:49 <bd_> they don't necessarily have an incentive to ensure that other nodes receive messages from other nodes quickly
269 2012-06-03 06:14:54 <amiller> so only a) and b) are worth looking at
270 2012-06-03 06:15:18 <amiller> the proof in the bitcoin whitepaper is about some non-existent-in-academia setting in which implicitly there are no message delays
271 2012-06-03 06:15:41 <bd_> message delays basically can result in blockchain forks
272 2012-06-03 06:15:59 <amiller> the forks are guaranteed to go away after some time
273 2012-06-03 06:16:09 <bd_> that time is the bound on propagation delay :)
274 2012-06-03 06:16:32 <amiller> forks can also be caused by the byzantine failures
275 2012-06-03 06:16:59 <bd_> In the nearly-half-are-malicious case, if all messages from the 'healthy' nodes are delayed beyond the point where nodes commit to the current block chain
276 2012-06-03 06:17:03 <amiller> and the byzantine adversary is given the advantage of zero message delays
277 2012-06-03 06:17:20 <bd_> then the malicious block chain could be accepted by a majority of nodes, despite better POWs potentially existing in the healthy nodes
278 2012-06-03 06:17:37 <amiller> so the byzantine consensus protocols have a couple of important qualities
279 2012-06-03 06:17:54 <amiller> one is that the nodes in a consensus protocol eventually 'decide'
280 2012-06-03 06:17:58 <bd_> the gossip protocol helps prevent this though, as it's very difficult to construct a scenario where all healthy node gossip is blocked but malicious<->healthy node gossip isn't
281 2012-06-03 06:18:19 <amiller> deciding is a one-time-only irreversible action, and the definition of the game is that there should never be a fork decision
282 2012-06-03 06:18:25 <amiller> the closest analogue to this in bitcoin-world is the 6 blocks rule
283 2012-06-03 06:18:57 <amiller> we might say that nodes 'decide' after 6 blocks, and we'd want the claim to be that any forks after 6 blocks can only occur with negligible probability
284 2012-06-03 06:19:39 <amiller> if negligible probability is defined as exponentially small for some security parameter k, then then the number of blocks required should hopefully be polynomial in k
285 2012-06-03 06:20:39 <amiller> the point of this puzzle is you have a fight between a 51% with message delays, and a 49% with no message delays
286 2012-06-03 06:21:09 <amiller> can you prove that the stronger 51% cpu will win out after some amount of time, despite the message delay
287 2012-06-03 06:21:21 <bd_> only if you put an upper bound on the delays
288 2012-06-03 06:21:59 <bd_> otherwise, you could construct a scenario in which delays are so high that every node in the 51% believes the only nodes that exist are itself and the 49% of evil nodes
289 2012-06-03 06:22:39 <amiller> right
290 2012-06-03 06:23:02 <amiller> so then there are two cases a) the upper bound is known, call it M, and b) it exists but it's not known
291 2012-06-03 06:23:18 <bd_> unknown isn't good enough
292 2012-06-03 06:23:29 <bd_> any healthy node N can make a guess at the upper bound B
293 2012-06-03 06:23:35 <amiller> i'm following along with this paper btw http://groups.csail.mit.edu/tds/papers/Lynch/jacm88.pdf
294 2012-06-03 06:23:40 <bd_> when B time elapses, N must decide whether it believes that a decision has been made
295 2012-06-03 06:24:06 <bd_> if it hasn't heard from ANY healthy nodes yet (because its guess for B was too small), it will believe that the unhealthy nodes have made whatever decision they claim to have made
296 2012-06-03 06:24:11 <amiller> bd_, how about if we assume that we know the total strength of the network ahead of time?
297 2012-06-03 06:24:32 <amiller> in most byzantine consensus protocols, the number N of nodes is a known parameter, of course the nodes are all known ahead of time, there are very few 'anonymous' node papers
298 2012-06-03 06:25:16 <bd_> I guess you could assume the entire network is unhealthy if you see less than 51% of the strength you expect before B
299 2012-06-03 06:25:53 <bd_> of course, what happens if the malicious nodes allow a few virtual networks to form - say, each one consists of all malicious nodes, and 50% of the healthy nodes
300 2012-06-03 06:26:07 <bd_> so each healthy node sees 74% of the network and declares itself in a healthy network
301 2012-06-03 06:26:11 <bd_> but actually there are two split networks
302 2012-06-03 06:26:16 <bd_> which come to different decisions
303 2012-06-03 06:26:25 <bd_> controlled by the unhealthy nodes
304 2012-06-03 06:26:55 <amiller> there's no way to observe more than 100% of the network in total
305 2012-06-03 06:27:05 <amiller> the 50% has to split its effort
306 2012-06-03 06:27:11 <amiller> er 49%*
307 2012-06-03 06:27:46 <bd_> hmm, I see
308 2012-06-03 06:27:50 <bd_> I suppose that would work
309 2012-06-03 06:27:52 <amiller> in byzantine protocol papers the network is N = 2f+1,   the bad guys are f,  the good guys are f+1
310 2012-06-03 06:28:01 <amiller> we have 100%, 49%, and 51% respectively!
311 2012-06-03 06:28:03 <bd_> but it's not relevant for bitcoin, as the strength of the network is also an unknown
312 2012-06-03 06:28:19 <amiller> deciding also isn't part of bitcoin either
313 2012-06-03 06:28:42 <amiller> so i think it's fair to address the theoretical distributed systems part of bitcoin this way
314 2012-06-03 06:29:03 <amiller> by saying that we can construct a protocol that decides as long as we know the strength of the network
315 2012-06-03 06:29:03 <bd_> deciding is part of bitcoin
316 2012-06-03 06:29:11 <bd_> the decision is simple - which transaction spent coin C?
317 2012-06-03 06:29:16 <bd_> there may be multiple candidates
318 2012-06-03 06:29:29 <amiller> i mean making a final irreversible decision
319 2012-06-03 06:29:35 <amiller> we're more concerned with the idea that it will converge
320 2012-06-03 06:29:42 <amiller> rather than that it will actually end
321 2012-06-03 06:29:42 <bd_> effectively, it does at some point
322 2012-06-03 06:30:03 <bd_> eg, after 6 blocks pass
323 2012-06-03 06:30:16 <amiller> if you know the size of the network, then you can decide irreversibly.
324 2012-06-03 06:30:23 <bd_> yes, but that's a very big if
325 2012-06-03 06:30:28 <amiller> well see the 6 blocks passing aspect of bitcoin implies some kind of network measurement
326 2012-06-03 06:30:30 <bd_> byzantine hosts may try to confound your estimates
327 2012-06-03 06:30:39 <bd_> eg, those 49% might only spend 1% of their effort for a while
328 2012-06-03 06:30:49 <bd_> and act as healthy nodes with that level of effort
329 2012-06-03 06:31:00 <bd_> when they split the network, they then expend all of their effort
330 2012-06-03 06:31:13 <amiller> sure, they might work invisibly
331 2012-06-03 06:31:34 <amiller> i mean i agree with you i think, that's another difficult point to resolve
332 2012-06-03 06:31:41 <amiller> i'm really just talking about a first step
333 2012-06-03 06:31:42 <bd_> I think it's a fundamental point, actually
334 2012-06-03 06:32:00 <bd_> it's one that bitcoin actually tries to estimate already, in fact
335 2012-06-03 06:32:19 <bd_> (in order to set the right level of POW)
336 2012-06-03 06:33:39 <amiller> i'll be happy if i can show that knowing the strength of the network is sufficient to do byzantine consensus in the a) synchronous and b) partial synchronous message delay scenarios
337 2012-06-03 06:33:46 <bd_> fair enough :)
338 2012-06-03 06:34:16 <amiller> but if it's possible to do it with even less than that, like if you don't know the strength of the network, that would be even better
339 2012-06-03 06:34:17 <amiller> hm
340 2012-06-03 06:34:35 <amiller> one thing i tried was if you only know the strength of the adversary but not the full size of the network
341 2012-06-03 06:34:49 <amiller> (if you know f but not N, in distributed systems)
342 2012-06-03 06:34:50 <bd_> well, then you know the size of the network
343 2012-06-03 06:35:01 <bd_> you can assume N >= 2f+epsilon
344 2012-06-03 06:35:13 <amiller> right but if you don't know N
345 2012-06-03 06:35:16 <bd_> otherwise the evil nodes control more than half of the network and you're doomed nayway
346 2012-06-03 06:35:19 <amiller> and you make a decision just based on f
347 2012-06-03 06:35:27 <amiller> then you're vulnerable to partitions
348 2012-06-03 06:35:53 <bd_> why's that?
349 2012-06-03 06:35:56 <amiller> knowing f but not N means your protocol can't distinguish between N=3f+1 and N=2f+!
350 2012-06-03 06:36:07 <amiller> or N=100f+1
351 2012-06-03 06:36:47 <amiller> but if you have a protocol that can come to a decision based on only knowing f, and it works in the 2f+1 case, then you could have dozens of those networks in the 100f+1 case, and they could all come to different decisions
352 2012-06-03 06:37:05 <amiller> maybe not if the message bound is known
353 2012-06-03 06:37:07 <bd_> oh, I see.
354 2012-06-03 06:37:26 <amiller> i think if you know the message bound then you don't need to know the size of the network
355 2012-06-03 06:37:32 <amiller> i must have only been worried about this for the partial synchrony case
356 2012-06-03 06:38:04 <bd_> ah, I see, the byzantine nodes could partition the network into multiple networks, each independently healthy but coming to different decisions, got you.
357 2012-06-03 06:38:26 <bd_> but yea, if you know the message bound, you just wait until that amount of time passes, and you know you've heard from all healthy nodes
358 2012-06-03 06:39:11 <bd_> so the network cannot be split
359 2012-06-03 06:41:21 <amiller> perhaps then the 6 blocks / 10 minutes then can be interpreted just as well as _either_ an assumption about message delays _or_ an assumption about the size of the network
360 2012-06-03 06:41:54 <bd_> message delays, really
361 2012-06-03 06:42:05 <bd_> with unbounded message delays, the byzantine partition will eventually reach 6 blocks
362 2012-06-03 06:42:14 <bd_> it just will take a little over twice as long as usual
363 2012-06-03 06:43:04 <bd_> actually, perhaps it's an assumption about both
364 2012-06-03 06:43:39 <bd_> you're assuming the total strength of the network is such that the byzantine nodes can't construct a 6-block chain in any amount of time shorter than the message delay bound
365 2012-06-03 06:43:44 <bd_> or something.
366 2012-06-03 06:44:35 <bd_> if your assumption about the message delay bound is wrong, you might be in a partition with the byzantine nodes, and they might be able to produce those 6 blocks before you can talk to another healthy node
367 2012-06-03 06:44:52 <amiller> i had to make a new thing to deal with this problem, when you don't know the message bound
368 2012-06-03 06:45:04 <bd_> if your assumption about network strength is wrong, the byzantine nodes again might be able to churn through 6 blocks really quickly
369 2012-06-03 06:45:08 <amiller> what i came up with is that you don't pick a particular difficulty threshold, i.e. 15 zeroes in front of your hashes
370 2012-06-03 06:45:26 <amiller> instead you sort of play the game for all numbers of zeroes
371 2012-06-03 06:45:36 <amiller> (if you get a perfect hash collision i guess you roll twice and so on)
372 2012-06-03 06:45:51 <amiller> no matter what the message delay bound is, eventually there is some event that happens infrequently enough
373 2012-06-03 06:46:08 <amiller> that the delayed 51% will pull ahead of the byzantine zero delay 49%
374 2012-06-03 06:46:24 <amiller> instead of making a decision based on a constant number of blocks,
375 2012-06-03 06:46:41 <bd_> not really
376 2012-06-03 06:46:50 <amiller> you make your decision based on the amount of time that's elapsed and the strength of the network
377 2012-06-03 06:47:10 <bd_> er, well, yes, you're making an assumption based on network strength again
378 2012-06-03 06:47:19 <amiller> but not on knowing the message bound
379 2012-06-03 06:47:45 <bd_> sure. this is like saying, "I'll accept this if I see it confirmed with 6 blocks in under 12 hours."
380 2012-06-03 06:48:14 <bd_> And if it takes more than 12 hours you assume there is a massive conspiracy and swear off bitcoin forever
381 2012-06-03 06:48:35 <bd_> (or increase the number of blocks you want to see)
382 2012-06-03 06:48:44 <amiller> right
383 2012-06-03 06:49:09 <bd_> problem is, difficulty estimates in bitcoin aren't fixed, so once you go beyond the point of a difficulty recalc you're screwed :)
384 2012-06-03 06:49:29 <amiller> well so that's how it can work if you know the network strength but not the message delay bound
385 2012-06-03 06:49:36 <bd_> indeed.
386 2012-06-03 06:50:19 <amiller> and if you know the message delay bound but not the network strength.... i forget what happened in that case
387 2012-06-03 06:50:47 <amiller> maybe that's where you know the size of the adversary but maybe not the total size of the network (you know f but not N)
388 2012-06-03 06:51:00 <bd_> it's not that hard if you know the message delay bound
389 2012-06-03 06:51:11 <bd_> and membership is fixed
390 2012-06-03 06:51:21 <bd_> first, give each node a cryptographic identity
391 2012-06-03 06:51:34 <bd_> now, in each round, all nodes vote on the decision, and send their votes to all other nodes
392 2012-06-03 06:51:44 <bd_> signed with their crypto identity
393 2012-06-03 06:51:59 <bd_> then they wait $messageDelayBound
394 2012-06-03 06:52:06 <bd_> and go with whatever got the most votes
395 2012-06-03 06:52:21 <bd_> assuming all healthy nodes vote the same way, this should work
396 2012-06-03 06:52:59 <bd_> ie, they have some kind of deterministic protocol to choose the winner... wait
397 2012-06-03 06:53:02 <bd_> nevermind ><
398 2012-06-03 06:53:34 <bd_> this is what I get for thinking up protocols on the fly at 1:53 am
399 2012-06-03 06:54:42 <amiller> well, i follow where you were going with that - it's easy if membership is fixed and message delay bound is known
400 2012-06-03 06:54:58 <bd_> fixed per-round
401 2012-06-03 06:55:02 <bd_> you can always vote in new members
402 2012-06-03 06:55:14 <bd_> it just needs to be part of the protocol
403 2012-06-03 06:55:23 <amiller> i can't figure out if it's worthwhile to look at where you know just the size of the adversary but not the total size of membership
404 2012-06-03 06:55:50 <amiller> like if f is part of the protocol but not N
405 2012-06-03 06:56:09 <bd_> how would f be part of the protocol?
406 2012-06-03 06:56:24 <bd_> I mean, realistically. Just an assumed upper bound?
407 2012-06-03 06:57:13 <amiller> well to be realistic i need to connect it back to the more fuzzy case with bitcoin in practice
408 2012-06-03 06:57:16 <amiller> the 6 blocks rule is negotiable
409 2012-06-03 06:57:29 <amiller> you're supposed to consider the cost of what you're doing
410 2012-06-03 06:58:07 <amiller> so i might say i know that it would cost more than $1000 to wind the network back more than 20 minutes
411 2012-06-03 06:59:21 <bd_> ah, indeed
412 2012-06-03 06:59:32 <amiller> that's analogous to an assessing f.
413 2012-06-03 08:44:08 <TD> good afternoon
414 2012-06-03 08:54:59 <molecular> hello
415 2012-06-03 09:06:07 <gribble> New news from bitcoinrss: Diapolo opened pull request 1412 on bitcoin/bitcoin <https://github.com/bitcoin/bitcoin/pull/1412>
416 2012-06-03 09:35:28 <grondilu> Has anyone ported Genjix's bitcoin.sql to MySQL?
417 2012-06-03 09:47:33 <gribble> New news from bitcoinrss: Diapolo opened pull request 1413 on bitcoin/bitcoin <https://github.com/bitcoin/bitcoin/pull/1413>
418 2012-06-03 10:16:16 <Tuxavant> Looking to hire a tutor to walk me thru building OpenSSL with EC, write a spec file, and get it into a personal or generic 3rd party repository. fedora centric. Just need someone to guide me and bounce questions off when I fail. $20 USD in Bitcoins for your trouble. Assistance via PM and mail.
419 2012-06-03 11:07:46 <BlueMatt> luke-jr: how much not-submitting-valid-blocks do you think you are seeing on eligius?
420 2012-06-03 11:07:59 <BlueMatt> s/think/estimate/
421 2012-06-03 12:19:20 <luke-jr> BlueMatt: impossible to detect, but hopefully none
422 2012-06-03 12:20:33 <luke-jr> BlueMatt: however, Bitcoin in general has seen a lot of attacks aimed at pools that don't benefit the attacker directly, so it seems inevitable
423 2012-06-03 12:20:57 <luke-jr> (or rather, there is a known way to detect it, but it's expensive an an arms race when implemented)
424 2012-06-03 12:23:25 <BlueMatt> luke-jr: I was just wondering if you had compared expected generation based on share count to actual generation
425 2012-06-03 12:23:36 <BlueMatt> and then compared to probability that your luck is at n
426 2012-06-03 12:24:00 <luke-jr> BlueMatt: blocks are not that predictable.
427 2012-06-03 12:24:20 <BlueMatt> no, but you can get a probability that there is nothing nefarious going on
428 2012-06-03 12:24:28 <luke-jr> not significantly
429 2012-06-03 12:24:41 <luke-jr> most of Eligius's runtime, we've been very "lucky"
430 2012-06-03 12:25:06 <luke-jr> at present, we're about 600 BTC "unlucky"
431 2012-06-03 12:25:18 <luke-jr> past "luck" peaked around 1200 BTC or so
432 2012-06-03 12:25:34 <luke-jr> there is no meaningful way to get statistics when it swings this far
433 2012-06-03 12:25:40 <BlueMatt> yea, afaik most pools tend to have pretty close to 100% "luck" over the long-run, so I have a hard time believing share withholding is an issue thats worth solving, really
434 2012-06-03 12:26:20 <BlueMatt> over a year or more, you would notice anything more than like 5% of shares withheld, I would think
435 2012-06-03 12:26:22 <BlueMatt> s/would/could
436 2012-06-03 12:26:24 <luke-jr> if it isn't, it will someday.
437 2012-06-03 12:26:52 <luke-jr> especially as centralized pooled mining phases out
438 2012-06-03 12:36:32 <TD> hmm
439 2012-06-03 12:36:39 <TD> luke-jr: p2pool seems to have stopped growing
440 2012-06-03 12:36:45 <TD> luke-jr: what makes you think centralized mining will phase out?
441 2012-06-03 12:37:05 <luke-jr> TD: it's inevitable IMO. and Eligius hasn't stopped growing.
442 2012-06-03 12:37:24 <luke-jr> p2pool is just one pool. hardly a good statistic :P
443 2012-06-03 12:37:28 <TD> heh
444 2012-06-03 12:37:53 <TD> i guess they have exhausted the supply of people willing to dick about with python programs. either that or their consistently <100% luck is a put-off
445 2012-06-03 12:37:54 <Eliel> luke-jr: looking at the hashrate graph, I find it hard to figure out how you come to that conclusion.
446 2012-06-03 12:38:34 <luke-jr> TD: possible their consistently <100% luck is the attack ;)
447 2012-06-03 12:38:42 <TD> huh
448 2012-06-03 12:38:55 <TD> somebody might be attacking p2pool? i guess so .... only people i can imagine who'd do that are centralized pool operators
449 2012-06-03 12:38:56 <luke-jr> Eliel: well, it was growing about 100 GH/s per month until that downtime thing
450 2012-06-03 12:38:59 <TD> everyone else stands to benefit
451 2012-06-03 12:39:46 <talpan> Hello. How can I get the bitcoin address of the sender, like satoshidice is doing it?
452 2012-06-03 12:40:09 <luke-jr> TD: dunno,  haven't thought it through
453 2012-06-03 12:40:12 <TD> ok
454 2012-06-03 12:40:52 <luke-jr> TD: there *are* non-obvious ways to profit from block withholding attacks, however
455 2012-06-03 12:41:01 <BlueMatt> TD/luke-jr: the discussion in #p2pool focuses on the huge orphan rates, which appears to be the source of much of the problem
456 2012-06-03 12:41:01 <TD> hmm
457 2012-06-03 12:41:02 <luke-jr> but PPLNS is immune to the one I know of in detail
458 2012-06-03 12:41:06 <TD> ok
459 2012-06-03 12:41:09 <TD> BlueMatt: thanks
460 2012-06-03 12:41:13 <luke-jr> BlueMatt: really? interesting
461 2012-06-03 12:41:16 <TD> BlueMatt: general perf issues then?
462 2012-06-03 12:41:21 <luke-jr> I'd expect lower orphan rates really
463 2012-06-03 12:41:22 <TD> talpan: satoshidice uses bitcoinj
464 2012-06-03 12:41:38 <BlueMatt> it appears that people are just running bitcoind on low-power systems which cant handle it/are poorly peered
465 2012-06-03 12:41:39 <TD> talpan: so they're just using the API that library provides. if you want to do it with bitcoind i think you have to wait for some more RPCs to be added?
466 2012-06-03 12:42:12 <luke-jr> TD: does BitcoinJ ignore transactions that don't meet bitcoind fee rules?
467 2012-06-03 12:42:31 <luke-jr> BlueMatt: so kinda an accidental form of block withholding?
468 2012-06-03 12:42:33 <BlueMatt> luke-jr: I dunno, I just dont see it as nearly worth hard-forking to avoid the withhold-share issue...unless some data comes up which shows that its likely actually occurring in a meaningful way, that is
469 2012-06-03 12:42:35 <talpan> Thank you, that's the reason I couldn't figure it out :) Thanks
470 2012-06-03 12:42:53 <TD> bitcoinj has no fee support at all right now :( users fork the library to add it. it's one of my next top priorities. at any rate, bitcoinj cannot know the fee of an arbitrary transaction without recursively downloading all the connected transactions.
471 2012-06-03 12:42:56 <TD> which it currently does not do
472 2012-06-03 12:42:58 <BlueMatt> luke-jr: no, just working on stale work, you could correlate the two, but...not really
473 2012-06-03 12:43:17 <luke-jr> BlueMatt: my point is that it's almost guaranteed to happen *someday* and therefore if we act now, we can leisurely schedule it for some time in the future
474 2012-06-03 12:43:29 <BlueMatt> I disagree
475 2012-06-03 12:43:35 <BlueMatt> its guaranteed to happen on some scale, sure
476 2012-06-03 12:43:46 <BlueMatt> but if its 1% of most pools...so what?
477 2012-06-03 12:43:47 <BlueMatt> anyway, gotta go
478 2012-06-03 13:22:50 <luke-jr> ironically, it seems the "sendrawtx" JSON-RPC command doesn't work?
479 2012-06-03 13:23:10 <wizkid057> ACK...
480 2012-06-03 13:24:13 <wizkid057> it seems to verify the transaction is valid, has valid inputs, etc, gives the txid back as a result... but it never hits the network
481 2012-06-03 13:28:29 <wladston> how bitcoind accounts works ? anyone if familiar with it ?
482 2012-06-03 13:28:59 <wladston> I wanted to know if it keeps track of confirmed and unconfirmed balances across internal transfers
483 2012-06-03 13:30:46 <luke-jr> sipa: where can I get seeds.txt that hasn't been filtered? -.-
484 2012-06-03 13:31:12 <luke-jr> sipa: & and why are you filtering the latest 0.4 and 0.5? :/
485 2012-06-03 13:35:11 <wladston> for example, if wallet A as 10 confirmed, and 5 unconfirmed, and it sends 11 to wallet B
486 2012-06-03 13:35:22 <wladston> how much confirmed and unconfirmed will wallet B have ?
487 2012-06-03 13:46:12 <gribble> New news from bitcoinrss: wizkid057 opened issue 1414 on bitcoin/bitcoin <https://github.com/bitcoin/bitcoin/issues/1414>
488 2012-06-03 14:00:17 <wladston> from my tests here, it looks like transfer between internal wallets will not respect the confirmed/unconfirmed status from the money being transfered
489 2012-06-03 14:01:18 <wladston> for example, I have 1 confirmed and 1 unconfirmed at account w1, and I issue a move of 2 from w1 to w2.
490 2012-06-03 14:01:43 <wladston> w2's balance will show 2 confirmed, with is just plain wrong
491 2012-06-03 14:02:58 <wladston> should I fill a bug ?
492 2012-06-03 14:04:19 <wizkid057> not sure I understand what you've been getting at...
493 2012-06-03 14:04:25 <wizkid057> you cant "send" confirmations...
494 2012-06-03 14:12:47 <wladston> wizkid057: You can't send confirmations, but the money you send from an account is either confirmed or not, right ?
495 2012-06-03 14:13:39 <wizkid057> once coins are sent, the transaction must be confirmed, regardless of whether the funds you're sending are confirmed already or not
496 2012-06-03 14:15:54 <wladston> wizkid057: I'm refering to internal transactions, between internal bitcoind accounts.
497 2012-06-03 14:16:02 <wizkid057> doesnt matter
498 2012-06-03 14:16:22 <wizkid057> the network still needs to confirm you moved the funds between accounts
499 2012-06-03 14:16:27 <wizkid057> thus, resetting to 0
500 2012-06-03 14:16:56 <wladston> wizkid057: it doesn't. the internal accounts moves are not brodcasted to the bitcoin network.
501 2012-06-03 14:17:20 <wladston> wizkid057: see here https://en.bitcoin.it/wiki/Accounts_explained
502 2012-06-03 14:17:56 <sipa> wizkid057: transfers between acounts are just adjusting local balances
503 2012-06-03 14:18:27 <wizkid057> oh, we're talking about accounts, not addresses
504 2012-06-03 14:18:30 <sipa> luke-jr: small bug, i lost my database, so i had to reset
505 2012-06-03 14:18:39 <luke-jr> sipa: oh :<
506 2012-06-03 14:19:05 <luke-jr> sipa: it's sending replies from the wrong source IP for me :/
507 2012-06-03 14:19:15 <wladston> guys, can you see the bug I'm taking about ?
508 2012-06-03 14:19:34 <sipa> luke-jr: how so?
509 2012-06-03 14:19:39 <wladston> I'm currently baking a paper describing a fix for it.
510 2012-06-03 14:20:36 <wladston> I took about 4 days thinking about the problem every day to come up with the solution
511 2012-06-03 14:21:01 <luke-jr> sipa: it's using the first IPv4 address on the interface, rather than the one that received the request
512 2012-06-03 14:21:40 <luke-jr> wladston: accounts don't have coins, just balances.
513 2012-06-03 14:22:12 <wladston> luke-jr: but it does have the confirmation status for the balances
514 2012-06-03 14:22:24 <luke-jr> wladston: confirmations apply to coins, not balances
515 2012-06-03 14:22:52 <wladston> luke-jr: so why the balance method has a minimum confirmation parameter ?
516 2012-06-03 14:23:02 <luke-jr> no idea
517 2012-06-03 14:23:10 <sipa> wladston: because it allows you to filter coins sent to it
518 2012-06-03 14:23:11 <wladston> that is a bug, then
519 2012-06-03 14:23:27 <sipa> this does not make much sense when you also have moves active
520 2012-06-03 14:23:35 <wladston> sipa: I did not send any coins to the wallet address
521 2012-06-03 14:23:45 <wladston> sipa: but the balance is still positive
522 2012-06-03 14:23:50 <wladston> from my move
523 2012-06-03 14:24:03 <sipa> moves are just always counted
524 2012-06-03 14:24:11 <wladston> sipa: why ?
525 2012-06-03 14:24:21 <sipa> how can you not count them?
526 2012-06-03 14:24:38 <wladston> sipa:  you can correctly track their confirmation status and count that
527 2012-06-03 14:24:50 <sipa> moves.are always infinitely confirmed
528 2012-06-03 14:24:54 <luke-jr> wladston: moves don't have confirmations in any sense of the word
529 2012-06-03 14:25:18 <luke-jr> all it does is set AccountFrom -= amount, AccountTo += amount
530 2012-06-03 14:25:23 <luke-jr> you can even make accounts negative
531 2012-06-03 14:25:24 <sipa> it's just decreasing the balance of one account, and increasing that of another
532 2012-06-03 14:25:35 <wladston> do you think this is a good design choice, to have moves always count as confirmed balance even when it comes from an unconfirmed source ?
533 2012-06-03 14:25:48 <luke-jr> wladston: it doesn't "come from" anywhere
534 2012-06-03 14:25:56 <sipa> moves do not come from any source, except the account
535 2012-06-03 14:26:09 <sipa> there are absolutrly no coins involved in a move
536 2012-06-03 14:26:43 <luke-jr> sipa: maybe we should add "changebalance" to assign/increment/decrement accounts without any interaction with other accounts ;)
537 2012-06-03 14:26:45 <luke-jr> just to stress this
538 2012-06-03 14:26:51 <wladston> I get what you are saying. It's just irrelevant for you to see if a move involves real confirmed coins or not
539 2012-06-03 14:27:01 <luke-jr> wladston: no
540 2012-06-03 14:27:06 <luke-jr> there are NO COINS INVOLVED
541 2012-06-03 14:27:17 <sipa> as i said, combining moves with confirmations selections leads to meaningless results, but it is well-defined
542 2012-06-03 14:27:22 <luke-jr> you can have an empty wallet and still move 100 BTC from one account to another
543 2012-06-03 14:27:31 <luke-jr> the from account will then have a negative balance
544 2012-06-03 14:28:02 <wladston> I understand. I was in need of a way to move REAL COINS within internal accounts
545 2012-06-03 14:28:16 <wizkid057> wladston: sendtoaddress ?
546 2012-06-03 14:28:27 <luke-jr> wladston: that could get expensive.
547 2012-06-03 14:28:30 <wladston> without broadcasting the transaction
548 2012-06-03 14:28:34 <wizkid057> sipa: I notice you ACK'd the sendrawtx command pull req... however, it doesnt broadcast no matter what I do. did you have a different experience?
549 2012-06-03 14:28:35 <wladston> just internally
550 2012-06-03 14:28:42 <luke-jr> oh
551 2012-06-03 14:28:45 <luke-jr> I think I see what you mean
552 2012-06-03 14:28:49 <sipa> as are in no way tied to an account, tht doesn't make sense
553 2012-06-03 14:29:02 <luke-jr> sipa: they could be
554 2012-06-03 14:29:17 <luke-jr> he basically wants a completely new account design
555 2012-06-03 14:29:19 <sipa> yes, incoming coins are, actually
556 2012-06-03 14:29:30 <sipa> i think he wants multiwallet support
557 2012-06-03 14:29:39 <luke-jr> sipa: you can't move individual coins between wallets
558 2012-06-03 14:29:40 <sipa> instead of accounts within a wallet
559 2012-06-03 14:29:40 <wladston> luke-jr: I did this design internally in my application
560 2012-06-03 14:30:10 <sipa> luke-jr: it requires a bitcoin transaction of course
561 2012-06-03 14:30:13 <luke-jr> sipa: no
562 2012-06-03 14:30:22 <sipa> right, i see, there is another design possible
563 2012-06-03 14:30:30 <luke-jr> it might, if he has a single 5 BTC coin and wants to move only 2.5 BTC to another account
564 2012-06-03 14:30:40 <sipa> where coins remain tied to the account they were received with
565 2012-06-03 14:31:00 <wladston> sipa: EXACTLY
566 2012-06-03 14:31:11 <wladston> sipa: that's what I did here
567 2012-06-03 14:31:13 <luke-jr> wladston: I think it'd be ugly for users
568 2012-06-03 14:31:22 <wladston> the interface would be the same
569 2012-06-03 14:31:25 <wizkid057> seems like it'd be an over-complication of the system
570 2012-06-03 14:31:27 <wladston> same api calls
571 2012-06-03 14:31:30 <sipa> accounts are already very hard to grasp
572 2012-06-03 14:31:32 <luke-jr> "why do I need a transaction to move 5 BTC, but not to move 5.4933344 BTCG?"
573 2012-06-03 14:32:14 <luke-jr> actually
574 2012-06-03 14:32:39 <luke-jr> I suppose a sufficiently complex implementation could flag a coin as "partly account A, and partly account B" and then use it in the next transaction to split it
575 2012-06-03 14:32:50 <wladston> my problem is the following : I want to accept zero confirmation coins, and let users play with then, sending then to each other (internally). But I only want to let anyone withdraw coins if the source was confirmed.
576 2012-06-03 14:32:52 <wizkid057> ...
577 2012-06-03 14:33:23 <sipa> wizkid057: to be honest, i only looked at the code and compiled it
578 2012-06-03 14:33:26 <wladston> luke-jr: yes! but it's actually no that complex to implement
579 2012-06-03 14:33:59 <sipa> wizkid057: i suppose it just broadcasts once, as it doesn't get associated with your wallet, it isn't rebroadcasted
580 2012-06-03 14:34:13 <wladston> so my users would know if the coins they are playing with are confirmed or not.
581 2012-06-03 14:34:18 <sipa> Ukto: what do you need me for?
582 2012-06-03 14:36:04 <wizkid057> sipa: well, to test before I posted my issue, i connected bitcoind to only blockchain.info, sent a raw tx, didnt appear on there... second, connected two local bitcoinds together using addnode, sent coins to an address known to one from the other using sendrawtx and it never showed, either... so, i'm at a loss
583 2012-06-03 14:36:07 <luke-jr> wladston: do you have a patch?
584 2012-06-03 14:36:48 <wladston> luke-jr: I developed that at the application level, using raw address calls.
585 2012-06-03 14:37:19 <wladston> luke-jr: my design could easily be ported inside bitcoind. the thing is, I never even looked at bitcoind source :)
586 2012-06-03 14:37:34 <wizkid057> i admittedly do not believe i've studied the bitcoin client sufficiently enough to fix the issue myself
587 2012-06-03 14:37:57 <wladston> luke-jr: the technique a bit hard to grasp, so I'm writing the scientific-looking paper first
588 2012-06-03 14:40:34 <wizkid057> oddly, the TX "is" in the local "getmemorypool" for that client
589 2012-06-03 14:40:41 <wizkid057> but not the receiving client
590 2012-06-03 14:42:26 <Ukto> sipa: is adding 1~3 seconds to report work from a miner to a pool worthy of being the excuse of rejects?
591 2012-06-03 14:42:46 <Ukto> esp. with LP enabled
592 2012-06-03 14:51:58 <sipa> wizkid057: i'll have a look at it later
593 2012-06-03 14:52:19 <wizkid057> np, thx
594 2012-06-03 14:52:21 <sipa> Ukto: hmm, there are a lot of people here who know more about mining
595 2012-06-03 14:52:45 <BlueMatt> wizkid057: are you sure you arent making an orphan?
596 2012-06-03 14:52:51 <BlueMatt> wizkid057: what does debug.log say?
597 2012-06-03 14:54:24 <sipa> if it is in his mempool, it is not an orphan
598 2012-06-03 14:54:41 <BlueMatt> oh, didnt read that, sorry
599 2012-06-03 14:55:11 <sipa> and if it is in his mempool, it should certainly be relayed
600 2012-06-03 14:55:28 <BlueMatt> mempool doesnt relay, but there is a RelayInventory call in sendrawtx
601 2012-06-03 14:56:01 <wizkid057> guess i should recompile with debugging :P
602 2012-06-03 14:56:04 <BlueMatt> anyway, also, check chub
603 2012-06-03 14:59:58 <MrTiggr> *cough* ..... longtime watcher first-time poster ...
604 2012-06-03 15:00:03 <MrTiggr> mebbe u guys can help me .......
605 2012-06-03 15:00:04 <MrTiggr> <MrTiggr> to stop exfiltration of wallets from servers
606 2012-06-03 15:00:06 <MrTiggr> <MrTiggr> see fingerprint
607 2012-06-03 15:00:08 <MrTiggr> <MrTiggr> STOP
608 2012-06-03 15:00:10 <MrTiggr> <MrTiggr> HALT
609 2012-06-03 15:00:12 <MrTiggr> <MrTiggr> DIE
610 2012-06-03 15:06:24 <Tuxavant> MrTiggr, it's a noble cause, but all it takes to circumvent is encoding the file before exfilration
611 2012-06-03 15:07:12 <Tuxavant> if I was exfiltrating any data, I'd be XORing it before hand and I'm not even that smart
612 2012-06-03 15:09:57 <MrTiggr> of, i would be too
613 2012-06-03 15:10:15 <MrTiggr> but you can make common base64/xor patter matches if you have a fingerprint
614 2012-06-03 15:10:28 <MrTiggr> i know,i know ... they cud just tomwilliams the keys out too
615 2012-06-03 15:10:32 <MrTiggr> and thats harder still
616 2012-06-03 15:10:59 <MrTiggr> but - for noobie/mybitcoinica operators, a set of snorts wud be a good sart no ?
617 2012-06-03 15:12:14 <MrTiggr> (pardon my fingrfail tonight .. its late)
618 2012-06-03 15:13:00 <luke-jr> sipa: https://github.com/sipa/bitcoin-seeder/pulls
619 2012-06-03 15:17:34 <Tuxavant> MrTiggr, I think it's worth doing if for nothing but the experience, but I'm just not a fan of signature based systems so I dismiss it easily... But dont let me discourage you from giving it a shot. It's a berkly DB so you might find quite a lot of collisions unless you dig deep into the bitcoin specific tables
620 2012-06-03 15:21:38 <luke-jr> MrTiggr: nevermind that none of the cracking has stolen wallet.dat files&
621 2012-06-03 15:22:25 <luke-jr> sipa: OK, I just switched my DNS seed from a CNAME to jgarzik to an actual bitcoin-seeder instance
622 2012-06-03 15:24:16 <MrTiggr> OK, thanks guys ... what about using snort rules to detect the exfiltration of your own keys then ?   id NEVER suggest including your keys in a sort rule .. but .. what about PART of it ? .... luke-jr Tuxavant ?
623 2012-06-03 15:24:47 <luke-jr> MrTiggr: all the crackers have used bitcoind to send a transaction to an address they control
624 2012-06-03 15:25:13 <luke-jr> they don't care about private keys
625 2012-06-03 15:25:18 <MrTiggr> luke-jr: so it will contain your own source id
626 2012-06-03 15:25:28 <BlueMatt> luke-jr: sure they do, there has been malware that steals all of wallet.dat
627 2012-06-03 15:25:40 <BlueMatt> luke-jr: I havent actually seen malware that steals coins directly
628 2012-06-03 15:25:40 <luke-jr> BlueMatt: I'm not talking about virii
629 2012-06-03 15:25:41 <MrTiggr> snort rules outbound to detect use of your wallet id ?
630 2012-06-03 15:25:48 <MrTiggr> im just brainstorming here
631 2012-06-03 15:25:54 <MrTiggr> sick of mopping up the mess
632 2012-06-03 15:25:58 <MrTiggr> wud rather find a prevention
633 2012-06-03 15:26:05 <luke-jr> BlueMatt: only big merchant/exchange/etc sites would be using firewalls of this calibur
634 2012-06-03 15:26:06 <BlueMatt> MrTiggr: sadly wallets are pretty commonly-used fingerprints, they are a bdb databse
635 2012-06-03 15:26:20 <BlueMatt> MrTiggr: you might try fingerprinting private keys
636 2012-06-03 15:26:21 <MrTiggr> yer .. but theres gotta be some set of unique rules
637 2012-06-03 15:26:28 <MrTiggr> hrrm
638 2012-06-03 15:26:50 <BlueMatt> oh, wait, do we store CSecrets now, or private keys still?
639 2012-06-03 15:27:16 <BlueMatt> luke-jr: no, they should be storing their wallet offline, so no firewall is required...
640 2012-06-03 15:28:04 <luke-jr> BlueMatt: exchanges at least need online wallets
641 2012-06-03 15:28:30 <luke-jr> MrTiggr: besides, they should be using wallet encryption
642 2012-06-03 15:29:20 <MrTiggr> oh, i KNOW THEY should be
643 2012-06-03 15:29:23 <MrTiggr> but srs
644 2012-06-03 15:29:31 <MrTiggr> we know this is the wild-west luke-jr
645 2012-06-03 15:29:35 <MrTiggr> ;)
646 2012-06-03 15:30:09 <BlueMatt> luke-jr: no they dont
647 2012-06-03 15:30:28 <luke-jr> MrTiggr: someone who doesn't, won't run a firewall rule either
648 2012-06-03 15:30:30 <BlueMatt> luke-jr: they can sneakernet withdraws
649 2012-06-03 15:30:37 <luke-jr> BlueMatt: LOL
650 2012-06-03 15:30:46 <BlueMatt> I said can, not should
651 2012-06-03 15:30:56 <BlueMatt> though I would argue for large withdraws, they probably should
652 2012-06-03 15:31:51 <MrTiggr> guys, dont shoot the messenger ... like i sed, im sick of mopping up the mess, wondering if thres a way to stop it in a preventative way IDS/IPS style
653 2012-06-03 15:32:01 <MrTiggr> all good points
654 2012-06-03 15:33:06 <BlueMatt> anyway, the point is, its quite hard, probably impossible to do even a good job stopping it ids style, doing it properly to begin with isnt /that/ hard, and saves all the headache later on
655 2012-06-03 15:33:46 <BlueMatt> sadly, no one seems to understand that just having one online wallet and one offline one with less cash isnt enough to do it "properly"
656 2012-06-03 15:33:54 <MrTiggr> kk, thnx ... better not waste my bandwidth there .. is the general response ... D:
657 2012-06-03 15:33:57 <BlueMatt> atleast not several major people who whould
658 2012-06-03 15:33:59 <MrTiggr> BlueMatt: i know :(
659 2012-06-03 15:34:10 <BlueMatt> I do think its a cool idea
660 2012-06-03 15:34:13 <MrTiggr> you can lead a horse to water... but you cant make him enjoiy the view
661 2012-06-03 15:34:23 <MrTiggr> ;)
662 2012-06-03 15:34:27 <BlueMatt> yep
663 2012-06-03 15:52:48 <Matt_von_Mises> For transaction output values, how is the sign represented? The serialisation code goes over my head completely.
664 2012-06-03 16:41:57 <luke-jr> sipa: ping
665 2012-06-03 16:53:41 <luke-jr> sipa: bayleef's client is stuck on block download, and his debug.log suggests he isn't even TRYING to get more: http://failbox.co.cc/debug.log
666 2012-06-03 16:54:03 <luke-jr> hmm
667 2012-06-03 16:54:17 <luke-jr> bayleef: I presume you tried restarting your client?
668 2012-06-03 16:55:04 <luke-jr> bayleef: if not, try that first; if so, try https://bitcointalk.org/?topic=82610
669 2012-06-03 16:56:54 <gmaxwell> luke-jr: how long has he been stuck?
670 2012-06-03 16:57:16 <gmaxwell> luke-jr: you'll stop pulling until the next block if the prior peer you were pulling from goes away.
671 2012-06-03 16:57:40 <luke-jr> "It also hasn't downloaded any blocks for several days."
672 2012-06-03 16:58:01 <gmaxwell> oh.
673 2012-06-03 16:58:34 <luke-jr> [18:22:08] <bayleef> So I'm running bitcoind 0.6.2, and it says that either me or someone else needs to upgrade. It also hasn't downloaded any blocks for several days. Any idea what's going on?
674 2012-06-03 16:58:36 <luke-jr> in #bitcoin
675 2012-06-03 16:59:01 <luke-jr> gmaxwell: BTW, I PM'd you
676 2012-06-03 17:08:40 <bayleef> luke-jr: Restarting just gave me the same results, building the test bitcoind now
677 2012-06-03 17:17:10 <BlueMatt> bayleef: have you tried running with -checkblocks ?
678 2012-06-03 17:18:00 <luke-jr> BlueMatt: that makes sense if the blocks are being rejected
679 2012-06-03 17:18:04 <luke-jr> but they're not even downloading
680 2012-06-03 17:19:57 <BlueMatt> luke-jr: Im not sure about the previous run, but the one start of bitcoin I see in that debug.log looks like a bad peer
681 2012-06-03 17:20:15 <BlueMatt> afaict it never responds to "askfor block 00000000000006da5742   0"
682 2012-06-03 17:21:56 <BlueMatt> in fact, that pattern appears before the restart...
683 2012-06-03 17:21:58 <BlueMatt> hmmm...
684 2012-06-03 17:27:26 <BlueMatt> on a related note, can we start logging getblocks the same way we do askfor ?
685 2012-06-03 17:44:47 <BlueMatt> bayleef: can you run with -debug for a minute or two and post the end of debug.log?
686 2012-06-03 17:49:11 <bayleef> sorry for the slow response... I ran the new bitcoind, it started upgrading my database, and everything went all unresponsive and stuff. Bitcoind seems to have died
687 2012-06-03 17:50:24 <BlueMatt> yea next-test can be a bit...unstable
688 2012-06-03 17:51:49 <bayleef> So what now? Should I try again? From looking at the debug log it seems to have been in, as GitHub would say, hardcore upgrading action
689 2012-06-03 17:52:09 <BlueMatt> can you try again with 0.6.2 with -debug
690 2012-06-03 17:52:22 <bayleef> kk
691 2012-06-03 17:53:30 <BlueMatt> luke-jr: why do you still list spesmilo in your sig...whens the last time it was updated?
692 2012-06-03 18:05:04 <bayleef> have an 11mb debug.log http://failbox.co.cc/debug.log.lzma
693 2012-06-03 18:06:49 <BlueMatt> thanks
694 2012-06-03 18:07:10 <luke-jr> ouch, forgot next-test had jgarzik's blockchain upgrade stuff
695 2012-06-03 18:07:36 <luke-jr> BlueMatt: you have binaries of master, right?
696 2012-06-03 18:08:00 <luke-jr> BlueMatt: dunno, sigs can't be updated without losing images anymore :/
697 2012-06-03 18:08:23 <BlueMatt> luke-jr: master bins are at http://jenkins.bluematt.me/job/Bitcoin/ws/
698 2012-06-03 18:08:46 <luke-jr> bayleef: all the same warnings as next-test apply, but try BlueMatt's build there
699 2012-06-03 18:09:10 <BlueMatt> do we not print args in debug.log?
700 2012-06-03 18:09:24 <BlueMatt> we should
701 2012-06-03 18:10:08 <luke-jr> inclduing those from config file
702 2012-06-03 18:10:18 <BlueMatt> yea
703 2012-06-03 18:10:31 <BlueMatt> well minus rpcpassword, rpcuser, etc
704 2012-06-03 18:11:48 <BlueMatt> luke-jr: that said, making him upgrade his db did result in an interesting debug.log..."ProcessBlock: ORPHAN BLOCK, prev=00000000000000000000"
705 2012-06-03 18:12:02 <BlueMatt> genisis is considered orphan...
706 2012-06-03 18:13:04 <luke-jr> BlueMatt: that was a bug in jgarzik's branch :p
707 2012-06-03 18:13:20 <luke-jr> actually
708 2012-06-03 18:13:29 <luke-jr> the build I linked shouldn't have had it&
709 2012-06-03 18:13:37 <BlueMatt> oh...
710 2012-06-03 18:14:31 <Nachtwind> hi.. is there any known problem with windows qt bitcoin and sendmany? It just doesnt seem to work. Downloaded the most recent binary
711 2012-06-03 18:15:15 <luke-jr> confirmed: it's not in 20120519
712 2012-06-03 18:15:26 <luke-jr> bayleef: what did you download exactly? O.o
713 2012-06-03 18:15:43 <luke-jr> Nachtwind: be more specific on "doesnt seem to work"
714 2012-06-03 18:16:24 <Nachtwind> trying to use sendmany via php using the normal rpcclient library and all i get is 500 Server error with the same code people on linux can use
715 2012-06-03 18:17:24 <meLon> I'm having a hell of a time running bitcoind as cli daemon under a different user using this init.d script: https://bitcointalk.org/?topic=965.0b  Even after changing the CHUID settings, the script attempts to find bitcoind.conf in *my* directory vs the bitcoin user, so I am assuming the CHUID setting isn't working/it's running as the wrong user.  Any suggestions?
716 2012-06-03 18:17:28 <luke-jr> Nachtwind: what's the command you're using?
717 2012-06-03 18:17:55 <luke-jr> meLon: init.d scripts need to be run as root
718 2012-06-03 18:18:02 <bayleef> Oops... Cloned and built this git://gitorious.org/~Luke-Jr/bitcoin/luke-jr-bitcoin.git
719 2012-06-03 18:18:29 <Nachtwind> $cBitcoin -> sendmany("", $to); where array is just $to[address] = floatval(amount)
720 2012-06-03 18:18:34 <luke-jr> bayleef: oh, you're compiling? just build git master then
721 2012-06-03 18:18:42 <meLon> That's why I'm really confused luke-jr.  I'm running sudo /etc/init.d/bitcoind restart, but it's looking in *my* home dir for the conf file >_<
722 2012-06-03 18:18:49 <BlueMatt> meLon: try sudo -H
723 2012-06-03 18:19:14 <meLon> Thanks BlueMatt, now it's using root's home directory >_<
724 2012-06-03 18:19:20 <luke-jr> lol
725 2012-06-03 18:19:32 <luke-jr> that means it's runnign as root
726 2012-06-03 18:19:33 <meLon> I've created the user 'bitcoin' who I would like to run bitcoind, instead of my user
727 2012-06-03 18:19:41 <BlueMatt> dont run it as root...
728 2012-06-03 18:20:07 <meLon> CHUID=bitcoin:bitcoin
729 2012-06-03 18:20:18 <BlueMatt> is it running as user bitcoin:bitcoin?
730 2012-06-03 18:20:49 <meLon> I am unable to determine that 100% BlueMatt, but I believe it is not, since it's looking in other home directories for the .conf file
731 2012-06-03 18:21:01 <tcatm_> What initscript are you using?
732 2012-06-03 18:21:08 <BlueMatt> home directory != user its running as (as sudo -H just proved ;) )
733 2012-06-03 18:21:11 <luke-jr> [20:17:00] <meLon> I'm having a hell of a time running bitcoind as cli daemon under a different user using this init.d script: https://bitcointalk.org/?topic=965.0b  Even after changing the CHUID settings, the script attempts to find bitcoind.conf in *my* directory vs the bitcoin user, so I am assuming the CHUID setting isn't working/it's running as the wrong user.  Any suggestions?
734 2012-06-03 18:22:56 <meLon> http://pastie.org/private/xafcb9fqpepmgiynkrtr1q
735 2012-06-03 18:23:36 <BlueMatt> meLon: can you change the initscript to just run like id or whoami?
736 2012-06-03 18:23:37 <talpan> don't run bitcoin as root
737 2012-06-03 18:23:50 <meLon> Im not trying to talpan...
738 2012-06-03 18:23:55 <tcatm_> meLon: try prefixing start-stop-daemon with HOME=/home/bitcoin/ (or wherever it should look)
739 2012-06-03 18:23:58 <BlueMatt> actually, nvm you arent running as root
740 2012-06-03 18:24:08 <BlueMatt> or it wouldnt get Permission denied
741 2012-06-03 18:24:42 <talpan> ah sorry, it is in .root: permission denied, you don't have the right do read/write .root
742 2012-06-03 18:24:51 <meLon> BlueMatt: I did 'echo $NAME' and 'sudo /etc...' reported *my* name whereas 'sudo -H ...' reported *root*
743 2012-06-03 18:24:52 <tcatm_> start-stop-daemon doesn't change enviroment settings so $HOME is likely still /root when running the binary
744 2012-06-03 18:24:54 <talpan> *root
745 2012-06-03 18:25:40 <tcatm_> Or better yet: Specify the exact location of the datadir
746 2012-06-03 18:25:51 <meLon> If I knew how to do that....
747 2012-06-03 18:26:07 <tcatm_> (and use SELinux to restrict access to wallet.dat to the bitcoind started by init!)
748 2012-06-03 18:26:29 <meLon> Is the post I linked not the most up-to-date/recommended init script?
749 2012-06-03 18:26:30 <talpan> bitcoind -datadir=/what/ever
750 2012-06-03 18:26:58 <talpan> or something similiar, but you have to use -datadir
751 2012-06-03 18:28:27 <meLon> DAEMON=/usr/bin/$name
752 2012-06-03 18:28:40 <meLon> DAEMON_ARGS=-daemon -datadir=/home/bitcoin/.bitcoin
753 2012-06-03 18:30:19 <meLon> Not working :\n3659267
754 2012-06-03 18:30:31 <BlueMatt> bayleef: do you mind running bitcoind with -detachdb, then quitting (just so it finishes loading and then quits, dont need more) and then lzmaing your .bitcoin/blk* and uploading it somewhere?
755 2012-06-03 18:31:09 <luke-jr> BlueMatt: seems to me the obvious thing to do first is him try git master to see if sipa's fix helps
756 2012-06-03 18:31:18 <BlueMatt> which one?
757 2012-06-03 18:31:32 <BlueMatt> sorry, I wanst aware there was a related fix, try master first
758 2012-06-03 18:31:34 <luke-jr> 385f730 (sipa/unstuck, origin-pull/1315/head) Hopefully final fix for the stuck blockchain issue
759 2012-06-03 18:31:43 <BlueMatt> oh, I thought that was 0.6.2
760 2012-06-03 18:31:56 <luke-jr> no, 0.6.2 was just addrman
761 2012-06-03 18:32:11 <BlueMatt> bayleef: nevermind, can you try downloading bitcoind/-qt[.exe] from http://jenkins.bluematt.me/job/Bitcoin/ws/ and trying that?
762 2012-06-03 18:32:16 <luke-jr> at that time, the unstuck didn't work, so it got deferred
763 2012-06-03 18:32:36 <BlueMatt> bayleef: if that doesnt get you unstuck, would you mind running with -detachdb
764 2012-06-03 18:32:44 <BlueMatt> and uploading
765 2012-06-03 18:37:44 <wizkid057> not sure if its related, but, I had an issue with bitcoind downloading blocks when getting my raspberry pi version compiled
766 2012-06-03 18:37:53 <wizkid057> was because of a problem with my libdb
767 2012-06-03 18:39:10 <meLon> Sweet luke-jr BlueMatt.  I was able to get it to work, and now I just have to figure out this boost permission issue :D  Thanks
768 2012-06-03 18:39:24 <meLon> EXCEPTION: N5boost12interprocess22interprocess_exceptionE :P
769 2012-06-03 18:44:09 <tcatm_> What's the current state of multiple wallets support in bitcoin-qt?
770 2012-06-03 18:44:37 <talpan> "none", you to rename each wallet and restart bitcoin
771 2012-06-03 18:44:55 <talpan> 1. shut down bitcoin, 2. rename wallet, 3. start bitcoin
772 2012-06-03 18:46:02 <meLon> You guys ever seen this? http://pastie.org/4021380
773 2012-06-03 18:47:02 <talpan> Permission denied? mhm
774 2012-06-03 18:47:22 <talpan> check the log, may be there is some more information
775 2012-06-03 18:47:45 <meLon> /var/log/bit* doesn't exists :\n3659461
776 2012-06-03 18:47:50 <meLon> rgr
777 2012-06-03 18:49:49 <meLon> I dont see it anywhere :.  Not in my or bitcoin ~/.bitcoin
778 2012-06-03 18:49:55 <meLon> nor*
779 2012-06-03 18:50:07 <talpan> is .bitcoin writeable=
780 2012-06-03 18:50:13 <talpan> ? by the user who runs bitcoind
781 2012-06-03 18:51:29 <meLon> lmao
782 2012-06-03 18:51:49 <meLon> <3 talpan I'd untar'd the .bitcoin directory and forgotten to change the permissions of the files
783 2012-06-03 18:52:01 <talpan> ;) you're welcome
784 2012-06-03 18:52:29 <bayleef> Should I just upload debug.log? Or something else too?
785 2012-06-03 18:52:52 <meLon> This is really weird.  I had to change /home/bitcoin/.bitcoin permissions to get it to run, but now it's telling me to put bitcoin.conf in /home/*my* wth
786 2012-06-03 18:53:16 <talpan> debug.log should be enough
787 2012-06-03 18:53:24 <luke-jr> bayleef: try git master first
788 2012-06-03 18:53:27 <talpan> are you using a script to start bitcoind?
789 2012-06-03 18:53:29 <tcatm_> I wrote a small patch yesterday that adds a -wallet=<file> option. Maybe someone has use for it :) https://github.com/tcatm/bitcoin/commit/3d9db0e416e5538f5442870c05f1cd03f3c072dd
790 2012-06-03 18:53:46 <luke-jr> tcatm_: that could be dangerous I think
791 2012-06-03 18:53:58 <bayleef> luke-jr: This is with master, actually
792 2012-06-03 18:54:19 <luke-jr> bayleef: it's still stuck?
793 2012-06-03 18:54:24 <tcatm_> luke-jr: Technically it should be safe.
794 2012-06-03 18:54:44 <luke-jr> tcatm_: what if the normal wallet.dat didn't detach (eg, a crash) and you use -wallet with a different one?
795 2012-06-03 18:56:28 <bayleef> luke-jr: yep, still on block 181808
796 2012-06-03 18:57:00 <tcatm_> luke-jr: Not sure. I thought bdb might use the filename within its log files?
797 2012-06-03 18:57:05 <talpan> did you download the latest blockchain and just put it in the bitcoin directory? had this problem once
798 2012-06-03 18:57:13 <luke-jr> bayleef: OK, run with -detachdb like BlueMatt said, start & shutdown (takes a while), then upload blkindex somewhere
799 2012-06-03 18:57:24 <meLon> Okay guys.  sudo su bitcoin - ; bitcoind getinfo works.    bitcoind getinfo as *my* user fails and tells me I need to have a .config file.  How can I work around this?
800 2012-06-03 18:57:37 <talpan> http://eu1.bitcoincharts.com/blockchain/blockchain-2012-06-03.tar
801 2012-06-03 18:57:46 <luke-jr> meLon: create the config file&
802 2012-06-03 18:57:58 <luke-jr> talpan: don't do that.
803 2012-06-03 18:58:03 <talpan> do you have a config file in /home/*user*/.bitcoin/bitcoin.conf?
804 2012-06-03 18:58:04 <talpan> why not?
805 2012-06-03 18:58:09 <luke-jr> talpan: it's not secure
806 2012-06-03 18:58:30 <meLon> No I do not, because I didn't want ot cause complications and stuff and wanted to be 100% sure that the daemon was loading the correct config
807 2012-06-03 18:58:39 <talpan> in which way? in a way that the source could be wrong or that is causes a crash?
808 2012-06-03 18:58:58 <luke-jr> talpan: in a way that you could be exploited
809 2012-06-03 18:59:08 <talpan> meLon: you have to, copy the current bitcoin.conf to the right directory
810 2012-06-03 18:59:13 <tcatm_> I just verified that bdb is actually using the filename so if wallets are located within $datadir everything should be safe.
811 2012-06-03 18:59:21 <luke-jr> bitcoind just trusts blk000x is correct, and won't check it unless you manually tell it to
812 2012-06-03 18:59:51 <luke-jr> tcatm_: common use case I see, is wallet.dat from another directory - will that work? ;)
813 2012-06-03 18:59:56 <talpan> okay, point taken.
814 2012-06-03 19:00:30 <meLon> Ahhh, interensting talpan. Sounds like a 'workaround'.  Is that the intended functionality?
815 2012-06-03 19:00:39 <talpan> just run bitcoind and change the path with -datadir, it should redownload everything
816 2012-06-03 19:00:44 <talpan> meLon: it is
817 2012-06-03 19:00:50 <tcatm_> luke-jr: I haven't tried yet and I wouldn't recommend it anyway.
818 2012-06-03 19:01:17 <talpan> with the latest client the blockchain-download is incredible fast
819 2012-06-03 19:02:17 <luke-jr> s/fast/slow imo
820 2012-06-03 19:03:17 <meLon> I am interested in connecting to more than 8 peers.  Is it port 8334 which should be forwarded?  I compiled with USE_UPNP=1, but I don't have UPNP set up correctly atm
821 2012-06-03 19:03:33 <BlueMatt> luke-jr: bayleef blkindex.dat and blk0001.dat, that is
822 2012-06-03 19:03:57 <Matt_von_Mises> I'll ask again in case anyone around now knows. For transaction output values, how is the sign represented?
823 2012-06-03 19:04:03 <luke-jr> BlueMatt: we should write a debug tool that builds blkdat0001.dat from a blkindex.dat :/
824 2012-06-03 19:04:24 <luke-jr> meLon: 8333
825 2012-06-03 19:04:33 <luke-jr> Matt_von_Mises: what sign?
826 2012-06-03 19:04:43 <BlueMatt> tcatm_: btw, when 0.7 comes out, can you stop including blkindex.dat in the blockchain copies (and in the README, put: use -loadblock=blk0001.dat)
827 2012-06-03 19:04:57 <tcatm_> BlueMatt: Sure.
828 2012-06-03 19:05:06 <Matt_von_Mises> luke-jr: Output values are signed integers.
829 2012-06-03 19:05:15 <luke-jr> tcatm_: might make blkindex.dat a separate download, just in case people using 0.4-0.6 want to use them
830 2012-06-03 19:05:35 <luke-jr> tcatm_: also, be sure to add detachdb=1 to the config file making those&
831 2012-06-03 19:05:40 <luke-jr> tcatm_: does it enforce P2SH btw?
832 2012-06-03 19:05:49 <bayleef> Have another debug log: http://failbox.co.cc/debug-detachdb.log
833 2012-06-03 19:05:51 <luke-jr> Matt_von_Mises: no, they're not really.
834 2012-06-03 19:06:06 <tcatm_> Does stock 0.6.2 enforce P2SH?
835 2012-06-03 19:06:20 <luke-jr> yes
836 2012-06-03 19:06:30 <luke-jr> stock 0.6.2 also requires -detachdb already I think
837 2012-06-03 19:06:39 <BlueMatt> it does