1 2012-06-13 00:07:57 <bayleef> And of course, another bitcoind on another network gave not but a passing glance at block 176948 before admitting it. Was kinda hoping it wouldn't work
  2 2012-06-13 00:13:01 <bayleef> wonder what would happen if I deleted the blockchain, and resynched it just from that one machine
  3 2012-06-13 00:17:49 <luke-jr> bayleef:
  4 2012-06-13 00:17:51 <luke-jr> ?
  5 2012-06-13 00:21:44 <bayleef> luke-jr: I have a server, just stuck bitcoind 0.6.2 on it and watched it go right on past the block I'm stuck on.
  6 2012-06-13 00:21:58 <luke-jr> bayleef: what version do you use?
  7 2012-06-13 00:23:00 <bayleef> locally I've tried with 0.6.0, 0.6.2 and latest from bitcoin master. Currently on master
  8 2012-06-13 07:48:48 <lukys> Is there any way I can change the working directory of Bitcoin-qt?
  9 2012-06-13 07:50:25 <sipa> lukys: -datadir=x
 10 2012-06-13 07:51:38 <lukys> Of course. It only just occurred to me to use the terminal.
 11 2012-06-13 07:53:24 <lukys> Does that set it permanently?
 12 2012-06-13 07:54:49 <sipa> no
 13 2012-06-13 07:58:29 <lukys> Well, I'll add the argument to the launcher then, but is there no way I can set it permanently in the config file or anything?
 14 2012-06-13 07:58:51 <lukys> I don't have the space on this filesystem.
 15 2012-06-13 07:59:56 <sipa> yes, you can set it permanently using datadir=X in ~/.bitcoin/bitcoin.conf
 16 2012-06-13 08:03:29 <lukys> When I've set that, I presume it's then safe to delete the wallet and everything else from the original directory?
 17 2012-06-13 08:03:36 <lukys> Apart from the config files.
 18 2012-06-13 08:03:37 <SomeoneWeird> no
 19 2012-06-13 08:03:41 <SomeoneWeird> dont delete your wallet
 20 2012-06-13 08:03:53 <SomeoneWeird> you'll lose all yours coins then
 21 2012-06-13 08:04:05 <lukys> But the wallet is in the new directory.
 22 2012-06-13 08:04:12 <SomeoneWeird> did you copy it over?
 23 2012-06-13 08:04:20 <SomeoneWeird> like, the exact same one?
 24 2012-06-13 08:04:31 <lukys> It copied automatically when I set the new directory.
 25 2012-06-13 08:04:50 <SomeoneWeird> it might've created a new one
 26 2012-06-13 08:04:56 <SomeoneWeird> not sure, you'll have to check
 27 2012-06-13 08:05:03 <SomeoneWeird> but i wouldn't delete it quite yet
 28 2012-06-13 08:05:37 <sipa> it created a new one
 29 2012-06-13 08:06:01 <lukys> Hmm, interesting. The new wallet is 73.7 kB, whereas the old one is 49.2 kB.
 30 2012-06-13 08:06:06 <SomeoneWeird> yep
 31 2012-06-13 08:06:18 <sipa> bdb has strange ways
 32 2012-06-13 08:06:20 <SomeoneWeird> so, copy the old one and overwrite the new one (backup the new one if you want to)
 33 2012-06-13 08:07:16 <sipa> it certainly did not copy anything
 34 2012-06-13 08:11:21 <lukys> Ok, copied and deleted old files.
 35 2012-06-13 08:13:37 <lukys> Just so you know, I don't have any coins yet, so this is all safe for the moment.
 36 2012-06-13 11:21:58 <jgarzik> so, stupid question
 37 2012-06-13 11:22:07 <jgarzik> sipa: where is CTxDestination defined?
 38 2012-06-13 11:23:24 <sipa> jgarzik: script.h
 39 2012-06-13 11:26:58 <jgarzik> sipa: where did CBitcoinAddress::GetHash160 go?
 40 2012-06-13 11:27:51 <sipa> jgarzik: decode it, and get the hash from the CTxDestination
 41 2012-06-13 11:29:07 <jgarzik> sipa: I have what is returned from ExtractDestinations()...  so a list of CTxDestination.  how to get hash160 from each of those?
 42 2012-06-13 11:29:08 <sipa> oh, you can use GetKeyID directly
 43 2012-06-13 11:29:54 <sipa> if the string encodes a key id instead of a destination
 44 2012-06-13 11:30:30 <sipa> the few times where there was actual need to write cases for each of the destinations i used a visitor
 45 2012-06-13 11:31:27 <jgarzik> sipa: see FilterMatchTxOut(), https://github.com/bitcoin/bitcoin/pull/1386/files
 46 2012-06-13 11:31:34 <jgarzik> sipa: have to rewrite that
 47 2012-06-13 11:32:25 <bost> hi
 48 2012-06-13 11:32:26 <sipa> jgarzik: i wonder if it isn't better to write a filter that filters the hash of the scriptOut directory
 49 2012-06-13 11:32:28 <jgarzik> sipa: it grabs possible match data from the CTxOut, and scans for matches
 50 2012-06-13 11:32:34 <sipa> jgarzik: without trying to decide it for addresses
 51 2012-06-13 11:33:01 <sipa> the other way around is easier: you can still specify an address to match on, turn that into a script, and add its hash to the filter
 52 2012-06-13 11:33:26 <sipa> s/directory/directly/
 53 2012-06-13 11:33:37 <sipa> s/decide/decode/
 54 2012-06-13 11:33:56 <bost> guys 'bout a 15 minutes ago I launched 'bitcoind -upgradewallet' but the process is still not finished...
 55 2012-06-13 11:34:02 <jgarzik> sipa: well the main use cases are (a) watching for activity on a bitcoin address or (b) watching for activity on a script hash.  I'm not sure the current code even gets (a) correct...  what is the best way to do (a) in current codebase?
 56 2012-06-13 11:34:04 <bost> do you have any idea why?
 57 2012-06-13 11:34:21 <sipa> bost: it should be instantaneous
 58 2012-06-13 11:34:24 <jgarzik> "current code" == #filter, "current codebase" == #master, to be clear
 59 2012-06-13 11:34:30 <jgarzik> that was a bit confusing
 60 2012-06-13 11:34:33 <sipa> bost: but you won't see any output from that
 61 2012-06-13 11:34:39 <bost> sipa: yea, I expected the same
 62 2012-06-13 11:34:47 <bost> sipa: but... :(
 63 2012-06-13 11:34:51 <sipa> bost: how do you infer it hasn't finished?
 64 2012-06-13 11:35:35 <sipa> jgarzik: would my suggestion not cover all cases, and be more efficient as well?
 65 2012-06-13 11:35:49 <bost> bost@desktop-64:~$ ps -ef | grep bitcoin
 66 2012-06-13 11:35:59 <sipa> bost: it doesn't exit after upgrading
 67 2012-06-13 11:36:04 <sipa> bost: it just does that at startup
 68 2012-06-13 11:36:45 <bost> sipa: ?? it exits at startup ?? what means that? is't it a contradiction?
 69 2012-06-13 11:37:09 <sipa> bost: -upgradewallet just means you allow bitcoin to upgrade the wallet to the latest format
 70 2012-06-13 11:37:19 <sipa> bost: for the rest it just runs the bitcoin daemon as usual
 71 2012-06-13 11:37:31 <bost> sipa: oh, you mean it simply stops before doing the job, right?
 72 2012-06-13 11:37:36 <sipa> no
 73 2012-06-13 11:37:51 <sipa> it just upgrades the wallet and then continues whatever it would do if -upgradewallet wasn't specified
 74 2012-06-13 11:38:09 <sipa> jgarzik: address -> script is easy; script -> address requires using extractaddresses; so i'd say just match on the scriptPubKey's hash in your bloom filter
 75 2012-06-13 11:38:34 <sipa> jgarzik: and when someone asks for a match on a particular address, turn that address into a scriptPubKey, hash it, and add that to your filter
 76 2012-06-13 11:38:35 <bost> sipa: uff! so it means the upgrade went (hopefully) ok, and now it just runs, right?
 77 2012-06-13 11:38:43 <sipa> bost: yes
 78 2012-06-13 11:39:00 <sipa> bost: the upgrade really is just writing "wallet version is X" really
 79 2012-06-13 11:39:34 <sipa> is it unclear? the help says "upgrade wallet at startup" - it doesn't say it will exit after upgrading
 80 2012-06-13 11:39:56 <sipa> use getinfo if you want to see whether it's running
 81 2012-06-13 11:40:12 <jgarzik> sipa: the filter must do both
 82 2012-06-13 11:40:23 <jgarzik> sipa: the first when adding hashes to the filter, and the second when matching
 83 2012-06-13 11:40:43 <jgarzik> sipa: the branch no longer builds due to the Extract* changes
 84 2012-06-13 11:40:43 <sipa> jgarzik: under "address" i understand both pubkeyhash addresses and scripthash addresses
 85 2012-06-13 11:40:45 <bost> sipa: uhm.. it's like with taking credit 1.000.000 $
 86 2012-06-13 11:41:00 <sipa> jgarzik: i don't see the problem
 87 2012-06-13 11:41:08 <jgarzik> sipa: your idea is fine, but does not address the build problem
 88 2012-06-13 11:41:12 <bost> sipe: and then just to say 'but I didnt say I gonna pay it back'
 89 2012-06-13 11:41:22 <sipa> jgarzik: it would mean you don't call extractaddresses at all
 90 2012-06-13 11:41:35 <jgarzik> sipa: the match must extract hashes somehow from the script
 91 2012-06-13 11:41:36 <sipa> jgarzik: you just do Hash(scriptPubKey) and match on that
 92 2012-06-13 11:42:07 <sipa> the script's hash != the script's destination (which also happens to be a hash)
 93 2012-06-13 11:42:16 <jgarzik> clearly
 94 2012-06-13 11:44:06 <bost> sipa: ok, thank you anyway
 95 2012-06-13 11:44:56 <sipa> jgarzik: not sure if you're with me now, or if there's something i'm missing?
 96 2012-06-13 12:03:17 <gavinandresen> Anybody willing to help me make up my mind on a 'signrawtx' RCP call detail?  sipa or gmaxwell ?
 97 2012-06-13 12:03:53 <sipa> gavinandresen: i'll have a look
 98 2012-06-13 12:04:22 <gavinandresen> Cool, here's what I'm waffling over:  You've got   signrawtx "hex-encoded serialized transaction to be partially signed"
 99 2012-06-13 12:04:54 <gavinandresen> (sorry, interrupted...)
100 2012-06-13 12:05:33 <sipa> remind me: could you start from an unsigned raw tx, add a signature for the last txin, and then add signatures for the previous txins?
101 2012-06-13 12:06:02 <sipa> or does the signing replace-magic require the earlier txins?
102 2012-06-13 12:06:04 <gavinandresen> signrawtx adds as many signatures as it can
103 2012-06-13 12:06:30 <gavinandresen> If you call it twice, it'll replace earlier signatures it made with new ones
104 2012-06-13 12:06:47 <kinlo> will it submit the transaction too?
105 2012-06-13 12:06:50 <gavinandresen> (or if you had the same private key in two wallets)
106 2012-06-13 12:06:55 <gavinandresen> No, sendrawtx sends it
107 2012-06-13 12:06:55 <sipa> i'm just talking about the message signature semantics
108 2012-06-13 12:06:59 <kinlo> good
109 2012-06-13 12:07:07 <sipa> not about signrawtx now
110 2012-06-13 12:07:18 <kinlo> I'd really like commands to do just one command and one command only
111 2012-06-13 12:07:38 <gavinandresen> sipa: hmm?
112 2012-06-13 12:08:08 <sipa> gavinandresen: i remember the hash being signed is calculated from some post-processed version of the serialized transaction
113 2012-06-13 12:08:21 <sipa> gavinandresen: what i'm wondering about is whether the signatures depend on one another
114 2012-06-13 12:09:04 <gavinandresen> No, the scriptSig's for the transaction being signed are cleared, and the one being signed is replaced by the previous scriptPubKey to compute the hash
115 2012-06-13 12:09:21 <sipa> ok, good
116 2012-06-13 12:09:36 <gavinandresen> And that actually gets to what I'm waffling over
117 2012-06-13 12:09:38 <sipa> so it's even independent from the other txins entirely
118 2012-06-13 12:09:41 <sipa> ?
119 2012-06-13 12:09:56 <sipa> oh, no, only the scriptSigs are cleared
120 2012-06-13 12:10:01 <sipa> pity
121 2012-06-13 12:10:12 <gavinandresen> So:  imagine you've created a raw transaction that depends on another transaction that isn't yet in the blockchain
122 2012-06-13 12:10:44 <sipa> right, i suppose the serialization would need to contain its dependencies?
123 2012-06-13 12:10:51 <gavinandresen> sipa:  other stuff is cleared for signature hash modes, I'm concentrating on SIGHASH_ALL right now....
124 2012-06-13 12:10:58 <sipa> right
125 2012-06-13 12:11:02 <sipa> ok, go on
126 2012-06-13 12:11:41 <gavinandresen> So, I want signrawtx to be:   signrawtx  "hex of tx being signed"   [ "previous transaction", "previous transcation" ]
127 2012-06-13 12:12:08 <gavinandresen> In other words, you can pass in the previous transactions in case bitcoind doesn't know about them yet (because they're not in the blockchain or are other, not-yet-sent raw txns)
128 2012-06-13 12:12:39 <gavinandresen> The question is what the format of that second param should be.  Easiest might be to just have the full, previous raw transaction
129 2012-06-13 12:13:00 <gavinandresen> ... but that's more data than is strictly necessary.  You really just need to know   (hash, output, scriptPubKey)
130 2012-06-13 12:13:14 <kinlo> no intermediate format like bip 10 describes?
131 2012-06-13 12:13:38 <gavinandresen> kinlo: in the RPC interface?  No, why would we write more possibly-buggy code for Yet Another Encoding?
132 2012-06-13 12:14:22 <sipa> hmmm
133 2012-06-13 12:14:25 <kinlo> I feel there is a place for some kind of generic transaction format that can be used in many places
134 2012-06-13 12:14:51 <gavinandresen> kinlo: okey doke, feel free to build that on top of the RPC interface
135 2012-06-13 12:14:56 <sipa> i agree there, actually; but let's talk about that later
136 2012-06-13 12:15:33 <sipa> so, assume i have in my wallet a transaction that is not in the blockchain
137 2012-06-13 12:15:43 <sipa> i want to create a raw transaction that spends one of its outputs
138 2012-06-13 12:16:04 <sipa> i believe there should be an interface that gives both the created raw transaction, plus its unconfirmed dependencies
139 2012-06-13 12:16:30 <gavinandresen> yup.  ./bitcoind gettransaction <txid>     .... gets you .... stuff.... but not the entire raw tx.
140 2012-06-13 12:17:04 <sipa> sec, brb
141 2012-06-13 12:20:58 <jgarzik> sipa: sorry, had to disappear due to baby, in middle of conversation.  poo-us interruptus
142 2012-06-13 12:21:04 <gavinandresen> For reference:  https://gist.github.com/2839617    (Proposed listunspent / createrawtx / signrawtx / sendrawtx commands, I've implemented list/create, am in the middle if sign)
143 2012-06-13 12:21:48 <jgarzik> Let me first ask the audience the dumb question, to verify my knowledge:  is the following the standard transaction most often used?   OP_DUP OP_HASH160 b5cd7aaed869cd5ccb45868e8666e7e934a23736 OP_EQUALVERIFY OP_CHECKSIG  And what is that a hash of?
144 2012-06-13 12:22:07 <gavinandresen> yes.  Hash of the full public key
145 2012-06-13 12:22:46 <sipa> that hex value is the hex encoding of the 160-bit hash of the public key, which is also encoded in base58 in the normal address that corresponds to that key
146 2012-06-13 12:22:51 <gavinandresen> That hash is different from the transaction id (hash of the entire transaction) and different yet again from the hash used by OP_CHECKSIG
147 2012-06-13 12:24:04 <sipa> gavinandresen: just a short comment; if you're creating such a low-level interface, i feel you should have the ability to add arbitrary txout scripts (not just those that have well-defined addresses)
148 2012-06-13 12:24:35 <gavinandresen> sipa: sure, but the signrawtx won't know how to sign non-standard inputs
149 2012-06-13 12:24:56 <jgarzik> I just want to make sure the filter is properly matching on p2sh destination or bitcoin address destination, and nothing else
150 2012-06-13 12:25:02 <luke-jr> gavinandresen: maybe a generic interface to get any data at all signed?
151 2012-06-13 12:25:13 <gavinandresen> sipa:  it will leave non-standard inputs, or inputs that it doesn't have previous transactions for, alone.
152 2012-06-13 12:25:23 <sipa> gavinandresen: it's your responsibility to only add outputs that the other party understands
153 2012-06-13 12:25:47 <gavinandresen> jgarzik: which pull request?
154 2012-06-13 12:25:47 <jgarzik> vis a vis code and sipa's comments
155 2012-06-13 12:25:51 <jgarzik> gavinandresen: filter*
156 2012-06-13 12:26:04 <sipa> jgarzik: why only match those two? you'll run into troubles when another address type is added which doesn't have a nicely-packaged 160-bit value in it
157 2012-06-13 12:26:15 <luke-jr> gavinandresen: your interface supports adding inputs that the client doesn't have in its wallet, no?
158 2012-06-13 12:26:32 <sipa> just matching on the scriptPubKey's hash is easier, faster and more flexible
159 2012-06-13 12:26:58 <sipa> luke-jr: i suppose those are supposed to already be in the raw transaction?
160 2012-06-13 12:28:10 <gavinandresen> (catching up, page-swapping the filter* RPC stuff into my brain....)
161 2012-06-13 12:29:35 <gavinandresen> luke-jr: sure, createrawtx creates inputs by refererring to (txid, n), so as long as you know the previous transactions txid ....
162 2012-06-13 12:31:12 <gavinandresen> jgarzik:  RE: filter:   my comment was just that if somebody used the uncommon <pubkey> OP_CHECKSIG form of transaction then the filter* API can't trigger when it is spent.
163 2012-06-13 12:32:08 <gavinandresen> jgarzik: that's probably OK... although <pubkey> OP_CHECKSIG is the common form for coinbase transactions, so if you want to trigger when you spend generated coins you'd be out of luck.
164 2012-06-13 12:32:22 <sipa> my point is this: addresses are in essence a shorthand notation for a particular subset of scriptPubKey's
165 2012-06-13 12:32:40 <sipa> i believe the receiver is always the one who gets to decide which scriptPubKey is used for payments to him
166 2012-06-13 12:33:00 <sipa> so it shouldn't be a problem to just match on the scriptPubKey directly
167 2012-06-13 12:33:28 <sipa> if someone wants to be informed about payments to a particular address, just expand that address to its corresponding scriptPubKey, and add that to your filter rule set
168 2012-06-13 12:34:03 <gavinandresen> sipa:  but the filter won't trigger if you pay to DUP HASH160 <hash> ....
169 2012-06-13 12:34:13 <gavinandresen> ... because the scriptPubKey isn't revealed until the spend
170 2012-06-13 12:34:22 <gavinandresen> excuse me, the full pubkey
171 2012-06-13 12:34:32 <sipa> i'm not talking about the pubkey
172 2012-06-13 12:34:42 <sipa> s/scriptPubKey/the script in the txout/
173 2012-06-13 12:34:57 <gavinandresen> Sorry, misinterpreted
174 2012-06-13 12:35:26 <sipa> if you accept payments to you using some weird non-standard script, and your payers and miners have no problem with that, no problem
175 2012-06-13 12:35:38 <gavinandresen> I think in general people will want to be informed of payments both to AND from
176 2012-06-13 12:35:45 <sipa> sure, sure
177 2012-06-13 12:35:56 <sipa> i was just simplifying to just send case here
178 2012-06-13 12:37:33 <sipa> gavinandresen: could your createrawtx be used to not specify any txin at all (or too few txins for the specified output) ?
179 2012-06-13 12:37:52 <ThomasV> 0.6.2 does not seem to delete old unprocessed transactions from its memory pool..
180 2012-06-13 12:37:53 <jgarzik> sipa: I am fine with filter matching based on Hash(scriptPubKey)
181 2012-06-13 12:37:54 <luke-jr> gavinandresen: FWIW, the BIP16 backports bugged on block 178118; I found the fix in the original OP_EVAL commit >_<
182 2012-06-13 12:38:08 <jgarzik> sipa: but it is a user interface issue, that some users will want to say "filter <bitcoinaddress>"
183 2012-06-13 12:38:22 <luke-jr> (the semantics of IsPushOnly changed: the older clients checked the 200 byte size in there too!)
184 2012-06-13 12:38:23 <gavinandresen> luke-jr: could you send me email about that?
185 2012-06-13 12:38:26 <jgarzik> so maybe we need 'filteraddhash' and 'filteraddaddress'
186 2012-06-13 12:38:29 <sipa> jgarzik: so, when they do, you expand that address into a script, hash it, and add it to the filter set
187 2012-06-13 12:38:33 <luke-jr> gavinandresen: sure
188 2012-06-13 12:38:36 <jgarzik> and 'filteraddaddress' would generate scripts
189 2012-06-13 12:38:40 <jgarzik> and add their hashes
190 2012-06-13 12:38:53 <jgarzik> 'filteraddhash' just directly adds the hash
191 2012-06-13 12:38:57 <ThomasV> bitcoincharts's list of unprocessed transactions shows "There are 8976 unconfirmed transactions "
192 2012-06-13 12:39:05 <ThomasV> is that normal?
193 2012-06-13 12:39:19 <ThomasV> I am having the same issue with my own bitcoind
194 2012-06-13 12:39:29 <ThomasV> about 6000 tx after a few days
195 2012-06-13 12:39:35 <sipa> jgarzik: or even filteraddscript (filteraddscripthash is more flexible, but you can't accept funds to unknown scripts anyway)
196 2012-06-13 12:40:36 <jgarzik> sipa: I think if they can generate a script, they can do the hash
197 2012-06-13 12:40:43 <sipa> agree
198 2012-06-13 12:40:51 <jgarzik> sipa: OTOH, a formal bitcoin address is most likely to be available to dumb, text-based programs
199 2012-06-13 12:41:01 <sipa> jgarzik: i think the UI/RPC implementation can safely be limited to just filteraddaddress for now
200 2012-06-13 12:41:14 <jgarzik> hmmm, ok
201 2012-06-13 12:41:43 <gavinandresen> ... but filteraddaddress won't detect spends FROM that address, will it?
202 2012-06-13 12:41:57 <jgarzik> gavinandresen: it looks at CTxOut
203 2012-06-13 12:42:00 <ThomasV> tcatm: ping
204 2012-06-13 12:42:14 <gavinandresen> jgarzik: previous transaction's CTxOut ?
205 2012-06-13 12:42:29 <jgarzik> gavinandresen: no, just current tx's CTxOut.
206 2012-06-13 12:42:39 <sipa> i don't like the phrase "from an address" ;)
207 2012-06-13 12:42:53 <gavinandresen> Right, current tx's CTxOut is only where the funds are going, not where they're from
208 2012-06-13 12:43:35 <jgarzik> block logic: for each tx { for each txout { check for match } }
209 2012-06-13 12:43:44 <jgarzik> obvious reduction, for TX memory pool match check
210 2012-06-13 12:44:10 <gavinandresen> It doesn't just match against the entire serialized transaction ?
211 2012-06-13 12:44:28 <gavinandresen> (can I match against the txins ?)
212 2012-06-13 12:45:23 <sipa> if you want to observe spends from things that are yours, i think you want a trigger generated by your wallet when one of its coins gets marked spent
213 2012-06-13 12:46:22 <sipa> rather than a manual check on spends from particular addresses - that's logic wallets already have
214 2012-06-13 12:46:23 <gavinandresen> If it matched against the entire transaction, then I could filter on the txid and get triggered when any output of that txid was spent
215 2012-06-13 12:47:14 <gavinandresen> (because the txins are (txid, n, scriptSig))
216 2012-06-13 12:47:34 <sipa> right, but if you want that you'll need special cases for all potential matches
217 2012-06-13 12:47:44 <sipa> as the bloom filter needs a hash to lookup
218 2012-06-13 12:48:06 <sipa> so you need to extract data from the serialized tx, hash it, and ask the filter for a match
219 2012-06-13 12:48:16 <sipa> i think only a few things are worth matching for
220 2012-06-13 12:48:38 <gavinandresen> ah, right.....
221 2012-06-13 12:49:13 <gavinandresen> I forgot it's not full matching
222 2012-06-13 12:49:13 <sipa> but matching on an txid sounds like a useful extension
223 2012-06-13 12:49:35 <sipa> but far less useful than matching on a txout script
224 2012-06-13 12:50:58 <jgarzik> hmmm
225 2012-06-13 12:51:09 <gavinandresen> ok, I like filteraddaddress.     Maybe filteraddtxid  later
226 2012-06-13 12:51:28 <jgarzik> people might use txid match for exchanging offline/OOB transactions, indeed
227 2012-06-13 12:51:45 <ThomasV> can someone please enlighten me? the memory pool of my bitcoind seems to keep growing day after day
228 2012-06-13 12:51:58 <jgarzik> I think filteraddhash, too.  it is easy to do, and can be used by power users
229 2012-06-13 12:52:01 <ThomasV> is that normal?
230 2012-06-13 12:52:20 <sipa> ThomasV: satoshidice...?
231 2012-06-13 12:52:45 <gavinandresen> ThomasV: either satoshidice or somebody might be mounting a slow "try to fill up your memory" attack.  Restarting will flush the pool.
232 2012-06-13 12:53:09 <gavinandresen> ThomasV: ... or git HEAD has code to keep the pool from growing forever
233 2012-06-13 12:53:23 <ThomasV> what is new in git head?
234 2012-06-13 12:54:04 <ThomasV> I don't think I am a target for attack
235 2012-06-13 12:54:24 <gavinandresen> IPv6 support, gettransaction showing non-wallet transactions, debug console in the GUI (right?), bug fixes....
236 2012-06-13 12:54:49 <ThomasV> and satoshi dice tx eventually make it in the chain, so there's no reason to have 6000 of them in the mempool
237 2012-06-13 12:55:12 <sipa> ThomasV: they tend to create very long chains of unconfirmed transactions
238 2012-06-13 12:55:22 <sipa> luke-jr: the memory pool is useful for non-miner!
239 2012-06-13 12:55:33 <luke-jr> sipa: is it?
240 2012-06-13 12:56:04 <sipa> well, maybe not actually
241 2012-06-13 12:56:09 <ThomasV> gavinandresen: I mean, what is the specific change that keeps the pool from growing? what was changed?
242 2012-06-13 12:56:18 <ThomasV> how was it addressed?
243 2012-06-13 12:56:51 <sipa> luke-jr: i wanted to say observing 0-conf transactions
244 2012-06-13 12:56:52 <gavinandresen> There's now a maximum pool size, and a random transaction is evicted from the pool to make room if that maximum is hit
245 2012-06-13 12:57:10 <luke-jr> sipa: ok, that's a corner case that could use -miner :p
246 2012-06-13 12:57:14 <ThomasV> how much is the max?
247 2012-06-13 12:57:24 <sipa> luke-jr: but it's not even true
248 2012-06-13 12:57:46 <sipa> gavinandresen: i wonder, do we evict depending transactions in the memory pool too?
249 2012-06-13 12:57:51 <ThomasV> why not evict the older tx?
250 2012-06-13 12:57:57 <gavinandresen> static const unsigned int MAX_ORPHAN_TRANSACTIONS = MAX_BLOCK_SIZE/100;
251 2012-06-13 12:58:43 <gribble> New news from bitcoinrss: sgaltsev opened issue 1447 on bitcoin/bitcoin <https://github.com/bitcoin/bitcoin/issues/1447>
252 2012-06-13 12:58:49 <gavinandresen> ThomasV: random makes it harder for an attacker to manipulate the pool for some attack
253 2012-06-13 12:59:03 <ThomasV> ic
254 2012-06-13 12:59:38 <gavinandresen> No memory pool if you're not mining is an interesting idea....
255 2012-06-13 12:59:56 <luke-jr> gavinandresen: I share sipa's concern about evicting transactions the new one depends on
256 2012-06-13 13:00:30 <gavinandresen> wait, hang on, memory pool != orphan pool ....
257 2012-06-13 13:00:46 <sipa> oh
258 2012-06-13 13:01:01 <ThomasV> I would think that 144*2 is indeed long enough
259 2012-06-13 13:02:28 <gavinandresen> yeah, strike everything I just said
260 2012-06-13 13:02:36 <ThomasV> k
261 2012-06-13 13:03:58 <gavinandresen> Size of the mempool is limited only by transaction fees and the "free transaction rate limiter"
262 2012-06-13 13:04:18 <ThomasV> what's that?
263 2012-06-13 13:04:41 <TD> jgarzik: this is the RPCs you are adding for bloom filtering?
264 2012-06-13 13:05:00 <TD> jgarzik: are you going to add p2p protocol commands for filtering connections too? we need that for the last chunk of SPV client scalability work
265 2012-06-13 13:05:12 <gavinandresen> -limitfreerelay option will limit the number of "free" (less than MIN_RELAY_TX_FEE) transactions added to the pool or relayed to other peers
266 2012-06-13 13:05:49 <jgarzik> TD: not sure what you mean
267 2012-06-13 13:06:29 <TD> jgarzik: you were talking about what to filter on (tx outs, etc)
268 2012-06-13 13:06:30 <jgarzik> TD: yeah, 'filter{addaddress,clear,clearall}' is for bloom filtering, matching on CTxOut (or, proposed, txid)
269 2012-06-13 13:06:31 <sipa> jgarzik: the ability to tell a peer "only send me transactions that interact with scripts/addresses in <bloom data>"
270 2012-06-13 13:06:44 <TD> jgarzik: right but are these RPCs or p2p commands or both?
271 2012-06-13 13:06:53 <jgarzik> sipa: c.f. "filtering connections"
272 2012-06-13 13:06:59 <jgarzik> TD: RPCs
273 2012-06-13 13:07:06 <TD> jgarzik: what about txins? otherwise you cannot find transactions that spend your own coins
274 2012-06-13 13:07:17 <jgarzik> TD: gavinandresen raised that point, yes
275 2012-06-13 13:07:17 <TD> ok. would it be much harder to add p2p support too?
276 2012-06-13 13:07:23 <ThomasV> that does not sound like a strong constraint
277 2012-06-13 13:07:25 <TD> ok great. now i'm caught up :)
278 2012-06-13 13:07:39 <sipa> right, for the connection filtering you also need matching on prevout's scripts
279 2012-06-13 13:07:57 <sipa> for RPC notifications, maybe there are better solutions
280 2012-06-13 13:08:01 <jgarzik> ahhhh, connection between txs
281 2012-06-13 13:08:08 <jgarzik> I thought TD meant network connections via TCP/IP
282 2012-06-13 13:08:10 <TD> it might be easier to match on txins rather than txouts
283 2012-06-13 13:08:19 <TD> jgarzik: i do. for mobile clients, i don't want to download whole blocks
284 2012-06-13 13:08:35 <sipa> TD: you mean filtering based on prevout txid's, basically?
285 2012-06-13 13:08:35 <TD> jgarzik: but rather get sent a header + filtered transactions + merkle branches linking them to the header, based on a filter
286 2012-06-13 13:08:56 <jgarzik> sipa: that's a good idea, hmmm
287 2012-06-13 13:09:06 <TD> sipa: i mean the input scripts. like, i want to say "send me any transaction that contains my public keys in the inputs" as well. filtering on prevtx outs might be equivalent
288 2012-06-13 13:09:10 <TD> didn't think about it
289 2012-06-13 13:09:58 <sipa> filtering on prevout txid's is certainly easier than filtering on prevout scripts
290 2012-06-13 13:10:11 <sipa> as you don't need to have the prevout script present
291 2012-06-13 13:11:13 <yellowhat> is there any push/callback in the RPC API to be notified of various events? or is it all request-response based?
292 2012-06-13 13:11:33 <sipa> you could even define some implicit bahaviour that any transaction relayed from the SPV peer gets added to the watch-txid set
293 2012-06-13 13:12:34 <TD> sipa: but that doesn't help me find transactions spending my keys. i don't know the hashes of the transactions of interest
294 2012-06-13 13:12:36 <jgarzik> for P2P, you'd want 'filteraddblah', 'filterclear' ... to magically apply to existing P2P commands like getblock/getdata?  or create a new 'getfiltereddata' P2P command?
295 2012-06-13 13:12:51 <sipa> TD: you have a list of your own 'coins', no?
296 2012-06-13 13:13:04 <ThomasV> gavinandresen: what is the typical 'steady state' number of tx in a daemon's pool, assuming it's always on?
297 2012-06-13 13:13:15 <TD> jgarzik: hmm, good question. i guess magically apply. i don't want to hear about broadcast transactions that are irrelevant to me too
298 2012-06-13 13:13:51 <sipa> TD: if you have a list of your own coins, you know their txids, so you could build an initial bloom filter for those, and send it to the peer
299 2012-06-13 13:14:01 <TD> sipa: yeah, i guess so, i was thinking in the case where you want to download broadcast transactions whilst still catching up with the chain
300 2012-06-13 13:14:19 <TD> you know your keys then but not every transaction. at least not right away.
301 2012-06-13 13:14:39 <sipa> right
302 2012-06-13 13:15:06 <sipa> everytime you hear about a transaction, if it's a spend to yourself, you'd want to have it retroactively added to the peer's txid filter set
303 2012-06-13 13:15:45 <gavinandresen> the "catching up" case is definitely the tricky bit
304 2012-06-13 13:17:54 <jgarzik> create a new mempool P2P command, which serves up the current mempool, and make sure filter* P2P commands apply to it
305 2012-06-13 13:18:19 <sipa> seems we're going to need matching on prevout scripts anyway
306 2012-06-13 13:18:35 <gavinandresen> So: bitcoind DOES fetch the previous scriptPubKeys to validate signatures, so matching against them shouldn't be too much of a burden
307 2012-06-13 13:18:49 <jgarzik> thanks to bdb cache :)
308 2012-06-13 13:19:05 <sipa> ... hopefully
309 2012-06-13 13:19:19 <jgarzik> thanks to bdb cache and satoshidice :)  </edit>
310 2012-06-13 13:20:29 <jgarzik> more seriously, both mempool and block checks, for filtering, already hit the database for TX validation, so it should be already in-cache for both those cases
311 2012-06-13 13:21:27 <sipa> yes, indeed
312 2012-06-13 13:21:45 <gavinandresen> Lemme think... if bitcoind was catching up before a checkpoint I don't think it would need to fetch previous transaction scriptPubKeys.
313 2012-06-13 13:22:10 <sipa> it always needs to fetch the prevout
314 2012-06-13 13:22:11 <jgarzik> I'm pretty sure filtering is skipped for IBD
315 2012-06-13 13:22:24 <sipa> as there is no way to know prevout's amount otherwise
316 2012-06-13 13:22:34 <sipa> even during IBD
317 2012-06-13 13:24:13 <jgarzik> quick vote on RPC/P2P UI minor issue:  (a) filterclearall or (b) filterclear '*'
318 2012-06-13 13:24:18 <gavinandresen> sipa: I'd have to stare at the code, but it seems rearranging the valid-transaction-checks and skipping the "ins > outs" check if before a checkpoint could optimize that away
319 2012-06-13 13:24:26 <jgarzik> filter names are limited to [a-zA-Z0-9_]
320 2012-06-13 13:24:44 <sipa> gavinandresen: that would speed up block download a lot
321 2012-06-13 13:25:10 <sipa> jgarzik: i wouldn't use filter names in P2P
322 2012-06-13 13:25:17 <sipa> for RPC, sure
323 2012-06-13 13:25:51 <jgarzik> sipa: limit P2P to a single filter?  fair enough
324 2012-06-13 13:25:56 <gavinandresen> jgarzik: filterclear '*'   is my vote.
325 2012-06-13 13:25:57 <sipa> jgarzik: hell no
326 2012-06-13 13:26:03 <wumpus> yellowhat: currently it is all request/response based, async notifications with jsonrpc are not obvious, I guess longpolling would be the simplest
327 2012-06-13 13:26:14 <jgarzik> sipa: single filter per connection, I mean
328 2012-06-13 13:26:25 <sipa> jgarzik: just send a bloom set
329 2012-06-13 13:26:55 <jgarzik> sipa: right...  "filter" == "bloom set" here.  does an individual P2P connection get more than one bloom set, like RPC?
330 2012-06-13 13:27:00 <sipa> i thought that was the point, to be able to compactly represent a filter, without compromising all privacy
331 2012-06-13 13:27:05 <Diablo-D3> yay bloomfilters
332 2012-06-13 13:27:07 <sipa> ooh, ok
333 2012-06-13 13:27:08 <wumpus> just one bloom set of a fixed size... otherwise, the amount of data per connection could be arbitrary
334 2012-06-13 13:27:11 <Diablo-D3> the stupidest thing ever invented them
335 2012-06-13 13:27:18 <jgarzik> it can be useful to know _which_ set is matched
336 2012-06-13 13:27:20 <Diablo-D3> yet Im already using them in lugh
337 2012-06-13 13:27:25 <jgarzik> but maybe for P2P that is overdesign
338 2012-06-13 13:27:36 <wumpus> you could check that on your own side
339 2012-06-13 13:27:48 <sipa> you should always be checking on your side anyway
340 2012-06-13 13:27:54 <sipa> as bloom filters can have false positives
341 2012-06-13 13:27:54 <wumpus> right
342 2012-06-13 13:27:58 <jgarzik> yes
343 2012-06-13 13:28:09 <Diablo-D3> false positives yes, false negatives no
344 2012-06-13 13:28:21 <sipa> and fixed size... unsure; ideal bloom filters have 50% of bits set
345 2012-06-13 13:28:32 <Diablo-D3> sipa: theres a mathematical formula for that
346 2012-06-13 13:28:34 <sipa> fixed maximum size maybe
347 2012-06-13 13:28:36 <wumpus> at least severly limited size
348 2012-06-13 13:28:37 <sipa> Diablo-D3: i know
349 2012-06-13 13:28:51 <Diablo-D3> I need to fix my bloom filter impl though
350 2012-06-13 13:28:58 <Diablo-D3> its only setting one bit
351 2012-06-13 13:29:28 <sipa> i implemented a bloom filter that had a weird extra requirement that you could not set a single bit; you always had to set two subsequent ones
352 2012-06-13 13:29:35 <sipa> doing the math for that was fun :)
353 2012-06-13 13:30:26 <TD> jgarzik: yes, being able to read the peers mempool would be awesome
354 2012-06-13 13:30:35 <yellowhat> i was thinking about bloomfilters - i understand that you could easily check all incoming transactions with them but are there even good data structures to index the existing blockchain for bloom filter queries?
355 2012-06-13 13:30:37 <TD> jgarzik: but being able to stay connected and only see invs for transactions of interest would also be helpful
356 2012-06-13 13:30:38 <wumpus> but a fixed size would give the least hassle with memory, no dynamic on the fly allocation based on connection state etc...
357 2012-06-13 13:30:42 <gavinandresen> http://code.google.com/p/bloom/  looks like a nice implementation, by the way
358 2012-06-13 13:30:56 <jgarzik> gavinandresen: license probs
359 2012-06-13 13:30:57 <TD> not the biggest deal right now but in future, it'd help reduce power usage and make bitcoin apps more responsive as they can stay connected to the p2p network permanently
360 2012-06-13 13:30:58 <jgarzik> IMO
361 2012-06-13 13:31:02 <TD> (or to aggregating proxies)
362 2012-06-13 13:31:06 <sipa> bloom filters are not hard to implement
363 2012-06-13 13:31:39 <sipa> yes, a getmemorypool *P2P* command would be nice
364 2012-06-13 13:31:49 <sipa> that just replies with a set of invs
365 2012-06-13 13:32:52 <jgarzik> TD: steps would be (1) sync up blocks, (2) filter* to set up bloom filter (3) getmemorypool P2P command to notice missed TX's [if they pass the filter], and (4) new network TXs are sent as usual [if they pass the filter]
366 2012-06-13 13:32:55 <jgarzik> sipa: ^^
367 2012-06-13 13:33:16 <jgarzik> maybe switch #2 and #1 for race reasons
368 2012-06-13 13:33:26 <wumpus> not hard to implement, however for a project like bitcoin it'd be preferable to use a well-tested implementation, if available with the right license of course
369 2012-06-13 13:33:49 <gavinandresen> RE: p2p or rpc bloom filters:  are there fixed params for the bloom filters that will work "good enough" for all applications, or do we need to specify the filter params in the p2p/rpc calls?
370 2012-06-13 13:33:49 <sipa> wumpus: if possible, sure
371 2012-06-13 13:34:37 <jgarzik> well ultimately you are just building the initial size of the filter, after which it is set in stone until reset
372 2012-06-13 13:34:50 <jgarzik> the bit accuracy etc. falls out from that guesstimate
373 2012-06-13 13:35:25 <gavinandresen> who is "you" ?  bitcoind or the caller ?
374 2012-06-13 13:35:26 <sipa> there are two parameters to be decided initially: number of hashes and number of bits in the filter
375 2012-06-13 13:35:57 <sipa> there exist some sort of cascading bloom filters that are somewhat more dynamic, iirc
376 2012-06-13 13:36:31 <gavinandresen> I'm looking at the params at http://code.google.com/p/bloom/source/browse/trunk/bloom_filter.hpp    :   projected_element_count and false_positive_probability
377 2012-06-13 13:36:52 <sipa> right - number of hashes and bits in the filter follow from those
378 2012-06-13 13:37:10 <wumpus> bloom filter is small, so why not send the entire filter anew on changes instead of trying to make it dynamic
379 2012-06-13 13:37:11 <gavinandresen> right, I'm just asking whether the caller can set them in the RPC/p2p
380 2012-06-13 13:37:24 <sipa> i think he should
381 2012-06-13 13:37:33 <sipa> with a max on the filter size
382 2012-06-13 13:38:09 <sipa> but depending on whether you have few or many things in the filter, or want low or high false positive chance, the parameters can vary
383 2012-06-13 13:38:23 <wumpus> and a max on the number of hashes... I suppose that affects CPU usage?
384 2012-06-13 13:38:33 <sipa> true
385 2012-06-13 13:38:43 <sipa> but more hashes is not always better
386 2012-06-13 13:38:52 <sipa> while a larger filter is
387 2012-06-13 13:38:55 <wumpus> a server with a large amount of peers might run into problems otherwise, or it might even be used as ddos
388 2012-06-13 13:38:58 <wumpus> right
389 2012-06-13 13:40:30 <sipa> but false positive change may be very high in our case
390 2012-06-13 13:40:48 <sipa> even 50% false positives means only a x2 factor in throughput
391 2012-06-13 13:43:11 <yellowhat> i even think the number of hashes should not be a parameter, it should be fixed.
392 2012-06-13 13:44:46 <sipa> if we set the filter size to 64KiB
393 2012-06-13 13:45:13 <sipa> then the optimal number of hashes is 360000 / n, with n the expected number of elements in the set
394 2012-06-13 13:45:24 <wumpus> if parameters can be fixed, they should be
395 2012-06-13 13:47:17 <gavinandresen> fewer knobs are better, that's why I asked if there's one set of params that could reasonably handle all the use cases we can think up.
396 2012-06-13 13:48:12 <TD> jgarzik: LGTM
397 2012-06-13 13:48:36 <TD> gavinandresen: clients will want to adjust the FP rate themselves for privacy
398 2012-06-13 13:48:55 <TD> wumpus: checking a bloom filter is very fast
399 2012-06-13 13:49:04 <TD> wumpus: i don't think it's a DoS risk
400 2012-06-13 13:49:20 <wumpus> I know, I think they're a great idea for this
401 2012-06-13 13:49:24 <sipa> it requires N hashes
402 2012-06-13 13:49:31 <sipa> but those hashes don't need to be cryptographic
403 2012-06-13 13:51:00 <gavinandresen> TD: agreed, false-positive rate should be exposed
404 2012-06-13 13:51:15 <sipa> is 64 KiB a reasonable max size for a filter?
405 2012-06-13 13:51:49 <BlueMatt> TD: is boom FP deterministic? how does FP rate provide privacy?
406 2012-06-13 13:52:33 <yellowhat> if you send only a small bloom filter you get lots of false positives and the bitcoind does not know which ones you are going to filter client-side
407 2012-06-13 13:52:34 <TD> BlueMatt: you don't want the peer to know your addresses. so you can make an inaccurate bloom filter and maybe get sent some stuff you don't care about
408 2012-06-13 13:52:36 <TD> which you throw awy
409 2012-06-13 13:52:40 <TD> exactly
410 2012-06-13 13:52:46 <TD> it lets you precisely trade off bandwidth vs privacy
411 2012-06-13 13:53:04 <wumpus> which is very neat
412 2012-06-13 13:53:06 <TD> for a mobile app, maybe on 3G you pick very precise filters, and when you're on wifi you pick very vague filters
413 2012-06-13 13:53:15 <sipa> a 64 KiB bloom filter with 300000 elements and 1 hash function has a false positive change of 43%
414 2012-06-13 13:53:40 <jgarzik> I suppose for P2P they are simply uploading a prebuilt filter, yes?
415 2012-06-13 13:53:40 <yellowhat> the number of elements = the number of addresses the cleint is interested in, right?
416 2012-06-13 13:53:41 <TD> 3G is encrypted and heavily NATd already, so realistically the privacy loss is limited to your carrier, who (at this point in time) certainly does not care about bitcoin. on open wifi at a bitcoin conference maybe you do care :)
417 2012-06-13 13:53:50 <jgarzik> vs. RPC, which exposes individual element addition
418 2012-06-13 13:53:51 <sipa> note that that is 43% of *all* transactions coming through
419 2012-06-13 13:53:53 <TD> jgarzik: yeah
420 2012-06-13 13:54:08 <BlueMatt> TD: obviously, sorry, I clearly dont know how bloom works, give me a few minutes
421 2012-06-13 13:54:26 <wumpus> the precise filters *themselves* are marginally bigger to send
422 2012-06-13 13:55:03 <jgarzik> as its a prebuilt filter, that might expose some implementation details, I'm guessing
423 2012-06-13 13:55:16 <TD> BlueMatt: np. they're a somewhat obscure data structure. i first encountered them when learning about bigtable. before that i'd never heard of them
424 2012-06-13 13:55:20 <wumpus> just an array of bits
425 2012-06-13 13:56:16 <BlueMatt> trying to learn about bloom filters while trying to listen to Jamie Dimon testifying isnt easy...
426 2012-06-13 13:56:37 <TD> BlueMatt: executive summary is this
427 2012-06-13 13:56:38 <sipa> with a 64 KiB filter and 16 hash functions, you could store up to 10000 elements with negligable FP
428 2012-06-13 13:56:53 <TD> BlueMatt: a filter is just a large array of bits. you build them up by switching bits on in particular patterns
429 2012-06-13 13:57:17 <TD> once built, you can in constant time test if an element was inserted into the filter, where "element" is defined by some hash function
430 2012-06-13 13:57:20 <TD> it can have false positives
431 2012-06-13 13:57:23 <TD> but not false negative
432 2012-06-13 13:57:46 <BlueMatt> ahh...ok, so its pretty simple...
433 2012-06-13 13:57:48 <TD> by choosing the size of the filter and other things, you can select your preferred FP / size tradeoff
434 2012-06-13 13:58:09 <sipa> but if you have a 64 KiB filter, and want to store 100000 elements, 16 hash functions will kill it
435 2012-06-13 13:58:28 <TD> eg, BigTable uses them to optimize out disk seeks. to read a bit of data you need to find which file it's in. it could be in one of several tablets. each tablet has a bloom filter associated with it. if you get an FP it means an unneeded disk seek but otherwise it's a fast way to avoid checking every file
436 2012-06-13 13:58:37 <TD> because the bloom filter can be stored in memory
437 2012-06-13 13:59:33 <sipa> if you want to store 100k elements in a 64KiB filter, you want only 4 hash functions (8% FP) and not 16 (46% FP)
438 2012-06-13 14:02:29 <Diablo-D3> hrm
439 2012-06-13 14:02:38 <Diablo-D3> sipa: btw, more hash functions doesnt improve performance
440 2012-06-13 14:02:40 <Diablo-D3> you only need two
441 2012-06-13 14:02:51 <gavinandresen> Seems to me we want two params: false positive rate and expected number of items, and derive everything else from those.  With reasonable limits on those, so I can't say "I'm going to store 10 million items and I want a 0% false positive rate, please"
442 2012-06-13 14:02:52 <Diablo-D3> and the second one can be an extra weak one that just continues processing from the first one one additional step
443 2012-06-13 14:03:37 <jgarzik> gavinandresen: the main limit is final allocated size.  you can limit FP rate / number of items via that.
444 2012-06-13 14:03:53 <gavinandresen> jgarzik: ACK
445 2012-06-13 14:03:54 <jgarzik> probably just need nitems inside 32-bit integer.
446 2012-06-13 14:05:54 <Diablo-D3> td, blueMatt, sipa: whats the math on a 64 bit filter with 2 functions, how many entries can I have until I hit, say, 50% failure
447 2012-06-13 14:05:57 <jgarzik> filterinit <nitems> <FP rate from 0.0 to 1.0> ; returns byte (not bit) size, as a verification check
448 2012-06-13 14:06:04 <jgarzik> filterload <data>
449 2012-06-13 14:06:17 <jgarzik> filterclear
450 2012-06-13 14:06:56 <jgarzik> maybe 'filteradd <hash data>' for manual building
451 2012-06-13 14:09:03 <sipa> Diablo-D3: 64 bit, not kilobit?
452 2012-06-13 14:09:05 <gavinandresen> jgarzik: ACK.  I think.  what is   the <data> in filterload ?   set of bit-masks, hex-encoded ?
453 2012-06-13 14:09:21 <jgarzik> bits in the bloom filter
454 2012-06-13 14:09:41 <jgarzik> a big chunk of data (64k as sipa was saying, for example)
455 2012-06-13 14:09:51 <Diablo-D3> sipa: bit.
456 2012-06-13 14:09:53 <gribble> New news from bitcoinrss: Diapolo opened pull request 1448 on bitcoin/bitcoin <https://github.com/bitcoin/bitcoin/pull/1448>
457 2012-06-13 14:10:17 <Diablo-D3> sipa: using only two hashes
458 2012-06-13 14:10:56 <gavinandresen> jgarzik: right, set of bits, padded to 8-bit-length?   And somewhere the hash functions used will be defined?
459 2012-06-13 14:11:21 <jgarzik> gavinandresen: yeah they would certainly have to be fixed in stone, for everybody to generate and understand the same data
460 2012-06-13 14:11:30 <jgarzik> on client and server side both
461 2012-06-13 14:11:46 <wumpus> yes, better get that decision right in one go :)
462 2012-06-13 14:11:48 <sipa> i wonder whether setting a nonce to be used for the hash functions is useful
463 2012-06-13 14:13:08 <sipa> Diablo-D3: 1 element: 0.1% FP; 2: 0.4%, 4: 2%, 8: 5%, 16: 15%, 32: 40%
464 2012-06-13 14:13:51 <Diablo-D3> wow 32? better than I thought
465 2012-06-13 14:13:54 <sipa> jgarzik: what if the size or hashes limit is reached when filterinit is called?
466 2012-06-13 14:14:05 <BlueMatt> http://upload.wikimedia.org/wikipedia/commons/e/ef/Bloom_filter_fp_probability.svg
467 2012-06-13 14:14:16 <sipa> jgarzik: best-effort, or failure?
468 2012-06-13 14:14:17 <BlueMatt> #bits defined per line, n == numer of items
469 2012-06-13 14:14:32 <jgarzik> sipa: failure.  client must know how to match server bit-for-bit here.
470 2012-06-13 14:14:41 <BlueMatt> Diablo-D3: ^
471 2012-06-13 14:14:50 <jgarzik> IMHO:)
472 2012-06-13 14:14:54 <sipa> jgarzik: wow no - the filter itself should encode its parameters
473 2012-06-13 14:15:11 <sipa> filter init calculation requires floating point math
474 2012-06-13 14:15:18 <sipa> you cannot expect others to redo that correctly
475 2012-06-13 14:15:21 <Diablo-D3> p is bits, n is items?
476 2012-06-13 14:15:27 <jgarzik> sipa: that's why it returns size (see above)
477 2012-06-13 14:15:30 <BlueMatt> Diablo-D3: p is prob, n is items
478 2012-06-13 14:15:40 <Diablo-D3> oh
479 2012-06-13 14:15:50 <BlueMatt> Diablo-D3: first line is 8 bits, second 12, third 16...
480 2012-06-13 14:15:50 <sipa> jgarzik: not what i mean
481 2012-06-13 14:15:55 <Diablo-D3> oh wtf
482 2012-06-13 14:15:55 <jgarzik> sipa: but given size, client must bit-for-bit create the binary bloom filter bits properly, for loading via filterload
483 2012-06-13 14:15:59 <Diablo-D3> that only goes up to 36 bits?!
484 2012-06-13 14:16:24 <sipa> Diablo-D3: 36 bits filter = half a gigabyte
485 2012-06-13 14:16:26 <BlueMatt> yea and with 1*10^9 items it has prob < 1*10^-10
486 2012-06-13 14:16:33 <sipa> Diablo-D3: your use case is 6 bits
487 2012-06-13 14:16:46 <Diablo-D3> sipa: erm
488 2012-06-13 14:16:49 <sipa> (it's the size of the *indices* in the table)
489 2012-06-13 14:16:54 <sipa> not the size of the table
490 2012-06-13 14:16:58 <Diablo-D3> maybe I described my problem wrong
491 2012-06-13 14:17:08 <sipa> ok, how large is your filter?
492 2012-06-13 14:17:23 <Diablo-D3> 2 hashes, filter is 64 bits, input of each item is 64 bits
493 2012-06-13 14:17:40 <sipa> do you mean the filter data structure uses 64 bits of memory?
494 2012-06-13 14:17:53 <Diablo-D3> yes, its a uint64_t.
495 2012-06-13 14:18:11 <sipa> ok, so the indices in your table are 6 bits large
496 2012-06-13 14:18:34 <sipa> as in, you will calculate a hash, and use the 6 lower bits of that hash to find an entry in the 64-bit vector
497 2012-06-13 14:18:56 <sipa> jgarzik: the serialized filter should encode its own size, number of hashes, and nonce (maybe)
498 2012-06-13 14:19:05 <Diablo-D3> sipa: actually, I was %ing that.
499 2012-06-13 14:19:32 <sipa> jgarzik: if you calculate size and number of hashes from expected number of elements and FP chance, you'll always get rounding errors (obviously)
500 2012-06-13 14:19:38 <sipa> jgarzik: that's not a problem
501 2012-06-13 14:19:41 <Diablo-D3> filter |= 1L << (hash % 64)
502 2012-06-13 14:19:54 <sipa> jgarzik: but after a certain point you cannot approximate the requested parameters anymore
503 2012-06-13 14:20:16 <sipa> Diablo-D3: yes, m=64 in your case, so log_2(m) is 6
504 2012-06-13 14:20:24 <Diablo-D3> sipa: ahh, I get it now
505 2012-06-13 14:20:25 <sipa> Diablo-D3: and k=2
506 2012-06-13 14:20:44 <Diablo-D3> and also I lied
507 2012-06-13 14:20:50 <Diablo-D3> 
508 2012-06-13 14:20:52 <Diablo-D3> thats what Im doing
509 2012-06-13 14:21:04 <sipa> Diablo-D3: don't do that
510 2012-06-13 14:21:19 <Diablo-D3> why not? it seems to generate far less false positives
511 2012-06-13 14:21:23 <sipa> using the entire 64 bits for both hashes has better performance
512 2012-06-13 14:22:06 <sipa> Diablo-D3: you're just implementing two k=1 m=32 filters instead of one k=2 m=64 one
513 2012-06-13 14:22:24 <sipa> gtg
514 2012-06-13 14:22:33 <Diablo-D3> with what I wrote, two threads that try to maximally conflict are doing 15-16 million tx/sec on one or two threads
515 2012-06-13 14:23:00 <Diablo-D3> doing % 64 on both does...
516 2012-06-13 14:23:22 <Diablo-D3> 15-16 on single thread, 14-15 on two thread
517 2012-06-13 14:23:29 <Diablo-D3> due to false positives
518 2012-06-13 14:23:47 <sipa> how many elements do you put in it?
519 2012-06-13 14:23:52 <Diablo-D3> very few
520 2012-06-13 14:24:01 <sipa> how many, typically?
521 2012-06-13 14:24:16 <Diablo-D3> well, lets say Im writing to one memory location in each thread, and I have two cores (ergo two threads)
522 2012-06-13 14:24:21 <Diablo-D3> thats two items
523 2012-06-13 14:24:43 <Diablo-D3> so its memory locations * threads running txes at any given moment in that process
524 2012-06-13 14:24:59 <sipa> you could easily use more hash functions and that would reduce false positives further
525 2012-06-13 14:25:01 <gribble> New news from bitcoinrss: laanwj opened pull request 1449 on bitcoin/bitcoin <https://github.com/bitcoin/bitcoin/pull/1449>
526 2012-06-13 14:25:13 <Diablo-D3> yeah, but then I just fill it up with bits
527 2012-06-13 14:25:19 <sipa> you should
528 2012-06-13 14:25:28 <sipa> bloom filters work best when they're half full
529 2012-06-13 14:25:36 <Diablo-D3> yeah, but what if someone breaks out a 64 core machine?
530 2012-06-13 14:25:48 <Diablo-D3> or writes to a lot of seperate memory locations
531 2012-06-13 14:26:02 <sipa> then you should use less hash functions
532 2012-06-13 14:26:07 <Diablo-D3> !#%QEARTFDSVCX
533 2012-06-13 14:26:08 <gribble> Error: "#%QEARTFDSVCX" is not a valid command.
534 2012-06-13 14:26:10 <Diablo-D3> but you just said use more!
535 2012-06-13 14:26:22 <sipa> if you're only putting in two elements, yes, definitely
536 2012-06-13 14:26:32 <Diablo-D3> I dont know what the user is going to do
537 2012-06-13 14:26:38 <Diablo-D3> I just know what the typical and worst cases are
538 2012-06-13 14:26:40 <sipa> the optimal number is 44/n, with n the number of elements in it
539 2012-06-13 14:28:10 <Diablo-D3> btw, one hash, 15-16 on single thread, 12 on two :<
540 2012-06-13 14:38:12 <BlueMatt> is there a bip for bloom filters in p2p, or is someone gonna write one?
541 2012-06-13 14:38:18 <BlueMatt> jgarzik: I guess?
542 2012-06-13 14:40:13 <gribble> New news from bitcoinrss: Diapolo opened pull request 1451 on bitcoin/bitcoin <https://github.com/bitcoin/bitcoin/pull/1451> || Asrael opened issue 1450 on bitcoin/bitcoin <https://github.com/bitcoin/bitcoin/issues/1450>
543 2012-06-13 14:45:20 <gribble> New news from bitcoinrss: Diapolo opened issue 1452 on bitcoin/bitcoin <https://github.com/bitcoin/bitcoin/issues/1452>
544 2012-06-13 14:54:27 <Habbie> hi - my testnet-in-a-box connects to IRC and finds the rest of the testnet, even though irc=0 is set in 1/bitcoin.conf; is this a known issue?
545 2012-06-13 14:54:55 <Habbie> nevermind - my bitcoin binaries are very old
546 2012-06-13 15:32:36 <Habbie> my testnet-in-a-box is not generating blocks - i am mining at approx 3 mhash/sec; how often should i expect a block, on average?
547 2012-06-13 15:33:19 <D34TH> 3 mhash?
548 2012-06-13 15:33:22 <Habbie> or well, i have one block, that was generated half an hour ago, apparently - but the tx in it is marked immature (obviously)
549 2012-06-13 15:33:37 <D34TH> well the network speed is a bit quicker
550 2012-06-13 15:33:38 <D34TH> so
551 2012-06-13 15:33:47 <D34TH> if your solo you wont find any
552 2012-06-13 15:34:07 <Habbie> if i can't find any, what's the point of testnet in a box?
553 2012-06-13 15:34:25 <Habbie> "difficulty" : 0.06249911,
554 2012-06-13 15:34:35 <D34TH> your not synced up
555 2012-06-13 15:34:43 <Habbie> testnet in a box is not supposed to sync up
556 2012-06-13 15:34:45 <D34TH> "difficulty":16.0000000,
557 2012-06-13 15:35:07 <D34TH> oh your on your own, it will be a while at 3 mh/s
558 2012-06-13 15:35:17 <Habbie> do you know how to calculate how long?
559 2012-06-13 15:36:18 <Habbie> according to the online (non-testnet) difficulty calculator i should be getting a block every 1-2 minutes
560 2012-06-13 15:36:21 <Habbie> obviously, that isn't happening :)
561 2012-06-13 15:36:29 <D34TH> unlucky?
562 2012-06-13 15:36:34 <Habbie> could be
563 2012-06-13 15:36:40 <Habbie> but i've been at it for 30 minutes ;)
564 2012-06-13 15:38:12 <Habbie> i've done setgenerate true on the daemon now; 1 mhash/sec there
565 2012-06-13 15:38:42 <Habbie> should yield a block every 5 minutes according to non-testnet rules
566 2012-06-13 15:43:53 <Habbie> ah!
567 2012-06-13 15:43:55 <Habbie> generated a block
568 2012-06-13 15:43:58 <Habbie> with the 3mhash/sec miner
569 2012-06-13 15:50:35 <someone42> does anyone know of any non-testnet blocks which contain P2SH transactions?
570 2012-06-13 16:02:44 <luke-jr> someone42: 178118 has one that breaks 0.4 and 0.5
571 2012-06-13 16:34:59 <someone42> luke-jr: i can't seem to find any P2SH transactions in block 178118
572 2012-06-13 16:37:30 <luke-jr> 177618 sorry
573 2012-06-13 16:37:41 <luke-jr> http://blockchain.info/tx-index/4530401/968a692ab98b1f275c635c76be003ab1db9740d0b62f338b270115342ca42f5b
574 2012-06-13 16:39:59 <someone42> thanks
575 2012-06-13 16:47:24 <Habbie> bah, 120 confirms to mature a block
576 2012-06-13 16:47:27 <Habbie> this will take a while ;)
577 2012-06-13 16:49:37 <gavinandresen> Habbie: do you need new blocks?  the first testnet-in-a-box node has a wallet with a few thousand coins in it
578 2012-06-13 17:21:31 <ssfd> anyone buying
579 2012-06-13 17:37:53 <Habbie> gavinandresen, i downloaded testnet3-box.zip from http://sourceforge.net/projects/bitcoin/files/Bitcoin/testnet-in-a-box/ and the balance for both datadir=1 and =2 is 0, and the only block i see is the genesis block
580 2012-06-13 17:38:15 <gavinandresen> Habbie: are you running git HEAD ?
581 2012-06-13 17:38:45 <Habbie> gavinandresen, 0.6.2 from the ubuntu ppa
582 2012-06-13 17:39:07 <gavinandresen> For 0.6.2 you'll need the old testnet-box.zip
583 2012-06-13 17:39:19 <Habbie> ah!
584 2012-06-13 17:39:34 <gavinandresen> (we're resetting testnet for the 0.7 release, please excuse the mess)
585 2012-06-13 17:40:18 <Habbie> that's what testnet is for ;)
586 2012-06-13 17:40:37 <Habbie> 3650 BTC
587 2012-06-13 17:40:40 <Habbie> thanks!
588 2012-06-13 17:42:59 <Habbie> gavinandresen, just for my curiosity, would testnet3-box + git head also have given me a bunch of BTC?
589 2012-06-13 17:43:09 <gavinandresen> Habbie: yes
590 2012-06-13 17:43:12 <Habbie> thanks
591 2012-06-13 18:15:52 <jgarzik> heavy TX day?
592 2012-06-13 18:15:57 <jgarzik> 2400 tx's in pool right now
593 2012-06-13 18:17:30 <Joric> fucking satoshidice!
594 2012-06-13 18:20:21 <sipa> i wonder... i suppose transfer size for bloom filters is more of a problem than memory size?
595 2012-06-13 18:21:04 <sipa> i believe you can create a very sparsely used filter, but use a compressed encoding
596 2012-06-13 18:22:28 <BlueMatt> if you're looking for low fp %, Id think so
597 2012-06-13 18:23:15 <sipa> sounds like a fun experiment :)
598 2012-06-13 18:23:24 <BlueMatt> but I wouldnt think you always want a low fp % here, 50% or more for some more privacy-conscious clients seems reasonable...
599 2012-06-13 18:23:44 <sipa> then just use a smaller one
600 2012-06-13 18:23:56 <sipa> high is strictly easier
601 2012-06-13 18:24:05 <sipa> high fp, i mean
602 2012-06-13 18:24:18 <BlueMatt> well Im saying if you use a smaller one, making a fancy encoding scheme isnt worth it
603 2012-06-13 18:24:34 <sipa> of course
604 2012-06-13 18:24:48 <sipa> but sometimes you want low fp
605 2012-06-13 18:25:02 <BlueMatt> and even a fancy encoding scheme may not be worth anything for high fp
606 2012-06-13 18:25:25 <BlueMatt> though for mobiles (who are probably the ones who will want low fp) saving bw may be worth it...
607 2012-06-13 18:26:09 <sipa> my point is this: bloom filters are intended to minimize memory usage, not to minimize an encoded representation of it
608 2012-06-13 18:27:24 <BlueMatt> yea, for mobiles who want low fp, you may be able to save significant transfer space
609 2012-06-13 18:28:44 <Habbie> what's fp?
610 2012-06-13 18:28:50 <sipa> false positive rate
611 2012-06-13 18:28:51 <BlueMatt> false positive rate
612 2012-06-13 18:29:24 <sipa> and those who want low fp, are exactly those for whom a small transfer size matters
613 2012-06-13 18:29:30 <BlueMatt> yep
614 2012-06-13 18:29:33 <Habbie> ah, right
615 2012-06-13 18:30:26 <sipa> i have some errands first, but i'll do the math on how small they can theoretically get later
616 2012-06-13 18:51:42 <BlueMatt> jgarzik: so I get this right, filter* will filter out blocks and send only block headers + individual CMerkleTxs (or similar encoding)
617 2012-06-13 18:51:45 <BlueMatt> ?
618 2012-06-13 18:52:31 <BTCHero> I just created a new receive addy and it shows transactions from months ago that I am sure I didn't do.  Could anyone lend some insight as to what is happening here?
619 2012-06-13 19:09:43 <nanotube> BTCHero: can you check if 1FYoNrSGDeAEFq9AqjLJuPn32in1L56U3o , 1Lh7MxCnVCv2c52hpXUzrHN3ridc9nc11e, or 1M9L4XWBxLihmiVsqio1cNaatWYoEFYB9G any of them are yours also?
620 2012-06-13 19:10:03 <danieldaniel> nanotube: Lol@telling him to come here, then answering him here :D
621 2012-06-13 19:10:22 <BTCHero> yeah just a sec, wallet is on other comp
622 2012-06-13 19:10:50 <nanotube> danieldaniel: just had an idea :)
623 2012-06-13 19:10:55 <danieldaniel> nanotube: :)
624 2012-06-13 19:12:26 <BTCHero> Yeah the first one is, it must have just used it as a temp and then gave it to me anyway
625 2012-06-13 19:12:33 <BTCHero> I haven't checked the others
626 2012-06-13 19:12:40 <sipa> jgarzik: i don't see how filterinit can be P2P command
627 2012-06-13 19:13:12 <BlueMatt> win32 auto-update anyone? https://github.com/bitcoin/bitcoin/pull/1453
628 2012-06-13 19:13:18 <sipa> BlueMatt: that's the idea
629 2012-06-13 19:13:22 <nanotube> BTCHero: ye if the first one is yours, that means it was one of your change addrs or something. because first one plus your 'mystery' addr both used as inputs in the same tx. http://blockexplorer.com/tx/be9316945b10203021ca50bed84f010d5a0f0fc2736f1680ac9dc31abccc13f7#i4991862
630 2012-06-13 19:14:08 <BlueMatt> oh, oops need to change path-specs...
631 2012-06-13 19:15:05 <BTCHero> Thanks for clearing that up... But they should implement something to avoid this.  I'm glad it wasn't really acollision.  Also, the wallet should report when the txn actually occurred instead of when it saw it.
632 2012-06-13 19:15:11 <BTCHero> but i don't really know what I am talking about
633 2012-06-13 19:16:00 <gribble> New news from bitcoinrss: TheBlueMatt opened pull request 1453 on bitcoin/bitcoin <https://github.com/bitcoin/bitcoin/pull/1453>
634 2012-06-13 19:16:33 <sipa> what is http://localhost:8080/latestversion.txt ?
635 2012-06-13 19:16:44 <nanotube> BTCHero: well i dunno what really happened - see if you can get one of the real devs to figure it out :)
636 2012-06-13 19:16:45 <sipa> does gitian have a http interface...?
637 2012-06-13 19:18:17 <BlueMatt> oops...should be bitcoin.org/latestversion.txt
638 2012-06-13 19:18:21 <BTCHero> nanotube: hopefully they see this and do something. It freaked me out a little, and I sort of understand how it works
639 2012-06-13 19:18:28 <BlueMatt> guess I pulled the trigger on the pull a few minutes too early
640 2012-06-13 19:18:56 <BTCHero> AND if it reported when a txn actually occurd I would have seen that I did a txn at the exact same time and not been too worried and assumed it was a recycled addy
641 2012-06-13 19:19:38 <nanotube> well, maybe BlueMatt will take a look </ping> :)
642 2012-06-13 19:20:14 <sipa> BlueMatt: ... it doesn't use gitian?
643 2012-06-13 19:20:17 <BlueMatt> it does
644 2012-06-13 19:20:25 <sipa> but just for updating, not for checking?
645 2012-06-13 19:20:31 <BlueMatt> just for updating, yea
646 2012-06-13 19:20:34 <sipa> hmmm
647 2012-06-13 19:20:40 <BlueMatt> because otherwise we download the latest-zip every time we launch...
648 2012-06-13 19:20:47 <BTCHero> BlueMatt: are there any plans to not use previously used addys?
649 2012-06-13 19:21:02 <BlueMatt> nanotube/BTCHero: whats up?
650 2012-06-13 19:21:04 <gribble> New news from bitcoinrss: TheBlueMatt opened pull request 38 on bitcoin/bitcoin.org <https://github.com/bitcoin/bitcoin.org/pull/38>
651 2012-06-13 19:21:15 <sipa> nanotube: can you explain BTCHero's problem?
652 2012-06-13 19:21:26 <nanotube> BlueMatt: he hit newaddress, and the address apparently was one that was already used in a transaction a few months ago.
653 2012-06-13 19:21:29 <nanotube> sipa: ^
654 2012-06-13 19:21:38 <nanotube> seems like it was one of the change addresses or something
655 2012-06-13 19:21:40 <BTCHero> I made a new receive addy and it was already used, it freaked me out a little because my wallet didn't have the real time of the txn in it
656 2012-06-13 19:21:46 <nanotube> so the question is, is that normal behavior?
657 2012-06-13 19:21:49 <sipa> a change address is never reused
658 2012-06-13 19:21:51 <nanotube> that 'new address' would give you a used one?
659 2012-06-13 19:21:52 <BlueMatt> thats...odd...
660 2012-06-13 19:22:08 <BTCHero> I thought so
661 2012-06-13 19:22:08 <sipa> it shouldn't ever give you an address it has used itself before
662 2012-06-13 19:22:27 <sipa> if you would dump your wallet.dat file, extract addresses in it, and manually use a keypool address from it
663 2012-06-13 19:22:28 <nanotube> (i did ask him if he's sure that he didn't make a mistake and clicked on an addr than was not the one that was freshly generated..>)
664 2012-06-13 19:22:40 <sipa> in that case, i suppose you'll later reuse the same address
665 2012-06-13 19:23:08 <BTCHero> no, I do nothing out of the ordinary
666 2012-06-13 19:23:19 <BTCHero> and I am very confident that I clicked new address
667 2012-06-13 19:23:45 <sipa> bitcoin-qt doesn't hilight or jump to the newly added address, afaik
668 2012-06-13 19:24:39 <BTCHero> entered the label and everything then checked it on blockchain.info to see if a txn came through I was expecting and saw one from months ago
669 2012-06-13 19:25:24 <BTCHero> If I search the address at the top, nothing appears either so I didn't change the name of a previous receive addy
670 2012-06-13 19:26:20 <BTCHero> I don't know how interested you are, if at all.  But if you want more info let me know.
671 2012-06-13 19:26:35 <BlueMatt> sipa: there, bit better now, just have to add the (binary :( ) gitian-updater.exe and double-check the like 5 licenses to make sure we can redist
672 2012-06-13 19:27:18 <BTCHero> Oh another thing that might be the cause, This was a backup wallet that I moved to this computer
673 2012-06-13 19:27:31 <BTCHero> that must be what happened?
674 2012-06-13 19:31:47 <justmoon> is there a blockexplorer yet for testnet3?
675 2012-06-13 19:32:54 <nanotube> hey justmoon :) not that i know of.
676 2012-06-13 19:33:12 <nanotube> BTCHero: is that the only address with that label?
677 2012-06-13 19:33:42 <justmoon> nanotube: hey, how's it goin'? I had to think of you the other day about when you introduced yourself at the new york conference and everyone was cheering :D
678 2012-06-13 19:34:14 <BTCHero> yep, it was very specific
679 2012-06-13 19:34:32 <sipa> hey there justmoon :)
680 2012-06-13 19:34:47 <BlueMatt> sipa: updated the list of deps on https://github.com/bitcoin/bitcoin/pull/1453 including the list of binaries that it depends on, is that list unreasonable, should time be spent trying to gitian-ify some of those?
681 2012-06-13 19:34:59 <justmoon> sipa: I fixed the microphone stand - drilled a hole, put a bolt through, rock solid now
682 2012-06-13 19:35:06 <sipa> justmoon: hahah!