1 2012-05-17 00:05:04 <gribble> New news from bitcoinrss: rebroad opened pull request 1326 on bitcoin/bitcoin <https://github.com/bitcoin/bitcoin/pull/1326>
  2 2012-05-17 00:45:53 <gmaxwell> I noticed that we're not using stack-protector with the mingw builds. We probably should be. (I'm not sure if relro/pie works on mingw but if so we should do that oo)
  3 2012-05-17 00:46:54 <gmaxwell> ... though I noticed this while trying to fix a library I work on where fstack-protector is giving me problems with mingw32  it ends up adding a dependency on libssp-0.dll, which wouldn't really be a problem for bitcoin, but it's annoying for my other package.
  4 2012-05-17 00:47:09 <gmaxwell> Anyone aware of any history w/ fstack-protector and mingw on bitcoin?
  5 2012-05-17 02:04:08 <jgarzik> Here is a C++/Boost demo program, which appends a line of text to a gzip-compressed file: https://gist.github.com/2716209
  6 2012-05-17 02:04:27 <jgarzik> is there anyone online that can build and test this program on Windows/mingw?
  7 2012-05-17 02:04:51 <jgarzik> even building on mingw, without testing, would be useful
  8 2012-05-17 02:17:31 <gmaxwell> jgarzik: it builds...
  9 2012-05-17 02:19:08 <gmaxwell> (e.g. in fedora 17 with mingw32* installed, i686-w64-mingw32-g++  -O2 -Wall -g -o boostgz.exe boostgz.cc /usr/i686-w64-mingw32/sys-root/mingw/lib/libboost_iostreams-gcc47-mt-1_48.dll.a  builds it
 10 2012-05-17 02:34:07 <jgarzik> gmaxwell: thanks
 11 2012-05-17 02:34:29 <jgarzik> that means using zlib stuffs does not require an additional link target in Windows
 12 2012-05-17 02:35:12 <gmaxwell> yea, it picks up the dependency somehow... I guess via boost.
 13 2012-05-17 02:35:36 <jgarzik> yep
 14 2012-05-17 02:36:03 <gmaxwell> wine reports it needs, libbz2-1.dll, libstdc++-6.dll, zlib1.dll, and boost_iostreams-gcc47-mt-1_48.dll
 15 2012-05-17 02:36:12 <jgarzik> peers.dat is on average 80%+ compressible with zlib
 16 2012-05-17 02:36:44 <gmaxwell> Hm. thats a little surprising.
 17 2012-05-17 02:37:18 <gmaxwell> (I mean, the ipv6 encoding must pad it up a good bit.. but still)
 18 2012-05-17 02:37:30 <jgarzik> so bdb or non-bdb, you're blipping close to 1MB of data to disk periodically.  in terms of bytes-written in a 10 minute window, CAddrDB probably writes more than other bitcoin sources
 19 2012-05-17 02:37:46 <jgarzik> lotta empty buckets would be my guess
 20 2012-05-17 02:38:02 <gmaxwell> yea, but older nodes will fill those up... making the savings go away.
 21 2012-05-17 02:38:03 <jgarzik> 32-bit ints with less than 8 bits used
 22 2012-05-17 02:38:34 <jgarzik> agree the savings will probably diminish
 23 2012-05-17 02:38:46 <gmaxwell> there probably isn't a great need to write it more than a couple times a day..
 24 2012-05-17 02:39:02 <gmaxwell> jgarzik: would you like an addrman addr.dat from a matureish node?
 25 2012-05-17 02:39:22 <jgarzik> the thread that calls DumpAddresses() does a sleep(100000)
 26 2012-05-17 02:40:21 <jgarzik> but some other source seems to trigger it more frequently than every 1666 minutes
 27 2012-05-17 02:40:56 <Diablo-D3> jgarzik: did you just say interrupt a thread? bad jgarzik
 28 2012-05-17 02:41:19 <jgarzik> gmaxwell: nah, too lazy to write a patch to make it work
 29 2012-05-17 02:41:50 <jgarzik> gmaxwell: my current solution is more optimal than BDB even without gzip, so no big deal
 30 2012-05-17 02:42:47 <jgarzik> current HEAD serializes entire CAddrman to a single, huge BDB key/value database record
 31 2012-05-17 02:44:59 <jgarzik> gmaxwell: what is the byte size of that mature addr.dat?
 32 2012-05-17 02:45:05 <jgarzik> octet count
 33 2012-05-17 02:45:19 <gmaxwell> oh, you don't want it in 9 bit bytes??
 34 2012-05-17 02:45:30 <jgarzik> ;)
 35 2012-05-17 02:45:58 <gmaxwell> one here is 4153344 octets.
 36 2012-05-17 02:46:19 <gmaxwell> might shrink if I vacuum it with dbdump
 37 2012-05-17 02:47:43 <jgarzik> gmaxwell: any easy way to view the size of the 'addrman' value in there?
 38 2012-05-17 02:48:09 <gmaxwell> lemme shut down the node and dbdump it.. the big record should be obvious.
 39 2012-05-17 02:50:12 <gmaxwell> well, there is one line in the dump file that has 1426038 hex characters... (the rest have two dozen or so)
 40 2012-05-17 02:51:33 <gmaxwell> hm, no my editor can't count.
 41 2012-05-17 02:51:47 <gmaxwell> oh yes it can.
 42 2012-05-17 02:52:32 <jgarzik> ok, so that's the amount of data a node currently shoves to disk periodically
 43 2012-05-17 02:52:52 <jgarzik> *amount of address data
 44 2012-05-17 02:58:17 <jgarzik> luke-jr: not if lib A internally includes lib B, no
 45 2012-05-17 02:59:36 <gmaxwell> I assume we already have a zlib (ugh) dependency from the png icons.
 46 2012-05-17 03:00:05 <jgarzik> openssl too
 47 2012-05-17 03:00:11 <gmaxwell> Though for some reason we've been fortunate enough to escape people insisting we rebuild for every weekly zlib remote code vulnerability.
 48 2012-05-17 03:00:34 <luke-jr> 37 KB 009494e944a5ed54ca611e9eff6361e0d3e85411d9013d9558514dd8011b2fd0 bin-osx-miniupnpc-1.6.tbz2
 49 2012-05-17 03:00:54 <luke-jr> err
 50 2012-05-17 03:01:00 <luke-jr> alreayd did that one <.<
 51 2012-05-17 03:01:08 <luke-jr> 76 KB d3c460a55983f33d2892dbb6240293bbb12f3055f0c67333c5930b4d3e2b6a24 bin-osx-zlib-1.2.6.tbz2 <-- new one
 52 2012-05-17 03:08:57 <luke-jr> rebuilt miniupnpc, and it seems to be making deterministic properly
 53 2012-05-17 03:09:04 <luke-jr> even if the cross-compiler itself isn't.
 54 2012-05-17 03:20:21 <gmaxwell> awesome, 30MiB of the chain is SatoshiDice, tycho's spamming has some competition!
 55 2012-05-17 03:21:23 <luke-jr> >_<
 56 2012-05-17 03:21:26 <luke-jr> at least they pay fees?
 57 2012-05-17 03:22:06 <gmaxwell> Yea, 36.8 BTC so far in fact.
 58 2012-05-17 03:24:49 <gmaxwell> If we assume 40k full nodes thats about 1.1 TB of data... to represent a bunch of silly people losing their money.
 59 2012-05-17 03:25:15 <gmaxwell> They could have just sent the money to the well known bitcoin eater address and saved a bunch of time and storage.
 60 2012-05-17 03:25:23 <Diablo-D3> lol
 61 2012-05-17 03:26:33 <ThomasV> if they pay fees then they are useful
 62 2012-05-17 03:27:13 <ThomasV> it's good to see the fee/generation ratio increase
 63 2012-05-17 03:27:20 <gmaxwell> The operator has made 108 btc profit. or about 8.5e11 BTC per byte they added to the global storage under the 40k assumption.
 64 2012-05-17 03:27:47 <gmaxwell> er 8.5e-11 (obviously)
 65 2012-05-17 03:27:52 <ThomasV> lol
 66 2012-05-17 03:28:44 <gmaxwell> Well, at least the txns tend to be fairly prunable on the dice side.
 67 2012-05-17 03:28:51 <gmaxwell> (not so much on the customer side)
 68 2012-05-17 05:12:11 <gribble> New news from bitcoinrss: jddebug opened issue 1327 on bitcoin/bitcoin <https://github.com/bitcoin/bitcoin/issues/1327>
 69 2012-05-17 07:57:43 <ronaz> j/b 31
 70 2012-05-17 07:59:31 <sipa> ?
 71 2012-05-17 08:06:09 <ronaz> i do that a lot. keyboard on my lap and browsing through channels.
 72 2012-05-17 08:14:08 <SomeoneWeird> Guys in your opinion, what's the best way to auth against an api?
 73 2012-05-17 08:15:14 <freewil> using an hmac of the request is nice and simple
 74 2012-05-17 08:15:20 <gribble> New news from bitcoinrss: laanwj opened pull request 1329 on bitcoin/bitcoin <https://github.com/bitcoin/bitcoin/pull/1329> || ahf opened pull request 1328 on bitcoin/bitcoin <https://github.com/bitcoin/bitcoin/pull/1328>
 75 2012-05-17 08:15:50 <SomeoneWeird> hmm
 76 2012-05-17 08:17:36 <freewil> you can then require a nonce in the request and include that as part of the hmac
 77 2012-05-17 08:17:51 <freewil> then that same request cant be used again
 78 2012-05-17 08:17:57 <freewil> a new hmac would have to be generated
 79 2012-05-17 08:19:45 <SomeoneWeird> basically what i was thinking is that each user gets an id and a secret key
 80 2012-05-17 08:20:05 <SomeoneWeird> on each request they post the id, along with a hash of their secret key and the requested url
 81 2012-05-17 08:20:39 <SomeoneWeird> which means that the secret key doesnt get transferred, and that if thats intercepted it can only be used for that 1 url
 82 2012-05-17 08:21:17 <freewil> yeah thats pretty much what hmac is
 83 2012-05-17 08:21:43 <sipa> SomeoneWeird: hmac is really a hash with a shared secret key
 84 2012-05-17 08:21:55 <SomeoneWeird> ahah alright, cool
 85 2012-05-17 08:21:57 <sipa> and it more secure than just hashing the key and the data together
 86 2012-05-17 08:22:05 <SomeoneWeird> hmm
 87 2012-05-17 08:22:38 <freewil> if you add in the nonce concept then even if the request is intercepted it cant be used again even for just that one url
 88 2012-05-17 08:23:15 <freewil> because whoever interecepted it would have to use a new nonce with a higher value and would need to know the secret key
 89 2012-05-17 08:23:28 <SomeoneWeird> right, and whats the best way to generate/validate nonces?
 90 2012-05-17 08:23:50 <freewil> i would just use an int, recommend the user start with 0
 91 2012-05-17 08:23:53 <sipa> SomeoneWeird: use the date/time, plus a random number
 92 2012-05-17 08:23:58 <freewil> and then they have to send an ever incrementing value
 93 2012-05-17 08:24:09 <freewil> what sipa said is better
 94 2012-05-17 08:24:09 <sipa> meh, that requires statefulness
 95 2012-05-17 08:24:34 <freewil> amen sipa
 96 2012-05-17 08:24:42 <sipa> here's the hmac implementation i wrote for use in deterministic wallets: https://github.com/sipa/bitcoin/blob/cfb185adf4617bc5db8385be5a40cdb228864c17/src/hmac.cpp
 97 2012-05-17 08:25:27 <ahf> Hi
 98 2012-05-17 08:26:03 <SomeoneWeird> so how do i verify that it hasn't been used before and is only for that request?
 99 2012-05-17 08:28:04 <sipa> because you never generate the same nonce twice
100 2012-05-17 08:28:22 <freewil> you would have to store the hash after each request/response and check if it's been used before each one
101 2012-05-17 08:28:28 <sipa> meh
102 2012-05-17 08:29:02 <sipa> if the timestamp is in there with millisec precision, and you add a 32-bit or 64-bit random number to it, the chance for a repeated nonce are almost zero
103 2012-05-17 08:29:27 <freewil> sipa: the concern is though if someone intercepts the request and tries to send it again
104 2012-05-17 08:29:29 <sipa> but maybe you just want oauth?
105 2012-05-17 08:30:14 <SomeoneWeird> not sure, trying to work out the best method
106 2012-05-17 08:30:47 <SomeoneWeird> basically the api is gunna be paid access, so I want a way thats simple to use, but still secure against intercepts and stuff
107 2012-05-17 08:31:13 <freewil> i would use the nonce concept, store the hashes and SSL ontop of that
108 2012-05-17 08:31:34 <SomeoneWeird> yeah sounds like the way to go
109 2012-05-17 08:34:04 <sipa> i'd use oauth
110 2012-05-17 08:34:53 <sipa> if it's for a website that provides an API service, at least
111 2012-05-17 08:36:17 <SomeoneWeird> SaaS
112 2012-05-17 08:36:35 <SomeoneWeird> so yeah
113 2012-05-17 08:57:39 <gribble> New news from bitcoinrss: fanquake opened pull request 1330 on bitcoin/bitcoin <https://github.com/bitcoin/bitcoin/pull/1330>
114 2012-05-17 09:55:34 <davout> hey there
115 2012-05-17 09:56:08 <davout> is it possible/logical that "getinfo" and "getbalance '*' 1" report different balances ??
116 2012-05-17 10:34:16 <davout> is it possible/logical that "getinfo" and "getbalance '*' 1" report different balances ??
117 2012-05-17 11:08:11 <TD> anyone got any idea of where to link this page from? https://en.bitcoin.it/wiki/Securing_online_services
118 2012-05-17 11:22:53 <gmaxwell> TD: Don't link it, it understates the bitcoin risk.
119 2012-05-17 11:23:02 <TD> how so
120 2012-05-17 11:24:37 <gmaxwell> Starting with the first sentence, ", in Bitcoin the risk is entirely theoretical and has yet to be observed in the wild"   The mybitcoin money loss was claimed to be a result of accepting unconfirmed transactions through their shopping cart interface.  And we observed quite a few duplicate input transactions circulating on the network which older nodes would display as forever unconfirmed which could have been used for that kind of attack.
121 2012-05-17 11:26:00 <gmaxwell> TD: please also read http://eprint.iacr.org/2012/248.pdf Which specifically criticizes the 'bitcoin developers' for overstating bitcoin transaction security due to claims you made on the FAQ page on the wiki.
122 2012-05-17 11:28:06 <TD> the mybitcoin case was very, very far from clear. the whole murkyness around that means it's hard to conclude anything. i think scam is a far more likely explanation. and yes, there have been double spending transactions observed. what i mean is that no credible merchant has come forward and said "I lost money because of THIS double spend right here"
123 2012-05-17 11:28:15 <TD> so i think it's totally fair to say the risk to merchants of that is very low
124 2012-05-17 11:28:23 <TD> not zero! that's why it's covered on the page
125 2012-05-17 11:28:26 <TD> if the risk was zero why even mention it ?
126 2012-05-17 11:28:33 <TD> interesting paper though.
127 2012-05-17 11:29:45 <luke-jr> makomk: shh
128 2012-05-17 11:32:02 <gmaxwell> TD: Retrospectively, if was a normal user and read that page I got ripped off I would research more and conclude that it overtly lied to me to make bitcoin sound safer than it was, because it make such an absolute statement when many people do believe that people have been exploited in that way.
129 2012-05-17 11:32:13 <gmaxwell> For the actual text "For merchants shipping things immediately, like digital downloads, measuring the propagation of a transaction across the network and waiting until most nodes have reported it is probably good enough. This allows you to accept the payment within seconds. Whilst there's a higher risk of transaction reversal here, the attack is still expensive and the attacker cannot choose the timing of the attack. Therefore watching out fo
130 2012-05-17 11:33:11 <gmaxwell> I don't agree with this recoomendation.  For the first point the software to 'measuring the propagation of a transaction' doesn't exist today and the of person who needs this FAQ entry is probably not going to write it.
131 2012-05-17 11:33:31 <TD> some people believe _one_ operation _may_ have been exploited in that way, but that belief is based only on the assertions of somebody who then immediately vanished and could not be located at all
132 2012-05-17 11:33:32 <gmaxwell> Also, it fails to make it clear that it's recommending creating a fresh payment address right before the transaction.
133 2012-05-17 11:33:35 <Eliel> TD: about the mybitcoin case, there were some messages in threads I found reading the past messages of the BitcoinATM guy that lead me to believe that mybitcoin really was emptied with a double-spend attack.
134 2012-05-17 11:33:50 <Eliel> on bitcointalk
135 2012-05-17 11:34:01 <TD> Eliel: which ones?
136 2012-05-17 11:34:13 <Eliel> messages from before mybitcoin went offline
137 2012-05-17 11:34:22 <TD> gmaxwell: actually the latest bitcoinj will let you measure that :-) but yes, the page has some recommendations that require software that does not exist (like the supervisor)
138 2012-05-17 11:34:30 <gmaxwell> TD: it isn't only based on that assertion, it's also based on the fact that at the time there were a large number of high value repeated input never-confirmable transactions circulating on the network.
139 2012-05-17 11:34:34 <TD> gmaxwell: i think it's still worth discussing them
140 2012-05-17 11:36:13 <TD> i remember people asking at the time which re-orgs were reversing the spends
141 2012-05-17 11:36:22 <gmaxwell> It also sound like listening to make sure nodes have heard the transaction is a defense against the finney attack. It is not. Nor is the finny attack expensive.
142 2012-05-17 11:36:29 <TD> as i think they said they were waiting for at least one block, right?
143 2012-05-17 11:36:38 <Eliel> TD: I think it was these threads: https://bitcointalk.org/index.php?topic=32807.0 https://bitcointalk.org/index.php?topic=33020.msg415098#msg415098
144 2012-05-17 11:37:04 <gmaxwell> TD: There wasn't reorgs needed to reverse the spends. According to nanotube (IIRC) they accepted zero confirm transactions on their shopping cart interface.
145 2012-05-17 11:37:14 <TD> it's not expensive? once you find a mined block you have to make a payment to an address that you've already been issued.
146 2012-05-17 11:37:32 <TD> unless you're finding mined blocks all the time (a major pool operator) that means there will be significant delays between when you start an order and when you complete it
147 2012-05-17 11:38:23 <luke-jr> TD: or unless you can buy a mined block within 10 minutes on average
148 2012-05-17 11:38:34 <gmaxwell> TD: The block you mine pays your coins to yourself.  You just need to be able to automate the entire order process to fire off as soon as you mine a block.  The GPUMAX service allows anyone to purchase hashing power on bitcoin on demand.
149 2012-05-17 11:39:08 <TD> the process of starting an order issues you an address. you can't mine the block without an address from the merchant
150 2012-05-17 11:39:11 <luke-jr> gmaxwell: I think TD's point is that merchants can require payment within 2 minutes of creating a new address
151 2012-05-17 11:39:22 <TD> yes. when i used bitpay yesterday the invoice had a 15 minute countdown
152 2012-05-17 11:39:34 <TD> unless i can mine a block on demand, pretty much, i wouldn't be able to outrun the expiration of the invoice
153 2012-05-17 11:39:38 <luke-jr> 15 minutes is probably good enough if GPUMAX remains < 25%
154 2012-05-17 11:39:56 <gmaxwell> TD: (1) I pointed out above that the recommendation text did not at all make the fresh address assumption clear, (2) You don't actually have to do that.
155 2012-05-17 11:40:13 <luke-jr> oh wait
156 2012-05-17 11:40:41 <gmaxwell> TD: say I have a single 10 BTC input.  I start working on a block with a txn to send that 10BTC to myself. I don't know anything about my victims addresses.
157 2012-05-17 11:40:48 <luke-jr> well, merchants can check the inputs to their transactions and demand confirmations on those
158 2012-05-17 11:41:30 <gmaxwell> When I find a block my script rapidly places an order on your site for a 1BTC song.  It uses that same 10BTC input.  As soon as you've started the download, I announce the block.
159 2012-05-17 11:42:00 <gmaxwell> The block reverses the announced transaction, even if it was well forwarded.
160 2012-05-17 11:42:26 <gmaxwell> My loss is only the increase in the chance of being orphaned.
161 2012-05-17 11:43:23 <gmaxwell> Fresh addresses are good, but they're good against isolation attacks, not finney attacks.
162 2012-05-17 11:43:42 <Diablo-D3> gmaxwell: hey
163 2012-05-17 11:43:52 <TD> ok, good point
164 2012-05-17 11:43:56 <Diablo-D3> how hard is it to get a large number of IPs in unique subnets?
165 2012-05-17 11:44:13 <TD> you mean with a botnet?
166 2012-05-17 11:44:51 <Diablo-D3> no
167 2012-05-17 11:45:05 <Diablo-D3> not proxied either
168 2012-05-17 11:45:17 <Diablo-D3> although mass cheap VPSes seems to be the only way
169 2012-05-17 11:45:32 <gmaxwell> "For merchants that are shipping physical goods you probably don't have to worry. Reversing a transaction gets harder and harder with time. After only one hour it becomes computationally infeasible to reverse a Bitcoin transaction. Inserting a small delay before shipping isolates you from this risk entirely. "  < this needs to tell you that you actually have to check again before shipping. :)
170 2012-05-17 11:45:39 <gmaxwell> (otherwise it's great)
171 2012-05-17 11:45:42 <TD> well, you could use tor
172 2012-05-17 11:47:04 <luke-jr> tor = proxy
173 2012-05-17 11:47:21 <gmaxwell> "For automated trading platforms that may pay out some other currency in response to Bitcoin deposits, you should also delay payment for a short time until reversal becomes infeasible."    ---> "I did what it said, I waited!" "How many confirms?" "Huh? I waited 10 seconds."   (that advice is not sufficiently hardened against users doing only what you tell them)
174 2012-05-17 11:47:31 <Diablo-D3> luke-jr: yeah, but thats only good for torred bitcoins
175 2012-05-17 11:47:43 <Diablo-D3> what I was thinking about is a possible attack where a single node pretends to be ALL nodes
176 2012-05-17 11:48:25 <luke-jr> Diablo-D3: that's Gavin's "sybil attack"
177 2012-05-17 11:48:36 <Diablo-D3> ahh.
178 2012-05-17 11:48:36 <TD> right now i wouldn't want anyone without a good understanding of bitcoin to run a trading platform
179 2012-05-17 11:48:45 <TD> i guess it can have a "not for n00bs" warning at the top :-)
180 2012-05-17 11:48:48 <Diablo-D3> luke-jr: at least Im thinking along the right paths
181 2012-05-17 11:48:55 <gmaxwell> And while I generally agree that reorg attacks are normally costly there are some situations that make reorg attacks less costly... so we should be careful.
182 2012-05-17 11:50:08 <luke-jr> it would be interesting to declare the generation void on blocks that had competitors before being built on top of
183 2012-05-17 11:50:21 <luke-jr> but I guess there's a lot of forking potential in that
184 2012-05-17 11:50:43 <gmaxwell> E.g. a bank site that accepted at 1 confirm and didn't notice reversals could be exploited every few days using the natural forks formed from orphaning.
185 2012-05-17 11:50:49 <luke-jr> (that is, if you reorg, you're guaranteed to lose the generation on the blocks you made)
186 2012-05-17 11:50:59 <TD> gmaxwell: this is a good paper
187 2012-05-17 11:51:09 <TD> though it's interesting they only seem to focus on timing attacks
188 2012-05-17 11:51:16 <helo> gmaxwell: that attack you just detailed is called 'finney'?
189 2012-05-17 11:51:35 <gmaxwell> helo: where you mine a block and then quickly buy something before announcing? yes.
190 2012-05-17 11:51:37 <helo> with the 10btc for a download
191 2012-05-17 11:52:33 <nanotube> gmaxwell: indeed, i only used the mybitcoin shopping cart once for a small transaction with some site, but my funds became available within a couple seconds, as soon as i sent the transaction. which leads me to believe that mybitcoin accepted 0conf tx - at least for small tx.
192 2012-05-17 11:52:45 <TuxBlackEdo> you know if the goverment really wanted to screw up bitcoin they could ask steam to push a global update which includes a mining screen saver.. this program would mine until a retarget and then desert the bitcoin network leaving everyone with impossible to solve difficulty, miners leave, value drops, bitcoin will need a hard fork... rinse and repeat.
193 2012-05-17 11:53:08 <gmaxwell> I used to be less concerned about it, but I think GPUmax has made quite a bit more achivable. In theory you have to only outlay a bit over 50 BTC, which you'll mostly get back from the block solution..
194 2012-05-17 11:53:24 <helo> i think they could just use their new cracking facility to do that themselves
195 2012-05-17 11:53:48 <TD> gmaxwell: assuming nobody else mines the same block whilst you're interacting with the merchant
196 2012-05-17 11:53:53 <TD> but yes
197 2012-05-17 11:54:20 <gmaxwell> TD: yes, but the probability of that is easy to model and is small if you can complete the transaction in a few seconds.
198 2012-05-17 11:54:22 <TD> gpumax does make it a lot more feasible. we used to talk about "black market mining rigs" but i suppose the sale of arbitrary shares makes the black-marketness irrelevant. nobody knows what you're doing with the shares
199 2012-05-17 11:54:32 <TD> ok
200 2012-05-17 11:54:40 <TD> i wrote the wiki page in question :) i'll go back and try to improve it later
201 2012-05-17 11:54:47 <TD> need to push a new gmail release now ...
202 2012-05-17 11:54:59 <luke-jr> TD: when will GMail be open source? :p
203 2012-05-17 11:55:10 <TD> get back to me on that one :-)
204 2012-05-17 11:55:15 <gmaxwell> GPUMax claims to have some protection against being used for attack... but it's not clear to me what they do. I know they can't prevent it from being used for finney attacks, though they might be able to block it being used for deep reorgs.
205 2012-05-17 11:55:21 <TD> some stuff we developed for it is open source actually
206 2012-05-17 11:55:46 <gmaxwell> td: Okay, you should read though the rest of that paper when you have a chance. :)
207 2012-05-17 11:56:22 <TD> i think it's covering solutions that were discussed last year at least, but in a more thorough way
208 2012-05-17 11:56:23 <TuxBlackEdo> it's funny that people try to find weakness in the code, when it's weakness could just be human. murphy's law. bitcoin's next big problem won't be a sybil attack or even some double spend attack, it will be some random steam/adobe/microsoft update which will just screw up bitcoin's difficulty.. won't even take that long and nobody will care (and the government will say, "sorry we just needed to do it because it was a terrorist projec
209 2012-05-17 11:56:53 <TuxBlackEdo> it won't be a sophisticated attack
210 2012-05-17 11:56:59 <gmaxwell> I think they focus on timing attacks because the text you wront on our FAQ made a big deal about listening for conflicting txns making you safe to accept unconfirmeds. (I've since removed that and replaced it with more conservative text)
211 2012-05-17 11:57:39 <gmaxwell> TuxBlackEdo: having higher difficulty alone doesn't hurt bitcoin.
212 2012-05-17 11:57:41 <TuxBlackEdo> i will bet you the first attack that actually will hurt bitcoin's reputation of stability and security won't be a sophisticated attack
213 2012-05-17 11:58:20 <TuxBlackEdo> gmaxwell, higher difficulty and then leave on a retarget
214 2012-05-17 12:00:01 <TuxBlackEdo> just like when it was more profitable to mine namecoins and a lot of people switched to mining nmc and selling it for btc, turns out when they left on the retarget it almost took 1-2 months for the difficulty to drop
215 2012-05-17 12:00:09 <TuxBlackEdo> same attack could be made against bitcoin
216 2012-05-17 12:00:35 <TuxBlackEdo> and there's no protection against it (other then the 4x difficulty increase limit)
217 2012-05-17 12:00:39 <TD> gmaxwell: well, good to know i successfully trolled somebody into doing useful research :-)
218 2012-05-17 12:03:07 <TD> gmaxwell: done at the ETH, cool. i'll have to invite those guys along to the next bitcoin meetup in zurich
219 2012-05-17 12:03:15 <gmaxwell> TD: unforunately the research caused some mass media to report that bitcoin is less secure than previously thought!
220 2012-05-17 12:04:11 <TD> the original "wait for a bunch of announcements" plan came from satoshi himself, didn't it
221 2012-05-17 12:04:31 <TD> it's useful to know that the conflict alerts are important
222 2012-05-17 12:08:52 <TuxBlackEdo> and don't think the government won't force steam/adobe/microsoft to push a mining client, previous presidents have said they would use "any means necessary" to combat a threat
223 2012-05-17 12:09:56 <TuxBlackEdo> especially since you are taking about the implications of bitcoin (a new currency) which could be framed as a threat
224 2012-05-17 12:22:56 <helo> that would overheat some people's machines, and raise their power bills... surely the government would just use their own hardware
225 2012-05-17 12:23:42 <TuxBlackEdo> if you are playing video games, you will be expecting 100% gpu usage anyways
226 2012-05-17 12:24:53 <TuxBlackEdo> plus it wouldn't be forver, just long enough to get it to a couple of retargets and then the software would auto destruct
227 2012-05-17 12:25:56 <TuxBlackEdo> if anything, this needs to be recognized as a weakness of bitcoin, even if it seems unlikely
228 2012-05-17 12:27:23 <gmaxwell> TuxBlackEdo: what you're saying doesn't make a lot of sense.
229 2012-05-17 12:27:33 <TuxBlackEdo> what part?
230 2012-05-17 12:27:58 <gmaxwell> Every part. For one, increasing the difficulty doesn't break bitcoin, it's alread two million times higher than when we started.
231 2012-05-17 12:28:25 <TuxBlackEdo> what if it were a billion times higher?
232 2012-05-17 12:28:31 <gmaxwell> Secondly that the goverment would engage in some kind of crazy unlawful technical attack against Bitcoin.
233 2012-05-17 12:29:09 <gmaxwell> (As a harmed party I'd certantly want to be a party to the resulting civil suit)
234 2012-05-17 12:29:21 <gmaxwell> Thirdly, that all the desktops everwhere would make it a 'billion times higher'  they wouldn't.
235 2012-05-17 12:29:21 <TuxBlackEdo> if the government sees bitcoin as a threat a president will say "we will combat these threats to our currency by any means necessary"
236 2012-05-17 12:29:42 <gmaxwell> Forthly, that it would only take a couple of retargets the change is clamped.
237 2012-05-17 12:30:19 <TuxBlackEdo> same with namecoin, and it had a retarget lasting 2-3months IIRC
238 2012-05-17 12:30:35 <gmaxwell> TuxBlackEdo: if they'd declare bitcoin unlawful your threat isn't the technical attack, it's the outlawing. Even outlaws have no use for outlaw currency, and bitcoin's security requires conspicious resource expendature.
239 2012-05-17 12:30:57 <TuxBlackEdo> that retarget lasted so long because it got multiplied x4 because it was more profirable to mine nmc rather then btc
240 2012-05-17 12:31:02 <gavinandresen> gmaxwell sipa jgarzik TD: I've got a scary commit that needs lots of review
241 2012-05-17 12:31:20 <gavinandresen> https://github.com/gavinandresen/bitcoin-git/commit/8b666b866d94a1538fc88c53ffd2e43c7e156f77
242 2012-05-17 12:31:20 <TD> the best kind ....
243 2012-05-17 12:31:45 <TD> good idea!
244 2012-05-17 12:32:20 <gavinandresen> It is actually a variation on code Satoshi sent me a year ago....
245 2012-05-17 12:32:53 <TuxBlackEdo> gmaxwell, like i said, presidents have said they would combat threats "by any means necessary", if it means forcing a company to push a malicious update it can and will be done
246 2012-05-17 12:33:29 <gmaxwell> TuxBlackEdo: _wooosh_ (the point flying over your head)
247 2012-05-17 12:33:36 <neofutur> TuxBlackEdo: and then'more people will move to linuxif they want to use or mine bitcoin ?
248 2012-05-17 12:33:40 <ageis> https://github.com/gavinandresen/bitcoin-git/commit/8b666b866d94a1538fc88c53ffd2e43c7e156f77
249 2012-05-17 12:33:44 <ageis> oops sry
250 2012-05-17 12:33:45 <neofutur> wheres the problem ?
251 2012-05-17 12:34:11 <TuxBlackEdo> gmaxwell, i think i heard you say this before "if the US makes it illegal, it doesn't make it illegal anywhere"
252 2012-05-17 12:34:14 <gmaxwell> TuxBlackEdo: bitcoin can't really survive being outlawed in major nations, the technical attack is moot.
253 2012-05-17 12:34:31 <gmaxwell> TuxBlackEdo: you've certantly not heard me saying that.
254 2012-05-17 12:34:44 <gmaxwell> You've heard me say "even outlaws have little use for outlaw currency"
255 2012-05-17 12:35:22 <gmaxwell> gavinandresen: I was contemplating that you could continue to check twice and assert on disagreement as a test of the initial commit but since invalid signatures are uncommon I don't know that its a useful test.
256 2012-05-17 12:37:30 <TuxBlackEdo> yes but lets say hypothetically that the US does outlaw bitcoin but this event starts the straisand effect. Then the US government frames bitcoin as a global thread and vows to disrupt the bitcoin network by any means necessary.. All I am saying is that an attack doesn't have to be sophisticated like a sybil/finney attack
257 2012-05-17 12:37:38 <gmaxwell> gavinandresen: So, weird situation: New TXN comes in node validates it before the memory pool. Cosmic ray flips a bit and the validation fails. Later a block is mined with that transaction. Cached failure means we'll never reconverge with the network.
258 2012-05-17 12:38:06 <TuxBlackEdo> s/thread/threat
259 2012-05-17 12:38:19 <gavinandresen> gmaxwell: cache is cleared once an hour, though....
260 2012-05-17 12:39:05 <gavinandresen> I was thinking of reports of "bitcoin has a memory leak" writing that code, though, not cosmic rays
261 2012-05-17 12:39:13 <gmaxwell> TuxBlackEdo: they don't have to be sophicated like a crazy compromise every computer attack either. They just have to catch some miners, gather them up on stage and shoot them in the head. (Or more realistically put them in prison). Don't hypothize weird technical attacks when boring things will do.
262 2012-05-17 12:40:03 <TuxBlackEdo> gmaxwell, i am just thinking back to the sony trojan horse and how nobody even remembers that today
263 2012-05-17 12:40:15 <TD> gavinandresen: you could pre-allocate the cache at startup
264 2012-05-17 12:40:38 <TD> gavinandresen: that avoids the case where on very constrained the system the caches grows and makes the system start thrashing
265 2012-05-17 12:41:04 <TD> but is this the best way to solve it?
266 2012-05-17 12:41:12 <gavinandresen> TD: very memory-constrained systems might want to run with -maxsigcachesize=500
267 2012-05-17 12:41:18 <TD> can't you just check when validating a block of the tx was in the memory pool and if so, assume the signatures are valid?
268 2012-05-17 12:43:03 <gavinandresen> The big advantage of doing it way down at the CKey::Verify level is I have more confidence it is always correct
269 2012-05-17 12:43:06 <gmaxwell> gavinandresen: so the locking here will probably frustrate future multithreaded signature validation.
270 2012-05-17 12:43:41 <gavinandresen> gmaxwell: ... then future multithreaded signature validation can modify to have a per-thread cache....
271 2012-05-17 12:44:38 <gavinandresen> and actually, the lock isn't held during the expensive ECDSA_verify bit
272 2012-05-17 12:45:50 <gavinandresen> TD: the miner advantages to caching at this low level is it works for transactions/blocks that you create yourself and put into your own mined blocks, works if you're getting hit with a lot of validatemessage requests, etc.
273 2012-05-17 12:45:59 <gavinandresen> ^miner^minor
274 2012-05-17 12:46:43 <TD> that's true
275 2012-05-17 12:47:23 <TuxBlackEdo> think about how easy it is for the US government to force verisign to close down '.com's, it would not be insane to think they couldn't also force a big company (adobe/steam) to release a mining program to the masses... call it government cyber warfare against bitcoin to protect a nation's currency
276 2012-05-17 12:48:32 <TuxBlackEdo> the great thing is that it wouldn't even need to run on people's computers forever, just enough to cause a couple retargets and then leave the network.
277 2012-05-17 12:49:13 <gavinandresen> Satoshi's suggested version and comments:  https://gist.github.com/5a4fe622536d19264cba
278 2012-05-17 12:52:23 <TD> maybe comment what the std::pair<bool,bool> is (or use a real struct) in return type of GetSigCache
279 2012-05-17 12:53:10 <TD> i'm not sure about the hourly cache clearance
280 2012-05-17 12:53:43 <TD> what if a node starts to rely on the performance win of the cache. ie somebody pushes its resource allocation down to the observed resource usage ... after an hour the performance might suddenly drop and the node tips over the edge
281 2012-05-17 12:54:02 <TD> same for when it gets full
282 2012-05-17 12:54:16 <TD> i wonder what the overhead of an lru cache would be here
283 2012-05-17 12:54:46 <jgarzik> TXs get into a block, then they can be removed too.
284 2012-05-17 12:55:15 <jgarzik> set the history size at X (either number of TXs, or amount of memory used).  class automatically prunes older stuff.
285 2012-05-17 12:55:43 <TD> bbiab
286 2012-05-17 12:56:51 <gavinandresen> RE: LRU class:  I like going for the simplest possible thing that will work unti it is shown not to work.
287 2012-05-17 12:57:47 <gavinandresen> RE: clearing the cache once an hour:   I'm not sure about that, either....
288 2012-05-17 12:58:23 <gmaxwell> gavinandresen: the simplest would be a fixed side preallocated buffer which is circularly utilized and searched with a linear scan.  But then is has a potentially unbounded lifetime.
289 2012-05-17 12:58:34 <gavinandresen> Also RE: history class:  one of the purposes of a signature cache is to prevent possible DoS exploits that involve asking bitcoind to verify a bunch of signatures....
290 2012-05-17 12:59:08 <jgarzik> history class solves that problem.  it's simply a cache that is bounded by time.
291 2012-05-17 12:59:18 <jgarzik> (or number of TXs, if you choose that bound)
292 2012-05-17 12:59:26 <gavinandresen> ... so I'd prefer to keep the cache policy unpredictable, so an attacker has to generate as many unique signatures as they're asking to verify.  So they lose....
293 2012-05-17 13:00:21 <jgarzik> <shrug>  pick your bound, byte size, number of TXs, time, whatever
294 2012-05-17 13:00:53 <jgarzik> 20MB works just as well as "2000 TXs" or "1 hour"
295 2012-05-17 13:01:02 <jgarzik> (though I think satoshi's 1 hour is quite short)
296 2012-05-17 13:02:01 <gmaxwell> "I switch between generating bad signatures which are cheap and flush your cache; and reusing real ones after the cache is flushed"
297 2012-05-17 13:02:33 <gavinandresen> gmaxwell: bad signatures trigger the DoS peer ban
298 2012-05-17 13:02:48 <gmaxwell> Right. duh, nevermind!
299 2012-05-17 13:03:27 <gavinandresen> is there a cheap way of generating insecure-but-valid ECDSA sigs?
300 2012-05-17 13:04:11 <gavinandresen> actually.... I think there is.
301 2012-05-17 13:20:45 <davout> is it just me or is this : https://bitcointalk.org/index.php?topic=82175 a massive bug ?
302 2012-05-17 13:21:30 <davout> basically, getbalance (apparently used by getinfo) and getbalance '*' 1 report (vastly) different amounts....
303 2012-05-17 13:22:10 <gmaxwell> sipa: dunno if you say the comment I left on #1315  but I'm seeing a high rate of forced pull from a small number of peers.
304 2012-05-17 13:24:44 <davout> anyone ?
305 2012-05-17 13:25:01 <gmaxwell> davout: I can't make sense of whats being said there.
306 2012-05-17 13:25:15 <gmaxwell> Start with the first post.
307 2012-05-17 13:25:17 <davout> in the forum thread ?
308 2012-05-17 13:25:17 <Diapolo> hello everyone
309 2012-05-17 13:25:23 <gmaxwell> "The correct value should be the one that getbalance yields."
310 2012-05-17 13:25:31 <davout> ok
311 2012-05-17 13:25:34 <gmaxwell> there are two getbalance calls there.
312 2012-05-17 13:25:35 <gmaxwell> 1810.91340447
313 2012-05-17 13:25:37 <gmaxwell> 1776.64506472
314 2012-05-17 13:25:42 <davout> yeah
315 2012-05-17 13:25:42 <gmaxwell> Both look reasonable.
316 2012-05-17 13:25:46 <davout> the correct value
317 2012-05-17 13:26:15 <davout> that should appear when doing getinfo and getbalance (without additional args) should be equal to getbalance '*' 1
318 2012-05-17 13:26:27 <davout> (balance of all combined accounts with one conf minimum)
319 2012-05-17 13:26:32 <davout> right ?
320 2012-05-17 13:26:57 <davout> the problem is that it's absolutely not the value that getbalance reports when it's called without parameters
321 2012-05-17 13:27:09 <gmaxwell> It's more complicated than 'one conf minimum' due to the IsMine check.
322 2012-05-17 13:27:13 <davout> does it make better sense now ?
323 2012-05-17 13:27:51 <davout> yes, does that mean the behaviour is correct ? am i missing something ?
324 2012-05-17 13:28:19 <gmaxwell> I believe that '*' actually skips the ismine check. Though I'm a bit surprised to see it give a /larger/ number as a result.
325 2012-05-17 13:28:33 <davout> well, look at the last post
326 2012-05-17 13:28:46 <davout> calling getbalance '' 1 (default account)
327 2012-05-17 13:29:03 <davout> yields the value that i think is correct, (ie. not the one that getinfo reports)
328 2012-05-17 13:29:15 <davout> this is driving me nuts
329 2012-05-17 13:29:35 <davout> my database says the bitcoin client should have a much larger balance than what getinfo reports
330 2012-05-17 13:29:54 <gmaxwell> the getinfo result seems to have changed a large amount between posts.
331 2012-05-17 13:30:14 <davout> yeah, it's instawallet's wallet, there's lots of activity, that's why it changes fast
332 2012-05-17 13:30:56 <gmaxwell> In any case, I am not surprised to see them return different amounts. I am surprised that '*' is the bigger amount.
333 2012-05-17 13:31:36 <davout> according to what I've read, getbalance '*' 1 and getbalance should report strictly equal amounts
334 2012-05-17 13:31:44 <davout> also seems quite logical
335 2012-05-17 13:31:57 <gmaxwell> Well, I think what you read was wrong. There is a comment in the code that says that, and I'm pretty sure that it's wrong.
336 2012-05-17 13:32:16 <gmaxwell> (I make no comment on it being _logical_)
337 2012-05-17 13:32:41 <davout> agreed, logic and common sense are not always relevant
338 2012-05-17 13:32:43 <gmaxwell> the normal getbalance (no arguments) does a computationally expensive accounting for unconfirmed change.
339 2012-05-17 13:32:46 <davout> however, reading the help output
340 2012-05-17 13:32:51 <gmaxwell> The getbalance '*' does not.
341 2012-05-17 13:32:54 <davout> getbalance [account] [minconf=1]
342 2012-05-17 13:33:00 <davout> If [account] is not specified, returns the server's total available balance.
343 2012-05-17 13:34:00 <gmaxwell> davout: Yes, but say you have a single 10000 btc input. You spend 1 btc, producing a transaction with 9999 BTC in change to you and 1 BTC out. What should your balance be?  The transaction is not yet confirmed.
344 2012-05-17 13:34:18 <gmaxwell> There isn't room in the margins of the help to explain all the vulgatities of whats going on behind the scenes.
345 2012-05-17 13:34:59 <davout> ok, so you're saying that the behaviour is normal ?
346 2012-05-17 13:35:18 <gmaxwell> no no, I can't explain your behavior. I'm saying it being somewhat weird is normal.
347 2012-05-17 13:35:38 <davout> oh yeah, I'm used at some bitcoin weirdness, definitely
348 2012-05-17 13:35:48 <gmaxwell> What I don't get is that my understanding would make getbalance '*' smaller because it doesn't do the recursive validation.
349 2012-05-17 13:35:52 <davout> believe me, I've been digging around this one for a couple of days now
350 2012-05-17 13:36:10 <davout> i'm simply saying that I'm reasonably sure that there's a nasty bug somewhere in getbalance
351 2012-05-17 13:36:29 <gmaxwell> E.g. In my hypothetical case I expect getbalance '*' to ruturn 0 and getbalance to return 9999.
352 2012-05-17 13:36:35 <davout> i see
353 2012-05-17 13:36:45 <davout> the information i have are basically the following
354 2012-05-17 13:36:50 <davout> * bitcoind
355 2012-05-17 13:37:18 <davout> * a mysql db of everything that goes in and out of the aforementioned bitcoind (modulo what i manually send to cold storage)
356 2012-05-17 13:37:31 <davout> mysql says that getbalance '*' 1 is correct
357 2012-05-17 13:37:52 <davout> (which for this matter equals getbalance '' 1 since i'm not using the accounts system at all)
358 2012-05-17 13:38:59 <Eliel> sounds like it'd be worth figuring out which inputs are the "stuck" ones.
359 2012-05-17 13:40:16 <gmaxwell> getbalance '' 1 also does not do the IsMine() check.
360 2012-05-17 13:40:59 <davout> i created this wallet by importing a massive amount of private keys into it
361 2012-05-17 13:41:05 <davout> approx 150k
362 2012-05-17 13:41:09 <gmaxwell> davout: do users mine to some of these wallets?
363 2012-05-17 13:41:15 <davout> yes
364 2012-05-17 13:41:24 <davout> but that's ok it's handled at the app level
365 2012-05-17 13:41:30 <davout> i see your point
366 2012-05-17 13:41:44 <gmaxwell> well the code for handling it looks somewhat wrong.
367 2012-05-17 13:41:49 <gavinandresen> davout:   see https://github.com/bitcoin/bitcoin/issues/172
368 2012-05-17 13:41:50 <davout> but the amount of mined funds in the wallet cannot account for the massive delta
369 2012-05-17 13:41:57 <gavinandresen> (may or may not be relevant)
370 2012-05-17 13:42:01 <davout> handling what ?
371 2012-05-17 13:42:10 <gavinandresen> davout: never mind, I see you already found that issue....
372 2012-05-17 13:42:19 <gmaxwell> davout: GetAccountBalance's handing of generated coin.
373 2012-05-17 13:42:25 <davout> haha yes, i already begged you :)
374 2012-05-17 13:43:44 <davout> the big difference i'm seeing in reported balances cannot be accounted for solely by the fact that some generations might be immature
375 2012-05-17 13:43:54 <davout> (let me just confirm that with my DB)
376 2012-05-17 13:44:30 <davout> couldn't there be some sort of subtle bug with the IsMine() and imported private keys ?
377 2012-05-17 13:46:20 <gmaxwell> Ah. I see how that might work.
378 2012-05-17 13:46:37 <davout> howso ?
379 2012-05-17 13:48:13 <Eliel> davout: can you identify the set of outputs that aren't being counted in the balance?
380 2012-05-17 13:48:19 <Eliel> is it stable?
381 2012-05-17 13:48:24 <gmaxwell> getbalance '*' doesn't do the IsMine() stuff in order to account for change but it doesn't have to because it trusts the transaction entry's net-Received figures.
382 2012-05-17 13:48:26 <davout> Eliel: how can i do that ?
383 2012-05-17 13:48:51 <Eliel> davout: hmm... I guess it'd require patched bitcoind.
384 2012-05-17 13:49:05 <davout> gmaxwell: i don't really see what you mean
385 2012-05-17 13:49:11 <davout> brb
386 2012-05-17 14:02:01 <sipa> gmaxwell: rebroad's comment makes sense
387 2012-05-17 14:02:42 <gmaxwell> davout: my current best guess it that you have imported keys which had previous traffic on them.
388 2012-05-17 14:03:05 <gmaxwell> davout: and the change is not being accounted for by getbalance because your node can't tell that its change.
389 2012-05-17 14:03:34 <gmaxwell> or something of that ilk.
390 2012-05-17 14:03:50 <gmaxwell> (It's sort of hard without actually having such a wallet in front of me)
391 2012-05-17 14:05:27 <gmaxwell> sipa: for some reason my mental model of that commit had it also checking the inv size.
392 2012-05-17 14:17:31 <jgarzik> inline void Sleep(int64 n) { boost::thread::sleep(boost::get_system_time() + boost::posix_time::milliseconds(n>315576000000LL?315576000000LL:n)); }
393 2012-05-17 14:17:36 <jgarzik> really?
394 2012-05-17 14:17:47 <jgarzik> that's #ifdef !WIN32, too
395 2012-05-17 14:17:56 <jgarzik> what about....   #define Sleep sleep
396 2012-05-17 14:17:58 <jgarzik> sheesh
397 2012-05-17 14:18:08 <davout_> gmaxwell: that's what happened
398 2012-05-17 14:18:55 <jgarzik> 05/17/12 16:12:25 Flushed 11702 addresses to peers.dat  70ms
399 2012-05-17 14:18:56 <jgarzik> 05/17/12 16:14:05 Flushed 11704 addresses to peers.dat  86ms
400 2012-05-17 14:19:15 <jgarzik> sipa: ^^  DumpAddresses() seems to be called rather frequently
401 2012-05-17 14:19:19 <davout_> but the change is probably not the explanation since i exported all the keys, imported them in the destination client, and finally made a transaction from the old client spending everything to a new address generated on the new client
402 2012-05-17 14:19:55 <jgarzik> sipa: more frequently than Sleep(100000) might suggest
403 2012-05-17 14:20:29 <Diapolo> jgarzik: I added 2 comments here: https://github.com/bitcoin/bitcoin/commit/768e5d52fb295b000940f6a806c3d4bfc3e4f54d
404 2012-05-17 14:20:45 <davout> and even if I was missing the change TXes I don't see how that could lead to different reported balances for getbalance and getbalance '*' 1
405 2012-05-17 14:22:35 <sipa> jgarzik: every 100s
406 2012-05-17 14:22:56 <sipa> i guess every 300s is fine as well
407 2012-05-17 14:23:46 <jgarzik> sipa: is Sleep() measured in milliseconds or something?   hmmmm
408 2012-05-17 14:24:26 <jgarzik> bleh, what an annoying unit usage
409 2012-05-17 14:24:37 <gribble> New news from bitcoinrss: roques opened pull request 1331 on bitcoin/bitcoin <https://github.com/bitcoin/bitcoin/pull/1331>
410 2012-05-17 14:24:38 <Diapolo> jgarzik: will you add code to remove the old addr.dat? Seems it stays in the datadir currently.
411 2012-05-17 14:25:12 <jgarzik> sipa: DumpAddresses() writes > 1MB at each call, both before and after my patch
412 2012-05-17 14:25:29 <jgarzik> Diapolo: so... can you be talked into making a pull req that fixes win32?  :)
413 2012-05-17 14:26:00 <Diapolo> sure that are things I'm able to do :) no crazy protocol stuff ^^
414 2012-05-17 14:28:38 <davout> anyone have an idea why bitcoind reports vastly different balances when i use getbalance and getbalance '*' 1 ?
415 2012-05-17 14:30:45 <sipa> jgarzik: or keep a counter of modified addrman records
416 2012-05-17 14:30:59 <seco> lurking around the blockchain i wonder if that guy owning this address runs a bitcoin-qt with only one receiving address in the wallet, or tries to spam somehow the chain: http://blockchain.info/address/1VayNert3x1KzbpzMGt2qdqrAThiRovi8
417 2012-05-17 14:31:14 <sipa> seco: deepbit
418 2012-05-17 14:31:56 <seco> meh; is there a wikipage of known cases how blockchain got spammed in the past, and probably listnings of punishments or something like that?
419 2012-05-17 14:32:11 <sipa> davout: there is a difference between those; a very early github issue even
420 2012-05-17 14:32:23 <sipa> but i can't remember exactly
421 2012-05-17 14:32:54 <davout> sipa: https://github.com/bitcoin/bitcoin/issues/172
422 2012-05-17 14:32:57 <jgarzik> sipa: good idea
423 2012-05-17 14:33:11 <jgarzik> Diapolo: thanks!
424 2012-05-17 14:33:28 <davout> sipa: i'm encountering an issue that looks like this on instawallet's wallet
425 2012-05-17 14:33:39 <Diapolo> jgarzik: you're welcome
426 2012-05-17 14:34:14 <davout> the balance reported by getinfo/getbalance differs greatly from the one reported if i do getbalance '*' 1
427 2012-05-17 14:34:41 <gribble> New news from bitcoinrss: Diapolo opened pull request 1332 on bitcoin/bitcoin <https://github.com/bitcoin/bitcoin/pull/1332>
428 2012-05-17 14:34:52 <davout> if i do a listaccounts the reported balance for account '' matches what i have in the associated DB
429 2012-05-17 14:35:09 <davout> but the balance reported by getinfo is like 1000 BTC off
430 2012-05-17 14:36:08 <gavinandresen> I was hoping one of those large-number-of-keys sites would invest in some engineering effort and submit patches ....
431 2012-05-17 14:36:12 <jgarzik> it becomes cumbersome to scan each new block for transactions associated with keys in wallet X
432 2012-05-17 14:36:15 <davout> haha
433 2012-05-17 14:36:22 <jgarzik> so I like the concept of "hot and cold keys"
434 2012-05-17 14:36:49 <davout> yea, my plan is to make instawallet evolve into a site where private keys are inferred from the URL
435 2012-05-17 14:37:06 <davout> so virtually any instawallet URL is a valid wallet
436 2012-05-17 14:37:09 <jgarzik> scan for activity on cold keys once every X blocks (144?), rather than once every block or TX via the map in RAM
437 2012-05-17 14:37:30 <davout> jgarzik: what are you calling "cold keys" exactly ?
438 2012-05-17 14:38:01 <Eliel> I'd guess addresses unlikely to receive coins.
439 2012-05-17 14:38:03 <jgarzik> davout: a site-defined policy, e.g. "a key that was generated > 24 hours ago, and has not seen any activity in 24 hours"
440 2012-05-17 14:38:28 <jgarzik> if you generate new address for each transaction, you wind up with a -ton- of addresses that are unlikely to be used again
441 2012-05-17 14:38:40 <jgarzik> those are cold keys, after their primary website use is done
442 2012-05-17 14:38:49 <davout> yea, that's more of a pruning/archiving issue i reckon
443 2012-05-17 14:38:52 <jgarzik> yes
444 2012-05-17 14:38:59 <davout> i really like the create a key from the URL
445 2012-05-17 14:39:01 <jgarzik> right now the bitcoin client does not prune etc.
446 2012-05-17 14:39:11 <davout> therefore there's just no limit on the number of wallets
447 2012-05-17 14:39:18 <davout> when a client connects on a wallet URL
448 2012-05-17 14:39:38 <jgarzik> larger sites IMO should be using a key-server
449 2012-05-17 14:39:42 <davout> the backend computes the address, shows the related balance, and monitors for incoming TXes while the client is online
450 2012-05-17 14:39:43 <gribble> New news from bitcoinrss: rebroad opened pull request 1333 on bitcoin/bitcoin <https://github.com/bitcoin/bitcoin/pull/1333>
451 2012-05-17 14:40:05 <jgarzik> something that does nothing but generate keys:  (a) retrieves a public key for use on a website, (b) sends associated private key into secure storage
452 2012-05-17 14:40:17 <jgarzik> that way the website may generate new addresses, but never touches private keys
453 2012-05-17 14:40:28 <jgarzik> sipa's HD stuff does similar things
454 2012-05-17 14:40:54 <jgarzik> I really should write up a mailing list post about large sites and bitcoin key management
455 2012-05-17 14:41:40 <davout> i couldn't agree more
456 2012-05-17 14:41:43 <jgarzik> bitcoin architecture does permit secure separation between money-handling website and wallet/blockchain backend.
457 2012-05-17 14:42:00 <davout> it would relieve me of a lot of stress :)
458 2012-05-17 14:42:03 <jgarzik> you don't _have_ to store private keys in bitcoind at all, to run a secure site that generates new payment addresses
459 2012-05-17 14:42:15 <davout> yep
460 2012-05-17 14:42:51 <davout> i still have my reported-balances-are-completely-fucked-up problem
461 2012-05-17 14:43:06 <jgarzik> for larger sites, bitcoind must become what BlueMatt's CBlockStore does internally:  essentially create a blockchain server, and handle key management and payment management externally.
462 2012-05-17 14:43:52 <davout> yea, but that must also fit in a good procedure framework when multiple people are supposed to be able initiate payments
463 2012-05-17 14:43:59 <jgarzik> bitcoind need only (a) watch for public key activity and (b) accept raw transactions from external source
464 2012-05-17 14:44:14 <jgarzik> then the site's database software can generate payments, using secure access to private keys
465 2012-05-17 14:44:25 <jgarzik> no single wallet to get hacked and stolen
466 2012-05-17 14:44:47 <gribble> New news from bitcoinrss: rebroad reopened pull request 1305 on bitcoin/bitcoin <https://github.com/bitcoin/bitcoin/pull/1305>
467 2012-05-17 14:45:14 <davout> if anyone feels like solving a riddle : https://bitcointalk.org/index.php?topic=82175.0
468 2012-05-17 14:47:21 <Diapolo> jgarzik: What about adding a boost remove in CAddrDB::CAddrDB() for the old addr.dat, maybe with if (exists)?
469 2012-05-17 14:48:30 <jgarzik> Diapolo: my only worry there is downgrading.  I don't see a burning need to delete it... might come in handy if you back down to an older version
470 2012-05-17 14:49:31 <jgarzik> sipa: yeah we could DumpAddresses() every 300 seconds... gmaxwell had suggested something on the order of several hours instead
471 2012-05-17 14:49:36 <Diapolo> jgarzik: That should be no problem, when that one is missing, a new addr.dat will be created, right?
472 2012-05-17 14:49:46 <jgarzik> sipa: which makes a certain amount of sense
473 2012-05-17 14:50:06 <jgarzik> Diapolo: yes
474 2012-05-17 14:50:43 <jgarzik> Diapolo: over time, fewer people will even know addr.dat existed.  the less code in current bitcoin spend dealing with old-bitcoin formats, the better.
475 2012-05-17 14:51:12 <jgarzik> time is the best janitor
476 2012-05-17 14:51:17 <jgarzik> or entropy
477 2012-05-17 14:54:55 <gavinandresen> I vote for leaving the old addr.dat, and telling people who notice/care they can delete it.
478 2012-05-17 14:55:18 <jgarzik> yep
479 2012-05-17 14:55:25 <gavinandresen> the percentage of people affected will become smaller and smaller over time, assuming bitcoin is successful.
480 2012-05-17 14:55:31 <gavinandresen> if bitcoin is not successful, then I don't care.
481 2012-05-17 15:10:10 <davout> can anyone point me a direction to diagnose why my bitcoind is reporting vastly different balances depending on whether i query it with "getbalance" and "getbalance '*' 1" ?
482 2012-05-17 15:11:29 <forrestv> davout, perhaps you have orphaned transactions?
483 2012-05-17 15:12:06 <forrestv> i'm not actually sure about the semantics.. but one may include orphaned transactions and the other might not?
484 2012-05-17 15:13:34 <davout> no, i don't have orphans
485 2012-05-17 15:14:02 <davout> i did 'listtransactions '*' 100000000'
486 2012-05-17 15:14:15 <davout> grepping 0-conf transactions
487 2012-05-17 15:18:47 <Eliel> perhaps not a good idea to run that on the production server though. A copy of the wallet would probably do fine on another system.
488 2012-05-17 15:19:38 <Eliel> you kind of need to know the outputs that are making the difference for any meaningful debug :)
489 2012-05-17 15:19:45 <davout> so i'd printf outs
490 2012-05-17 15:19:58 <davout> into a couple of files
491 2012-05-17 15:20:01 <davout> and diff them out
492 2012-05-17 15:20:07 <Eliel> pretty much
493 2012-05-17 15:20:09 <davout> sounds like a plan
494 2012-05-17 15:20:12 <gavinandresen> yes, do what Eliel says.
495 2012-05-17 15:20:43 <davout> wouldn't that be a little impractical with IW's wallet ? (like +150k keys)
496 2012-05-17 15:21:01 <gavinandresen> yes, you'll notice I didn't volunteer to help you figure out what's going on....
497 2012-05-17 15:21:45 <Eliel> well, I suppose you could also try subsets of that wallet to see which addresses are the problem addresses :)
498 2012-05-17 15:21:47 <davout> gavinandresen: hah, i did notice, but i can't really expect anyone to solve *my* problems, can i
499 2012-05-17 15:22:06 <davout> no it's cool i'll do the outputs printf thing
500 2012-05-17 15:22:09 <Eliel> that doesn't require code changes :)
501 2012-05-17 15:22:18 <gmaxwell> jgarzik: That Sleep hack is my misdeed unfortunately boost's sleep doesn't if the sleep spans 2038, which was causing wallet unlocks with far future times to lock again instantly.
502 2012-05-17 15:22:52 <gmaxwell> jgarzik: I could have avoided it purely in the unlock code, but I was concerned that other callers of Sleep (now or in the future) would blow up in similar ways.
503 2012-05-17 15:22:53 <davout> thanks gavinandresen and Eliel for the pro-tips ! much appreciated :D
504 2012-05-17 15:23:05 <gavinandresen> davout: the problem probably has something to do with 'change' addresses
505 2012-05-17 15:23:32 <gavinandresen> ... so while you're dumping data you probably want to dump whether or not bitcoin considers each output 'change' or not.
506 2012-05-17 15:23:45 <gmaxwell> (and the fact that Sleep takes ms unlike sleep(3) is certantly not my bad. :) )
507 2012-05-17 15:23:58 <gavinandresen> Sending coins to change addresses might cause confusion, too (wouldn't be surprised if instawallet users used services that did that)
508 2012-05-17 15:24:13 <Eliel> gavinandresen: is there a built in function to print debug data of outputs?
509 2012-05-17 15:24:36 <gavinandresen> Eliel: I don't think so
510 2012-05-17 15:24:53 <davout> i'll try to hack something out
511 2012-05-17 15:25:33 <davout> i might ask if you can take a quick look at my patch in a couple of hours if you're still around to see if i'm actually printfing the right stuff
512 2012-05-17 15:25:50 <gmaxwell> davout: I'd dump all the data each of the functions uses to make a determination, then walk through the steps in each function while looking at the data until I spot the difference.  Failing that I'd remove addresses one at a time from the wallet until they agree.
513 2012-05-17 15:26:31 <gmaxwell> Once identifying the last address removed that makes them differ I'd look at all its transactions and see whats special about it. (or if there are many, I'd start removing the transactions)
514 2012-05-17 15:26:44 <Diapolo> jgarzik: okay :)
515 2012-05-17 15:27:16 <davout> they're like over 150k txes and addresses in this wallet, i think i can only go for an automated approach
516 2012-05-17 15:27:26 <jgarzik> gmaxwell: that Sleep() is ifdef unix
517 2012-05-17 15:27:31 <davout> like dumping summed outs and diffing
518 2012-05-17 15:27:37 <jgarzik> gmaxwell: just call sleep()
519 2012-05-17 15:27:45 <davout> i assume that's how it works
520 2012-05-17 15:27:51 <gmaxwell> jgarzik: we have a lot of subsecond sleeps through the code.
521 2012-05-17 15:27:59 <davout> getbalance will sum spendable outs, right ?
522 2012-05-17 15:28:10 <gmaxwell> (e.g. a lot of the network code had 100ms / 500 ms sleeps IIRC)
523 2012-05-17 15:28:21 <gmaxwell> But doh on the ifdef unix.
524 2012-05-17 15:29:37 <jgarzik> gmaxwell: sure we can use nanosleep then
525 2012-05-17 15:29:46 <jgarzik> gmaxwell: but as well, most of those sub-second sleeps are bogus
526 2012-05-17 15:29:58 <jgarzik> gmaxwell: the network code is riddled with silly pauses
527 2012-05-17 15:30:27 <jgarzik> gmaxwell: in one loop, we do nothing but sleep for 0.5 seconds, then check fShutdown
528 2012-05-17 15:31:54 <gmaxwell> jgarzik: Agreed. We often have threads that sit around doing nothing but that.
529 2012-05-17 15:32:30 <gavinandresen> I would very much like to see a refactor to use boost::thread and doing proper interruption/join....
530 2012-05-17 15:33:15 <sipa> gavinandresen: ack
531 2012-05-17 15:33:32 <Diapolo> sounds good :)
532 2012-05-17 15:34:24 <gmaxwell> gavinandresen: hopefully it actually works.
533 2012-05-17 15:34:39 <sipa> that boost::sleep(now + boost::milliseconds(x))... i think that does 2 pointless calls to gettimeofday or similars
534 2012-05-17 15:35:22 <gmaxwell> yea, ever looked at a strace of bitcoin? we're doing an ungodly number of gettimeofdays.
535 2012-05-17 15:35:49 <gmaxwell> Last time I did it it was hard to see any other syscalls. They scrolled off too fast.
536 2012-05-17 15:35:51 <sipa> jgarzik: i suppose writing addrman data once per hour wouldn't hurt either (though maybe do an extra dump dns seeding, or after the gets getaddr return)
537 2012-05-17 15:35:56 <gavinandresen> we could create a cache and only call gettimeofday if more than N microseconds have ... oh, wait....
538 2012-05-17 15:36:06 <sipa> gavinandresen: lol
539 2012-05-17 15:36:09 <gmaxwell> hahah
540 2012-05-17 15:36:14 <gmaxwell> use the TSC! (ugh)
541 2012-05-17 15:36:15 <gavinandresen> :)
542 2012-05-17 15:36:45 <sipa> gavinandresen: w00t for non-portable code
543 2012-05-17 15:36:58 <Diapolo> Is anyone running an IPv6 node I could try a connect with?
544 2012-05-17 15:37:34 <gmaxwell> sipa: making addrman only run once per hour just has the downside of a little risk of forgetting some of work you did finding good peers when.
545 2012-05-17 15:37:58 <sipa> gmaxwell: what about doing it at every connect?
546 2012-05-17 15:38:38 <gmaxwell> sipa: what about every successful addition of the node to the good pool so long as it's at least been a minute?
547 2012-05-17 15:39:27 <sipa> gmaxwell: that will make it write it close to once every minute
548 2012-05-17 15:40:23 <gavinandresen> I'd vote for writing it once a day or at clean shutdown.
549 2012-05-17 15:40:51 <gmaxwell> gavinandresen: replace that or with and.
550 2012-05-17 15:41:10 <sipa> gavinandresen is using a mathematical 'or', i think
551 2012-05-17 15:41:15 <gavinandresen> right, and.   Even better if you can write it just before clean shutdown :)
552 2012-05-17 15:41:30 <sipa> though that will slow down shutdown
553 2012-05-17 15:41:38 <gavinandresen> how slow is the write?
554 2012-05-17 15:41:42 <sipa> 50ms
555 2012-05-17 15:41:48 <gavinandresen> meh.
556 2012-05-17 15:41:50 <gmaxwell> 'meh'
557 2012-05-17 15:41:50 <sipa> ;)
558 2012-05-17 15:42:57 <sipa> i'd say once every 500 updated records, once per day, and at shutdown
559 2012-05-17 15:43:11 <gmaxwell> I don't like adding more work to shutdown.  https://lwn.net/Articles/191059/
560 2012-05-17 15:44:06 <gmaxwell> Since we have to survive crashes anyways, there is an argument that you should just handle shutdowns by crashing.  (BDB makes this less attractive for us, so I don't think we should actually do it but the argument has merit and should influence our decisions)
561 2012-05-17 15:44:51 <sipa> since jgarzik's peers.dat, it shouldn't every get corrupted, and even if it does, it won't bother the user
562 2012-05-17 15:45:03 <gmaxwell> (the usenix paper linked from there is interesting as I recall)
563 2012-05-17 15:52:15 <sipa> gmaxwell: interesting... crash + recover being faster than shutdown/restart
564 2012-05-17 15:53:45 <seco> make crash the new shutdown xD xD
565 2012-05-17 15:54:17 <sipa> where are the DOS days when you could just shut down the computer by pulling the power cable?
566 2012-05-17 15:54:58 <gmaxwell> sipa: not only that, it means your recovery code actually gets tested.
567 2012-05-17 15:55:03 <sipa> indeed
568 2012-05-17 15:55:22 <sipa> it would result in a very stable system, if you can pull it off
569 2012-05-17 15:56:12 <gribble> New news from bitcoinrss: rebroad opened pull request 1334 on bitcoin/bitcoin <https://github.com/bitcoin/bitcoin/pull/1334>
570 2012-05-17 15:56:49 <gmaxwell> I don't have any hope of doing that in bitcoin due to bdb but perhaps if we manage a process split between wallets and the blockchain the wallet side could be crash-only. ... though we'd probably still use bdb there for transaction tracking.
571 2012-05-17 15:58:30 <gmaxwell> wumpus: I'm guessing you're not around now?  I want to nitpick the out of sync message.
572 2012-05-17 15:58:55 <gmaxwell> "The displayed information reflects an older state. To get more recent information, the local block chain needs to be synced with the network. This process starts automatically after a network connection is established." is what diapolo has from your suggestion.
573 2012-05-17 15:59:44 <gmaxwell> I prefer ""The displayed information may be out of date. Your wallet automatically synchronizes with the Bitcoin network after a connection is established, but this process has not completed yet.", avoiding 'state', 'block chain' and the action phrase 'to get more'.  Thoughts?  This is all bikeshedding.
574 2012-05-17 16:07:59 <wumpus> gmaxwell: yes, I also think that's better
575 2012-05-17 16:08:20 <wumpus> especially avoiding the "block chain" is nice
576 2012-05-17 16:11:20 <gribble> New news from bitcoinrss: rebroad opened pull request 1335 on bitcoin/bitcoin <https://github.com/bitcoin/bitcoin/pull/1335>
577 2012-05-17 16:11:45 <Eliel> gmaxwell: I'd disagree about that being bikeshedding. It's important to leave unnecessary complexity out of user interfaces.
578 2012-05-17 16:13:39 <wumpus> it's only bikeshedding if it results in long discussions
579 2012-05-17 16:14:46 <sipa> i personally would try to avoid discussions about whether it is bikeshedding or not
580 2012-05-17 16:14:52 <Eliel> :D
581 2012-05-17 16:15:11 <sipa> because metabikeshedding is the ultimate form of it
582 2012-05-17 16:15:38 <wumpus> hah, metametabikeshedding
583 2012-05-17 16:22:56 <eian> I'm connected to 2000 nodes and only half of them are relaying transactions - does this sound right?
584 2012-05-17 16:24:39 <sipa> some may be SPV nodes?
585 2012-05-17 16:25:30 <eian> I'll need to look into this
586 2012-05-17 16:25:40 <gavinandresen> I run my nodes with a lower-than-default limitfreerelay
587 2012-05-17 16:26:07 <gavinandresen> eian: how are you determining whether or not they're relaying transactions?
588 2012-05-17 16:26:23 <sipa> and why are you connected to 2000 nodes?
589 2012-05-17 16:26:45 <eian> gavin, I'm storing everything they are sending into a sql database
590 2012-05-17 16:26:46 <Diablo-D3> heh
591 2012-05-17 16:26:52 <Diablo-D3> if I could run bitcoin on alpine
592 2012-05-17 16:26:58 <Diablo-D3> I'd be connected to over 2000 nodes
593 2012-05-17 16:27:10 <eian> diable, what is alpine?
594 2012-05-17 16:27:16 <eian> diablo*
595 2012-05-17 16:27:19 <Diablo-D3> a linux distro
596 2012-05-17 16:27:57 <gavinandresen> if we get many more people doing the "connect to everybody" then we might need to do something about it....
597 2012-05-17 16:28:31 <eian> gavin, I'm doing research to see if there is collusion on the network
598 2012-05-17 16:28:53 <eian> part of my masters thesis
599 2012-05-17 16:28:55 <sipa> eian: the non-relaying nodes, did they connect to you, or did you connect to them?
600 2012-05-17 16:28:57 <gavinandresen> eian: excellent.
601 2012-05-17 16:29:00 <eian> I connected to them
602 2012-05-17 16:29:28 <sipa> hmm, then they can't be SPV
603 2012-05-17 16:30:01 <eian> I'll have better data available in the next few weeks - the issues I'm seeing may be in my software
604 2012-05-17 16:34:28 <jgarzik> <gmaxwell> [...] perhaps if we manage a process split between wallets and the blockchain [...]
605 2012-05-17 16:34:32 <jgarzik> funny, I was just thinking that
606 2012-05-17 16:34:47 <jgarzik> thinking more about BlueMatt's CBlockStore (besides the awful, misleading name)
607 2012-05-17 16:35:25 <jgarzik> why not follow BlueMatt's lead on message dispatch...  and create a hard [yet internal, initially] separation of "one or more wallet servers" and "blockchain server"
608 2012-05-17 16:35:57 <sipa> jgarzik: that's the plan
609 2012-05-17 16:35:58 <jgarzik> and communicate via RPC message (i.e. as it is initially internal only, the RPCs would be simple message classes passed back and forth via memory
610 2012-05-17 16:36:18 <jgarzik> CBlockStore does not really have a messaging concept
611 2012-05-17 16:36:34 <sipa> jgarzik: have you looked at the implementation?
612 2012-05-17 16:36:48 <jgarzik> async signals (interrupts) systems wind up creating a huge interrupt table (ie. CBlockStore's signals stuff)
613 2012-05-17 16:37:06 <jgarzik> sipa: yes, interrupt hooks, basically -- not RPC messaging
614 2012-05-17 16:38:07 <sipa> jgarzik: anyway, at least in broad outline that was the plan; it was originally called CHub, as it would be a hub that managed the communication between blockchain, wallets and network
615 2012-05-17 16:38:26 <jgarzik> if you create 10 wallet servers, each would create an internal RPC_Connection object (analagous to a TCP connection) to the blockchain server
616 2012-05-17 16:38:51 <sipa> exactly
617 2012-05-17 16:39:21 <wumpus> well if you have interrupt hooks, and peddle around information in calls instead of directly digging in data structures, it's pretty easy step to rpc
618 2012-05-17 16:39:55 <sipa> jgarzik: is your complaint the fact that there are separate signals for each message, instead of one listener for a catch-all message?
619 2012-05-17 16:41:18 <sipa> (apart from the fact that for now blockstore is also managing the blockstore)
620 2012-05-17 16:41:43 <gribble> New news from bitcoinrss: laanwj opened pull request 1336 on bitcoin/bitcoin <https://github.com/bitcoin/bitcoin/pull/1336>
621 2012-05-17 16:43:06 <jgarzik> sipa: it tries to do way too much
622 2012-05-17 16:43:09 <jgarzik> +    std::queue<CBlock*> queueCommitBlockCallbacks;
623 2012-05-17 16:43:22 <sipa> huh?
624 2012-05-17 16:43:39 <wumpus> that's a lot of queues
625 2012-05-17 16:44:15 <jgarzik> it's not just a lot of queues, it's an example of lumping a large amount of vaguely related activity into a single message dispatcher class
626 2012-05-17 16:44:49 <sipa> where do you see that code?
627 2012-05-17 16:44:51 <jgarzik> RPC systems do not normally encode _every_ detail about _every_ message into a single class
628 2012-05-17 16:45:04 <sipa> i replaced that a long time ago by one queue
629 2012-05-17 16:45:05 <jgarzik> sipa: 6bb36f2343ef5c00666520d3ae84eabf0204f28b
630 2012-05-17 16:45:58 <sipa> what branch/repository is that in?
631 2012-05-17 16:48:03 <sipa> since my rewrite it's just std::queue<CBlockStoreCallback*> queueCallbacks;
632 2012-05-17 16:50:27 <wumpus> btw, talking about async notifications, can someone comment on https://github.com/bitcoin/bitcoin/pull/1205 ?
633 2012-05-17 16:51:00 <sipa> wumpus: hmm, i thought i already tested and ACKed it, actually
634 2012-05-17 16:51:46 <wumpus> nope... only diapolo replied on it afaik
635 2012-05-17 16:53:20 <wumpus> though it could be github lost it, wouldn't been the first time
636 2012-05-17 16:58:21 <sipa> wumpus: i remember the wallet and keystore-specific signals used to be in ui_interface as well?
637 2012-05-17 16:58:32 <sipa> it's better now
638 2012-05-17 16:58:35 <jgarzik> sipa: I was looking at the commits in BlueMatt's pull req #771
639 2012-05-17 16:58:39 <wumpus> yes
640 2012-05-17 16:59:04 <sipa> jgarzik: i'm sure the old code is still there, but look at the result :)
641 2012-05-17 16:59:20 <wumpus> first I simply converted the ui_interface functions to signals, after that, I realized it's better to put them on the object that emits them
642 2012-05-17 16:59:30 <jgarzik> sipa: in general, callbacks are a huge mess, because callbacks break the wall-of-separation between (say) wallet server's thread and blockchain server's thread
643 2012-05-17 17:00:12 <sipa> jgarzik: the callbacks are done from blockstore's thread
644 2012-05-17 17:00:22 <wumpus> yeah, I'm happy I can use qt for interthread message passing
645 2012-05-17 17:00:33 <jgarzik> sipa: yes... TO which thread?
646 2012-05-17 17:01:00 <sipa> threads and objects are separate
647 2012-05-17 17:01:08 <sipa> there is no thread per object
648 2012-05-17 17:01:35 <sipa> you just don't want the callback to wallet to happen inside a call that started in block processing
649 2012-05-17 17:01:46 <sipa> because that flips the order in which locks are acquired
650 2012-05-17 17:01:56 <sipa> (and results in deadlocks)
651 2012-05-17 17:02:09 <jgarzik> sipa: you don't want the blockchain server calling a callback, whose callback implementation is wallet code that normally runs in a wallet thread
652 2012-05-17 17:02:46 <jgarzik> sipa: take a look at RDMA design.  set up mailboxes, and do message passing like that
653 2012-05-17 17:03:03 <sipa> the wallet has no special thread, but its code normally runs from UI or RPC thread
654 2012-05-17 17:03:20 <sipa> and it can also receive signals from the dispatcher thread
655 2012-05-17 17:03:21 <jgarzik> sipa: that's beside the point
656 2012-05-17 17:03:29 <sipa> that's all what happens
657 2012-05-17 17:03:44 <jgarzik> sipa: callback systems create a mess for threads, especially if you are trying to separate for security
658 2012-05-17 17:04:24 <sipa> before blockstore, callbacks happened directly and ad-hoc from block processing to wallet
659 2012-05-17 17:04:30 <sipa> THAT is a problem for threading
660 2012-05-17 17:04:55 <jgarzik> sipa: nothing I said is an argument for avoiding all change ;-)
661 2012-05-17 17:05:08 <jgarzik> sipa: obviously the current system wants updating
662 2012-05-17 17:05:08 <sipa> well, do a better suggestion
663 2012-05-17 17:05:27 <wumpus> right, the callbacks don't change the distribution of work over threads, it just formalizes/decouples it a bit.. going to full message passing would be even nicer but is a big step
664 2012-05-17 17:05:30 <jgarzik> sipa: c.f. RDMA design.  I'd be happy to code it.
665 2012-05-17 17:05:41 <jgarzik> that's what we need -- full message passing
666 2012-05-17 17:06:18 <jgarzik> otherwise the boundaries will continue to be violated as they are today
667 2012-05-17 17:06:20 <sipa> my only complaint about blockstore right now is that it still does block processing, instead of having the block processing be a signal listener
668 2012-05-17 17:06:46 <sipa> but that's just one step further (and changing cblockstore's name, as it is not storing any blocks)
669 2012-05-17 17:07:56 <sipa> but in essence, blockstore is a queue of callbacks, that are handled one by one, and other threads can safely and without risk for deadlocks put messages on the queue
670 2012-05-17 17:08:40 <wumpus> aren't there any frameworks that could be used to make it easier? does boost have a message passing library? at least not having to hand-roll everything would be a win
671 2012-05-17 17:08:42 <jgarzik> sipa: you need multiple message queues ("mailboxes" in RDMA parlance), one for blockchain server, one for each wallet server, ...
672 2012-05-17 17:09:05 <sipa> jgarzik: this has nothing to do with communication between wallet and UI/RPC
673 2012-05-17 17:09:33 <sipa> if you want a messaging system that is general enough that it can help implementing both layers, please
674 2012-05-17 17:09:43 <wumpus> or we could just switch to erlang :p
675 2012-05-17 17:10:05 <sipa> and blockstore (the communication between block db, wallets and network) would, imho, then just be a specialization of that
676 2012-05-17 17:11:05 <sipa> wumpus: haha
677 2012-05-17 17:12:28 <jgarzik> sipa: where is the latest code, if not #771?
678 2012-05-17 17:12:45 <jgarzik> maybe you have already fixed the OneBigClassThatDoesEverything issue :)
679 2012-05-17 17:13:23 <sipa> no
680 2012-05-17 17:13:25 <sipa> #711
681 2012-05-17 17:13:36 <sipa> and no, it's still one class that is the dispatcher for everything
682 2012-05-17 17:14:20 <sipa> but the dispatcher should deal with all relevant messages in that area, imho
683 2012-05-17 17:14:31 <sipa> it should not actually contain code to deal with the messages, just pass them along
684 2012-05-17 17:14:41 <sipa> and that is something it still does, right now
685 2012-05-17 17:15:34 <sipa> jgarzik: note that cblockstore has been refactored many times over, and maybe the order of transactions in the github page is not the actual logical commit order anymore
686 2012-05-17 17:15:59 <sipa> eh, #771
687 2012-05-17 17:18:06 <wumpus> if there are merges involved, the commit order on the github page will be really weird
688 2012-05-17 17:18:42 <sipa> jgarzik: hmm, if i understand your idea correctly, you end up with a handler thread for every object in the message-passing system (every wallet is one thread, blockchain db is one thread, ...) and you just pass messages between their queues, and they each handle their received messages in order?
689 2012-05-17 17:19:06 <sipa> that sounds very nice, but hard to combine with signals whose responses need some form of aggregation
690 2012-05-17 17:19:41 <wumpus> you'd need a combiner abstraction as well
691 2012-05-17 17:21:11 <sipa> in contrast to that, cblockstore is the handler thread for everything, and the messages come from independent threads outside of the handling system (UI thread, network thread, ...)
692 2012-05-17 17:21:41 <sipa> but cblockstore does deal with the aggregation of data as well (which is quite simple in our case)
693 2012-05-17 17:22:37 <sipa> your idea certainly sounds more general, but it may be overkill, imho
694 2012-05-17 17:53:53 <gribble> New news from bitcoinrss: rebroad opened pull request 1337 on bitcoin/bitcoin <https://github.com/bitcoin/bitcoin/pull/1337>
695 2012-05-17 17:54:31 <Diapolo> wumpus: nice icon there :)
696 2012-05-17 18:33:25 <luke-jr> well, bdb zlib and miniupnpc successfully built for OS X deterministicly&
697 2012-05-17 18:33:38 <luke-jr> openssl and libpng giving me trouble tho :/
698 2012-05-17 18:34:36 <gribble> New news from bitcoinrss: rebroad opened pull request 1338 on bitcoin/bitcoin <https://github.com/bitcoin/bitcoin/pull/1338>
699 2012-05-17 18:39:54 <gribble> New news from bitcoinrss: rebroad opened pull request 1339 on bitcoin/bitcoin <https://github.com/bitcoin/bitcoin/pull/1339>
700 2012-05-17 18:41:17 <Diapolo> I feel spamed ^^.
701 2012-05-17 18:42:23 <sipa> to spame is always fun
702 2012-05-17 18:46:37 <Diapolo> I can't keep up to read rebroads pulls ... I think too much too quick and no one read longer pull reqs.
703 2012-05-17 18:46:46 <Diapolo> +s
704 2012-05-17 18:50:00 <gribble> New news from bitcoinrss: rebroad opened pull request 1340 on bitcoin/bitcoin <https://github.com/bitcoin/bitcoin/pull/1340>
705 2012-05-17 19:00:06 <gribble> New news from bitcoinrss: rebroad opened pull request 1341 on bitcoin/bitcoin <https://github.com/bitcoin/bitcoin/pull/1341>
706 2012-05-17 19:05:18 <gribble> New news from bitcoinrss: rebroad opened pull request 1342 on bitcoin/bitcoin <https://github.com/bitcoin/bitcoin/pull/1342>
707 2012-05-17 19:10:25 <gribble> New news from bitcoinrss: rebroad opened pull request 1343 on bitcoin/bitcoin <https://github.com/bitcoin/bitcoin/pull/1343>
708 2012-05-17 19:20:32 <gribble> New news from bitcoinrss: rebroad opened pull request 1344 on bitcoin/bitcoin <https://github.com/bitcoin/bitcoin/pull/1344>
709 2012-05-17 19:28:43 <eian> how are people calculating the aggregate hashing power of the network?  Bitcoin charts reports that it is currently 150 petaflops... I'd like to run through the numbers myself to verify this claim
710 2012-05-17 19:29:06 <sipa> eian: the bitcoin network does 0 flop/s
711 2012-05-17 19:29:25 <eian> What?
712 2012-05-17 19:29:33 <eian> Did I word that wrong?
713 2012-05-17 19:29:36 <luke-jr> yes
714 2012-05-17 19:29:43 <luke-jr> flop = floating-point operation
715 2012-05-17 19:29:48 <luke-jr> bitcoin is strictly integer, not FP
716 2012-05-17 19:30:15 <sipa> exactly
717 2012-05-17 19:30:19 <eian> I'm just trying to validate the claim I keep hearing that bitcoin is the "largest distributed computing project"
718 2012-05-17 19:30:27 <eian> I'm looking at http://bitcoincharts.com/bitcoin/
719 2012-05-17 19:30:30 <sipa> it probably is
720 2012-05-17 19:30:38 <gribble> New news from bitcoinrss: laanwj opened pull request 1346 on bitcoin/bitcoin <https://github.com/bitcoin/bitcoin/pull/1346> || Diapolo opened pull request 1345 on bitcoin/bitcoin <https://github.com/bitcoin/bitcoin/pull/1345>
721 2012-05-17 19:31:00 <luke-jr> tcatm: wtf?
722 2012-05-17 19:31:47 <sipa> you can assume a particular hardware distribution for the bitcoin network, map the hash speed to iops/s, map that to hardware units, and calculate the flop/s of that combined assumed hardware
723 2012-05-17 19:31:57 <Diapolo> the gribble links are unusable for me there is always an > appended to the end -_-
724 2012-05-17 19:32:16 <sipa> Diapolo: what irc client?
725 2012-05-17 19:32:22 <eian> sipa, feels like a whole bunch of assuming.
726 2012-05-17 19:32:28 <eian> but thanks
727 2012-05-17 19:32:32 <sipa> eian: it is
728 2012-05-17 19:32:40 <eian> I thought there was some way to calculate based on the difficulty
729 2012-05-17 19:32:58 <sipa> you can calculate the hash rate quite accurately
730 2012-05-17 19:33:00 <Eliel> eian: unfortunately, the type of calculation is rather different.
731 2012-05-17 19:33:24 <sipa> but even calculating the iops/s rate is hard
732 2012-05-17 19:33:27 <eian> Alright, I'll do some reading.  I was hoping someone had a formula somewhere I could look at
733 2012-05-17 19:33:43 <eian> thanks
734 2012-05-17 19:34:12 <Diapolo> https://webchat.freenode.net/?channels=bitcoin-dev&uio=d4
735 2012-05-17 19:34:17 <eian> wait a minute
736 2012-05-17 19:34:24 <eian> sipa, are you the guy behind sipa.be?
737 2012-05-17 19:34:30 <sipa> eian: yes
738 2012-05-17 19:34:35 <eian> :) cool website
739 2012-05-17 19:35:11 <sipa> thanks; it's not very maintained though
740 2012-05-17 19:37:58 <Diapolo> sipa: any idea why the web-irc client does this?
741 2012-05-17 19:38:16 <sipa> no, but try using a real client :)
742 2012-05-17 19:39:00 <Diapolo> I don't want to spend time on IRC stuff ^^ just checkin in without any knowledge what IRC is :D.
743 2012-05-17 19:39:22 <sipa> Diapolo: http://silverex.org has a good one for windows, imho
744 2012-05-17 19:40:01 <Diapolo> I'm not sure but has pidgin one integrated?
745 2012-05-17 19:40:13 <sipa> yes, i think so
746 2012-05-17 19:41:36 <Diapolo> I use that for ICQ, maybe I should try it.
747 2012-05-17 19:41:49 <sipa> ICQ still exists?
748 2012-05-17 19:41:55 <Diapolo> sure
749 2012-05-17 19:42:18 <PiZZaMaN2K> lol yup
750 2012-05-17 19:42:31 <sipa> i had id 36609172, but didn't use it after 2000-2001 somewhere
751 2012-05-17 19:42:52 <Eliel> I had an id too but ... I don't even remember it anymore :)
752 2012-05-17 19:42:52 <sipa> always remembered the number though
753 2012-05-17 19:43:09 <PiZZaMaN2K> i had 36157453 and still use it lol
754 2012-05-17 19:43:16 <PiZZaMaN2K> *have
755 2012-05-17 19:43:29 <PiZZaMaN2K> sad that i remember the id number lol
756 2012-05-17 19:43:46 <sipa> around 2000 everyone started switching to msn here
757 2012-05-17 19:44:10 <sipa> and since some years that's dead too... facebook and gtalk
758 2012-05-17 19:44:39 <PiZZaMaN2K> well the entire AIM staff was layed off
759 2012-05-17 19:44:50 <Diapolo> Never in my live I will login to Facebook!
760 2012-05-17 19:44:55 <PiZZaMaN2K> so my bet is that'll die next
761 2012-05-17 19:45:34 <Diapolo_onPidgin> seems to work :D
762 2012-05-17 19:46:16 <Diapolo_onPidgin> Can someone please open a pull so I can test the gribble links :-P, rebroad?
763 2012-05-17 19:47:13 <sipa> 23:30:38 <@gribble> New news from bitcoinrss: laanwj opened pull request 1346 on bitcoin/bitcoin  <https://github.com/bitcoin/bitcoin/pull/1346> || Diapolo opened pull request 1345 on bitcoin/bitcoin  <https://github.com/bitcoin/bitcoin/pull/1345>
764 2012-05-17 19:47:47 <Diapolo_onPidgin> works
765 2012-05-17 19:48:01 <Diapolo_onPidgin> sipa: thanks
766 2012-05-17 19:54:45 <Diapolo> seems like I can't connect via SSL what port is standard for ssl and IRC?
767 2012-05-17 19:55:14 <wumpus> freenode uses port 7070 for SSL
768 2012-05-17 19:59:35 <Diapolo1> wumpus: thanks
769 2012-05-17 20:02:24 <sipa> Diablo-D3: connection problems?
770 2012-05-17 20:02:45 <sipa> eh
771 2012-05-17 20:02:59 <sipa> Diapolo: connection problems?
772 2012-05-17 20:03:12 <moa7> got stuck updating blockchain on block 180380
773 2012-05-17 20:03:17 <moa7> any ideas?
774 2012-05-17 20:04:02 <Diapolo> sipa: no I switched to IRC via Pidgin and enabled SSL, now everything is fine.
775 2012-05-17 20:06:33 <gribble> New news from bitcoinrss: rebroad opened pull request 1347 on bitcoin/bitcoin <https://github.com/bitcoin/bitcoin/pull/1347>
776 2012-05-17 20:07:26 <Diapolo> wumpus: Btw. why uses Qt the tag notr on style-sheets? seems weird
777 2012-05-17 20:07:39 <rebroad> good evening all
778 2012-05-17 20:07:43 <rebroad> sorry for all the pull requests...
779 2012-05-17 20:07:55 <rebroad> although... I am a little confused why some have been closed without discussion
780 2012-05-17 20:08:22 <rebroad> didn't realise bitcoin on github had become a one person show..
781 2012-05-17 20:08:35 <Diapolo> hi there
782 2012-05-17 20:08:38 <rebroad> (unless any discussion was done on here!)
783 2012-05-17 20:09:19 <rebroad> I do think the renaming of mapAlreadyAskedFor was an important one.. .this bug (referenced here https://github.com/bitcoin/bitcoin/pull/1347) I think would have been less likely to occur with more intuitive naming.
784 2012-05-17 20:10:10 <gmaxwell> Eliel> gmaxwell: I'd disagree about that being bikeshedding.   < It's the Nth post on that topic already. I always feel a little guilty when I'm helpful over minor stuff like that, because it does make discussions go on forever. No worries though, because the guilt never stops me.
785 2012-05-17 20:10:10 <rebroad> with that name, people think it's ok to map if it's already asked for.. it's not.. it should only be mapped if you are actually still Waiting For the inv in question.
786 2012-05-17 20:10:27 <Diapolo> I feel crit by the number of pulls ... I don't even want to read all of them :D. Perhaps a bit more serialisation on pull-creation would be helpful.
787 2012-05-17 20:11:08 <rebroad> Diapolo, crit?
788 2012-05-17 20:12:27 <Diapolo> sorry old WoW-speaking "critically hit" in terms of too much input in a short period of time
789 2012-05-17 20:13:26 <gmaxwell> rebroad: I think jeff closed a couple that were probably not worth the effort reviewing. Please don't take that as an insult your efforts are greatly appreciated.
790 2012-05-17 20:13:26 <rebroad> Diapolo, ah. I understand... anything I can do to help?
791 2012-05-17 20:13:42 <rebroad> gmaxwell, i appreciate you saying that
792 2012-05-17 20:16:02 <rebroad> out of interest..  wha does one need to do to get more involved with the development, e.g. access to helping review issues, etc..?
793 2012-05-17 20:16:58 <gmaxwell> rebroad: even most simple commits I merge take me on the order of half an hour work. I test pulls on multiple nodes (sometimes several at a time though), and I do a reading by stepwise abstraction where I walk through every line of code around the change and write out in english what it did before and after.  (and even with doing this I've still screwed up and pulled some mildly broken stuff due to bad assumptions on the code I wasn't looking