1 2012-10-24 00:01:30 <gmaxwell> 'rainbow tree'
  2 2012-10-24 00:10:32 <xblitz> gmaxwell: hey good job.. you've made the other fork on par with the one im currently on ;)
  3 2012-10-24 00:11:39 <gmaxwell> xblitz: yea, I only bothered putting 600MH/s on it, cause that was enough to overtake before bedtime.
  4 2012-10-24 00:12:09 <gmaxwell> I'm now pinned to that chain.
  5 2012-10-24 00:12:16 <xblitz> how many blocks will it take for my node to use that one instead? 6?
  6 2012-10-24 00:12:39 <gmaxwell> as soon as it has greater sum difficulty it will switch. lemme see..
  7 2012-10-24 00:13:14 <gmaxwell> (it's _very_ important that the decision be stateless: e.g. reorg has to not care about what chain you had before)
  8 2012-10-24 00:13:43 <gmaxwell> 10/24/12 01:53:06 SetBestChain: new best=0000000059d94765aaeb  height=33651  work=1686775377454366  date=10/24/12 01:55:04
  9 2012-10-24 00:13:49 <gmaxwell> whats your highest work number?
 10 2012-10-24 00:14:09 <xblitz> 10/24/12 01:53:07 received block 0000000059d94765aaeb 10/24/12 01:53:07 InvalidChainFound: invalid block=0000000059d94765aaeb  height=33651  work=1686775377454366  date=10/24/12 01:55:04 10/24/12 01:53:07 InvalidChainFound:  current best=00000000fd280811b3e2  height=33651  work=1679019341087110  date=10/23/12 21:43:29 10/24/12 01:53:07 InvalidChainFound: Warning: Displayed transactions may not be correct! You may need to upgrade,
 11 2012-10-24 00:14:11 <xblitz> oh crap
 12 2012-10-24 00:14:21 <xblitz> 10/24/12 01:53:07 received block 0000000059d94765aaeb
 13 2012-10-24 00:14:27 <xblitz> 10/24/12 01:53:07 InvalidChainFound: invalid block=0000000059d94765aaeb  height=33651  work=1686775377454366  date=10/24/12 01:55:04
 14 2012-10-24 00:14:34 <xblitz> 10/24/12 01:53:07 InvalidChainFound:  current best=00000000fd280811b3e2  height=33651  work=1679019341087110  date=10/23/12 21:43:29
 15 2012-10-24 00:14:49 <gmaxwell> you're rejecting my chain still??? did you nuke your chain after initially rejecting that block?
 16 2012-10-24 00:15:08 <xblitz> i didnt do anything special
 17 2012-10-24 00:15:13 <xblitz> im just letting it run
 18 2012-10-24 00:15:17 <gmaxwell> in any case, I have more work now??? you'd be on my chain except you're broken. :P
 19 2012-10-24 00:15:31 <xblitz> im using the current node
 20 2012-10-24 00:15:42 <gmaxwell> xblitz: right so you ran old ultraprune, rejected that block.. it'll never test it again unless you reindex (not merged yet) or nuke your chain.
 21 2012-10-24 00:15:44 <xblitz> current git master
 22 2012-10-24 00:15:59 <xblitz> oh...  booh for that
 23 2012-10-24 00:16:27 <gmaxwell> sorry for not mentioning it before, it's something we've dealt with before so I assumed you knew.
 24 2012-10-24 00:16:30 <xblitz> thank goodness its only testnet
 25 2012-10-24 00:16:45 <gmaxwell> (the BDB maximum locks issue did this to a lot of nodes on mainnet)
 26 2012-10-24 00:17:08 <xblitz> ouch
 27 2012-10-24 00:17:24 <gmaxwell> crazy that work is 2^50.5 now.
 28 2012-10-24 00:18:18 <gmaxwell> four to five more bits for _testnet_ to have done enough operations to have cracked DES.
 29 2012-10-24 00:20:08 <xblitz> okay.. deleted the data.. its redownloading all
 30 2012-10-24 00:31:40 <gmaxwell> sipa: so pindexBest can change while CreateNewBlock runs and will trigger an assert in connect block. Preference for solving that by retrying vs taking a lock?
 31 2012-10-24 00:33:34 <twobitsprite> how is a new bitcoin account/user/client "bootstrapped" in to the network? Is there a central set of "root" servers, or is there some kind of "discovery" system used?
 32 2012-10-24 00:34:39 <gmaxwell> there are no accounts in the bitcoin system
 33 2012-10-24 00:34:51 <gmaxwell> sipa: retring seems to make more sense to me- don't want to mine an old block.
 34 2012-10-24 00:35:25 <gmaxwell> twobitsprite:  the only discovery needed is for bitcoin nodes to find each other. There are several methods used.
 35 2012-10-24 00:35:47 <gmaxwell> twobitsprite: the primary one is that nodes announce themselves and pass the announcements on.. but that only works once you're connected.
 36 2012-10-24 00:37:16 <gmaxwell> twobitsprite: to get connected the first time we have five methods... first, the user can provide a addr.txt file with nodes, or the user can manually -addnode peers, we also can connect to IRC to find nodes (off by default), or query a set of DNS names that resolve to a bunch of recently working nodes, or failing the above the software has a list of 500 randomly selected peers hard coded that it will use as a last resort.
 37 2012-10-24 00:38:14 <gmaxwell> beyond that there is no registration. or identication or user tracking. Nothing in the system knows when a user joins, or how many addresses they have. An address is a purely private thing until you ask someone to send to one of yours.
 38 2012-10-24 00:40:49 <twobitsprite> gmaxwell: interesting... thanks
 39 2012-10-24 00:40:57 <twobitsprite> 22:35 < gmaxwell> twobitsprite:  the only discovery needed is for bitcoin nodes to find each other. There are several methods used.
 40 2012-10-24 00:41:01 <twobitsprite> 22:35 -!- fiesh [~fiesh@p4FCB7245.dip.t-dialin.net] has joined #bitcoin-dev
 41 2012-10-24 00:41:03 <twobitsprite> 22:35 < gmaxwell> twobitsprite: the primary one is that nodes announce themselves and pass the announcements on.. but that only works once you're connected.
 42 2012-10-24 00:41:07 <twobitsprite> 22:37 < gmaxwell> twobitsprite: to get connected the first time we have five methods... first, the user can provide a addr.txt file with nodes, or the user can manually -addnode peers, we also can connect to IRC to find nodes (off by
 43 2012-10-24 00:41:11 <twobitsprite> default), or query a set of DNS names that resolve to a bunch of recently working nodes, or failing the above the software has a list of 500 randomly selected peers hard coded that it will use as a last resort.
 44 2012-10-24 00:41:16 <twobitsprite> 22:38 < gmaxwell> beyond that there is no registration. or identication or user tracking. Nothing in the system knows when a user joins, or how many addresses they have. An address is a purely private thing until you ask someone to send to one of yours.
 45 2012-10-24 00:41:20 <twobitsprite> crap, sorry
 46 2012-10-24 00:41:23 <twobitsprite> stupid putty...
 47 2012-10-24 00:44:23 <zveda> hey why doesn't bitcoin-qt use zbar
 48 2012-10-24 00:44:30 <zveda> to scan qr code from phone with webcame
 49 2012-10-24 00:44:38 <zveda> is ti released under lgpl
 50 2012-10-24 00:44:41 <zveda> it is
 51 2012-10-24 00:45:48 <zveda> even has qt bindings
 52 2012-10-24 00:52:06 <Luke-Jr> zveda: because you didn't write the code yet
 53 2012-10-24 00:52:34 <zveda> but like, it wont be a big waste of time if I do it ?
 54 2012-10-24 00:52:48 <Luke-Jr> I can only speak for myself :p
 55 2012-10-24 00:53:14 <Luke-Jr> zveda: if you implement it (reasonably), I'll support it :P
 56 2012-10-24 00:53:20 <zveda> mm ok
 57 2012-10-24 00:53:59 <jgarzik> sipa: sadly I think the fseek() in LoadExternalBlockFile() is artificially limiting that routine to 2GB
 58 2012-10-24 00:54:05 <jgarzik> due to 32-bit signed long
 59 2012-10-24 01:01:11 <Cory> Does the sum of two ECDSA private keys map to the sum of their public keys?
 60 2012-10-24 01:04:54 <forrestv> Cory, i .. think so
 61 2012-10-24 01:13:16 <Cory> I'm trying to understand https://bitcointalk.org/index.php?topic=84569.0 (Vanity pool), and it makes sense if that's the case.
 62 2012-10-24 01:15:30 <Cory> Do all public-key cryptography algorithms work that way?
 63 2012-10-24 01:15:42 <twobitcoins> gmaxwell: Regarding pindexBest and CreateNewBlock, there is a longstanding problem that callers of CreateNewBlock read from pindexBest and assume it corresponds to the block returned by CreateNewBlock, when it may not.
 64 2012-10-24 01:15:57 <twobitcoins> I fix it in my builds by modifying CreateNewBlock to move the assignment "pindexPrev = pindexBest" inside the LOCK2(cs_main, mempool.cs) and also to return it to callers so they can use the same CBlockIndex used by CreateNewBlock instead of reading pindexBest themselves.
 65 2012-10-24 01:16:04 <twobitcoins> I'm not sure if that fixes the problem you are seeing.
 66 2012-10-24 01:16:08 <forrestv> Cory, btw, it's elliptic curve arithmetic addition
 67 2012-10-24 01:16:22 <forrestv> Cory, no
 68 2012-10-24 01:19:44 <Cory> Thanks. :)
 69 2012-10-24 01:20:43 <gmaxwell> twobitcoins: er, advancing pindexPrev = pindexBest after the fact can leave you with invalid blocks.
 70 2012-10-24 01:21:56 <gmaxwell> jgarzik: should be easy to fix.
 71 2012-10-24 01:22:42 <gmaxwell> twobitcoins: e.g. if you pull transactions for block 3 and then mine block 4 you'll mine a bunch of double spends.
 72 2012-10-24 01:23:40 <twobitcoins> gmaxwell: The point is to guarantee that pindexPrev correctly matches the set of transactions.
 73 2012-10-24 01:23:55 <twobitcoins> So I assign pindexPrev = pIndexBest immediately after taking the lock, which is before the first use of pindexPrev.
 74 2012-10-24 01:24:35 <gmaxwell> Gotcha, yes I could do that... but I'd rather not lock, if a new block is coming in just now we don't want to block it, we want to discard our work and make new work.
 75 2012-10-24 01:24:56 <gmaxwell> (otherwise we may just end up mining against an old point in the chain)
 76 2012-10-24 01:26:02 <twobitcoins> The lock is already there, right?  The troublesome case is when pindexBest changes after assigning pindexPrev = pindexBest and before taking the lock.  By deferring the assignment to pindexPrev, that case is avoided.
 77 2012-10-24 01:29:47 <gmaxwell> twobitcoins: indeed, we take it to build the transactions and thats an issue, but even if I move it there it can move by the time it gets to ConnectBlock. (that lock is only held while its in the mempool)
 78 2012-10-24 01:30:01 <gmaxwell> hm. I'm surprised that this hasn't resulted in invalid blocks on the network.
 79 2012-10-24 01:30:36 <Luke-Jr> gmaxwell: find the rare stuck client bug?
 80 2012-10-24 01:30:56 <twobitcoins> As far as I can tell, the lock is held pretty much until the end of CreateNewBlock, including the call to ConnectBlock.
 81 2012-10-24 01:31:16 <gmaxwell> twobitcoins: you looking at the post ultraprune code?
 82 2012-10-24 01:32:38 <gmaxwell> oh indeed, I fail at indentation reading.
 83 2012-10-24 01:33:23 <gmaxwell> Then fine, moving it into the lock is fine, though I'd prefer to still toss the stale work. At least it won't be invalid.
 84 2012-10-24 01:34:22 <gmaxwell> Luke-Jr: no, just an obsecure mining race condition.
 85 2012-10-24 01:43:54 <twobitcoins> Work can become stale at any moment, so it's sort of futile to try to return non-stale work because it may become stale the next instant anyway.  CreateNewBlock has to return work based on the state at *some* instant.  If it becomes stale, the next call to CreateNewBlock will fix it.
 86 2012-10-24 01:45:36 <twobitcoins> For correctness though, CreateNewBlock should pass the correct pindexPrev to the caller as well as using it itself, otherwise the caller can operate on an inconsistent state of the world and mine invalid blocks.
 87 2012-10-24 01:52:55 <Luke-Jr> 
ACTION wonders if his LP pullreq fixes this

 88 2012-10-24 02:57:27 <MC1984> fuck yeah 3d printing
 89 2012-10-24 02:57:36 <MC1984> post-scarcity rising
 90 2012-10-24 02:58:36 <MC1984> but first, a long painful period of prohibition and a 'War on Plenty' akin to the War on Piracy of today
 91 2012-10-24 03:12:38 <MC1984> i think the end game of 3d printers is literally "push button, receive bacon"
 92 2012-10-24 03:12:45 <MC1984> a glorious future awaits
 93 2012-10-24 03:21:53 <gmaxwell> MC1984: we reached post-scarcity in the 1970s. The most important goods and tools today are pure information with zero marginal cost.
 94 2012-10-24 03:23:15 <MC1984> not quite
 95 2012-10-24 03:23:42 <MC1984> information post scarcity is becoming a reality and its causing a shitstorm
 96 2012-10-24 03:24:04 <MC1984> it will push out into the real world eventually
 97 2012-10-24 03:25:32 <MC1984> i often wonder if the lack of jobs is consequent of this
 98 2012-10-24 03:25:41 <MC1984> if its just the beginning
 99 2012-10-24 03:26:17 <MC1984> it seems near 100% unemployment is an end result of post scarcity
100 2012-10-24 03:26:29 <gmaxwell> this is way offtopic.
101 2012-10-24 03:26:35 <gmaxwell> #bitcoin?
102 2012-10-24 03:27:09 <MC1984> >implying we will become a civilisation of artists and philisophers when the need to work is removed, instead of sit around in our pants al day pushing buttons and receiving bacon
103 2012-10-24 04:01:35 <gmaxwell> twobitcoins: https://github.com/bitcoin/bitcoin/pull/1953 FYI
104 2012-10-24 04:10:08 <twobitcoins> gmaxwell: The comment says "Enforce on CreateNewBlock invocations which don't have a hash." but it seems like the code *doesn't* enforce in that case (if my brain is working properly).
105 2012-10-24 04:12:06 <gmaxwell> yea, fair enough. I meant don't but in hindsight thats kinda dumb, though harmless currently. Long month.
106 2012-10-24 04:15:31 <twobitcoins> gmaxwell: Yeah, not too important either way, just an inconsistency.
107 2012-10-24 04:19:27 <twobitcoins> gmaxwell: Also, I guess you didn't want to pass pindexPrev to the callers of CreateNewBlock?  It means the callers may still do things based on inconsistent state, like putting the wrong height into the scriptsig or calculating the wrong mintime in getblocktemplate.
108 2012-10-24 04:25:09 <gmaxwell> twobitcoins: haven't decided what I wanted to do with restructing that, nor have I been able to reproduce it yet, but moving the lock was simple and I'd already tested it. I haven't forgotten it though. :)
109 2012-10-24 04:29:17 <twobitcoins> gmaxwell: Okay, no problem.  :-)
110 2012-10-24 04:29:30 <twobitcoins> gmaxwell: Thanks for taking a look at these things.
111 2012-10-24 05:03:18 <sipa> jgarzik: yes, the fseek use could be avoided
112 2012-10-24 05:07:58 <sipa> gmaxwell: i thought i already moved the querying op pindexBest to below the LOCK2 in that block?
113 2012-10-24 05:08:13 <sipa> gmaxwell: oh, no i only suggested that to Diapolo
114 2012-10-24 05:16:34 <sipa> gmaxwell: oh, that's what you did
115 2012-10-24 06:28:06 <ibno> would it be possible to implement mental poker with bitcoin contracts?
116 2012-10-24 06:28:54 <ibno> has anyone tried?
117 2012-10-24 07:01:15 <zveda> sorta
118 2012-10-24 07:01:18 <zveda> not me but
119 2012-10-24 07:01:59 <zveda> ibno: I think I read bitzino are doing it
120 2012-10-24 07:02:56 <ibno> will they open source it?
121 2012-10-24 07:05:52 <zveda> no idea
122 2012-10-24 07:05:57 <zveda> hope so
123 2012-10-24 08:37:10 <TD> BlueMatt: ping
124 2012-10-24 09:12:21 <ibno> where can I download the testnet blockchain?
125 2012-10-24 09:13:03 <ibno> nvm, it goes pretty quickly via the client
126 2012-10-24 09:17:26 <BlueMattBot> Project Bitcoin build #116: STILL FAILING in 1 hr 57 min: http://jenkins.bluematt.me/job/Bitcoin/116/
127 2012-10-24 10:29:45 <ibno> how is blkindex.dat structured?
128 2012-10-24 10:37:41 <Luke-Jr> ibno: bdb 4.8 database
129 2012-10-24 10:37:46 <Luke-Jr> ibno: going away with 0.8
130 2012-10-24 10:44:45 <ibno> Luke-Jr: ok, what will replace it?
131 2012-10-24 10:45:38 <sipa> ibno: two leveldb databases
132 2012-10-24 10:50:10 <drizztbsd> sipa: is your tor hidden service always on?
133 2012-10-24 10:51:35 <sipa> drizztbsd: should be
134 2012-10-24 10:51:41 <TD> hety
135 2012-10-24 10:51:44 <TD> hey
136 2012-10-24 10:52:11 <drizztbsd> ok, thanks
137 2012-10-24 10:52:42 <drizztbsd> which is the advantage to use the hidden service vs exit node?
138 2012-10-24 10:53:23 <sipa> drizztbsd: the fact that the traffic doesn't need to leave the tor network
139 2012-10-24 10:53:55 <drizztbsd> is it possibile to support i2p?
140 2012-10-24 10:54:01 <drizztbsd> teoretically
141 2012-10-24 10:55:52 <sipa> not right now; i2p needs 256-bit routable addresses, and we only have IPv6 (which is 128 bits)
142 2012-10-24 10:56:08 <sipa> IPv4 and onion are both mapped into a subrange of IPv6, but for I2P that isn't possible
143 2012-10-24 10:56:29 <drizztbsd> ok, thanks
144 2012-10-24 11:10:44 <ibno> will everyone have to download the blockchain at new with the 0.8 release?
145 2012-10-24 11:10:49 <sipa> no
146 2012-10-24 11:11:02 <sipa> it will reindex the block chain you already have
147 2012-10-24 11:11:12 <sipa> but that isn't implemented entirely yet
148 2012-10-24 11:11:18 <ibno> ok
149 2012-10-24 11:22:00 <TD> sipa: what stops somebody sending a giant message to a peer, to make it OOM?
150 2012-10-24 11:24:40 <sipa> TD: there's a limit on message size, afaik
151 2012-10-24 11:25:32 <helo> gmaxwell: ~2 hours with -loadblock (same system that took ~4hr from network)
152 2012-10-24 11:32:20 <jgarzik> serialize.h:static const unsigned int MAX_SIZE = 0x02000000;
153 2012-10-24 11:32:21 <jgarzik> main.cpp:        if (nMessageSize > MAX_SIZE)
154 2012-10-24 11:34:50 <drizztbsd> -connect supports port?
155 2012-10-24 11:34:55 <drizztbsd> -connect=ip:port?
156 2012-10-24 11:35:08 <sipa> drizztbsd: yes
157 2012-10-24 11:35:26 <sipa> jgarzik, TD: so, max 32 MiB...
158 2012-10-24 11:35:29 <TD> ok
159 2012-10-24 11:37:46 <drizztbsd> curl -sk https://en.bitcoin.it/wiki/Fallback_Nodes\\#Tor_network | sed -n 's/^<td> \\(.*\\.onion.*\\) <\\/td>$/\\1/p' | while IFS=: read -r host port; do echo "-connect=`dig +short -p 9053 @127.0.0.1 "$host"`${port:+:}$port"; done | xargs bitcoin-qt
160 2012-10-24 11:37:54 <drizztbsd> very raw, but it works greatly :P
161 2012-10-24 11:38:55 <jgarzik> ;;seen gavinandresen
162 2012-10-24 11:38:56 <gribble> gavinandresen was last seen in #bitcoin-dev 20 hours, 35 minutes, and 37 seconds ago: <gavinandresen> sipa:  rather than making IBD faster, I think it would be better to work on a headers-first, backfill-in-background mode.  It doesn't matter if the backfill takes 1 hour or 11.
163 2012-10-24 11:39:02 <sipa> drizztbsd: bitcoin.sipa.be/seeds.txt has crawling results, including onion nodes
164 2012-10-24 11:39:24 <drizztbsd> uhm thanks
165 2012-10-24 11:42:08 <helo> that text file is in swahili according to google Oo
166 2012-10-24 11:44:22 <sipa> TD: who should I talk to, to fix that? :)
167 2012-10-24 11:44:38 <TD> hah
168 2012-10-24 11:44:57 <TD> i think that's a good guess at the language. i mean it might as well be swahili
169 2012-10-24 11:45:27 <sipa> there are only a few words in it, and all of them are english
170 2012-10-24 11:46:24 <sipa> even Japanese would be a better guess, with "Satoshi" occurring a few thousand times in the text
171 2012-10-24 11:47:20 <TD> language detection is based on very aggressive heuristics, probably also bloom filters
172 2012-10-24 11:47:31 <TD> if you give it basically random input you'll get basically random output
173 2012-10-24 11:47:48 <sipa> right
174 2012-10-24 11:47:54 <TD> it has to be rough because the langdet tables have to be in RAM all the time, so size really matters a lot
175 2012-10-24 11:49:04 <sipa> still, if you'd do very limited preprocessing before anything else, which drops all obvious non-text data, i think the guess should be good
176 2012-10-24 11:49:09 <sipa> then agian, hard problem indeed
177 2012-10-24 11:49:22 <TD> "obvious non text data" in a world with hundreds of languages? :)
178 2012-10-24 11:49:34 <TD> yeah it's sort of a hard CS problem. obviously server-side language detection is much more accurate
179 2012-10-24 11:49:46 <drizztbsd> curl -s http://bitcoin.sipa.be/seeds.txt | cut -f 1 | fgrep .onion pwnz
180 2012-10-24 11:49:53 <TD> huh
181 2012-10-24 11:50:01 <TD> bitcoin wiki has an SSL cert from the future?!
182 2012-10-24 11:50:07 <drizztbsd> lol
183 2012-10-24 11:50:21 <gmaxwell> TD: Bitcoin is alien technology after all.
184 2012-10-24 11:50:28 <TD> hah
185 2012-10-24 11:50:52 <TD> i've never see this error before
186 2012-10-24 11:51:01 <sipa> Issued On: 12/15/10
187 2012-10-24 11:51:14 <TD> so it should be valid
188 2012-10-24 11:51:28 <drizztbsd> curl: (60) SSL certificate problem: certificate has expired
189 2012-10-24 11:51:30 <TD> i wonder if it's chaining through an invalid cert or to an authority chrome doesn't recognize
190 2012-10-24 11:52:14 <TD> well shit
191 2012-10-24 11:52:19 <TD> startcoms certificate has expired
192 2012-10-24 11:52:30 <gmaxwell> 0_o
193 2012-10-24 11:52:34 <TD> boggle
194 2012-10-24 11:52:37 <TD> i wonder how that happens
195 2012-10-24 11:52:39 <gmaxwell> is there a new chaining cert that we should be using?
196 2012-10-24 11:52:46 <gmaxwell> s/we/the wiki/
197 2012-10-24 11:53:30 <sipa> TD: according to my browser, all three certs in the path are valid
198 2012-10-24 11:53:37 <TD> that's odd
199 2012-10-24 11:53:42 <TD> what browser/version do you have
200 2012-10-24 11:54:05 <sipa> Builtin Object Token:StartCom Certificate Authority: Not After: 9/17/36 (wtf... 2036?)
201 2012-10-24 11:54:37 <sipa> StartCom Class 2 Primare Intermediate Server CA: Not After 10/24/17
202 2012-10-24 11:54:40 <TD> https://forum.startcom.org/viewtopic.php?f=15&t=2534
203 2012-10-24 11:54:56 <TD> those are some weird date formats
204 2012-10-24 11:55:07 <TD> i think Tux has to get new certs that chain through a newer intermediate CA
205 2012-10-24 11:55:30 <TD> i'll pm mark
206 2012-10-24 11:59:07 <davout> hey
207 2012-10-24 11:59:24 <davout> can someone in the UK do me a tiny little favor ?
208 2012-10-24 12:06:11 <Icoin> hi everyone i run a bitcoind on a raspberrypi by now i reached 181631 blocks with the precompiled 3.2.4 client. I tried to compile the 7 version but i end up with lack of memory in the standard configuration, i try now to load the chain fully to see if that works, but it would be great if there where a source for the binary similar to the ubuntu ppa process just for debian wheezy raspberrypi edition. After successuful chainload i try to ad
209 2012-10-24 12:06:12 <Icoin> 512 mb swap and try to recompile the latest daemon again
210 2012-10-24 12:07:48 <helo> the reference client isn't really very well suited for a rpi
211 2012-10-24 12:08:31 <gmaxwell> nothing is really well suited for a rpi.
212 2012-10-24 12:08:35 <helo> it's optimized for machines with much faster procs, and much more ram
213 2012-10-24 12:09:04 <gmaxwell> In any case, I wouldn't even bother trying prior to 0.8.
214 2012-10-24 12:09:44 <gmaxwell> helo: bluematt has bitcoin running on some small arm sbcs... but they are ones which are like 16x faster than the rpi.
215 2012-10-24 12:09:53 <Icoin> gmaxwell: but you understand the significance of running a bitcoind on a pi ?
216 2012-10-24 12:10:08 <helo> 
ACTION does not

217 2012-10-24 12:12:15 <gmaxwell> Icoin: The pi is a really handicapped computer, it is an old in order single issue core, which most of the available cache handed over to the useless (to us) dsp on board... and a quite small amount of attached ram. It will never have good performance. And while I'm always amused to see bitcoin run on more limited hardware, it's not a good target for bitcoind.
218 2012-10-24 12:12:39 <gmaxwell> (or, apparently g++ :) )
219 2012-10-24 12:13:27 <sipa> well it'd be nice if it worked, but it's not a priority to make the user experience of bitcoin running on such low-spec deviced good
220 2012-10-24 12:13:29 <Icoin> gmaxwell: i understand, but its a awesome way for little students become familiar with the toppic
221 2012-10-24 12:13:57 <sipa> students that have a rpi, but no laptop or desktop system?
222 2012-10-24 12:13:58 <helo> hopefully students have access to more of a machine than a pi
223 2012-10-24 12:14:09 <gmaxwell> Icoin: and a way for them to get a low opinion of it when it doesn't work well on hardware that is too small to handle it.
224 2012-10-24 12:14:41 <Icoin> gmaxwell: when we follow the one computer per chide idea then not every child is able to handle a massive hardware infrastructure like in western countries
225 2012-10-24 12:14:54 <helo> wouldn't pi students focus more on the basics of embedded programming input/output?
226 2012-10-24 12:15:59 <Icoin> helo: well see it from the perspective that your parents needs to work for a month or so to be able to buy a computer like the rpi, is there an other hardware alternative ??
227 2012-10-24 12:16:36 <helo> children with raspberry pis will be using them to learn, rather than be trying to run a full bitcoin node
228 2012-10-24 12:16:49 <Icoin> helo: yes i agree
229 2012-10-24 12:17:02 <sipa> you'll want to run a lightweight client on a pi, not a full node
230 2012-10-24 12:17:07 <Icoin> helo: its not the purpose of the rpi bitcoind to run a full node
231 2012-10-24 12:17:09 <helo> it really isn't very educational to make; ./bitcoind
232 2012-10-24 12:17:39 <sipa> but i do expect an rpi to be fast enough to keep up-to-date, actually, though IBD will be slow
233 2012-10-24 12:18:02 <Icoin> helo: grins, well i wouldnt say so since not many even know there is such a thing like ./bitcoind
234 2012-10-24 12:18:07 <gmaxwell> sipa: I'd be somewhat surprised, it's surprisingly slow.
235 2012-10-24 12:18:49 <gmaxwell> For learning testnet is good, or even better??? but unfortunately if you can't compile it... well, not much room for learning.
236 2012-10-24 12:19:11 <sipa> at least for SHA256, it's an order of magnitude slower than a core2 duo 2.2 GHz
237 2012-10-24 12:19:28 <sipa> i expect a core2 duo to still do around a block per second these days
238 2012-10-24 12:19:40 <Icoin> gmaxwell: 3.2.4 was compiled and by now it works
239 2012-10-24 12:19:43 <sipa> the question is how fast I/O is
240 2012-10-24 12:19:54 <Icoin> at least the blockchain load
241 2012-10-24 12:19:57 <sipa> Icoin: if you're only up to 181k, you haven't hit the hardest part yet
242 2012-10-24 12:20:16 <gmaxwell> Icoin: it was a cross compiled binary. Most people don't compile _on_ such limited systems.
243 2012-10-24 12:20:27 <Icoin> sipa: yes i know but memory usage is only 40% by now
244 2012-10-24 12:20:42 <epscy> i got bitcoin running on a rpi
245 2012-10-24 12:20:43 <sipa> how long did it take to sync to 181k?
246 2012-10-24 12:20:51 <epscy> realised how slow it was
247 2012-10-24 12:20:56 <epscy> and stopped it
248 2012-10-24 12:21:00 <Icoin> epscy what version ?
249 2012-10-24 12:21:24 <epscy> 0.6 probably
250 2012-10-24 12:21:30 <epscy> i can't remember
251 2012-10-24 12:21:37 <Icoin> did you compile yourself ?
252 2012-10-24 12:21:43 <epscy> i think so
253 2012-10-24 12:21:54 <epscy> i am disspointed with the rpi
254 2012-10-24 12:21:54 <sipa> Icoin: how long did it take you to sync 181k blocks?
255 2012-10-24 12:22:13 <epscy> oh i copied the blockchain from elsewhere
256 2012-10-24 12:22:23 <epscy> i didn't wait for it to sync by itself
257 2012-10-24 12:22:44 <Icoin> sipa: by now like 3 days
258 2012-10-24 12:23:00 <epscy> what i have noticed with the rpi is whenever it writes to the sd card the whole system seems to freeze
259 2012-10-24 12:23:15 <epscy> tried with another sd card now so i don't think it is just me
260 2012-10-24 12:23:29 <sipa> Icoin: impressive, with 0.3.24 even (more recent versions should be faster)
261 2012-10-24 12:23:30 <epscy> this means the rpi is fine for doing stuff that runs in memory
262 2012-10-24 12:23:49 <epscy> but anything using the disk will always be annoying to use
263 2012-10-24 12:23:56 <Icoin> epscy: depends on the raspberian version you use, you ever tried razor ?
264 2012-10-24 12:24:05 <epscy> Icoin: nope?
265 2012-10-24 12:24:33 <Icoin> there are various compilations avalable
266 2012-10-24 12:24:40 <epscy> Icoin: got a link
267 2012-10-24 12:24:43 <epscy> ?
268 2012-10-24 12:24:58 <gmaxwell> epscy: the pandaboard has a similar problem with the sd card. Reading/Writing to it is fairly slow and it blocks in the kernel while doing it.
269 2012-10-24 12:25:11 <sipa> leveldb should help a lot on such storage
270 2012-10-24 12:25:26 <sipa> i should unpack my rpi one day...
271 2012-10-24 12:25:29 <Icoin> gmaxwell: use the latest sd cards avalable to solve this problem a bit
272 2012-10-24 12:25:59 <Icoin> i realized it is depending on the speed of the sdcard how fast your rpi realy is
273 2012-10-24 12:26:03 <gmaxwell> I solved it by running my pandaboard off network storage.
274 2012-10-24 12:26:04 <epscy> gmaxwell: would running of a usb flash disk help at all?
275 2012-10-24 12:26:13 <thermoman> Dependency changes: openssl 1.0.1c - do i need this version at runtime or only when compiling? running debian squeeze with openssl 0.9.8o
276 2012-10-24 12:26:25 <epscy> actually i have a small portable HDD
277 2012-10-24 12:26:35 <gmaxwell> epscy: thats what some people do on the pandaboards, but I thought I heard the rpi has usb controller speed issues too?
278 2012-10-24 12:26:38 <sipa> thermoman: when you compile yourself, you'll use whatever openssl is available at compile time
279 2012-10-24 12:26:44 <gmaxwell> thermoman: No.
280 2012-10-24 12:26:44 <Icoin> epscy: yeah that would work aswell
281 2012-10-24 12:26:46 <epscy> it would be interesting to see how it performs running off that
282 2012-10-24 12:27:08 <Icoin> epscy: please try when you have the patience for it :)
283 2012-10-24 12:27:12 <epscy> gmaxwell: oh i haven't heard about that
284 2012-10-24 12:27:16 <thermoman> sipa: then why is this mentioned in the changelog if it's non-relevant?
285 2012-10-24 12:27:33 <sipa> thermoman: because the changelog is about the release
286 2012-10-24 12:27:34 <epscy> i get the impression getting it to boot from a usb device is a fair bit of work
287 2012-10-24 12:28:26 <epscy> Icoin: the problem with the HDD is that it would require it's own power source too, so now i have two wall warts for a raspberry pi
288 2012-10-24 12:28:29 <thermoman> sipa: the changelos tells me the new release depends on openssl 1.0.1c - and you tell me i'm fine with 0.9.8c what i have. so i'm confused
289 2012-10-24 12:28:47 <epscy> and that point i start to wonder if the whole endeavour is worth it
290 2012-10-24 12:29:11 <sipa> thermoman: the source code doesn't require a specific openssl version, but released binaries have a dependency on that openssl version
291 2012-10-24 12:29:41 <thermoman> sipa: ok, so i can't run this release on debian squeeze, right?
292 2012-10-24 12:29:56 <thermoman> that's bad news
293 2012-10-24 12:30:03 <sipa> thermoman: are there even binaries for debian squeeze?
294 2012-10-24 12:30:17 <kinlo> isn't it possible to just make a static build?
295 2012-10-24 12:30:22 <thermoman> sipa: i just downloaded the 0.7.1 tarball
296 2012-10-24 12:30:27 <thermoman> there are binaries included
297 2012-10-24 12:30:37 <thermoman> these i would use
298 2012-10-24 12:30:38 <sipa> thermoman: well, if you compile it yourself it should work
299 2012-10-24 12:30:50 <Icoin> epscy: i gonna do some more test with the rpi, i like the idea probably the swap extension will solve the compilation problem
300 2012-10-24 12:31:06 <kinlo> in any case, I don't really the problem
301 2012-10-24 12:31:20 <kinlo> I'm using debian squeeze everywhere and it seems to work fine
302 2012-10-24 12:31:34 <kinlo> hmmmz, altough the only 0.7.x versions I'm running are source-compiled...
303 2012-10-24 12:33:58 <Icoin> epscy: if you do something with you rpi i suggest to use berryboot http://www.berryterminal.com/doku.php/berryboot
304 2012-10-24 12:34:32 <Icoin> epscy: try it with razor :)
305 2012-10-24 12:35:22 <Icoin> epscy: the process is very simple: To install extract the contents of the .zip file to a normal FAT formatted SD card, and put it in your Raspberry Pi. This can be simply done under Windows without any special image writer software.
306 2012-10-24 12:35:50 <BlueMatt> helo/gmaxwell/Icoin: I thought I'd heard of people running bitcoind fine on an rpi (skipping IBD by copying blkindex, etc from their own comp though)
307 2012-10-24 12:36:02 <epscy> Icoin: that does sound easy
308 2012-10-24 12:36:18 <Icoin> epscy: it is
309 2012-10-24 12:36:28 <thermoman> sipa, kinlo: started precompiled bitcoind on debian squeeze and it works - should i now compile it myself to be safe or is it good to go?
310 2012-10-24 12:37:00 <kinlo> the precompiled is the safe bet - if you start using your own you might get issues with the databases....
311 2012-10-24 12:37:14 <kinlo> I've seen berkleydb doing very strange things
312 2012-10-24 12:37:19 <thermoman> that's why i would really hate it to use the self compiled one
313 2012-10-24 12:37:21 <kinlo> actually quite happy it will be removed
314 2012-10-24 12:37:35 <thermoman> 
ACTION too - berkeley db performance is baaaad

315 2012-10-24 12:37:56 <thermoman> so you run 0.7.1 on squeeze with no problems?
316 2012-10-24 12:38:08 <thermoman> the binary release i mean
317 2012-10-24 12:41:22 <kinlo> I run 0.6.x on squeeze with no problems
318 2012-10-24 12:41:31 <kinlo> and patched/source compiled versions
319 2012-10-24 12:41:45 <kinlo> but since they are patched they are self-compiled so ...
320 2012-10-24 12:42:01 <thermoman> just ran precompiled bitcoind 0.7.1 and it downloads the blocks ... so i guess it's good to go
321 2012-10-24 12:42:23 <thermoman> what do you patch?
322 2012-10-24 12:47:38 <BlueMatt> sipa: not sure how I missed this, here is the real jenkins error: "port/port_win.cc:33:21: fatal error: windows.h: No such file or directory"
323 2012-10-24 12:48:14 <BlueMatt> also...why did make continue trying to build after a calling the leveldb make failed?
324 2012-10-24 12:50:45 <sipa> BlueMatt: oh, i copy-pasted that line the first time i mentioned the problem
325 2012-10-24 12:51:08 <BlueMatt> oh, ok Im just blind (again)
326 2012-10-24 12:51:29 <BlueMatt> though it does seem like the makefile is broken if it will try to continue after the leveldb stuff fails
327 2012-10-24 12:51:34 <sipa> agree
328 2012-10-24 12:51:47 <sipa> and no, can't expect anyone to see everything said on IRC
329 2012-10-24 12:51:52 <sipa> +problem
330 2012-10-24 13:40:06 <TD> BlueMatt: i have a question about initializing murmur
331 2012-10-24 13:40:11 <TD> the bootstrap formula is nHashNum * (MAX_UINT32 / (nHashFuncs - 1))
332 2012-10-24 13:40:21 <TD> so that means h1 will be zero for the first hash func
333 2012-10-24 13:40:46 <TD> is that an issue? it seems like it could be just as easy to root the hash func ID at 1 and then not subtract 1
334 2012-10-24 13:41:38 <TD> i think it's not really an issue
335 2012-10-24 13:45:31 <BlueMatt> TD: I dunno, I dont think the hash function is broken with a seed of 0, or I cant see any reason why it would be
336 2012-10-24 13:46:07 <stamit> you know, i kinda feel like reading the things on -otc now, without having to go through a proxy or a paid VPN or anything
337 2012-10-24 13:46:51 <TD> i wonder if the bitcoin wiki supports mathematics notation
338 2012-10-24 13:48:21 <ThomasV> how is a compact converted into bignum?
339 2012-10-24 13:49:31 <ThomasV> is there a doc about that?
340 2012-10-24 13:51:11 <stamit> i would also like my $126, my $70.25, my $30 and my 4.6765 BTC
341 2012-10-24 13:51:31 <stamit> *$70.23
342 2012-10-24 13:53:47 <stamit> there is also another 20 EUR, but maybe i'll be able to sort that out without help
343 2012-10-24 13:54:27 <ThomasV> ok, I guess this is the formula: https://en.bitcoin.it/wiki/Difficulty#How_is_difficulty_stored_in_blocks.3F
344 2012-10-24 13:56:28 <TD> phew
345 2012-10-24 13:56:30 <gmaxwell> sipa: In the main ultraprune commit,
346 2012-10-24 13:56:31 <gmaxwell> -    if (fHelp || params.size() > 1)
347 2012-10-24 13:56:34 <gmaxwell> Huh?
348 2012-10-24 13:56:34 <TD> BIP 37 for bloom filtering, uploaded
349 2012-10-24 13:57:28 <gmaxwell> ThomasV: yep.
350 2012-10-24 13:59:00 <ThomasV> gmaxwell: it works. but I find it strange that the interval is 2015 blocks long, and not 2016
351 2012-10-24 14:01:31 <gmaxwell> ThomasV: it measures 2016 back from the next block, which results in a flaw in the algorithim in fact??? where a majority attacker can create inflation???, because the difficulty windows do not overlap.
352 2012-10-24 14:01:49 <ThomasV> huh?
353 2012-10-24 14:02:58 <sipa> gmaxwell: what RPC?
354 2012-10-24 14:03:21 <sipa> TD: reading BIP37 - why limit the number of matching transactions to 255?
355 2012-10-24 14:06:01 <JWU_42> ay chance removeprivkey will be added anytime soon?
356 2012-10-24 14:06:06 <JWU_42> *any
357 2012-10-24 14:06:15 <JWU_42> to a stable release that is...
358 2012-10-24 14:08:49 <sipa> TD: what does "each hash and key in the output script" mean exactly? what about the output script in its entirety?
359 2012-10-24 14:11:38 <sipa> TD: hmm, is sharing parts of the merkle branches not worth it?
360 2012-10-24 14:14:50 <sipa> gmaxwell: in getblocktemplate? hmm, maybe a rebasing conflict...
361 2012-10-24 14:23:00 <JWU_42> ok - so patching bitcoind is my only bet it seems...
362 2012-10-24 14:24:09 <sipa> JWU_42: why do you want to remove a key?
363 2012-10-24 14:25:55 <JWU_42> sipa: I have decided to keep (not sell) coins I solo mine.  I created a new wallet as i dumped the privkey from the wallet associated with bitcoind.  Now want to remove that privkey from the "mining" wallet
364 2012-10-24 14:26:59 <JWU_42> maybe at some point in the very distant future these mined coins will have a greater value - so they go to cold storage...
365 2012-10-24 14:27:47 <jeremias> why not just transfer the coins...
366 2012-10-24 14:27:52 <JWU_42> so having dump without remove seems to allow the potential for misbehaviour
367 2012-10-24 14:28:12 <JWU_42> jeremias: then they are not "as virgin"
368 2012-10-24 14:28:17 <jeremias> lol
369 2012-10-24 14:28:17 <JWU_42> ;)
370 2012-10-24 14:28:26 <jeremias> well, if you care, then you can do it by hand
371 2012-10-24 14:28:38 <jeremias> that feature is definitely a very small niche
372 2012-10-24 14:28:38 <JWU_42> yeah - that is what it is coming down too
373 2012-10-24 14:29:03 <BlueMatt> TD[gone]: the type/size/etc of the filter created on receiving a filteradd message without a filterload should not (IMHO) be set in the spec, it should be up to the serving node to do something reasonable
374 2012-10-24 14:29:04 <jeremias> just because you have fetish with virgin coins, you want that feature
375 2012-10-24 14:29:12 <jeremias> 99% of users won't care
376 2012-10-24 14:29:55 <jeremias> but I guess with pywallet you can remove the privkey
377 2012-10-24 14:29:59 <jeremias> or patch your bitcoind
378 2012-10-24 14:30:46 <JWU_42> jeremias: I think it has utility beyond my peculiar circumstance
379 2012-10-24 14:30:59 <JWU_42> i just question why have dump without remove
380 2012-10-24 14:31:14 <jeremias> remove is definitely dangerous operation
381 2012-10-24 14:31:15 <JWU_42> the only reason to dump is to then import elsewhere
382 2012-10-24 14:31:16 <jeremias> and useless
383 2012-10-24 14:31:27 <JWU_42> I suspect...
384 2012-10-24 14:31:42 <jeremias> but yeah, even dump is a little bit questionable
385 2012-10-24 14:31:55 <jeremias> import should return the resulted bitcoin address, btw
386 2012-10-24 14:32:01 <jeremias> now it returns nothing
387 2012-10-24 14:32:04 <JWU_42> I suspect that is my main point and why i had used pywallet in the past
388 2012-10-24 14:32:51 <JWU_42> jeremias: yeah - now i have the 50 BTC in two wallets - which makes perfect sense until i can remove the key from one of the wallets
389 2012-10-24 14:33:20 <gmaxwell> sipa: yes, getblocktemplate
390 2012-10-24 14:33:32 <JWU_42> I found some comments via goolging about the removeprivkey being out there but seems to not be of high priority
391 2012-10-24 14:34:48 <gmaxwell> JWU_42: There is no remove private key because it has unexpected results, even beyond the normal danger of erasing data.
392 2012-10-24 14:35:22 <JWU_42> gmaxwell: OK :/
393 2012-10-24 14:37:25 <slush> gmaxwell: sipa: btw genjix didn't respond to my BIP number request.
394 2012-10-24 14:39:17 <sipa> slush: meh, pick BIP38, i suppose
395 2012-10-24 14:41:20 <gmaxwell> d
396 2012-10-24 14:41:24 <slush> sipa: yes, we have complete workflow described by BIP1, tuned by the community to perfectly fit all our needs. And few months later, this ends with "meh, pick BIP38" on the IRC <sarcasm/>
397 2012-10-24 14:41:52 <slush> sipa: I'll do so. I just hate burreaucracy
398 2012-10-24 14:42:09 <gmaxwell> slush: We'll sort it out.
399 2012-10-24 14:42:19 <gmaxwell> Would you prefer we replace genjix with a distributed algorithim?
400 2012-10-24 14:42:29 <gmaxwell> Merged mined BIPchain?
401 2012-10-24 14:42:37 <slush> gmaxwell: we should vote about bips with coinbase
402 2012-10-24 14:42:46 <slush> the best way is to create another bip for coinbase voting
403 2012-10-24 14:42:54 <gmaxwell> slush: the community is more than miners. :P
404 2012-10-24 14:43:34 <slush> gmaxwell: are you slowly moving bitcoin from capitalist (miners have the power) to communist (community have the power)? ;)
405 2012-10-24 14:43:41 <sipa> how about using randomly generated UUIDs for BIPs? ;)
406 2012-10-24 14:43:42 <gmaxwell> slush: but we do coinbase votes for things that are unsafe without super-majority enforcement... Eventually that will get a standalone spec.
407 2012-10-24 14:43:43 <sipa> *ducks*
408 2012-10-24 14:44:17 <sipa> oh no, just use the double-SHA256 of the markup code!
409 2012-10-24 14:44:36 <sipa> we'll add a nLockTime and nSequence to allow replacements!
410 2012-10-24 14:44:39 <slush> sipa: lol, let's imagine wars between BIP 05f75160-1dfa-11e2-81c1-0800200c9a66 and BIP  acd71f04-8bf9-44dd-9b7b-cdc78615c892
411 2012-10-24 14:44:59 <gmaxwell> sipa: just make the bip a bond in jeff's blockchain spammer, the private key that owns the bond can update the bit. :P  And to create a new bip you need a merged mined token of at least some difficulty. :P
412 2012-10-24 14:45:11 <slush> gmaxwell: I know about coinbase voting, of course. I was just kidding
413 2012-10-24 14:45:50 <gmaxwell> (ah, obviously you know, didn't know if you knew the key point was the security bit. Some people seemed to think we did it as a way to measure approval :P )
414 2012-10-24 14:45:50 <sipa> slush: i'm in favor of some formal process for guaranteeing unique BIP ids, and maybe very limited selection to rule the idiotic parts
415 2012-10-24 14:46:02 <sipa> slush: but if that process becomes a burden, better have not any at all
416 2012-10-24 14:46:13 <gmaxwell> Yea, I'd be fine with something that gives IDs but I really would prefer there exist some selection mechenism.
417 2012-10-24 14:46:49 <gmaxwell> Because otherwise we'll have 20 million  "stock split bitcoin!" BIPs. :P
418 2012-10-24 14:47:06 <slush> sipa: honestly, I think that this (dev) community is still so small that assigning BIP numbers on IRC works very well :)
419 2012-10-24 14:47:33 <orion> sipa: Thank you for merging my pull request.
420 2012-10-24 14:47:51 <gmaxwell> slush: agreed. Also, IRC works okay for "holy crap, dont do that!"
421 2012-10-24 14:48:04 <slush> :)
422 2012-10-24 14:49:10 <slush> "jeff's blockchain spammer" really entertain me :)
423 2012-10-24 14:49:33 <TD> sipa: i fixed the uint8_t thing
424 2012-10-24 14:49:52 <orion> sipa: What's a BIP?
425 2012-10-24 14:50:04 <sipa> orion: https://en.bitcoin.it/wiki/BIP_0001
426 2012-10-24 14:50:41 <sipa> TD: what if a filter is defined, and one requests a MSG_BLOCK?
427 2012-10-24 14:51:37 <TD> you get the full block
428 2012-10-24 14:52:24 <ibno> in the RPC console, what is n for the command: gettxout <txid> <n>?
429 2012-10-24 14:52:29 <gmaxwell> slush: I'm mostly ribbing him at this point. I don't like things that generate blockchain traffic which aren't currency, though I understand what he's doing and why.
430 2012-10-24 14:52:41 <sipa> ibno: the output index
431 2012-10-24 14:52:50 <gmaxwell> ibno: the output index. E.g. which of multiple outputs you want.
432 2012-10-24 14:53:04 <ibno> ok
433 2012-10-24 14:57:17 <orion> https://github.com/bitcoin/bitcoin/issues/1719 <-- what other processes does bitcoin need to talk to?
434 2012-10-24 14:57:43 <gmaxwell> sort of a bummer that it would repeat the merkle fragments over and over again. That means that if you set a very permissive filter you'll send MUCH more data than the whole block.
435 2012-10-24 14:58:44 <gmaxwell> orion: Itself for URL loading for example.
436 2012-10-24 14:58:48 <sipa> yes, you should never need more than (total_num_transactions - matching_num_transactions) merkle math entries
437 2012-10-24 14:58:51 <sipa> *path
438 2012-10-24 15:00:31 <sipa> (as you can always just send the hashes of all non-matching transactions; any path element provided higher up the tree is an optimization)
439 2012-10-24 15:30:33 <Shaded> Howdy, I'm looking for a big miner to comment in a news article
440 2012-10-24 15:30:44 <Shaded> Anyone here have a big mining rig?
441 2012-10-24 15:31:16 <ibno> I saw the video presentation by Mike Hearn about contracts, has anyone actually implemented the "micro-payment channel" that he mentions in the talk?
442 2012-10-24 15:32:04 <ibno> how much is used of the contract feature in bitcoin?
443 2012-10-24 15:32:20 <Shaded> ibno: Last I checked, not much has been used yet.
444 2012-10-24 15:32:34 <Shaded> At least from all the things I've read there haven't been any good examples yet
445 2012-10-24 15:32:38 <ibno> I seems that there's so much potential there
446 2012-10-24 15:32:45 <Shaded> Hard to see the potential though ;)
447 2012-10-24 16:08:27 <luke-jr_> Did ultraprune really touch the GBT RPC? #1955 smells like it somehow is accidentally reverting things
448 2012-10-24 16:09:24 <sipa> luke-jr_: yup, that's what happened probably
449 2012-10-24 16:10:01 <luke-jr_> sipa: :/
450 2012-10-24 16:10:33 <sipa> it's just that one line, right?
451 2012-10-24 16:10:58 <luke-jr_> sipa: I'm worried about possible things we missed
452 2012-10-24 16:11:23 <sipa> sure, there can be
453 2012-10-24 16:11:42 <sipa> it was a large change, and i've been rebasing it for a long time
454 2012-10-24 16:12:25 <gmaxwell> I went and looked for similar changes.
455 2012-10-24 16:12:43 <gmaxwell> didn't find any. .. but god knows. ... but thats what review for.
456 2012-10-24 16:12:53 <sipa> this bug in particular was probably caused by rebasing onto the moves to rpc* files
457 2012-10-24 16:12:57 <luke-jr_> I just glanced at that commit, and I see 2 "obvious" possibilities, but I don't know if they might have been intentional or not
458 2012-10-24 16:13:10 <sipa> well, please ask
459 2012-10-24 16:13:17 <luke-jr_> sipa: did you change it after rebasing?
460 2012-10-24 16:13:25 <luke-jr_> src/db.cpp
461 2012-10-24 16:13:27 <luke-jr_> -                            "main",    // Logical db name
462 2012-10-24 16:13:28 <luke-jr_> +                            fMockDb ? pszFile : "main", // Logical db name
463 2012-10-24 16:13:52 <luke-jr_> src/wallet.cpp
464 2012-10-24 16:13:54 <luke-jr_> -        // txdb must be opened before the mapWallet lock
465 2012-10-24 16:13:55 <luke-jr_> -        CTxDB txdb("r");
466 2012-10-24 16:14:05 <sipa> yes, that is intentional
467 2012-10-24 16:14:08 <sipa> both
468 2012-10-24 16:14:10 <luke-jr_> ok
469 2012-10-24 16:14:27 <luke-jr_> I'd be interested in gdd the pre-rebase with the rebase
470 2012-10-24 16:15:06 <luke-jr_> gdd=diff(diff(commit^..commit), diff(rebasedcommit^..rebasedcommit))
471 2012-10-24 16:15:10 <sipa> ha
472 2012-10-24 16:15:39 <luke-jr_> 
ACTION ponders the easiest way to get all the pre-rebased versions

473 2012-10-24 16:18:44 <sipa> i think it's too long ago for me to have them still
474 2012-10-24 16:19:06 <sipa> though i'll check the reflog
475 2012-10-24 16:25:42 <BlueMatt> sipa: re: why not de-duplicate merkle branch in bip 37: the target is individual spv nodes that are bw limited, for an average bitcoin node that is matching even a few % of a block - you are likely not an individual node on a cell connection, you have the bw you need to just not use a filter...
476 2012-10-24 16:26:50 <gmaxwell> BlueMatt: you don't know in advance which will be smaller.. esp if you've accidentally sent your filter a bit too broad for privacy sake.
477 2012-10-24 16:27:10 <sipa> BlueMatt: right, i'm not sure the extra complexity either, but there seem to be some outlier cases where it does result in excessive overhead imho
478 2012-10-24 16:27:33 <BlueMatt> sipa: it can, but if you end up hitting those cases, IMHO you shouldnt have been using a filter to begin with
479 2012-10-24 16:27:48 <BlueMatt> gmaxwell: if you set your filter too broad, you are going to fill your bw no matter what, so...
480 2012-10-24 16:29:18 <gmaxwell> BlueMatt: but it ends up being much worse... I'm not sure why you'd intentionally make it substantially ineffcient. Especially with things like contracts and multisig even a conservative filter may still catch a bunch of extra stuff.
481 2012-10-24 16:30:47 <sipa> well, you could define two versions of a filtered block; one that just sends all txids, and one that contains the matched merkle paths, and have the server decide which to send
482 2012-10-24 16:31:08 <sipa> but a generic solution seems neater, but harder
483 2012-10-24 16:31:45 <BlueMatt> yea, gets pretty tricky to prove correctness there, but...not sure
484 2012-10-24 16:35:43 <eian> does anyone have the gist post jgarzik made as a rebuttal to the shamir paper?
485 2012-10-24 16:36:41 <gmaxwell> https://gist.github.com/3901921
486 2012-10-24 16:36:47 <sipa> BlueMatt: the one thing that may convince me that the current solution is better, is the fact that it's easy for the verifier
487 2012-10-24 16:37:08 <sipa> while every generic solution pretty much implies rebuilding the entire merkle tree
488 2012-10-24 16:37:30 <gmaxwell> sending all the data is easier still. :P
489 2012-10-24 16:37:47 <BlueMatt> gmaxwell: and less efficient in the common case
490 2012-10-24 16:37:47 <eian> thanks
491 2012-10-24 16:37:52 <gmaxwell> and has better worst case bandwidth usage.
492 2012-10-24 16:38:02 <BlueMatt> sipa: that was the idea...Im just not so sure about how often you match more than 2-3 txes in a block when filtering is my question
493 2012-10-24 16:38:17 <sipa> also, why do the txids and the merkle root need to be transmitted?
494 2012-10-24 16:38:20 <BlueMatt> gmaxwell: but my point is that you shouldn't ever see the worst case unless you are doing something horribly wrong
495 2012-10-24 16:38:26 <sipa> they have to be recalculated anyway
496 2012-10-24 16:38:46 <gmaxwell> BlueMatt: in your primary application, but for example, someone making a bond tool might pull a lot more because the filter can't fully distinguish.. but are still only pulling a quarter of the block.
497 2012-10-24 16:38:49 <jgarzik> eian: it could use a rewrite at this point, as several additional criticisms wound up getting appended
498 2012-10-24 16:39:05 <eian> :P
499 2012-10-24 16:39:10 <sipa> BlueMatt: oh, by the way, even though i'm nitpicking about these encoding details, i'm very much in favor of the general idea
500 2012-10-24 16:39:16 <BlueMatt> gmaxwell: meh...then dont bother filtering, if you are only saving that much...
501 2012-10-24 16:39:35 <sipa> BlueMatt: the problem is what if you accidentally end up setting up a filter that matches way too much?
502 2012-10-24 16:39:39 <BlueMatt> gmaxwell: the biggest goal is not to relay so many TX invs, the block filtering is just a cool side-part...
503 2012-10-24 16:39:40 <gmaxwell> BlueMatt: saving 3/4 of the bandwidth isn't somethiing to sneeze at!
504 2012-10-24 16:39:48 <BlueMatt> sipa: generally, I dont think anyone would
505 2012-10-24 16:40:03 <sipa> people rarely want to do things accidentally :)
506 2012-10-24 16:40:07 <BlueMatt> gmaxwell: when the bw is max 10MB/10 minutes...it kinda is
507 2012-10-24 16:40:30 <gmaxwell> BlueMatt: and when you're scanning the history because it's a new node or the history.
508 2012-10-24 16:40:32 <BlueMatt> sipa: in general, I dont expect the fp rate to be user-configureable (beyond a generic slider that doesnt go beyond like 1%)
509 2012-10-24 16:41:15 <gmaxwell> 1MB/10 minutes, but a year of history to scan at that rate is 52GBytes. Sending 13 GBytes is much better.
510 2012-10-24 16:41:32 <gmaxwell> Plus protocols are forever.
511 2012-10-24 16:41:42 <gmaxwell> ::shrugs::
512 2012-10-24 16:41:44 <BlueMatt> sipa: txids because you arent actually sending the tx in that message, merkle root: because I was lazy and just sent the block header message using SER_HEADERONLY or whatever it is :)
513 2012-10-24 16:42:06 <BlueMatt> sipa: (because you usually have the tx already)
514 2012-10-24 16:42:46 <gmaxwell> If we're generally going to filter txid relaying, it might be useful to have a couple extra knobs to pick up some other usecases. E.g. "don't send me txn with fees under X btc/kb"
515 2012-10-24 16:43:14 <BlueMatt> gmaxwell: its a fair point, and the answer is mostly: I/others were thinking about the personal-use case of a phone on a mobile network/etc, not so much about bond-matching because those are likely full-verifying nodes anyway (id hope they want more security)
516 2012-10-24 16:44:19 <BlueMatt> gmaxwell: meh, not sure how much that is needed...
517 2012-10-24 16:44:24 <sipa> BlueMatt: well the merkle root there isn't going to kill anyway, and it's indeed somewhat nice to just send an extended block header
518 2012-10-24 16:44:45 <BlueMatt> sipa: yea...meh its only one more hash
519 2012-10-24 16:44:47 <sipa> BlueMatt: and i misread - i think the actual tx data was sent as well
520 2012-10-24 16:44:51 <sipa> so, nevermind those
521 2012-10-24 16:45:25 <sipa> and i understand that what i'm arguing about isn't useful for the intended use case
522 2012-10-24 16:45:30 <BlueMatt> sipa: the tx is send only if its not currently in the bitcoind's known invs for you, however that could be changed in the future if you are allowed to request txes in the most recent N blocks via P2P
523 2012-10-24 16:46:52 <sipa> wait... tx data is sent in the filtered block data?
524 2012-10-24 16:46:58 <sipa> or not?
525 2012-10-24 16:47:01 <BlueMatt> no, separate message
526 2012-10-24 16:47:18 <BlueMatt> (immediately after, but only if we think you dont already have it, which should be usually)
527 2012-10-24 16:47:23 <sipa> ok
528 2012-10-24 16:47:39 <BlueMatt> (potential issue is if you actually dont already have it you cant get it)
529 2012-10-24 16:47:47 <BlueMatt> (but thats only if you are broken so...meh)
530 2012-10-24 16:48:08 <sipa> disconnect, reconnect, re-request specific filtered block -> tx certainly sent
531 2012-10-24 16:48:19 <BlueMatt> yes
532 2012-10-24 16:48:25 <BlueMatt> true
533 2012-10-24 16:48:25 <sipa> good enough
534 2012-10-24 16:48:35 <BlueMatt> ugly hack but, sure
535 2012-10-24 16:48:40 <BlueMatt> or re-request entire block
536 2012-10-24 16:49:17 <sipa> also
537 2012-10-24 16:57:10 <sipa> gavinandresen: you think complicating is the right idea? that's what the sentence says, but not what it sounds like
538 2012-10-24 16:58:38 <sipa> gavinandresen: also, requiring that the invs remain available is harder than keeping track of what has already been sent
539 2012-10-24 16:58:45 <sipa> i know
540 2012-10-24 16:59:28 <gavinandresen> sipa: oops, yeah, NOT complicating....
541 2012-10-24 17:01:08 <gavinandresen> sipa: and right, I forgot ultraprune doesn't keep the txid --> txn index.
542 2012-10-24 17:03:52 <sipa> the alternative is including the transactions inline, but that doesn't conform to the "minimize bandwidth" goal
543 2012-10-24 17:10:02 <gmaxwell> BlueMatt: the use case is certantly greater than phones/mobile networks, it's just most important there. Your application should be _all_ SPV nodes. The largest cost difference between running a pruned full node and a SPV node is the bandwidth for the block data. If you're going to see all the block data then you should probably be a (pruned) full node.
544 2012-10-24 17:11:18 <sipa> signature verification CPU cost may also matter in some environments
545 2012-10-24 17:11:35 <sipa> though in general i assume those will be bandwidth-limited as well
546 2012-10-24 17:12:10 <BlueMatt> gmaxwell: Id say the CPU cost is the most for most environments
547 2012-10-24 17:12:36 <BlueMatt> (by far, Id think)
548 2012-10-24 17:12:38 <gmaxwell> BlueMatt: uh. It shouldn't be.
549 2012-10-24 17:12:55 <BlueMatt> for environments where the bw is limited, the cpu is limited more
550 2012-10-24 17:12:58 <BlueMatt> eg ibd
551 2012-10-24 17:13:03 <gmaxwell> Not to mention that you can smoothly reduce your cpu usage trading for security, but not so for data.
552 2012-10-24 17:13:15 <BlueMatt> for regular operation, neither is limited so...
553 2012-10-24 17:13:36 <BlueMatt> thats true
554 2012-10-24 17:13:53 <sipa> how long would an ECDSA verification take on a high-end ARM smartphone chip these days?
555 2012-10-24 17:13:54 <gmaxwell> BlueMatt: every enviroment is limited wrt bandwidth. If bitcoins load had been the maximum since start the chain would be 208 gb. When talking about protocol design you need to look a little further than now.
556 2012-10-24 17:14:38 <BlueMatt> gmaxwell: true, but bw is growing at nearly the rate of cpu power too, so...
557 2012-10-24 17:14:51 <gmaxwell> BlueMatt: hahaha
558 2012-10-24 17:15:03 <gmaxwell> No way. Bandwidth grows _substantially_ slower than cpu power and storage.
559 2012-10-24 17:15:10 <BlueMatt> ok, not _that_ much slower
560 2012-10-24 17:15:31 <BlueMatt> in a lot of places bw caps are growing fast
561 2012-10-24 17:15:54 <BlueMatt> in charlotte, the bw cap 4 years ago was ~10Mbps if you were lucky, now you can get 50...
562 2012-10-24 17:15:57 <gmaxwell> Communications has infrastructure driven timelines and cross-section bottlenecks. They're growing, but its linear progress with steps. It's not the kind of growth we have in cpus and storage.
563 2012-10-24 17:16:20 <BlueMatt> true, but its not hugely slower than cpus
564 2012-10-24 17:16:27 <BlueMatt> its just slower than cpus
565 2012-10-24 17:16:59 <BlueMatt> the infrastructure grows slowly, but the rate you can push through infrastructure at given cost grows at about the same rate
566 2012-10-24 17:17:12 <BlueMatt> so when you get new infrastructure, it can grow very fast
567 2012-10-24 17:17:50 <sipa> 10 years ago, cpu speed was... 10x-20x lower than now for the same price?
568 2012-10-24 17:19:22 <gmaxwell> For compute bound things like ecdsa, I expect substantially more than that??? I can measure those. :P
569 2012-10-24 17:20:06 <BlueMatt> same with ethernet, but internet, probably more like 5x, but its hard to measure thanks to the ever-increasing size itself (average speeds go down as size grows and the size of slow connections increases)
570 2012-10-24 17:20:39 <helo> it takes 1000 times longer to download all the things :(
571 2012-10-24 17:20:56 <BlueMatt> ...
572 2012-10-24 17:29:44 <gmaxwell> So on a E31230 I get 11223 validations per second, on a 164.7 on Pentium(R) 4 CPU 2.00GHz
573 2012-10-24 17:30:19 <gmaxwell> I'm reasonable sure that P4 cost a fair bit more than the E31230 cpu, but we'll ignore that.
574 2012-10-24 17:35:14 <gmaxwell> If we assume that maxsize blocks are 4000 checksigs, that E31230 can keep up with about 25mbit/sec of blockchain data, so indeed. Closer than I expected.
575 2012-10-24 17:35:31 <BlueMatt> :)
576 2012-10-24 17:35:42 <D34TH> gmaxwell, how does one find the validations per second
577 2012-10-24 17:36:03 <gmaxwell> BlueMatt: But it still shows validation has increased much faster than bandwidth and I expect it to do so.
578 2012-10-24 17:36:12 <BlueMatt> gmaxwell: true
579 2012-10-24 17:36:21 <BlueMatt> not insanely, but faster
580 2012-10-24 17:36:23 <gmaxwell> If we had gpu ecdsa code then I expect the discussion would be kind of moot it would be so fast. :P
581 2012-10-24 17:36:33 <gmaxwell> Small differences add up.
582 2012-10-24 17:36:48 <gmaxwell> D34TH: Hal wrote a little comparison tool for a faster implementation of the signature validation.
583 2012-10-24 17:37:04 <D34TH> i can test?
584 2012-10-24 17:37:04 <gmaxwell> er, which we should merge now. because it would make a visible improvement now that we have ultraprune.
585 2012-10-24 17:37:35 <gmaxwell> https://bitcointalk.org/index.php?topic=3238.msg45795#msg45795  (note speedup numbers it gives are calculated wrong)
586 2012-10-24 17:38:06 <gmaxwell> e.g. it reports a 20% speedup as a .20% speedup. :P
587 2012-10-24 17:39:21 <sipa> didn't he report in a later post that you could get up to 40% with another improvement?
588 2012-10-24 17:39:22 <D34TH> aww i got an error
589 2012-10-24 17:40:35 <gmaxwell> D34TH: gcc -o hal hal.c -lcrypto
590 2012-10-24 17:40:41 <D34TH> ahh
591 2012-10-24 17:40:43 <gmaxwell> (needs openssl with ecc support)
592 2012-10-24 17:40:45 <BlueMatt> gmaxwell: ok fair point on the gpu
593 2012-10-24 17:40:45 <D34TH> thats where i went wrong
594 2012-10-24 17:41:12 <BlueMatt> sipa: iirc that was batch checking with a fp rate of N
595 2012-10-24 17:41:16 <BlueMatt> s/fp/fn/
596 2012-10-24 17:41:20 <sipa> BlueMatt: no
597 2012-10-24 17:41:21 <BlueMatt> no, sorry fp
598 2012-10-24 17:41:27 <D34TH> oh god
599 2012-10-24 17:41:46 <gmaxwell> D34TH: the numbers it gives are second per check, 1/ to get checks per second
600 2012-10-24 17:41:49 <BlueMatt> hmm...missed that post somewhere though
601 2012-10-24 17:41:54 <D34TH> http://pastebin.com/LHnphzdW
602 2012-10-24 17:42:01 <gmaxwell> I think I missed it too.
603 2012-10-24 17:42:10 <gmaxwell> Oh I have an arm system to test on..
604 2012-10-24 17:42:15 <sipa> https://bitcointalk.org/index.php?topic=103172.msg1138884#msg1138884
605 2012-10-24 17:43:57 <sipa> by the way, iirc comparing the sigcheck speed by commenting out/in the script verification entirely, vs raw speed of ecdsa verification is still a factor 2
606 2012-10-24 17:43:59 <jgarzik> 
ACTION needs to find a big endian test system for pynode

607 2012-10-24 17:44:00 <D34TH> gmaxwell, i think my libcrypto is bork for this even though it compiles bitcoind just fine
608 2012-10-24 17:44:17 <D34TH> jgarzik, would a mac g3 do?
609 2012-10-24 17:44:49 <BlueMatt> mmm...fun, guess Ill have to look into that (Im still slowly beating java into submission to get remotely close to openssl in sig verif)
610 2012-10-24 17:44:56 <jgarzik> D34TH: probably, as long as it has...  > 500MB of RAM
611 2012-10-24 17:44:59 <D34TH> nope
612 2012-10-24 17:45:03 <D34TH> 64mb
613 2012-10-24 17:45:03 <jgarzik> drat
614 2012-10-24 17:45:05 <jgarzik> he
615 2012-10-24 17:45:22 <sipa> BlueMatt: how fast is it?
616 2012-10-24 17:45:23 <jgarzik> well... you could still try it, just reduce the pynode cache size from 750 to 10 ;p
617 2012-10-24 17:45:32 <jgarzik> and let it run for a few days
618 2012-10-24 17:45:36 <D34TH> lol
619 2012-10-24 17:45:46 <BlueMatt> sipa: its something like 3ms/sig vs .5 for openssl, but I just picked it back up like yesterday after a month off
620 2012-10-24 17:45:50 <D34TH> and you guys think gpu's produce heat
621 2012-10-24 17:45:55 <D34TH> that will fry bacon
622 2012-10-24 17:45:55 <jgarzik> heh
623 2012-10-24 17:45:56 <gmaxwell> jgarzik: worked fine before.
624 2012-10-24 17:45:59 <BlueMatt> (better than the original ~15ms though...)
625 2012-10-24 17:46:09 <D34TH> i still cant compile it
626 2012-10-24 17:46:09 <gmaxwell> jgarzik: want an account on a PPC G4 debian box?
627 2012-10-24 17:46:12 <jgarzik> gmaxwell: you've tried pynode on BE?
628 2012-10-24 17:46:14 <D34TH> my libcrypto is bork
629 2012-10-24 17:46:26 <gmaxwell> jgarzik: yes sir! And I told you I was.
630 2012-10-24 17:46:50 <D34TH> oh wait
631 2012-10-24 17:46:50 <jgarzik> gmaxwell: cool.  I think I remember that.  Vaguely.  Too much stuff in brain.
632 2012-10-24 17:46:52 <D34TH> derpyderp
633 2012-10-24 17:47:14 <D34TH> i built a windows bin if anybody wants to try
634 2012-10-24 17:47:51 <jgarzik> gmaxwell: if you've tried it, I'll be lazy and not trouble you for a login
635 2012-10-24 17:47:58 <D34TH> 21.94% speedup
636 2012-10-24 17:48:00 <jgarzik> thnx
637 2012-10-24 17:48:14 <BlueMatt> D34TH: yep, thats pretty much what it always is
638 2012-10-24 17:48:31 <D34TH> gmaxwell, i had to add -lgdi32
639 2012-10-24 17:48:46 <jgarzik> assuming you (a) downloaded the chain already and (b) reduce the block cache size to something very small, pynode should be quite low resource
640 2012-10-24 17:49:11 <jgarzik> bet it could run on an ARM netbook
641 2012-10-24 17:50:15 <BlueMatt> sipa: sadly, apart from the link you just gave, Im running out of options for speeding it up...afaict, its pretty much where its gonna be (minus opencl and some other crazy stuff)
642 2012-10-24 17:50:23 <gmaxwell> I should hope so, bitcoin works okay on such hardware.
643 2012-10-24 17:51:13 <sipa> BlueMatt: JNI :P
644 2012-10-24 17:51:31 <BlueMatt> sipa: yea...but afaim concerned JNI == give up :)
645 2012-10-24 17:51:55 <gmaxwell> My older pandaboard gives 240 check/s with hal's code (almost a 25% speedup over openssl on that system).
646 2012-10-24 17:52:09 <BlueMatt> (plus I dont expect this to get merged...unless someone knows a professional cryptographer to review it, aside from hal I'd think)
647 2012-10-24 17:52:21 <gmaxwell> (the pandaboard is a 1ghz dual core omap 4)
648 2012-10-24 17:52:54 <D34TH> gmaxwell, couldnt you take the ecdsa opencl from vanitygen?
649 2012-10-24 17:53:08 <sipa> D34TH: that can only do EC addition
650 2012-10-24 17:53:16 <sipa> you need quite a bit more for ECDSA validation
651 2012-10-24 17:53:35 <gmaxwell> Does anyone have a rpi to test that on, I'm in the mood for a laugh.
652 2012-10-24 17:53:52 <D34TH> i can boot my virtualrpi
653 2012-10-24 17:53:57 <sipa> i wonder whether my pi is already here
654 2012-10-24 17:53:59 <D34TH> through qemu
655 2012-10-24 17:54:12 <gmaxwell> yea, thats no good. :P
656 2012-10-24 17:54:15 <sipa> (not everything's moved to .ch yet)
657 2012-10-24 17:55:59 <sipa> yup, got it!
658 2012-10-24 17:58:24 <gmaxwell> https://bitcointalk.org/index.php?topic=103172.msg1138884#msg1138884 < 40% comment, btw.
659 2012-10-24 18:02:46 <D34TH> gmaxwell, https://github.com/bitcoin/bitcoin/pull/1940#issuecomment-9645026 good thing i didnt edit makefile.linux-mingw right?
660 2012-10-24 18:03:41 <sipa> gmaxwell: way ahead of you :)
661 2012-10-24 18:04:26 <sipa> good, found micro-usb power, sd card, ethernet cable, usb keyboard, ... only a screen left
662 2012-10-24 18:04:34 <BlueMatt> gmaxwell: sipa linked that while ago...
663 2012-10-24 18:04:45 <BlueMatt> sipa: screen? wtf do you need one of those for?
664 2012-10-24 18:04:48 <gmaxwell> sipa: oh, indeed you were! sorry, was benchmarking things and missed a screenfull. :P
665 2012-10-24 18:05:06 <sipa> BlueMatt: well, i don't suppose it will boot an SSH server by default?
666 2012-10-24 18:05:11 <D34TH> sipa, whats the amp output on the power?
667 2012-10-24 18:05:21 <BlueMatt> sipa: you dont have it set up to?
668 2012-10-24 18:05:54 <sipa> BlueMatt: i never booted my pi - i have no idea
669 2012-10-24 18:06:02 <BlueMatt> ahh, ok
670 2012-10-24 18:06:06 <sipa> D34TH: 5V 0.9A
671 2012-10-24 18:06:16 <D34TH> grab a whole amp if possible
672 2012-10-24 18:06:25 <D34TH> it will make a large difference
673 2012-10-24 18:06:30 <sipa> we'll see
674 2012-10-24 18:06:31 <D34TH> in the usb/ethernet
675 2012-10-24 18:23:43 <sipa> uh... http://bitcoin.stackexchange.com/questions/5167/synchronising-official-bitcoin-wallet-for-the-first-time-at-95-started-going-b
676 2012-10-24 18:23:50 <sipa> actually someone who can't keep up?
677 2012-10-24 18:25:29 <gmaxwell> sipa: doubt it, plus they never would have made it as far as 9500 remaining if they couldn't keep up.
678 2012-10-24 18:25:49 <gmaxwell> More likely that changing peer mixes moved the estimate forward once or twice.
679 2012-10-24 18:35:21 <D34TH> git head isnt compiling on mingw anymore
680 2012-10-24 18:35:29 <D34TH> give me a minute and ill look into t
681 2012-10-24 18:37:37 <D34TH> thats weird, looks just fine after i reran without making changes
682 2012-10-24 18:54:12 <sipa> gmaxwell: just implemented a more-or-less accurate guess for how small i could serialize full blocks using similar techniques as the one used in ultraprune's txout set encoding
683 2012-10-24 18:54:48 <gmaxwell> sipa: odd, I started writing a compressor/decompressor earlier today, though only got a few dozen lines into it.
684 2012-10-24 18:54:48 <sipa> not too much it seems
685 2012-10-24 18:54:53 <sipa> block 00000000000001a6090f037a564265ea8621d574371a621ec6561e7978d9df12: 496076 -> 373548
686 2012-10-24 18:55:06 <gmaxwell> ah you're talking block at a time.
687 2012-10-24 18:55:14 <gmaxwell> I was working on something for bootstrap.dat compression.
688 2012-10-24 18:55:25 <sipa> oh, that could get a lot more, i suppose
689 2012-10-24 18:55:52 <sipa> although the majority is still txin scripts, and those are not too compressible
690 2012-10-24 18:56:27 <sipa> seems pretty systematic a 25% reduction i get now
691 2012-10-24 18:59:18 <sipa> (it includes using pubkey compression, stripping DER overhead from signatures, varint encoding for all small integers, + everything we already have for txouts)
692 2012-10-24 18:59:34 <TD> might be an optimization worth mentioning on the scalability page
693 2012-10-24 19:01:24 <gavinandresen> TD and/or Matt:  what does block catch-up look like with Bloom filters, with respect to memory usage for the node that is being fetched from?
694 2012-10-24 19:02:00 <sipa> i don't see why there'd be more overhead than the size of the bloom filter - one per filtered connection
695 2012-10-24 19:02:05 <OneFixt> i'm getting "warning: displayed transactions may not be correct" message - upgrading the client didn't help
696 2012-10-24 19:02:13 <TD> you need to build a CMerkleBlock for each block, of course
697 2012-10-24 19:02:34 <gavinandresen> e.g. when catching up from (say) 10,000 blocks ago, does the node sending the blocks and filtered transactions add those transactions to the "already seen" list ?
698 2012-10-24 19:02:35 <TD> but worst case scenario that, what, doubles the size of the block in memory? so 1mb->2mb plus a bit for std::vector overheads
699 2012-10-24 19:02:39 <TD> doesn't seem worth worrying about
700 2012-10-24 19:02:53 <OneFixt> what's the best way to fix that? does it require redownloading the entire blockchain?
701 2012-10-24 19:03:31 <OneFixt> hm, blocks have stopped updating
702 2012-10-24 19:03:32 <TD> gavinandresen: i don't think so. relevant code is here
703 2012-10-24 19:03:34 <sipa> gavinandresen: setInventoryKnown is an mruset
704 2012-10-24 19:03:39 <TD> https://github.com/TheBlueMatt/bitcoin/commit/f2ecbb0627851b163fa23c1f94821c78d2fc0662#L0R2660
705 2012-10-24 19:03:53 <TD> OneFixt: what block are you up to? restarting your client may help
706 2012-10-24 19:04:02 <OneFixt> i tried that, i'll try again
707 2012-10-24 19:04:04 <gavinandresen> sipa: mruset with a bounded size?  (how big?)
708 2012-10-24 19:04:18 <OneFixt> TD: i'm on 204788
709 2012-10-24 19:04:41 <TD> OneFixt: alright. if another restart doesn't help then a debug.log would be useful
710 2012-10-24 19:04:44 <sipa> gavinandresen: SendBufferSize() / 1000
711 2012-10-24 19:04:52 <sipa> gavinandresen: (that many invs)
712 2012-10-24 19:05:08 <OneFixt> TD: oh, thanks, good point - i'll look in there if this doesn't fix it
713 2012-10-24 19:05:51 <gavinandresen> sipa: thanks
714 2012-10-24 19:06:09 <sipa> TD: merkle tree can't double the size in memory - it's just a extra vector with size 2*nTx
715 2012-10-24 19:06:37 <TD> 1 branch per tx
716 2012-10-24 19:06:38 <OneFixt> TD: https://www.privatepaste.com/6bde0ef8b5
717 2012-10-24 19:06:50 <TD> you said yourself, for a very deep tree the branch can be the same size as the tx
718 2012-10-24 19:06:58 <sipa> TD: the tree is serialized into a single vector
719 2012-10-24 19:07:17 <sipa> so every node is only represented once in it
720 2012-10-24 19:07:35 <sipa> s/serialized/flattened/
721 2012-10-24 19:07:47 <TD> i don't understand your point. gavin is asking about the worst case memory usage of a node serving a filtered client
722 2012-10-24 19:08:03 <TD> if the bloom filter matches every TX then the CMerkleBlock will contain a branch for every transaction
723 2012-10-24 19:08:13 <TD> and the CMerkleBlock is created from a CBlock
724 2012-10-24 19:08:17 <sipa> oh, you're talking about the CMerkleBlock
725 2012-10-24 19:08:19 <TD> yeah
726 2012-10-24 19:08:27 <sipa> i was talking about CBlock::vMerkleTree
727 2012-10-24 19:08:30 <TD> ah. no.
728 2012-10-24 19:08:54 <sipa> yes, sure, you'll always need memory to store the data that will be transmitted
729 2012-10-24 19:08:54 <TD> OneFixt: odd. it's not even trying to download blocks.
730 2012-10-24 19:09:03 <TD> OneFixt: i wonder if you keep connecting to stuck nodes or something
731 2012-10-24 19:09:09 <gavinandresen> I'm trying to think like an attacker who might try to use the new bloom filter stuff to make mischief.
732 2012-10-24 19:09:18 <TD> yeah
733 2012-10-24 19:09:35 <sipa> OneFixt: can you send a larger portion of your debug.log?
734 2012-10-24 19:09:38 <sipa> or upload it?
735 2012-10-24 19:09:38 <TD> i don't think it's very easy. one of the nice things about the filter construction is that testing membership is fast. some hashing operations + bit checks
736 2012-10-24 19:10:04 <TD> it's not like a static list where matching time would be linear in the number of included elements
737 2012-10-24 19:10:12 <gavinandresen> TD: yes, I REALLY like the constant time factor of bloom filters.
738 2012-10-24 19:10:47 <OneFixt> sipa: here is the log from the restart https://www.privatepaste.com/def61f7494
739 2012-10-24 19:12:01 <sipa> OneFixt: corrupted blockchain (index)
740 2012-10-24 19:12:26 <sipa> it thinks block 204789 contains a double spend
741 2012-10-24 19:12:36 <OneFixt> perhaps due to a pc crash?
742 2012-10-24 19:12:39 <sipa> maybe
743 2012-10-24 19:12:49 <gmaxwell> td: I'm disappointed by the gratuitous inefficiency of the redundant hashtree fragments; and I'd really like to understand a justification beyond what I got from bluematt??? which I felt was that being less efficient than grabbling whole blocks when pulling more than a small number of txn isn't required for mobile devices.
744 2012-10-24 19:12:54 <OneFixt> what's the fastest way to fix it without redownloading everything?
745 2012-10-24 19:14:04 <TD> if you think about the case where you're matching a significant proportion of all transactions in a block, you probably quickly approach the point where just downloading the whole thing is cheaper (considering that server-side filtering does have a cost)
746 2012-10-24 19:14:19 <TD> the code and encoding to collapse together partial tree segments is also likely to be quite involved.
747 2012-10-24 19:14:22 <TD> maybe we're wrong and it's trivial
748 2012-10-24 19:14:25 <TD> but i doubt it.
749 2012-10-24 19:14:27 <gmaxwell> OneFixt: shut down, move blk0*.dat out of the way, delete the blkindex and start with two -loadblock= pointed to the old blockfiles.
750 2012-10-24 19:14:38 <sipa> TD: is a server allowed to send a normal block when a filtered block is requested?
751 2012-10-24 19:14:52 <OneFixt> gmaxwell: thanks
752 2012-10-24 19:15:17 <sipa> TD: oh, it's not that hard to construct the necessary data
753 2012-10-24 19:15:22 <gmaxwell> OneFixt: this is faster than downloading but it still takes a while.
754 2012-10-24 19:15:43 <OneFixt> thanks, i'll try that right now
755 2012-10-24 19:15:46 <sipa> TD: but validating it is a lot trickier than just a chain of hashes
756 2012-10-24 19:16:07 <gmaxwell> Assembling it is trivial, validating requires some more code.
757 2012-10-24 19:16:07 <TD> i don't think so. perhaps that should be allowed by the BIP, though it seems unnecessary. filtering of blocks really isn't designed for people like SatoshiDice who actually can dominate transaction volumes. the vast, vast majority of wallets will never care about the vast majority of blocks
758 2012-10-24 19:16:46 <TD> if somebody can come up with a reason why a site might want to process some significant fraction of all network traffic but not just download full blocks, well, i'm all ears
759 2012-10-24 19:16:50 <TD> but the tradeoff doesn't feel right to me
760 2012-10-24 19:16:59 <sipa> TD: is a server allowed to send a normal block when a filtered block is requested?
761 2012-10-24 19:17:13 <TD> sipa: "i don't think so. perhaps that should be allowed by the BIP, though it seems unnecessary. "
762 2012-10-24 19:17:37 <gmaxwell> TD: Allowing that makes sense to me... Though I was also thinking about things like contract tools that may need very broad filter rules and catch??? say 10-20% of the transactions??? only to throw out most locally.
763 2012-10-24 19:18:43 <gmaxwell> Thats still a _major_ savings when going through the history, but not with the redundancy in the current BIP.
764 2012-10-24 19:18:55 <sipa> TD: construct a tree with a leaf for every transaction, containing a boolean in every node; initialize every node with true if it corresponds to a transaction or is the parent thereof; then iterate breadth-first through the tree and output the hash for every false node, while in the process marking them and all their children true
765 2012-10-24 19:19:14 <TD> yeah, things like bond clients may well want to match a lot of transactions but i doubt it'd ever be as high as 20% of a block. unless we all start using bitcoin in radically different ways to how it's used today, it's hard to imagine contract/smart property style txns dominating
766 2012-10-24 19:20:05 <gmaxwell> TD: I'd expect high numbers due to the non-specific rules required to match some non-txouts and filter pollution. Bloom filters rapidly start selecting everything when over filled.
767 2012-10-24 19:20:09 <TD> sipa: i would be tempted to just use a dictionary of hashes and represent independent branches as indexes into that dictionary
768 2012-10-24 19:20:19 <TD> seems a lot simpler to spec out and implement
769 2012-10-24 19:20:30 <sipa> ok, fine by me
770 2012-10-24 19:20:59 <sipa> but that may lead to a blow-up in verification time
771 2012-10-24 19:21:17 <TD> it's a few hashing operations.
772 2012-10-24 19:21:25 <TD> a single sigop would blow that away
773 2012-10-24 19:21:56 <sipa> ok, worst case, 4000 transactions and all over them match: 48000 hash operations
774 2012-10-24 19:22:20 <sipa> you are targetting clients that do not want to do sigops
775 2012-10-24 19:22:33 <OneFixt> gmaxwell: is there an option to disable logging while i'm importing the blocks?
776 2012-10-24 19:22:55 <sipa> OneFixt: you can symlink debug.log to /dev/null :)
777 2012-10-24 19:23:04 <OneFixt> on windows >_<
778 2012-10-24 19:23:24 <sipa> sure, first install an ubuntu VM, and ...
779 2012-10-24 19:23:33 <OneFixt> lol
780 2012-10-24 19:23:41 <sipa> seriously - no idea
781 2012-10-24 19:23:47 <OneFixt> np, thanks
782 2012-10-24 19:24:05 <TD> sipa: sure but they aren't that slow!
783 2012-10-24 19:24:45 <TD> heck ARMv8 actually has hardware opcodes for sha-2 hashing
784 2012-10-24 19:25:32 <gmaxwell> Bet those works great from java. :P
785 2012-10-24 19:25:45 <sipa> still, i don't think it's so hard to implement an optimal encoding/verification
786 2012-10-24 19:26:33 <gmaxwell> There may even me some compact algebraic code that can take an index list and expand it to the tree pattern that people can just ape.
787 2012-10-24 19:27:02 <gmaxwell> At least this is code that you're unlikely to get wrong and fail to notice.
788 2012-10-24 19:29:30 <TD> ok so 48,000 hash ops with each op being on 64 bytes (concatenation of two hashes) == 48k*64 / 1024 / 1024 == 2.92 megabytes of data to hash
789 2012-10-24 19:29:39 <TD> http://www.bolet.org/sphlib-report-round3.pdf
790 2012-10-24 19:29:45 <TD> this will take ~1 second to complete
791 2012-10-24 19:29:52 <TD> on an overclocked _scientific calculator-
792 2012-10-24 19:30:02 <TD> which is running at 75 Mhz
793 2012-10-24 19:30:21 <TD> seriously. if you want to argue for a more complicated encoding use bandwidth as a justification
794 2012-10-24 19:30:58 <TD> on any reasonable system sha2 can crunch hundreds of megabytes per second
795 2012-10-24 19:31:00 <sipa> i already did that - i was now arguing for just outputting the necessary merkle nodes, instead of an array an indexes into it
796 2012-10-24 19:31:17 <sipa> which isn't much of a problem, but just feels stupid
797 2012-10-24 19:32:45 <sipa> anyway, not worth it to keep discussing it
798 2012-10-24 19:33:08 <TD> btw i think this is the first discussion of the idea: https://bitcointalk.org/index.php?topic=7972.0