1 2012-01-03 00:00:03 <sipa> i made three critical blocks for cs_mapAddresses smaller, so concurrency should have improved as well
  2 2012-01-03 00:00:20 <BlueMatt-mobile> Fair enough
  3 2012-01-03 00:00:21 <gmaxwell> sipa: it appears to be _clearly_ faster, yes.
  4 2012-01-03 00:00:29 <sipa> noticably? nice
  5 2012-01-03 00:00:56 <BlueMatt-mobile> (as long as you arent hacking around with Sleep to fix deadlocks...)
  6 2012-01-03 00:01:19 <gmaxwell> BlueMatt-mobile: nah, the purpose of the sleep stuff was to simulate slow/unresponsive dns to make sure it wasn't getting stuck there.
  7 2012-01-03 00:01:25 <BlueMatt-mobile> logs, saw sleep and was concerned
  8 2012-01-03 00:01:43 <BlueMatt-mobile> Oh ok, yea didnt think so
  9 2012-01-03 00:01:53 <gmaxwell> One of sipas patches did.
 10 2012-01-03 00:04:12 <CIA-100> bitcoin: Pieter Wuille master * ra75d706 / src/net.cpp : Fix some address-handling deadlocks ... https://github.com/bitcoin/bitcoin/commit/a75d7066b824eb1e70fb5b0e0e7c3c122e00c4b6
 11 2012-01-03 00:04:13 <CIA-100> bitcoin: Pieter Wuille master * r4231eb2 / src/net.cpp : Merge pull request #738 from sipa/dnsseed-fix ... https://github.com/bitcoin/bitcoin/commit/4231eb217ca06e93cfb0875924b4383f92baf134
 12 2012-01-03 00:06:25 <gmaxwell> luke-jr: ^ thats a bug fix that should probably be pulled into whatever stable versions you have (at least after it seasons in head for a bit)
 13 2012-01-03 00:07:22 <luke-jr> gmaxwell: I'm looking at it now
 14 2012-01-03 00:07:31 <luke-jr> it looks like there's a non-fix tacked on the end of it tho
 15 2012-01-03 00:07:40 <luke-jr> specifically, it looks like it disables DNS seeds if IRC connects
 16 2012-01-03 00:08:16 <sipa> luke-jr: no
 17 2012-01-03 00:08:21 <BlueMatt-mobile> Dnsseed should still run if irc connects
 18 2012-01-03 00:08:28 <sipa> that code was already there, inside the critical block
 19 2012-01-03 00:08:35 <BlueMatt-mobile> Irc sucks and should be phased out...
 20 2012-01-03 00:08:35 <sipa> and it is about the hard-coded seeds, not dns seeds
 21 2012-01-03 00:08:41 <gmaxwell> Are you just mistaking an indenting change?
 22 2012-01-03 00:08:52 <sipa> it's a code-move-and-indent
 23 2012-01-03 00:08:55 <luke-jr> ah
 24 2012-01-03 00:09:01 <luke-jr> ok nm
 25 2012-01-03 00:09:42 <sipa> except for increasing concurrency a bit, there should be no functional changes
 26 2012-01-03 00:09:53 <luke-jr> k
 27 2012-01-03 00:10:14 <luke-jr> I'll push to 0.4.x for now, and merge up to 0.5.0.x and 0.5.x later
 28 2012-01-03 00:10:21 <CIA-100> bitcoin: Pieter Wuille 0.4.x * rb52b6f2e3801 bitcoind-stable/src/net.cpp: Fix some address-handling deadlocks http://tinyurl.com/6w9eqyx
 29 2012-01-03 00:13:02 <gmaxwell> Can CIA be made to report issue creation in here?
 30 2012-01-03 00:13:20 <BlueMatt-mobile> No afaik
 31 2012-01-03 00:13:32 <BlueMatt-mobile> Though an rss bot could pretty easily
 32 2012-01-03 00:14:11 <BlueMatt-mobile> (Github private feed following bitcoin/bitcoin)
 33 2012-01-03 00:14:58 <luke-jr> gmaxwell: great idea, but no thanks to GitHub
 34 2012-01-03 00:53:21 <luke-jr> sigh, bzr really does have a much saner design
 35 2012-01-03 01:29:43 <gmaxwell> sipa: so on tmpfs on the same system, the keypool fill takes 15 seconds.. so thats still insanely slow, but a big improvement from almost 40.
 36 2012-01-03 01:35:47 <jrmithdobbs> gmaxwell: w/e, i just had to wait for 1.44M to write at 17K/s I don't want to hear you're whining about slow
 37 2012-01-03 01:35:50 <jrmithdobbs> ;p
 38 2012-01-03 01:36:36 <taipres> heh
 39 2012-01-03 02:06:26 <Acciaio> hi all,how can I autenticate to mtgox api? example found here -> https://mtgox.com/api#Authentication dont work  :-(
 40 2012-01-03 02:08:09 <pumpkin> #mtgox
 41 2012-01-03 02:12:07 <Acciaio> #mtgox :Cannot join channel (+r) - you need to be identified with services
 42 2012-01-03 02:12:20 <Acciaio> :-(
 43 2012-01-03 02:12:53 <pumpkin> then register and then identify :) it takes a couple of minutes
 44 2012-01-03 02:17:51 <Acciaio> where I have to register? on mtgox? I have an account on mtgox and how can i identify?
 45 2012-01-03 02:34:06 <nanotube> Acciaio: with nickserv. ,,(sl freenode nickserv registration)
 46 2012-01-03 02:34:07 <gribble> http://freenode.net/faq.shtml | ? Just plug your nickserv ... the one you've got registered use ...
 47 2012-01-03 02:42:18 <Joric> guys do you mind
 48 2012-01-03 02:42:34 <Joric> check out this private key 5KFKMNB2P989YqLZKoWGLNdFvMPA2K7x3GY6cWiF1jGCGKDuvq
 49 2012-01-03 02:42:56 <Joric> the address should be 1DrGossc3QidjzgDXzveCAQGiPWsoiDZ8C
 50 2012-01-03 02:43:16 <Joric> neither pywallet nor bitaddress.org won't understand this key
 51 2012-01-03 02:43:35 <Joric> it might be something with base58 padding? i can't tell
 52 2012-01-03 02:44:30 <Joric> nevermind it was just a typing error
 53 2012-01-03 02:44:38 <rjk2> lol
 54 2012-01-03 02:44:44 <rjk2> sorry you can't use it now
 55 2012-01-03 02:44:48 <rjk2> :p
 56 2012-01-03 02:46:08 <Joric> meh 400 btc you can have it
 57 2012-01-03 02:46:39 <Joric> price falls anyway
 58 2012-01-03 02:52:07 <finway> So, No OP_EVAL, and no 15th Jan deadline ?
 59 2012-01-03 03:32:29 <AlonzoTG> om
 60 2012-01-03 03:32:53 <AlonzoTG> I just tried to build bitcoin yet again, same error I've been getting for many many months now: headers.h:39:20: fatal error: db_cxx.h: No such file or directory
 61 2012-01-03 03:32:56 <AlonzoTG> =|
 62 2012-01-03 03:33:27 <AlonzoTG> << gentoo linux; Yes, I have the correct library, Yes it's in a perfectly reasonable location, NO, bitcoin still doesn't have a build system that can find it.
 63 2012-01-03 03:38:19 <gmaxwell> AlonzoTG: Bitcoin has a makefile, not an automatic buildsystem. If you can't handle modifying it for your system, then perhaps you shouldn't be using gentoo.
 64 2012-01-03 03:39:00 <gmaxwell> And I seem to recall that you didn't, in fact, have the correct versions of the db-cxx headers when we last spoke.
 65 2012-01-03 04:01:13 <luke-jr> AlonzoTG: emerge bitcoind
 66 2012-01-03 04:01:29 <luke-jr> gmaxwell: there should be no need to modify the makefile anymore
 67 2012-01-03 04:01:47 <luke-jr> gmaxwell: also, his problem is not Gentoo-specific, but occurs in most distros
 68 2012-01-03 04:02:07 <luke-jr> (because bdb++ headers are not in the default include path)
 69 2012-01-03 05:06:27 <gribble> New news from bitcoinrss: TheBlueMatt pushed to master at bitcoin/gitian.sigs
 70 2012-01-03 05:11:14 <SomeoneWeird> heh
 71 2012-01-03 05:11:34 <BlueMatt> ;;bitcoinrss
 72 2012-01-03 05:11:35 <gribble> TheBlueMatt pushed to master at bitcoin/gitian.sigs <https://github.com/bitcoin/gitian.sigs/compare/33e5c35329...70b232f00d>
 73 2012-01-03 05:14:29 <nanotube> BlueMatt: i've changed config, so now the announcements will include url
 74 2012-01-03 05:14:42 <BlueMatt> nice
 75 2012-01-03 05:14:52 <BlueMatt> thanks nanotube
 76 2012-01-03 05:15:22 <nanotube> np. let's see how it works out :)
 77 2012-01-03 05:40:52 <gmaxwell> so... ltrace is interesting.
 78 2012-01-03 05:41:11 <gmaxwell> I set bitcoin to generate 1000 new keys then quit, ran it in ltrace:
 79 2012-01-03 05:41:26 <gmaxwell> ------ ----------- ----------- --------- --------------------
 80 2012-01-03 05:41:34 <gmaxwell> % time     seconds  usecs/call     calls      function
 81 2012-01-03 05:45:37 <gmaxwell> commenting out all mlock in the source makes it ~instantly fast.
 82 2012-01-03 05:45:48 <gmaxwell> jesus christ that was a pain in the ass to figure out.
 83 2012-01-03 05:45:54 <gmaxwell> oprofile sees nothing there.
 84 2012-01-03 05:46:06 <gmaxwell> valgrind/callgrind sees nothing (unsurprising)
 85 2012-01-03 05:46:21 <gmaxwell> strace saw nothing (I'm not sure why).
 86 2012-01-03 05:46:50 <gmaxwell> but ltrace sees it. mlock is #@$@# expensive, I assume because it has to twiddle the page tables then flush the tlbs.
 87 2012-01-03 05:47:10 <Diablo-D3> what are we doing?
 88 2012-01-03 05:47:26 <luke-jr> gmaxwell: in case you missed it, someone informed me that bitcoind mlocks *everything*
 89 2012-01-03 05:47:42 <Diablo-D3> who the fuck uses locks goddamnit
 90 2012-01-03 05:47:57 <Diablo-D3> no, luke just steals the namecoins
 91 2012-01-03 05:48:02 <gmaxwell> luke-jr: it does?
 92 2012-01-03 05:48:11 <luke-jr> gmaxwell: apparently
 93 2012-01-03 05:48:24 <Diablo-D3> I dont really doubt it
 94 2012-01-03 05:48:35 <Diablo-D3> DURR HURR, WE HAVE THREADS, LETS USE LOCKS
 95 2012-01-03 05:48:35 <luke-jr> gmaxwell: some Linux security framework complains about it mlocking too much, is how he noticed
 96 2012-01-03 05:49:00 <Diablo-D3> LOCKS LOCKS LOCKITY LOCKS
 97 2012-01-03 05:49:08 <gmaxwell> well doing so is enormously stupendously slow. if you really want all mlocked you do a mlockall(MCL_FUTURE) .. but the program will crash it it can't mlock more.
 98 2012-01-03 05:49:17 <gmaxwell> Diablo-D3: mlock isn't a lock.
 99 2012-01-03 05:49:26 <gmaxwell> Diablo-D3: man mlock
100 2012-01-03 05:49:29 <Diablo-D3> gmaxwell: its a memory lock
101 2012-01-03 05:49:31 <Diablo-D3> erg, m lock
102 2012-01-03 05:49:49 <Diablo-D3> its part of the suite of functions that people think are legitimate to EVER USE
103 2012-01-03 05:50:09 <gmaxwell> omg
104 2012-01-03 05:50:14 <gmaxwell> block chain syncup is 1000x faster.
105 2012-01-03 05:50:21 <SomeoneWeird> hah
106 2012-01-03 05:50:34 <gmaxwell> seriously, ... what took >3 hours before just finished in 30 seconds.
107 2012-01-03 05:50:44 <gmaxwell> omg
108 2012-01-03 05:55:02 <Diablo-D3> 3 HOURS?!
109 2012-01-03 05:55:06 <Diablo-D3> what the fuck takes 3 hours!
110 2012-01-03 05:55:40 <gmaxwell> Diablo-D3: syncing part of the blockchain.
111 2012-01-03 05:58:12 <gmaxwell> synced from zero to 140000 in seven minutes!
112 2012-01-03 05:58:54 <gmaxwell> (it got somewhat slower after that I assume because it's doing signature validations again, though its still processing many blocks per second)
113 2012-01-03 05:58:58 <Rabbit67890> PONG :VERNE.FREENODE.NET
114 2012-01-03 05:59:05 <Diablo-D3> gmaxwell: wait wait wait
115 2012-01-03 05:59:06 <Diablo-D3> as in
116 2012-01-03 05:59:12 <Diablo-D3> downloading the blocks?
117 2012-01-03 05:59:15 <Diablo-D3> in 7 minutes?
118 2012-01-03 05:59:36 <gmaxwell> Diablo-D3: yes.
119 2012-01-03 05:59:42 <Diablo-D3> godfuckingdamnit
120 2012-01-03 05:59:51 <Backburn> i just host the block chain on my CDN heh
121 2012-01-03 05:59:53 <gmaxwell> well, the last bit is going to take somewhat longer... it's still going.
122 2012-01-03 06:00:19 <gmaxwell> Backburn: network speed was never the issue, I've been saying this for months.
123 2012-01-03 06:01:01 <gmaxwell> (well, fsync is _still_ an issue even with the mlock problem removed, but right now I'm syncing to media that its not a problem on)
124 2012-01-03 06:01:37 <Diablo-D3> gmaxwell: its better than 9000 fucking hours
125 2012-01-03 06:01:56 <gmaxwell> only up to 144000 now.
126 2012-01-03 06:02:40 <Diablo-D3> stil, a total of what, 10 minutes? 15?
127 2012-01-03 06:02:52 <Diablo-D3> it took me 3 hours to just do about 5000 blocks
128 2012-01-03 06:03:49 <gmaxwell> Yea, I'll say when its done. Maybe 20 minutes.
129 2012-01-03 06:05:10 <gmaxwell> anyways.. removing mlock completely isn't an actual solution... (well, it's a workaround esp for nodes that don't deal with private data)
130 2012-01-03 06:10:28 <gmaxwell> :-/
131 2012-01-03 06:10:29 <gmaxwell> {
132 2012-01-03 06:10:52 <gmaxwell> I'm C++ incompetent, is _that_ how everything under the @#$@#$@ sun is ending up mlocking?
133 2012-01-03 06:15:08 <gmaxwell> BlueMatt: !
134 2012-01-03 06:17:29 <TuxBlackEdo> not sure if this is the right channel to ask, but anyone here know of/use ga-bitbot?
135 2012-01-03 06:18:15 <BlueMatt> gmaxwell: was just reading logs, wtf?
136 2012-01-03 06:18:30 <BlueMatt> gmaxwell: so removing the one mlock call in bitcoin makes chain download 10000x faster?
137 2012-01-03 06:20:09 <BlueMatt> (sorry, 4 mlock calls)
138 2012-01-03 06:20:15 <gmaxwell> COMPLETE SYNC IN 28 minutes!!!
139 2012-01-03 06:20:15 <midnightmagic> mlock is being applied to incoming block data, or mlock is being applied to keygen?
140 2012-01-03 06:20:24 <BlueMatt> no way in hell
141 2012-01-03 06:20:27 <gmaxwell> Yep!
142 2012-01-03 06:20:40 <gmaxwell> midnightmagic: to *
143 2012-01-03 06:21:04 <BlueMatt> how did it not used to be that fast (before we added encryption?)
144 2012-01-03 06:21:12 <BlueMatt> also, why is everyone finding bugs in my code today
145 2012-01-03 06:21:18 <midnightmagic> it makes sense to apply it to private data, else key recovery can retrieve it with swap tricks
146 2012-01-03 06:21:21 <BlueMatt> also, why are there so many bugs in my code that no one found until today
147 2012-01-03 06:21:24 <BlueMatt> also, wtf???
148 2012-01-03 06:21:36 <BlueMatt> midnightmagic: its written to only apply to private data...
149 2012-01-03 06:21:40 <gmaxwell> Because I started testing again.
150 2012-01-03 06:21:45 <gmaxwell> I have this effect on code bases.
151 2012-01-03 06:24:43 <BlueMatt> can anyone duplicate gmaxwell's results?
152 2012-01-03 06:25:11 <midnightmagic> as luck has it, I am *exactly right now* just raising up a fresh vm and about to compile stock btc on it.
153 2012-01-03 06:25:22 <midnightmagic> what's the patch
154 2012-01-03 06:25:54 <BlueMatt> in serialize.h and crypter.cpp, comment out mlock calls
155 2012-01-03 06:26:01 <BlueMatt> should be a total of 5
156 2012-01-03 06:26:23 <midnightmagic> k i'll head towards that..
157 2012-01-03 06:26:25 <BlueMatt> gmaxwell: well heres your damn bug, who the fuck put secure_allocator in CDataStream???
158 2012-01-03 06:26:40 <BlueMatt> oh, probably for key export...
159 2012-01-03 06:26:41 <BlueMatt> wtf?
160 2012-01-03 06:27:38 <gmaxwell> https://people.xiph.org/~greg/omg_go_fast.patch
161 2012-01-03 06:27:49 <gmaxwell> BlueMatt: thats what I said above!
162 2012-01-03 06:28:15 <gmaxwell> Keep in mind, that patch just removes all the mlock.. bad for encrypted wallet security.
163 2012-01-03 06:28:52 <BlueMatt> gmaxwell: what the stuff about CDataStream?
164 2012-01-03 06:28:55 <midnightmagic> bad for any private data security, including txn
165 2012-01-03 06:29:00 <gmaxwell> And you're not going to get the _28 minute_ sync unless you're on a fast network and have a fs/media where fsync doesn't hurt.  but it still should be enormously faster. (or at least it is for me even with a slow disk)
166 2012-01-03 06:29:06 <BlueMatt> midnightmagic: txn are not private...
167 2012-01-03 06:29:37 <gmaxwell> BlueMatt: see up 23:10 < gmaxwell> class CDataStream
168 2012-01-03 06:29:40 <midnightmagic> retrieving txn origins would be uncomfortable for someone who expected his swap not to be contaminated with evidence of txn sends..
169 2012-01-03 06:29:44 <midnightmagic> unenecrypted..
170 2012-01-03 06:29:48 <BlueMatt> gmaxwell: oh...sorry cant read apparently
171 2012-01-03 06:29:57 <gmaxwell> midnightmagic: well, that would be exposed already.
172 2012-01-03 06:30:13 <midnightmagic> where?
173 2012-01-03 06:30:16 <gmaxwell> midnightmagic: tons of stuff is _not_ mlocked.
174 2012-01-03 06:30:38 <gmaxwell> There is just enough stuff mlocked to create horiffic performance, but not enough to e.g. hide your addresses.
175 2012-01-03 06:31:02 <midnightmagic> assumption on my part then.
176 2012-01-03 06:31:04 <gmaxwell> midnightmagic: e.g. non of the bdb memory is mlocked for sure, plus lots of other stuff.
177 2012-01-03 06:31:52 <midnightmagic> mm
178 2012-01-03 06:32:03 <gmaxwell> midnightmagic: if that were a goal, I'm afraid the first step would be to throw out the software and rewrite it all.
179 2012-01-03 06:32:45 <gmaxwell> (there are allocations all over the @#$@ place, so the only way to do it would be to mlockall(mcl_future)  and that would crash on all systems because we use _far_ too much memory for that)
180 2012-01-03 06:33:02 <midnightmagic> i guess i'd assumed that similar practices to openssh/et al would be in satoshi's radar. guess he just wanted to get the software written.
181 2012-01-03 06:33:07 <BlueMatt> midnightmagic: txn are not encrypted in wallet
182 2012-01-03 06:33:27 <gmaxwell> hm. Res side of bitcoind of 70m now too.
183 2012-01-03 06:33:48 <gmaxwell> midnightmagic: ssh doesn't hide your identity in memory.. or the hosts you connect to.. only keying data.
184 2012-01-03 06:34:58 <midnightmagic> no, because there's not a lot of point in hiding a one-hop connection txn, and state must be preserved to help detect host key changes.
185 2012-01-03 06:35:33 <BlueMatt> gmaxwell: https://github.com/bitcoin/bitcoin/commit/223b6f1b#diff-18
186 2012-01-03 06:35:38 <BlueMatt> commit that killed it
187 2012-01-03 06:36:39 <gmaxwell> Thanks, I was just starting to look.. the bitcoin-qt stuff made a @#$@ mess of the history.
188 2012-01-03 06:37:01 <gmaxwell> oh no, that didn't do it.
189 2012-01-03 06:37:02 <BlueMatt> why the fuck were any functional changes made in that commit???
190 2012-01-03 06:37:12 <BlueMatt> it changed to secure_allocator in CDataStream
191 2012-01-03 06:37:27 <BlueMatt> oh, nope Im a dumbass
192 2012-01-03 06:37:27 <gmaxwell> No it didn't.
193 2012-01-03 06:37:28 <Joric> how to get back to old db version? i hate qt
194 2012-01-03 06:37:37 <gmaxwell> -    typedef vector<char, secure_allocator<char> > vector_type;
195 2012-01-03 06:37:38 <gmaxwell> +    typedef std::vector<char, secure_allocator<char> > vector_type;
196 2012-01-03 06:37:40 <BlueMatt> god why can I not focus at all today...
197 2012-01-03 06:38:10 <gmaxwell> yea, I'm trying to find it, but that commit is off on some crazy branch where Wladimir wrote every line of code.
198 2012-01-03 06:38:54 <BlueMatt> wait, nope I have that secure_allocator was in a commit by s_nakamoto
199 2012-01-03 06:39:04 <BlueMatt> (before secure_allocator mlocked)
200 2012-01-03 06:39:16 <BlueMatt> so its when secure_allocator was changed to mlock
201 2012-01-03 06:40:26 <gmaxwell> Yea, which is what you did, I guess.
202 2012-01-03 06:40:33 <BlueMatt> no, that wasnt me
203 2012-01-03 06:40:38 <BlueMatt> that is when SecureString was added
204 2012-01-03 06:41:08 <BlueMatt> c1aacf0b
205 2012-01-03 06:41:36 <gmaxwell> c1aacf0b
206 2012-01-03 06:41:38 <gmaxwell> yea.
207 2012-01-03 06:41:39 <BlueMatt> oh, but it was part of the encryption pull
208 2012-01-03 06:41:41 <BlueMatt> that I wrote...
209 2012-01-03 06:41:43 <BlueMatt> goddamit
210 2012-01-03 06:41:58 <BlueMatt> alway my goddamn fault of late it seems...
211 2012-01-03 06:42:12 <gmaxwell> This is what you get for being productive.
212 2012-01-03 06:42:21 <gmaxwell> If you were more like me you'd insert no faults!
213 2012-01-03 06:43:02 <BlueMatt> while you are being productive and benchmarking code, do you feel like benchmarking and faul-testing something Ive been working on?
214 2012-01-03 06:43:36 <midnightmagic> new machine raised, patch committed, building bitcoind
215 2012-01-03 06:43:38 <gmaxwell> In any case, options seem to be: write our own malloc() that uses a pre-mlocked arena... and/or seperate off the 'secure_allocator' with a 'really_secure_allocator' which is only used on the the keying material.
216 2012-01-03 06:45:04 <gmaxwell> Yea. Thats my preference too.. though, ... seeing how CDataStream works.. meh, it's already a performance problem even without the mlock making it horiffic. (it's creating a LOT of allocator calls)
217 2012-01-03 06:45:19 <BlueMatt> though can I ask why CDataStream gets secure_allocated to begin with?
218 2012-01-03 06:45:39 <gmaxwell> I don't know.
219 2012-01-03 06:45:43 <BlueMatt> dealloc used to clear it, but thats also a waste if we dont need it...
220 2012-01-03 06:46:22 <gmaxwell> yea, ... well. the code should be carefully audited to make sure private keys never get stored in one (e.g. in the process of signing) I guess.
221 2012-01-03 06:46:28 <gmaxwell> if not, then I don't think they need to clear.
222 2012-01-03 06:46:33 <gmaxwell> and yes, that should be a lot faster.
223 2012-01-03 06:46:46 <BlueMatt> does key export use CDataStream?
224 2012-01-03 06:46:52 <gmaxwell> But just going back to the old behavior would last least make us no less secure than we were.
225 2012-01-03 06:47:13 <BlueMatt> not a big deal, Ive got a plane ride to audit tomorrow
226 2012-01-03 06:47:42 <gmaxwell> probably, but if you're exporting.. you're probably getting that data all over unlocked memory in any case. ::shrugs:: .. though clearing is kinda prudent.. though oy, bad for performance:
227 2012-01-03 06:48:16 <gmaxwell> ends up freshinging it in cache just to throw it away.
228 2012-01-03 06:48:27 <midnightmagic> downloading blockchain.
229 2012-01-03 06:48:36 <midnightmagic> curheight 22k
230 2012-01-03 06:49:12 <BlueMatt> fix: https://github.com/TheBlueMatt/bitcoin/commit/03f238d86e497315e595f74cc4721a57a4b63dd0
231 2012-01-03 06:49:20 <BlueMatt> (to be pull-requested after audit)
232 2012-01-03 06:49:32 <gmaxwell> midnightmagic: 14:20 < gmaxwell> hmph. weird. seeing really slow syncup off the network. Three hours and twenty five minutes and it only made it up to block 36448,  with very low cpu/disk IO the whole time.
233 2012-01-03 06:49:52 <BlueMatt> CDataStream does not exist in wallet.*, thats a great sign
234 2012-01-03 06:50:26 <midnightmagic> curheight 43k
235 2012-01-03 06:51:36 <gmaxwell> midnightmagic: it'll slow down some and become cpu bound at 140k or so. (wherever the highest checkpoint is)
236 2012-01-03 06:52:27 <gmaxwell> (the code doesn't do ECDSA validation below the highest checkpoint)
237 2012-01-03 06:54:22 <gmaxwell> BlueMatt: the secure allocator probably need to be changed some too, but its less urgent.
238 2012-01-03 06:55:57 <gmaxwell> mlock is damn slow (0.28 seconds per call for me, according to ltrace).. that slow enough that its making keypool refill take too darn long.
239 2012-01-03 06:56:28 <BlueMatt> ok, only place CDataStream needs to be secure: reading wallet.dat in db.*
240 2012-01-03 06:56:47 <gmaxwell> BlueMatt: why does it need to be secure there?
241 2012-01-03 06:56:58 <BlueMatt> oh wait, it doesnt
242 2012-01-03 06:57:02 <gmaxwell> :)
243 2012-01-03 06:57:44 <gmaxwell> but if only a very few things are mlocking ... we can fix that pretty easily.
244 2012-01-03 06:58:32 <BlueMatt> https://github.com/bitcoin/bitcoin/pull/740
245 2012-01-03 06:59:04 <gmaxwell> BlueMatt: we really need to get the jenkins setup performing automated tests.. e.g. having it run a build that times bringing up a new node.
246 2012-01-03 06:59:24 <BlueMatt> gmaxwell: it does, but the vm is too unreliable to rely on chain download times
247 2012-01-03 06:59:42 <gmaxwell> It would have caught this.
248 2012-01-03 06:59:51 <BlueMatt> gmaxwell: it does that exact test...
249 2012-01-03 06:59:52 <gribble> New news from bitcoinrss: TheBlueMatt opened pull request 740 on bitcoin/bitcoin <https://github.com/bitcoin/bitcoin/pull/740>
250 2012-01-03 07:00:05 <BlueMatt> thanks gribble ,,(botsnack)
251 2012-01-03 07:00:07 <gribble> Forget the snack, just send me some bitcoins at 1MgD6rah5zUgEGYZnNmdpnXMaDR3itKYzU :)
252 2012-01-03 07:00:19 <midnightmagic> curheight 97k, 30% cpu
253 2012-01-03 07:00:30 <luke-jr> BlueMatt: that patch would be much nicer if you just typedef std::vector<char> vector_type; :P
254 2012-01-03 07:00:43 <BlueMatt> luke-jr: yea, but the resulting code wouldnt be
255 2012-01-03 07:00:51 <luke-jr> yes it would :p
256 2012-01-03 07:00:54 <BlueMatt> (or would be more standard with other bitcoin code)
257 2012-01-03 07:01:31 <luke-jr> the typedef means you don't need to change 20 lines if it every changes type
258 2012-01-03 07:01:48 <luke-jr> actually, that one should probably be a template with a typedef on it
259 2012-01-03 07:01:51 <BlueMatt> but you are defining std::vector<char>
260 2012-01-03 07:01:58 <luke-jr> &
261 2012-01-03 07:02:10 <BlueMatt> meh
262 2012-01-03 07:03:20 <BlueMatt> luke-jr: changed, happy?
263 2012-01-03 07:04:54 <gribble> New news from bitcoinrss: TheBlueMatt commented on pull request 740 on bitcoin/bitcoin <https://github.com/bitcoin/bitcoin/pull/740#issuecomment-3337231>
264 2012-01-03 07:05:06 <luke-jr> :p
265 2012-01-03 07:05:17 <midnightmagic> crazIness
266 2012-01-03 07:06:10 <BlueMatt> (on that subject can someone check my CDataStream grepping?)
267 2012-01-03 07:06:12 <midnightmagic> BlueMatt should go to sleep and stop worrying about things that can wait until tomorrow. :)
268 2012-01-03 07:10:28 <gribble> New news from bitcoinrss: gmaxwell commented on pull request 740 on bitcoin/bitcoin <https://github.com/bitcoin/bitcoin/pull/740#issuecomment-3337245>
269 2012-01-03 07:10:28 <midnightmagic> slowing now. 111k
270 2012-01-03 07:10:33 <midnightmagic> very very low cpu usage.
271 2012-01-03 07:11:07 <gmaxwell> midnightmagic: cpu use won't go up until 140xxx
272 2012-01-03 07:11:59 <gmaxwell> You might just be getting network throughtput bound now with the bigger blocks.
273 2012-01-03 07:17:57 <BlueMattBot> gmaxwell you may not issue bot commands in this chat!
274 2012-01-03 07:17:57 <gmaxwell> BlueMattBot: alas, your pull doesn't actually build for me.
275 2012-01-03 07:21:24 <midnightmagic> +1
276 2012-01-03 07:23:46 <gmaxwell> I'm betting that this mlock fix will also improve the performance I've had with mining..
277 2012-01-03 07:24:05 <gmaxwell> had some annoyance with bitcoin being unresponsive for a second or so when a new block comes in...
278 2012-01-03 07:24:43 <midnightmagic> 123k and I am definitely not running into bandwidth issues..  total incoming b/w is like 120k
279 2012-01-03 07:25:14 <midnightmagic> how big's your mining wallet?
280 2012-01-03 07:26:23 <gmaxwell> midnightmagic: many inputs.
281 2012-01-03 07:27:12 <midnightmagic> disk i/o is... 4MB/s and so on..
282 2012-01-03 07:27:29 <gmaxwell> midnightmagic: well, time to fix the _next_ issue.
283 2012-01-03 07:27:32 <gmaxwell> :)
284 2012-01-03 07:27:32 <midnightmagic> 125k, total bog.
285 2012-01-03 07:28:08 <midnightmagic> when i mine into an empty wallet, i don't have to worry about wallet flushes gumming me up
286 2012-01-03 07:28:28 <midnightmagic> accidental getinfo by some script i wrote that i never disabled.. etc.
287 2012-01-03 07:28:55 <gmaxwell> setting up an empty wallet node to mine aagainst was actually what inspired me to setup another node eariler today and I've been finding bugs since then. :)
288 2012-01-03 07:29:44 <midnightmagic> i only ever noticed the problem because my minin return during the merged-mining initial rush was..  pathetic. like.. 1% of what it should have been.
289 2012-01-03 07:30:12 <midnightmagic> totally screwed me because i'd calculated in advance what i was going to make a pre-sold them, expecting a crash in the price.
290 2012-01-03 07:30:38 <gmaxwell> midnightmagic: yea, I expected a crash too.. came much later than expected!
291 2012-01-03 07:30:51 <midnightmagic> wayyyy later. i didn't understand that at all
292 2012-01-03 07:31:12 <gmaxwell> I sold a bunnnnch and then it went way up later.
293 2012-01-03 07:31:20 <midnightmagic> lol
294 2012-01-03 07:31:29 <midnightmagic> stupid weird namecoin dynamics anyway
295 2012-01-03 07:33:34 <midnightmagic> kvm is way faster than i expected it to be
296 2012-01-03 07:33:48 <gmaxwell> midnightmagic: if you get bored of watching that node... undo the patch and nuke it... I'd not expect it to be synced before 12 hours was up. :)
297 2012-01-03 07:33:58 <gmaxwell> midnightmagic: yea, KVM is fast.
298 2012-01-03 07:34:06 <midnightmagic> ok, nuking.
299 2012-01-03 07:34:32 <midnightmagic> need a pristine base for qcow2 branchpoint anyway
300 2012-01-03 07:35:43 <cjdelisle> on any modernish processor, kvm is essentially native speed
301 2012-01-03 07:35:43 <gmaxwell> midnightmagic: Did I tell you I'm running p2pool now? I'm quite impressed with it. It's come a long way.
302 2012-01-03 07:36:37 <midnightmagic> very nice. :) i forget now, the individual payouts are by the individual right?
303 2012-01-03 07:36:58 <midnightmagic> forrestv isn't doing centralized payouts like i think he used to be?
304 2012-01-03 07:37:09 <gmaxwell> midnightmagic: Correct. It's eligius style payouts.
305 2012-01-03 07:37:25 <gmaxwell> The pool uses a merged mining chain to agree on shares for a PPLNS decision.
306 2012-01-03 07:37:46 <gmaxwell> The coinbase txn pays according to the consensus (and if not, your shares aren't valid for the pool)
307 2012-01-03 07:38:02 <midnightmagic> i must've asked this before..  how does it stop me from sending along non-solve shares but keeping the winnings for myself?
308 2012-01-03 07:38:10 <gmaxwell> the merged chain has a mean time of 10 seconds.. so lots of stales on it, but all that matters is your relative stale rate.
309 2012-01-03 07:38:42 <gmaxwell> midnightmagic: same way any pool does: you send your peers the failed candidates that prove that you were trying to pay according to the consensus.
310 2012-01-03 07:39:39 <gmaxwell> where 'failed' means not difficulty enough to be a solution but at least the p2pshare difficulty (adapts to keep the mean time to 10 seconds)
311 2012-01-03 07:40:34 <gmaxwell> Works ducky with namecoin merged mining too, though the merged mining part is just solo.  Even easier to setup than any other merged mining solution.
312 2012-01-03 07:40:37 <midnightmagic> and the solve has a pool midstate or whatever, so if you do find a solve, it already belongs to the pool
313 2012-01-03 07:40:46 <gmaxwell> midnightmagic: right.
314 2012-01-03 07:41:32 <midnightmagic> nice.  so I guess there's no more reason not to switch to p2pool..  all my original objections are gone. no more central payouts..
315 2012-01-03 07:41:51 <midnightmagic> and the more of us there are, the stronger our shared anonymity
316 2012-01-03 07:42:05 <gmaxwell> Indeed. I'm .. kind of identifyable on it right now.
317 2012-01-03 07:42:13 <midnightmagic> you are
318 2012-01-03 07:42:15 <midnightmagic> ?
319 2012-01-03 07:42:20 <midnightmagic> for your hashrate?
320 2012-01-03 07:42:40 <forrestv> currently 38% of the p2pool hashrate is going to .. 14ezpf2E6pK3A4EWibdPYWZSs442NtX1Ft
321 2012-01-03 07:42:43 <forrestv> i wonder who that is :P
322 2012-01-03 07:43:06 <gmaxwell> yea....
323 2012-01-03 07:43:08 <midnightmagic> is it possible to chew through addresses for shares?
324 2012-01-03 07:43:41 <gmaxwell> right now it pulls an address at startup, though I don't see any technical reason you couldn't split it up or rotate.
325 2012-01-03 07:44:12 <midnightmagic> lol so what is your p2pool hashrate then? :)
326 2012-01-03 07:44:13 <gmaxwell> would be kinda lame if enough was done to bloat the coinbase, but ::shrugs::
327 2012-01-03 07:44:47 <forrestv> midnightmagic, currently 25GH/s
328 2012-01-03 07:45:14 <midnightmagic> for the whole pool?
329 2012-01-03 07:45:31 <gmaxwell> midnightmagic: well, only have about 9GH/s on it. I guess I'll move the rest soon.
330 2012-01-03 07:46:09 <gmaxwell> Yea, it's currently small. But hey, improvement on solo.. but you still retain the same amount of txn selection autonomy. And you get fees, and the confidence that a pool operator isn't robbing you blind.
331 2012-01-03 07:46:33 <midnightmagic> mmm.. fees!  yeah for sure.
332 2012-01-03 07:46:45 <forrestv> midnightmagic, yeah, see http://u.forre.st/p2pool/600.png for history
333 2012-01-03 07:47:04 <midnightmagic> cool!
334 2012-01-03 07:47:10 <gmaxwell> you can see where I recently came in. :)
335 2012-01-03 07:47:37 <midnightmagic> i'll have to put some on it, but i guess i keep saying that and doing nothing..
336 2012-01-03 07:48:25 <midnightmagic> lol  yeah IMO that's the future of mining. at a certain point, really this or something like it should be integrated into -main
337 2012-01-03 07:49:34 <midnightmagic> apparently i have some miners somewhere still working on deepbit, but you think i can find them? trickling along..
338 2012-01-03 07:50:09 <forrestv> it's not yet a complete solution - the share difficulty is proportional to the pool size, which will make it difficult for small miners
339 2012-01-03 07:50:17 <forrestv> i'm working on techniques to make it more scalable, though
340 2012-01-03 07:50:43 <gmaxwell> forrestv: yea... and unfortunately the use-p2pool as a pool doesn't work great due to the fast chain's stales.
341 2012-01-03 07:51:06 <gmaxwell> otherwise I'd say that p2pool would be a boon for small centeralized pools too..
342 2012-01-03 07:52:42 <gmaxwell> forrestv: in any case, as is it's pretty much an ideal solution for someone who is big enough to be on the fence about solo mining.
343 2012-01-03 07:53:52 <forrestv> a separate p2pool with a share period of something like a minute would make using it as a pool work...
344 2012-01-03 07:54:20 <midnightmagic> can i set my own difficulty in it?
345 2012-01-03 07:54:57 <forrestv> midnightmagic, you mean in current-p2pool? no, it's automatically set to regulate the time between shares
346 2012-01-03 07:55:05 <midnightmagic> ok
347 2012-01-03 07:55:11 <midnightmagic> ah, so attractive.
348 2012-01-03 07:55:49 <gmaxwell> it's at 51 right now.
349 2012-01-03 08:00:38 <gmaxwell> Okay, way late to bed... hopefully I'll wake tomorrow to find all kinds of wonderful fixes and more things to measure.
350 2012-01-03 08:00:45 <midnightmagic> :)
351 2012-01-03 08:00:47 <gmaxwell> midnightmagic: how's that resync going?
352 2012-01-03 08:00:47 <midnightmagic> night man
353 2012-01-03 08:01:00 <midnightmagic> oh, I nuked it, it bogged down around 120k
354 2012-01-03 08:01:36 <midnightmagic> it would be nice if I could tell it to leech off my local trusted node so i didn't have to shut it down and copy the blockchain
355 2012-01-03 08:01:54 <gmaxwell> --connect but it still validates. :)
356 2012-01-03 08:12:18 <midnightmagic> LOL "Now FPGA mining at a whole 25 MHash/sec. Wow!" (makomk sig) awesome!
357 2012-01-03 08:13:04 <Diablo-D3> http://naurunappula.com/748185/tehosekoitin.jpg
358 2012-01-03 08:18:04 <midnightmagic> uh..  forrestv did you merge the getauxblock patches successfully with luke's coinbase uniqueless bugfix patch?
359 2012-01-03 08:18:46 <forrestv> midnightmagic, getauxblock is only present in namecoin. are you talking about getmemorypool?
360 2012-01-03 08:19:10 <midnightmagic> woops.
361 2012-01-03 08:19:28 <forrestv> pretty sure his patch wouldn't affect getmemorypool, because getmemorypool doesn't create a coinbase transaction, it just returns what you need to construct one
362 2012-01-03 08:20:10 <midnightmagic> no, i'm tired of being blocked by my procrastination in merging vince's bitcoind patch with the ongoing march of main bitcoind progress
363 2012-01-03 08:21:31 <midnightmagic> okay then.. hrm..
364 2012-01-03 09:24:46 <sipa> q wait what
365 2012-01-03 09:25:18 <sipa> CDataStream used the secure allocator?
366 2012-01-03 09:25:33 <sipa> since always?
367 2012-01-03 09:26:21 <sipa> why didn't we notice a performance regression when mlock was introduced?
368 2012-01-03 09:28:05 <sipa> and 28 minutes for block sync
369 2012-01-03 09:28:50 <sipa> it was way more than that before mlock was introduced, no?
370 2012-01-03 10:01:40 <gribble> New news from bitcoinrss: laanwj commented on issue 734 on bitcoin/bitcoin <https://github.com/bitcoin/bitcoin/issues/734#issuecomment-3338486>
371 2012-01-03 10:01:41 <gribble> New news from bitcoinrss: laanwj closed issue 734 on bitcoin/bitcoin <https://github.com/bitcoin/bitcoin/issues/734>
372 2012-01-03 10:31:59 <CIA-100> DiabloMiner: Patrick McFarland master * r0b8a06b / src/main/java/com/diablominer/DiabloMiner/DiabloMiner.java : Fix silly bug in multipool pool advancing - http://git.io/ImNT3Q https://github.com/Diablo-D3/DiabloMiner/commit/0b8a06b9f648a6b033acc722ea271833c603c948
373 2012-01-03 11:46:27 <luke-jr> sipa: since always
374 2012-01-03 11:47:21 <luke-jr> forrestv: actually, getmemorypool *strips* the coinbase from a normal work&
375 2012-01-03 11:47:50 <luke-jr> midnightmagic: I'm not sure why anyone would merge vince's patch :P
376 2012-01-03 13:25:25 <mega_p2k> hi, got a specific protocol question: can someone tell me under what conditions a valid transaction will be removed from the transaction pool while building a block?
377 2012-01-03 13:33:41 <lianj> mega_p2k: not sure, but a valid tx should not be removed
378 2012-01-03 13:35:16 <mega_p2k> lianj: I'm asking because I'd like to skip the getmemorypool call in my software and get the transactions off the p2p protocol instead
379 2012-01-03 13:36:06 <mega_p2k> is it also true that a bitcoind will only relay valid blocks and valid transactions to a connected peer?
380 2012-01-03 13:37:09 <mega_p2k> that way I could skip validation in my software knowing that I'm connected to a local - and therefore trusted - client
381 2012-01-03 13:37:30 <lianj> yes, to extend that it will only forward to types of valid tx script types atm. but protocol spec wise the other tx scripts are also valid
382 2012-01-03 13:38:02 <lianj> depends on how you define valid in your case but yes
383 2012-01-03 13:39:03 <mega_p2k> I would define valid as being allowed in a block
384 2012-01-03 13:40:05 <mega_p2k> but a valid tx should be a tx for which the inputs are valid
385 2012-01-03 13:40:10 <lianj> then the current client might not forward all valid ones to you, as it check for is_standard txs and only forward these
386 2012-01-03 13:40:30 <mega_p2k> well, how would I get other transactions then?
387 2012-01-03 13:40:54 <mega_p2k> just by examining a block?
388 2012-01-03 13:41:46 <lianj> yes, just wait for them to show up in latest blocks. for the official client i dont know how to get them. using the wire protocol on your own you would just get them (and have to validate them on your own)
389 2012-01-03 13:42:03 <mega_p2k> in all cases, I don't see any problem for me.
390 2012-01-03 13:42:11 <lianj> but non is_standard txs are rare cases
391 2012-01-03 13:42:21 <lianj> yep, you should be fine
392 2012-01-03 13:42:41 <mega_p2k> in case you don't know, I'm the author of a new pool mining software called ecoinpool
393 2012-01-03 13:43:27 <lianj> ah ok, hehe then you might what those other valid, but non is_standard txs too. but its not a must
394 2012-01-03 13:43:31 <mega_p2k> so for creating a block, I should be fine just taking any transactions the official client relays to me and putting them in a new block
395 2012-01-03 13:43:33 <lianj> *want
396 2012-01-03 13:44:14 <mega_p2k> yes, my point is that I don't require to get or validate non standard txs at all
397 2012-01-03 13:44:43 <lianj> aw :( but yes, then youre fine with the official client
398 2012-01-03 13:45:05 <mega_p2k> I just want to build a block, that's the single purpose of a pool software
399 2012-01-03 13:46:17 <lianj> i was under the impression that most mining pools still accept valid txs even though they are not is_standard ones
400 2012-01-03 13:47:00 <gavinandresen> There are three places where transactions are accepted:  when relaying, when deciding what to put into blocks, and when validating blocks...
401 2012-01-03 13:47:17 <gavinandresen> Most mining pools do not relay or mine non-standard transactions, but will accept them if somebody else has mined them
402 2012-01-03 13:47:22 <mega_p2k> lianj: yes, but how should non-standard txns find their way into a block that a mining pool software?
403 2012-01-03 13:48:33 <lianj> gavinandresen: ah, thanks. so makes the chance very low/hard for these non-standard txs to get mined at all
404 2012-01-03 13:48:41 <mega_p2k> gavinandresen: well yes, sure, they are accepted. but they won't be integrated into a new block per se.
405 2012-01-03 13:49:46 <gavinandresen> lianj: yes, it is hard to get non-standard transactions accepted into the block chain (unless you own a big mining pool or can convince a big mining pool to accept them)
406 2012-01-03 13:50:09 <mega_p2k> to my rationale, a non-standard txn can only find its way into a block if the mining software itself creates it and places it into a new block
407 2012-01-03 13:51:00 <gavinandresen> mega_p2k: that's not true, I believe the Eligius pool will mine non-standard transactions as long as they include a fee.
408 2012-01-03 13:51:01 <mega_p2k> the "mining software" could also just be bitcoind via getwork
409 2012-01-03 13:51:50 <mega_p2k> gavinandresen: does this imply that eligius' bitcoind will also forward non-standard transactions?
410 2012-01-03 13:51:52 <lianj> gavinandresen: aw :(  oh, now i like eligius even more
411 2012-01-03 13:52:13 <gavinandresen> mega_p2k: I think so.  You'd have to ask luke-jr for a definitive answer.
412 2012-01-03 13:52:20 <mega_p2k> ok
413 2012-01-03 13:53:03 <gavinandresen> mega_p2k: you might want to use getmemorypool instead of getwork, depending on how much control your pool software wants to have.
414 2012-01-03 13:54:19 <mega_p2k> gavinandresen: you joined a bit late. I was talking to lianj that I want to leave out calling an RPC call at all and instead take the forwarded transactions directly via the p2p protocol.
415 2012-01-03 13:54:58 <gavinandresen> mega_p2k: that'd work, too, if you trust the node forwarding you transactions to validate them before relaying them (the standard client always does).
416 2012-01-03 13:55:40 <mega_p2k> yes, the node which you can configure in my software must be a trusted one, preferably running on localhost
417 2012-01-03 13:55:52 <mega_p2k> this is a precondition, ofc
418 2012-01-03 13:56:41 <mega_p2k> this is until I implement the complete validation in my software
419 2012-01-03 13:58:42 <mega_p2k> but given that a local or trusted bitcoind will only forward valid blocks and valid transactions, I can skip validation for now
420 2012-01-03 13:59:54 <mega_p2k> I just need to calculate the fees based on the referenced transactions, which should be easy
421 2012-01-03 14:01:39 <mega_p2k> gavinandresen: just to be sure: is there really no way that a forwarded transaction from a trusted node will become invalid at some point?
422 2012-01-03 14:02:41 <gavinandresen> If it was part of a double-spend, and the "other" spend gets into the block-chain, then it will be invalid.
423 2012-01-03 14:04:08 <mega_p2k> I see. is it easy to detect that?
424 2012-01-03 14:04:30 <mega_p2k> I just have to compare inputs, right?
425 2012-01-03 14:04:51 <gavinandresen> Yes, txin[i].prevout.hash/prevout.n
426 2012-01-03 14:04:56 <[Tycho]> Hello.
427 2012-01-03 14:05:22 <gavinandresen> When a new block is created, remove any transactions that you were thinking of including that have matching prevout.hash/n ....
428 2012-01-03 14:05:46 <gavinandresen> (when you see a new block message from the network....)
429 2012-01-03 14:05:51 <gavinandresen> Hi [Tycho]
430 2012-01-03 14:05:59 <mega_p2k> gavinandresen: yep, sonds easy enough to me
431 2012-01-03 14:06:50 <mega_p2k> gavinandresen, lianj: thanks a lot
432 2012-01-03 14:07:33 <mega_p2k> gavinandresen: btw. we know each other, I'm p2k on github. looks like my mac deploy script did its job well.
433 2012-01-03 14:08:20 <gavinandresen> mega_p2k: yes-- I need to pull your updated doesn't-depend-on-macdeployqt version....
434 2012-01-03 14:09:12 <mega_p2k> I'm still to lazy to ask github's support for moving around my forks :)
435 2012-01-03 14:18:31 <sipa> ;;later tell BlueMatt your perf fix do't compile here
436 2012-01-03 14:18:32 <gribble> The operation succeeded.
437 2012-01-03 14:21:09 <CIA-100> bitcoin: Gavin Andresen master * r0fcf91e / (src/init.cpp src/irc.cpp src/net.cpp src/util.cpp src/util.h): Fix issue #659, and cleanup wallet/command-line argument handling a bit - http://git.io/Rwa9fQ https://github.com/bitcoin/bitcoin/commit/0fcf91ea1e23697736032caadc8e487e0ba6cfef
438 2012-01-03 14:21:51 <gribble> New news from bitcoinrss: gavinandresen pushed to master at bitcoin/bitcoin <https://github.com/bitcoin/bitcoin/compare/4231eb217c...0fcf91ea1e>
439 2012-01-03 14:22:46 <CIA-100> bitcoin: Gavin Andresen master * rdaad9a9 / (5 files in 4 dirs): Merge branch 'gitianfix' of https://github.com/TheBlueMatt/bitcoin - http://git.io/rlZnAw https://github.com/bitcoin/bitcoin/commit/daad9a9a71e9efc42e43245e374b4466dca6ebe6
440 2012-01-03 14:24:41 <CIA-100> bitcoin: Gavin Andresen master * r73e86ee / (src/script.cpp src/test/multisig_tests.cpp): Merge branch 'bugfix_multisig' of https://github.com/coderrr/bitcoin - http://git.io/czTgFA https://github.com/bitcoin/bitcoin/commit/73e86eedd5703e43bb8770a62acbb7c589a1905a
441 2012-01-03 14:26:53 <gribble> New news from bitcoinrss: gavinandresen pushed to master at bitcoin/bitcoin <https://github.com/bitcoin/bitcoin/compare/daad9a9a71...73e86eedd5>
442 2012-01-03 14:26:54 <gribble> New news from bitcoinrss: gavinandresen merged pull request 733 on bitcoin/bitcoin <https://github.com/bitcoin/bitcoin/pull/733>
443 2012-01-03 14:26:55 <gribble> New news from bitcoinrss: gavinandresen pushed to master at bitcoin/bitcoin <https://github.com/bitcoin/bitcoin/compare/0fcf91ea1e...daad9a9a71>
444 2012-01-03 14:27:54 <sipa> do we need both gribble and cia reporting pulls here?
445 2012-01-03 14:31:55 <gmaxwell> 02:28 < sipa> it was way more than that before mlock was introduced, no?
446 2012-01-03 14:32:10 <gmaxwell> ^ it was, but gavin removed the ecdsa sig validation before the highest checkpoint.
447 2012-01-03 14:32:20 <gmaxwell> thats why I note it gets slower after ~144k.
448 2012-01-03 14:34:15 <gmaxwell> And why didn't we notice?  I expect that we need to further improve the development process... we're not often resyncing nodes, and people expect it to be slow and getting slower.
449 2012-01-03 14:34:43 <gavinandresen> gmaxwell: I'm not going to regret pulling "no secure allocator for CDataStream" am I?  My rationale is: it is at BEST a teeny-tiny-marginal improvement in security, if it even has any effect at all...
450 2012-01-03 14:37:07 <gmaxwell> gavinandresen: I haven't personally audited the usage of CDataStream.  So long as it's not being used to handle private keys, I think the difference is pretty inconsequential.  Though Matt's patch goes further than the behavior before mlock made everything suck since it removes the post-zeroization.
451 2012-01-03 14:37:57 <gmaxwell> (this is also probably good for performance but I hadn't measured it when I checked out matt's patch didn't build for me and I didn't spare even a moment to try fixing it)
452 2012-01-03 14:38:09 <gavinandresen> Doesn't build for me either....
453 2012-01-03 14:39:29 <gmaxwell> If we wanted to be more security conservative we could replace all that with a properly performaing mlocked allocator (e.g. mlock in big infrequently allocated blocks and provide our own allocator on top of it), but I think that carries non-trivial software bug risk.
454 2012-01-03 14:40:28 <sipa> whatever we try to mlock, there will always be data leaking
455 2012-01-03 14:40:39 <sipa> so it is never more than a best-effort attempt, imho
456 2012-01-03 14:40:46 <gavinandresen> Not zeroing freed memory makes me nervous, that's a good building block for a remote code injection exploit
457 2012-01-03 14:41:20 <sipa> we could have a secure_alocator which zeroes, and a very_secure_allocator which mlocks
458 2012-01-03 14:41:25 <gmaxwell> yea, use after free...
459 2012-01-03 14:41:31 <user_> if i die. is it possible bitcooin automaticaly send my coins to a predefinied address
460 2012-01-03 14:41:45 <sipa> user_: how will bitcoin know you died?
461 2012-01-03 14:42:06 <justmoon> sipa: it will know ...if it killed him!!
462 2012-01-03 14:42:10 <gmaxwell> user_: It's possible to constrct arrangements like that but you need a trusted agent to confirm that you died.
463 2012-01-03 14:42:46 <gmaxwell> justmoon: I think if bitcoin was going to kill you it would start by sending all your coins to another address long before it got to the stabbing killing stage.
464 2012-01-03 14:42:52 <justmoon> sipa: unrelated question, are there any bitcoin meetups in belgium?
465 2012-01-03 14:43:00 <sipa> justmoon: not that i know of
466 2012-01-03 14:43:09 <justmoon> gmaxwell: lol, help, somebody make a patch!
467 2012-01-03 14:43:27 <sipa> gavinandresen: i prefer keeping the zeroing-freed-memory property as well
468 2012-01-03 14:43:51 <justmoon> sipa: do you want to maybe come to a swiss one sometime? I've got a belgian guy living in geneva who's a fan of yours :D
469 2012-01-03 14:44:18 <sipa> maybe, i even have family in switzerland :)
470 2012-01-03 14:44:37 <sipa> not sure exactly where, never visited them there
471 2012-01-03 14:44:55 <justmoon> sipa: switzerland is very small, they are probably my neighbours
472 2012-01-03 14:45:10 <justmoon> sipa: I'll put you on swiss meetup mailing list
473 2012-01-03 14:45:14 <sipa> good
474 2012-01-03 14:47:10 <gribble> New news from bitcoinrss: gavinandresen commented on pull request 740 on bitcoin/bitcoin <https://github.com/bitcoin/bitcoin/pull/740#issuecomment-3341417>
475 2012-01-03 14:48:13 <gmaxwell> Boost apparently does have an allocator, which you can override how it gets memory: http://www.boost.org/doc/libs/1_47_0/libs/pool/doc/interfaces/user_allocator.html
476 2012-01-03 14:49:00 <sipa> gmaxwell: looks exactly like what we need
477 2012-01-03 14:51:40 <gmaxwell> if we want to be properly security paranoid, and we're using a library allocator, we could potentially add canary functionality just like the stack protection in addition to the zeroize. I'd have to dump the allocator usage data to see how much overhead that might have.
478 2012-01-03 14:52:13 <gmaxwell> (e.g. add a word after (and perhaps before) every allocation which is checked on free)
479 2012-01-03 14:53:10 <gavinandresen> sounds like a very good idea, maybe turned on by compile-time or run-time debug switch....
480 2012-01-03 14:53:54 <gavinandresen> (if performance matters)
481 2012-01-03 14:54:41 <CIA-100> bitcoin: Gavin Andresen master * r112b0e9 / (15 files in 2 dirs): Merge pull request #741 from laanwj/typo734fix ... https://github.com/bitcoin/bitcoin/commit/112b0e97d47709f9f7fd0d9aad62a9063ba541f3
482 2012-01-03 14:55:49 <sipa> can anyone on OSX try to build my netbase patch?
483 2012-01-03 14:57:14 <gribble> New news from bitcoinrss: gavinandresen pushed to master at bitcoin/bitcoin <https://github.com/bitcoin/bitcoin/compare/73e86eedd5...112b0e97d4>
484 2012-01-03 14:57:15 <gribble> New news from bitcoinrss: gavinandresen merged pull request 741 on bitcoin/bitcoin <https://github.com/bitcoin/bitcoin/pull/741>
485 2012-01-03 14:57:27 <gavinandresen> sipa: ok, I'll try...
486 2012-01-03 14:57:36 <sipa> gavinandresen: thanks
487 2012-01-03 15:01:35 <gavinandresen> sipa:  I'm doing the rebasing work as I go... (I pulled an fTOR cleanup this morning)
488 2012-01-03 15:02:16 <gribble> New news from bitcoinrss: gmaxwell commented on pull request 740 on bitcoin/bitcoin <https://github.com/bitcoin/bitcoin/pull/740#issuecomment-3341703>
489 2012-01-03 15:05:30 <sipa> gavinandresen: ok, let me know if you have troubles
490 2012-01-03 15:05:40 <gavinandresen> sipa: compiling....
491 2012-01-03 15:08:02 <jgarzik> sipa: bdb ugh...  a75d7066b824eb1e70fb5b0e0e7c3c122e00c4b6
492 2012-01-03 15:08:18 <jgarzik> sipa: better to build a list, then shove it all into bdb with a single transaction
493 2012-01-03 15:08:33 <jgarzik> sipa: that _dramatically_ increases the db load, for the normal case (quick DNS lookups)
494 2012-01-03 15:09:24 <jgarzik> sipa: there is a reason why bdb txn commit was designed as it was, prior to a75d7066b824eb1e70fb5b0e0e7c3c122e00c4b6
495 2012-01-03 15:09:47 <sipa> jgarzik: that's possible, but it increases the time before you have any seeds
496 2012-01-03 15:09:54 <sipa> especially when there is a problem with one
497 2012-01-03 15:11:23 <sipa> anyway, i'm thinking about redoing the CAddress-database entirely
498 2012-01-03 15:11:47 <jgarzik> sipa: the normal case blasts a bunch of ~100-byte flushes to your disk
499 2012-01-03 15:11:51 <jgarzik> sipa: that's a regression
500 2012-01-03 15:13:03 <sipa> yes, but seeds come faster and with less contention on the db db lock
501 2012-01-03 15:13:12 <CIA-100> bitcoinjs/bitcoinjs-lib: booo master * r7675cf1 / src/wallet.js : src/wallet.js: retab file - http://git.io/M42BgQ https://github.com/bitcoinjs/bitcoinjs-lib/commit/7675cf14e4bd7643af61a95aa5231863434c5e0e
502 2012-01-03 15:13:13 <CIA-100> bitcoinjs/bitcoinjs-lib: booo master * r3445ae2 / src/wallet.js : src/wallet.js: use jshint - http://git.io/X5Znlw https://github.com/bitcoinjs/bitcoinjs-lib/commit/3445ae2a3649e40a4ec639654defed1113e3980a
503 2012-01-03 15:14:41 <jgarzik> sipa: not quite true -- it is noticeably slow here, because disk flush takes a longer than average time
504 2012-01-03 15:15:20 <sipa> hmm, would it help moving the CAddrDb declaration out of the loop?
505 2012-01-03 15:15:48 <sipa> i really prefer not needing to wait for all dns seeds
506 2012-01-03 15:16:23 <jgarzik> sipa: one commit per seed is better than one commit per address
507 2012-01-03 15:16:39 <sipa> so, it's one commit per seed noz
508 2012-01-03 15:16:47 <jgarzik> noz?
509 2012-01-03 15:16:49 <sipa> now
510 2012-01-03 15:16:50 <gmaxwell> jgarzik: Even on a super slow writing cheap ssd there didn't appear to be any performance problem from that change... though yea, one commit per seed would not insert any horrible delays.
511 2012-01-03 15:16:57 <gmaxwell> oh.
512 2012-01-03 15:17:27 <gribble> New news from bitcoinrss: gmaxwell commented on issue 739 on bitcoin/bitcoin <https://github.com/bitcoin/bitcoin/issues/739#issuecomment-3341895>
513 2012-01-03 15:17:34 <CIA-100> bitcoin: Gavin Andresen master * r8677f9c / src/init.cpp : I broke -testnet with my TOR option-parsing fixes. - http://git.io/soaXkQ https://github.com/bitcoin/bitcoin/commit/8677f9c751b397476593f60c747709c61d5b8f19
514 2012-01-03 15:18:53 <jgarzik> sipa: hrm, so it is.  will have to benchmark this performance regression further.  new bitcoin is definitely hitting db hard and flushing a lot more, making it noticably slower.  zero activity from other processes on the machine.
515 2012-01-03 15:19:24 <sipa> we're talking about *three* more db commits; in total
516 2012-01-03 15:19:31 <jgarzik> nod
517 2012-01-03 15:19:47 <jgarzik> sipa: any non-seed-related db changes come to mind, perchance?
518 2012-01-03 15:19:53 <jgarzik> otherwise it's bisect time :/
519 2012-01-03 15:20:26 <sipa> jgarzik: in the same git commit i moved some things out of db-locked code
520 2012-01-03 15:20:36 <gmaxwell> I expect with the mlock fix syncup will be fsync limited again. So sounds like a good time to think about reducing the db load independant of whatever regression jgarzik is seeing.
521 2012-01-03 15:21:08 <sipa> jgarzik: though it's hard to see how that could reduce performance
522 2012-01-03 15:21:18 <jgarzik> yeah
523 2012-01-03 15:21:23 <gavinandresen> I've thought several times of using a different DbEnv for wallet.dat versus the other .dats-- sync is really only important for wallet.
524 2012-01-03 15:21:31 <gmaxwell> jgarzik: sipa's patch creates much greater concurrency.. e.g. it doesn't sit blocked waiting on dns seed before it beging pulling blocks from a peer. Make it noticably faster to start syncing for me.
525 2012-01-03 15:21:32 <jgarzik> gavinandresen: that would be nice
526 2012-01-03 15:22:03 <sipa> gavinandresen: ACK, it's even required if we want to have multiple-wallets and wallets loaded from user-defined locations one day
527 2012-01-03 15:22:28 <jgarzik> gmaxwell: agree that sipa's commit no longer -appears- to be the source of a flush-based performance regression
528 2012-01-03 15:22:30 <gribble> New news from bitcoinrss: gavinandresen pushed to master at bitcoin/bitcoin <https://github.com/bitcoin/bitcoin/compare/112b0e97d4...8677f9c751>
529 2012-01-03 15:23:53 <sipa> gavinandresen: any result from compiling netbase?
530 2012-01-03 15:24:45 <gavinandresen> sipa: compiles fine after fixing merge conflicts
531 2012-01-03 15:25:19 <gmaxwell> gavinandresen: so the other advantage of that seperation would be that they could have seperate transaction logs.. right now you can think you're being clever and putting your wallet on a more reliable medium and get hosed by not taking the txn logs with it.
532 2012-01-03 15:25:42 <gavinandresen> sipa:  but I managed to get myself into merge/rebase hell by fixing an unrelated bug in the middle....
533 2012-01-03 15:26:01 <gmaxwell> gavinandresen: e.g. you've put datadir on tmpfs but symlinked the wallet onto a raid array. Works great until you have a power outage while it was writing the wallet.
534 2012-01-03 15:26:04 <sipa> ah
535 2012-01-03 15:26:24 <gavinandresen> gmaxwell: good point.
536 2012-01-03 15:27:09 <gmaxwell> Of course, without all the fsync crap on everything else there would be a lot less reason to run from tmpfs. :)
537 2012-01-03 15:30:14 <CIA-100> bitcoinjs/bitcoinjs-gui: booo master * re0eef3c / scripts/exitnode.js : scripts/exitnode: log socketio errors - http://git.io/6oXWSA https://github.com/bitcoinjs/bitcoinjs-gui/commit/e0eef3c4f07dd1268d5a4fa02044554be713da08
538 2012-01-03 15:30:15 <CIA-100> bitcoinjs/bitcoinjs-gui: booo master * racddc7b / scripts/vendor/bitcoinjs-lib : update bitcoinjs-lib submodule - http://git.io/eoFeFg https://github.com/bitcoinjs/bitcoinjs-gui/commit/acddc7bb43f5865e426b32d3ed34aac76b518ca1
539 2012-01-03 15:30:45 <justmoon> aw man, why do I always hit exactly five commits, where it spams like crazy... sorry :(
540 2012-01-03 15:31:33 <luke-jr> gavinandresen: Why does 0fcf91e change net.cpp?
541 2012-01-03 15:31:55 <luke-jr> actually, I think I understand the reasoning now&
542 2012-01-03 15:33:45 <sipa> gavinandresen: in 0fcf91e, in net.cpp: you declare fTOR locally, but don't use it anymore?
543 2012-01-03 15:34:03 <gavinandresen> sipa: oops
544 2012-01-03 15:34:33 <gavinandresen> You guys shouldn't let me write code any more, I'm going senile.....
545 2012-01-03 15:37:39 <sipa> gavinandresen: i've rebased my netbase now
546 2012-01-03 15:38:11 <sipa> ah damn, don't have the ssh key there to push
547 2012-01-03 15:38:24 <gavinandresen> sipa:  that's funny, I just finished rebasing....
548 2012-01-03 15:38:38 <sipa> don't worry; it wasn't much work :)
549 2012-01-03 15:41:05 <gavinandresen> sipa: rebased netbase compiles and runs fine on my mac
550 2012-01-03 15:42:59 <sipa> good!
551 2012-01-03 15:43:23 <sipa> ming/win may be a bit more trouble, as it uses getaddrinfo now, which requires some extra/other headers
552 2012-01-03 15:47:09 <justmoon> gavinandresen: has anyone thought of a good name for your proposal? (the OP_EVAL alternative with a magic scriptPubKey)
553 2012-01-03 15:47:18 <sipa> gavinandresen, jgarzik, gmaxwell: what do you think about this new handling of CAddress'es: classify each address in one of 64 buckets, based on Hash(nonce + addr >> 16); for each bucket, keep at most 64 already-succesfully-connected-to addresses (ordered by last succesfull try), and at most 64 not-yet-tried addresses
554 2012-01-03 15:47:48 <sipa> that means at most 8192 addresses, which are just kept in memory all the time, and occasionally (not synchronously) written to disk
555 2012-01-03 15:47:50 <gribble> New news from bitcoinrss: gavinandresen commented on pull request 735 on bitcoin/bitcoin <https://github.com/bitcoin/bitcoin/pull/735#issuecomment-3342374>
556 2012-01-03 15:48:13 <gavinandresen> justmoon: It needs a good name-- PayToScriptHash is the best I've been able to come up with
557 2012-01-03 15:48:35 <justmoon> k, I'll use that until further notice
558 2012-01-03 15:52:11 <gavinandresen> sipa:  do we have any data on how many connection attempts it takes to find a working peer if you haven't connected in a day/week/month ?
559 2012-01-03 15:52:44 <gavinandresen> sipa:  ... or, in other words, I'm not sure how we'd know if 8192 is too many or too few.
560 2012-01-03 15:52:56 <sipa> gavinandresen: stats from bitcoin-crawler:
561 2012-01-03 15:52:58 <sipa> 1641/16549 available (16268 tried in 1620s, 257 new, 24 active), 301029 banned;
562 2012-01-03 15:53:22 <sipa> that means it knows about 16500 sometimes-reachable nodes, of which 1641 are very reliable
563 2012-01-03 15:53:51 <sipa> and it also knows about 300000 nodes that have awful reliability (probably not reachable at all)
564 2012-01-03 15:54:06 <gmaxwell> sipa: I feel a little uneasy about the >>16 just because I don't like the fact that your neighbor can spin up 64 hosts and knock you out of nodes tables.
565 2012-01-03 15:54:32 <sipa> gmaxwell: he could only poison one of the 64 buckets
566 2012-01-03 15:54:42 <sipa> that's exactly the reason for the shift
567 2012-01-03 15:55:15 <sipa> oh, addr is in network byte order, so it should be <<
568 2012-01-03 15:55:32 <sipa> anyway, i just meant: the higher-order 16 bits of the IPv4 address
569 2012-01-03 15:55:53 <phantomcircuit> i still think the connection strategy needs to be updated
570 2012-01-03 15:56:01 <sipa> phantomcircuit: explain
571 2012-01-03 15:56:02 <phantomcircuit> 4 stable connections 4 roaming connections
572 2012-01-03 15:56:17 <sipa> hmm
573 2012-01-03 15:56:25 <gmaxwell> Yes, I agree but thats seperate.
574 2012-01-03 15:57:26 <gmaxwell> (I actually think it should be something like 'keep the 'best' two peers where best means they give you the most good blocks and txn' and roam the single oldest connection... but whatever, seperate subject)
575 2012-01-03 15:58:53 <phantomcircuit> well actually
576 2012-01-03 15:58:55 <gmaxwell> sipa: thats a great assumption if you assume that a single /16 is an administrative domain, but thats a poor assumption.  I wasn't disagreeing with the shift, I know why you have it there. I just feel uneasy about the amount.
577 2012-01-03 15:59:11 <phantomcircuit> i would say have about 100 roaming connections
578 2012-01-03 15:59:22 <phantomcircuit> but that will break on windows and destroy many NAT setups
579 2012-01-03 15:59:37 <gmaxwell> ugh. no. 8 connections is already a _lot_ from a graph connectivity perspective.
580 2012-01-03 15:59:50 <sipa> gmaxwell: no, i assume most administrative domains are smaller than a /16
581 2012-01-03 15:59:57 <phantomcircuit> gmaxwell, yeah i understand
582 2012-01-03 16:00:22 <phantomcircuit> if you had lots of roaming connections you would need to change the notification algorithm
583 2012-01-03 16:00:33 <phantomcircuit> doing so would obviously be dangerous though
584 2012-01-03 16:00:47 <gmaxwell> sipa: right, which means that I can zot out neighboring admin domains. How about something morally like this:
585 2012-01-03 16:01:30 <sipa> gmaxwell: not following
586 2012-01-03 16:01:57 <sipa> gavinandresen: i'll try to gather some real statistics, butbased on what i've seen with my crawler, i'd guess that 100 addresses is enough to find a good one after a week
587 2012-01-03 16:02:18 <sipa> but that's really just a guess based on how many different addresses i've seen in +- a week
588 2012-01-03 16:02:20 <phantomcircuit> sipa, if the average admin domain is smaller than a /16 then you can get neighboring domains blocked
589 2012-01-03 16:02:56 <gribble> New news from bitcoinrss: gavinandresen commented on issue 739 on bitcoin/bitcoin <https://github.com/bitcoin/bitcoin/issues/739#issuecomment-3342682>
590 2012-01-03 16:03:04 <gmaxwell> e.g. I can get hosted 'next door' to mtgox and get them blocked.
591 2012-01-03 16:03:11 <sipa> phantomcircuit: but all addresses under your control will for everyone always end up in only one bucket
592 2012-01-03 16:03:29 <phantomcircuit> uh
593 2012-01-03 16:03:31 <phantomcircuit> no
594 2012-01-03 16:03:55 <phantomcircuit> spammers are getting better
595 2012-01-03 16:03:57 <gmaxwell> Well I controll addresses on multiple 16s just fine. :)
596 2012-01-03 16:04:04 <phantomcircuit> i've got stuff here where the sa score is 1.7
597 2012-01-03 16:04:05 <phantomcircuit> :(
598 2012-01-03 16:04:20 <gmaxwell> sipa: (ignoring byteswap) hash(nonce + addr>>16 + hash(nonce+address&((1<<16)-1)&3)  e.g. /16 goes into 4 bins.
599 2012-01-03 16:04:29 <gmaxwell> ^ what do you think of something like that.
600 2012-01-03 16:04:35 <sipa> hmm, maybe the buckets for not-yet-tried-addresses should be based on where you got them from
601 2012-01-03 16:04:42 <sipa> instead of the addresses themselves
602 2012-01-03 16:04:48 <gmaxwell> yea, I think that someone suggested that before.
603 2012-01-03 16:04:55 <sipa> yes, i think i did
604 2012-01-03 16:05:31 <sipa> gmaxwell: looks good, a bit of spreading in it as well
605 2012-01-03 16:05:52 <gavinandresen> I'd be a much happier camper if somebody was simulating all this under plausible attack scenarios....
606 2012-01-03 16:06:21 <gavinandresen> (but I agree the addr message handling / addr.dat is broken enough to warrant fixing)
607 2012-01-03 16:06:33 <gmaxwell> gavinandresen: the problem with that is 'attacker gets a bunch of /16s and denies incoming to some nodes is plausable but we can't stop it.
608 2012-01-03 16:07:52 <gmaxwell> sipa: perhaps always take a few bits from where you learned it.
609 2012-01-03 16:11:19 <sipa> as soon as you go to using where you got an address from, a single address can of course end up in several bins
610 2012-01-03 16:13:54 <gmaxwell> meh. yea.
611 2012-01-03 16:14:20 <sipa> not really a problem, but that may warrant somewhat larger bins for the not-yet-tried ones
612 2012-01-03 16:14:25 <sipa> as they will have overlap
613 2012-01-03 16:14:51 <gmaxwell> Hm. Well, you need to lookup to see if you have it already... so you can just use the first you heard it from, no?
614 2012-01-03 16:15:15 <sipa> that's also possible
615 2012-01-03 16:15:26 <gmaxwell> though overlap might not be so bad.
616 2012-01-03 16:15:30 <sipa> and probably the easiest way
617 2012-01-03 16:15:44 <sipa> on the other hand: you can use overlap in your advantage: as a scoring function
618 2012-01-03 16:15:52 <sipa> that was suggested before as well, iirc
619 2012-01-03 16:15:58 <gmaxwell> (makes it harder to slay an address if there is some chance it'll stick around in an overlap)
620 2012-01-03 16:17:09 <sipa> what about: for tried addresses: use mostly upper bits of address itself, with a small influence from the lower-order bits (like your example above)
621 2012-01-03 16:17:44 <sipa> for non-tried addresses: use mostly upper bits of source, with a small influence from the address itself
622 2012-01-03 16:19:39 <sipa> meh, becomes hard if you want to do it right
623 2012-01-03 16:19:46 <gmaxwell> I wonder how much attack resistance we'd gain by using random eviction that can actually evict the new entry.
624 2012-01-03 16:20:17 <sipa> eviction when the bins get full, you mean?
625 2012-01-03 16:20:22 <CIA-100> bitcoin: Gavin Andresen master * raf8c56f / (src/bitcoinrpc.cpp src/main.h): Merge branch 'getblock' - http://git.io/reg12g https://github.com/bitcoin/bitcoin/commit/af8c56f8be7f01136c88e665c7580daf6f9c9d7d
626 2012-01-03 16:20:32 <gavinandresen> gribble, be quiet.
627 2012-01-03 16:20:36 <gmaxwell> e.g. when you're full and you go to add a new one pick one of the 65 possible addresses at random to evict. This way someone trying to flood out your bin can only increase their chances with more attacks.
628 2012-01-03 16:21:27 <sipa> sounds good
629 2012-01-03 16:21:33 <gmaxwell> (they might just evict themselves over and over agai.. and as they get closer to all of the bin they'll spend most of their time doing that)
630 2012-01-03 16:21:46 <sipa> i like having random factors in the algorithm, makes it hard to control by others
631 2012-01-03 16:22:06 <luke-jr> gavinandresen: wasn't the older dumpblock more powerful/useful? O.o
632 2012-01-03 16:23:22 <gribble> New news from bitcoinrss: gavinandresen pushed to master at bitcoin/bitcoin <https://github.com/bitcoin/bitcoin/compare/8677f9c751...af8c56f8be>
633 2012-01-03 16:23:59 <rjk2> when did gribble get spam capabilities?
634 2012-01-03 16:24:47 <gmaxwell> sipa: I dunno, seems easy enough, especially if you allow duplication in the ones that include the source... just don't bother to check if you've already got an address before adding it to the waiting pool.
635 2012-01-03 16:24:57 <CIA-100> bitcoin: Gavin Andresen master * r96d3bcb / (4 files): Merge pull request #731 from laanwj/txshowfix ... https://github.com/bitcoin/bitcoin/commit/96d3bcb99690726d4c0e28355cc87c25e14f4c8d
636 2012-01-03 16:25:12 <gmaxwell> Check the addresses against connected.. fails that stuff it in the waiting pool
637 2012-01-03 16:25:22 <CIA-100> bitcoin: Wladimir J. van der Laan 0.5.x * r20e3f2aefc60 bitcoind-stable/src/qt/ (15 files in 2 dirs): Fix typo (#734) http://tinyurl.com/8yhtqu9
638 2012-01-03 16:25:24 <CIA-100> bitcoin: Luke Dashjr 0.5.x * ra2e9767225d3 bitcoind-stable/src/ (7 files in 2 dirs): Merge branch '0.5.0.x' into 0.5.x http://tinyurl.com/6rf58sv
639 2012-01-03 16:25:41 <luke-jr> rjk2: BlueMatt
640 2012-01-03 16:27:19 <gmaxwell> rjk2: if it bugs you, filter it out.
641 2012-01-03 16:27:36 <rjk2> nah i like to hear from him occasionally
642 2012-01-03 16:28:01 <justmoon> you could filter on "news from bitcoinrss"
643 2012-01-03 16:28:23 <gribble> New news from bitcoinrss: gavinandresen pushed to master at bitcoin/bitcoin <https://github.com/bitcoin/bitcoin/compare/af8c56f8be...96d3bcb996>
644 2012-01-03 16:28:24 <gribble> New news from bitcoinrss: gavinandresen merged pull request 731 on bitcoin/bitcoin <https://github.com/bitcoin/bitcoin/pull/731>
645 2012-01-03 16:28:47 <rjk2> it just seems a little more verbose than expected =)
646 2012-01-03 16:28:47 <sipa> wow, it's mergin' day!
647 2012-01-03 16:28:58 <rjk2> ill look into a special ignore rule
648 2012-01-03 16:29:56 <luke-jr> gavinandresen: am I correct in assuming that coinbaser and signmessage are waiting on jgarzik and wumpus at this point?
649 2012-01-03 16:30:09 <gavinandresen> luke-jr: yup
650 2012-01-03 16:33:04 <luke-jr> wumpus: Does 56c6e36 hold true (not all tx'es with "from"/"to" field are necessarily IP tx'es) before OP_EVAL?
651 2012-01-03 16:33:44 <gribble> New news from bitcoinrss: gmaxwell commented on issue 739 on bitcoin/bitcoin <https://github.com/bitcoin/bitcoin/issues/739#issuecomment-3343118>
652 2012-01-03 16:35:35 <gavinandresen> luke-jr: I hate to feature-creep.... but I wonder if -blocknotify aught to pass the block hash to the command being run.  That'd make it play nicely with getblock.
653 2012-01-03 16:35:53 <luke-jr> gavinandresen: it does, if you put %s in the command
654 2012-01-03 16:36:43 <gavinandresen> So it does.... cool
655 2012-01-03 16:38:56 <justmoon> gmaxwell: do you have a branch where I can pull your secure allocator test version?
656 2012-01-03 16:39:10 <luke-jr> gavinandresen: don't suppose you know the answer to "from"/"to" since wumpus seems to not be here?
657 2012-01-03 16:39:17 <justmoon> (secure allocator disablement to be precise)
658 2012-01-03 16:39:22 <gmaxwell> justmoon: Er, I have a stupid patch that removes all the mlocks.
659 2012-01-03 16:39:32 <sipa> justmoon: just re-#define mlock/munlock in serialize.h
660 2012-01-03 16:39:35 <gmaxwell> BlueMatt: has the disablement but it doesn't build.
661 2012-01-03 16:39:50 <gmaxwell> https://people.xiph.org/~greg/omg_go_fast.patch
662 2012-01-03 16:39:52 <sipa> the real patch won't do that though; but is broken currently
663 2012-01-03 16:40:19 <gmaxwell> (that leaves the zeroization in.. which is what I expect we'll do long term)
664 2012-01-03 16:40:30 <gmaxwell> justmoon: if you'd like to benchmark it w and w/o that be fun.
665 2012-01-03 16:40:44 <sipa> 12000
666 2012-01-03 16:40:52 <justmoon> gmaxwell: i just want to run it on my standard benchmark system so I can get a sense where it is in comparison w/ bitcoinjs
667 2012-01-03 16:40:53 <gmaxwell> It might be that it was more of a speedup for me (due to cpu/kernel/libc version) than others.
668 2012-01-03 16:41:20 <gmaxwell> justmoon: oh well, there are still a lot of low hanging fruit in bitcoind.
669 2012-01-03 16:41:39 <sipa> 15000
670 2012-01-03 16:41:49 <shadders> luke-jr: miss me?  hope you've finally released yr toy pool sw while I was awol...
671 2012-01-03 16:41:55 <justmoon> gmaxwell: such as?
672 2012-01-03 16:42:02 <sipa> gmaxwell: still, you did a full sync in half an hour?
673 2012-01-03 16:42:05 <luke-jr> shadders: not really, but the other pool ops were getting desperate.
674 2012-01-03 16:42:07 <gmaxwell> justmoon: all the crazy pointless fsync behavior.
675 2012-01-03 16:42:07 <sipa> over lan?
676 2012-01-03 16:42:13 <gmaxwell> sipa: yes sir.
677 2012-01-03 16:42:13 <luke-jr> shadders: they're all defecting to an Erlang pool :P
678 2012-01-03 16:42:24 <sipa> gmaxwell: that's by far faster than any number i've ever seen
679 2012-01-03 16:42:48 <shadders> luke-jr: so I hear... Now I know how Moses felt after a few weeks on a mountain
680 2012-01-03 16:42:50 <gmaxwell> sipa: as I mentioned earlier, gavin made it faster recently with the ecdsa disabling.
681 2012-01-03 16:43:08 <sipa> yes, but still.. a few hours for a sync, no?
682 2012-01-03 16:43:23 <justmoon> gmaxwell: that doesn't do too much, 4.5 million ecdsa verifications take about 8 minutes of cpu time
683 2012-01-03 16:43:28 <sipa> not sure i tested it after that check was disabled
684 2012-01-03 16:43:47 <sipa> 21000
685 2012-01-03 16:44:02 <gmaxwell> Yes, thats what it's been.. but back in march? before the mlock stuff it was taking a half hourish on a system with a slower cpu. (If I go through logs I can probably find where I benchmarked it)
686 2012-01-03 16:44:32 <sipa> cya
687 2012-01-03 16:44:55 <justmoon> gmaxwell: wait, so the 30 min figure - is it for the current chain or back then?
688 2012-01-03 16:45:12 <gmaxwell> current chain.
689 2012-01-03 16:45:45 <gmaxwell> (This isn't surprising, as mentioned: faster system)
690 2012-01-03 16:45:57 <gmaxwell> (plus improvements)
691 2012-01-03 16:46:14 <justmoon> alright, I gotta run this for myself
692 2012-01-03 16:46:16 <justmoon> thanks for the help
693 2012-01-03 16:46:23 <gmaxwell> gavinandresen: Then perhaps we have another bug because I see it become obviously slower at about 140k
694 2012-01-03 16:46:49 <nanotube> well, i can filter out the 'pushed' rss items, since they're duplicated by cia. leaving the issue comments and pull requests that are not covered by cia ?
695 2012-01-03 16:46:56 <nanotube> what do you guys think?
696 2012-01-03 16:47:33 <gavinandresen> new pull requests are valuable.  I think issue comments are too much
697 2012-01-03 16:47:40 <gmaxwell> lemme run a syncup with timestamps.
698 2012-01-03 16:47:57 <gmaxwell> Can we make it just do new issues?
699 2012-01-03 16:48:25 <nanotube> yes we can
700 2012-01-03 16:48:29 <nanotube> just ... decide what you guys want :)
701 2012-01-03 16:48:36 <nanotube> gribble is here to serve
702 2012-01-03 16:49:09 <justmoon> gmaxwell: here is a partial download with bitcoinjs (with timestamps) if you're curious: https://gist.github.com/1556051
703 2012-01-03 16:49:55 <justmoon> i broke something that makes it run out of file descriptors, once that's fixed I'll run a full one
704 2012-01-03 16:50:15 <justmoon> (and no I'm not leaking file descriptors just opening too many...)
705 2012-01-03 16:53:12 <gmaxwell> Why aren't timestamps on in the debug log by default? They're very handy.
706 2012-01-03 16:53:29 <gmaxwell> (would be more handy if they had subsecond precision, but hey)
707 2012-01-03 16:53:59 <nanotube> ok, gribble will now only mention new pull requests and issues, and ignore comments and commits
708 2012-01-03 16:55:11 <gavinandresen> gmaxwell: they're not on by default for probably-paranoid-and-ill-considered privacy reasons:  what if an attacker gathered debug.logs from a bunch of machines and used the timestamps to figure out.... stuff....
709 2012-01-03 16:55:25 <gmaxwell> meh, they still preserve total otder.
710 2012-01-03 16:55:27 <gmaxwell> er order.
711 2012-01-03 16:56:02 <gavinandresen> ... so you're saying we should randomize OutputDebugStringF a little bit... I'll get right on that.
712 2012-01-03 16:56:04 <gavinandresen> :)
713 2012-01-03 16:56:21 <gmaxwell> god no. hah. I should watch what I say.
714 2012-01-03 16:56:47 <gmaxwell> too often in #bitcoin/forums I'll make a joke and someone will say "thats a fantastic idea!"
715 2012-01-03 16:58:07 <gmaxwell> Looks like I might beat my prior record...
716 2012-01-03 16:58:49 <luke-jr> gmaxwell: you were going to implement deterministic wallets, right?
717 2012-01-03 16:58:52 <luke-jr> :P
718 2012-01-03 17:05:40 <gmaxwell> I'll have cool syncup graphs in a bit.
719 2012-01-03 17:08:35 <justmoon> gmaxwell: I got 80000 blocks in ten minutes so far - that would still be significantly slower than bitcoinjs (about 50%)
720 2012-01-03 17:08:43 <justmoon> obviously the real test will be the larger blocks
721 2012-01-03 17:09:05 <justmoon> gmaxwell: have you run the unmodified client on the same system? how long does that take?
722 2012-01-03 17:20:06 <gmaxwell> justmoon: you're slow, I took 100 seconds to get 80k blocks.
723 2012-01-03 17:20:25 <justmoon> gmaxwell: how long does the unmodified client take for you?
724 2012-01-03 17:21:07 <justmoon> I think you are testing on >3 times faster hardware than I am :)
725 2012-01-03 17:21:28 <justmoon> I got to 110000 blocks after 20 minutes (bitcoinjs does it in 6)
726 2012-01-03 17:22:15 <gmaxwell> 01/03/12 17:51:33 SetBestChain: new best=00000000839a8e6886ab  height=1 work=8590065666
727 2012-01-03 17:22:18 <gmaxwell> 01/03/12 18:21:04 SetBestChain: new best=00000000000004c0de2c  height=160458 work=201667296186438652831
728 2012-01-03 17:22:27 <gmaxwell> Just finished measuring patched with timestamps.
729 2012-01-03 17:22:42 <forrestv> luke-jr, oh yeah, forgot about that, but that's just an implementation detail :P
730 2012-01-03 17:23:45 <justmoon> gmaxwell: unmodified client, how long does it take on that same system you're testing the patch on - seems like a percentage improvement would be a useful figure, even a rough one
731 2012-01-03 17:24:37 <gmaxwell> justmoon: stop!
732 2012-01-03 17:24:44 <gmaxwell> justmoon: :) you're asking the same question over and over again.
733 2012-01-03 17:24:57 <justmoon> gmaxwell: I'm known to be persistent :)
734 2012-01-03 17:24:58 <gmaxwell> justmoon: I'm measuring that now. But it takes a while to sync up in order to answer!
735 2012-01-03 17:25:06 <justmoon> gmaxwell: kk, sorry
736 2012-01-03 17:25:24 <gmaxwell> ask again _tomorrow_ thats how slow it is without the patch for me.
737 2012-01-03 17:25:25 <gmaxwell> :)
738 2012-01-03 17:25:33 <justmoon> hmm, weird
739 2012-01-03 17:25:34 <gmaxwell> (I'll give some halarious partial results in a bit)
740 2012-01-03 17:27:49 <luke-jr> gmaxwell: instead of comparing time from block 1 to 160458, compare number of blocks in 30 minutes ;)
741 2012-01-03 17:28:18 <luke-jr> maybe not perfect, but still a general idea
742 2012-01-03 17:28:58 <justmoon> I got 120000 blocks in 30 minutes with gmaxwell's patch and 145000 for bitcoinjs
743 2012-01-03 17:29:11 <imsaguy> gmaxwell: you're lucky vragnaroda only just now joined :p
744 2012-01-03 17:29:57 <vragnaroda> imsaguy: I still smacked him in #bitcoin.
745 2012-01-03 17:32:58 <justmoon> gmaxwell: OS and system specs would be good to know
746 2012-01-03 17:33:11 <justmoon> gmaxwell: maybe your OS has a bug in mlock/munlock?
747 2012-01-03 17:42:46 <luke-jr> IMO, mlock is basically useless so long as we're not secure-deleting unencrypted wallet files ;)
748 2012-01-03 17:45:21 <justmoon> kind of interesting, with node-bitcoin-p2p I don't care about zeroing memory, because it doesn't handle any secrets
749 2012-01-03 17:45:50 <BlueMatt> the reason for the memory zeroing is to prevent use-after-free attacks
750 2012-01-03 17:45:50 <luke-jr> wumpus: wumpus where are you? :P
751 2012-01-03 17:45:57 <BlueMatt> not to clear secrets
752 2012-01-03 17:46:02 <BlueMatt> (CDataStream never handles secrets
753 2012-01-03 17:46:02 <midnightmagic> luke-jr: unless someone cares about whether their swap is vulnerable to analysis, though right?
754 2012-01-03 17:46:03 <BlueMatt> )
755 2012-01-03 17:46:06 <BlueMatt> have to board bbl
756 2012-01-03 17:47:33 <justmoon> so what if an attacker uses-before-free...
757 2012-01-03 17:49:41 <BlueMatt-mobile> justmoon then zeroing wont help
758 2012-01-03 17:50:33 <LightRider> Is it possible to construct a bitcoin transaction such that anyone can spend the result?
759 2012-01-03 17:50:49 <BlueMatt-mobile> Yes very easily
760 2012-01-03 17:51:02 <BlueMatt-mobile> (With nonstd txn)
761 2012-01-03 17:51:03 <gmaxwell> justmoon: http://people.xiph.org/~greg/bitcoin-sync.png
762 2012-01-03 17:51:15 <gmaxwell> http://people.xiph.org/~greg/bitcoin-sync-speed.png
763 2012-01-03 17:51:36 <justmoon> lol
764 2012-01-03 17:51:44 <BlueMatt-mobile> Heh oh god
765 2012-01-03 17:51:52 <justmoon> gmaxwell: there is more to the story than just zeroing memory
766 2012-01-03 17:52:10 <luke-jr> midnightmagic: swap isn't special
767 2012-01-03 17:52:12 <BlueMatt-mobile> Kinda depressing that no one noticed that...
768 2012-01-03 17:52:13 <gmaxwell> justmoon: it's not zeroing memory thats the issue, it's calling mlock.
769 2012-01-03 17:52:16 <justmoon> for me your patch does maybe a 40% improvement
770 2012-01-03 17:52:24 <justmoon> at most
771 2012-01-03 17:52:25 <gmaxwell> justmoon: and mlock blows up the TLB.
772 2012-01-03 17:52:38 <justmoon> so something is going on on your system that isn't on mine
773 2012-01-03 17:52:43 <BlueMatt-mobile> gmaxwell can you benchmark how much time zeroing takes?
774 2012-01-03 17:52:46 <gmaxwell> justmoon: may be worse on AMD vs intel. ... though, I'm also pulling from an mlocked fixed node.
775 2012-01-03 17:52:52 <gmaxwell> (I'm still zeroing in mine)
776 2012-01-03 17:53:13 <BlueMatt-mobile> Can you test nonzerod
777 2012-01-03 17:53:24 <gmaxwell> BlueMatt-mobile: I would, but your patch didn't compile as was.
778 2012-01-03 17:53:29 <justmoon> gmaxwell: right... pulling from a fixed node would probably be a good idea, wouldn't it.... *facepalm*
779 2012-01-03 17:53:30 <gmaxwell> and I didn't feel like fixing it.
780 2012-01-03 17:53:40 <BlueMatt-mobile> gmaxwell fixed now
781 2012-01-03 17:53:54 <gmaxwell> okay. I'll let the embarassing mode run for a bit longer..
782 2012-01-03 17:53:59 <BlueMatt-mobile> (I REALLY need to stop coding at night
783 2012-01-03 17:54:01 <gmaxwell> justmoon: I'm also just a lot faster than you for some reason.
784 2012-01-03 17:54:02 <BlueMatt-mobile> )
785 2012-01-03 17:54:39 <justmoon> gmaxwell: hardware maybe? mine is a four year old core duo with a 5400rpm hard drive on its last leg :)
786 2012-01-03 17:55:13 <gmaxwell> oh. yea, you're getting fsync bound.
787 2012-01-03 17:55:48 <gmaxwell> This is on a X6 1055T with a super fast ssd and fs logging disabled.
788 2012-01-03 17:56:11 <justmoon> lol
789 2012-01-03 17:57:17 <gmaxwell> https://people.xiph.org/~greg/with.fix.debug.log.xz  < here is the debug log from that run.
790 2012-01-03 17:57:41 <gmaxwell> justmoon: but hey, 40% is still nice on a system that is otherwise 'slow'
791 2012-01-03 17:58:14 <justmoon> gmaxwell: yeah, I was just freaking out because if it was faster than bitcoinjs, then I just wasted two months of my life :)
792 2012-01-03 17:58:21 <luke-jr> lol
793 2012-01-03 17:58:40 <gmaxwell> justmoon: oh well, you may be in trouble if someone takes out the fsync crap. ;)
794 2012-01-03 17:59:15 <luke-jr> :D
795 2012-01-03 17:59:17 <justmoon> gmaxwell: maybe you could run bitcoinjs on your beastmachine sometime - I need to fix some bugs of my own first though
796 2012-01-03 17:59:28 <justmoon> also: challenge accepted :P
797 2012-01-03 18:00:06 <BlueMatt-mobile> justmoon tmpfs?
798 2012-01-03 18:01:36 <justmoon> BlueMatt-mobile: that machine has 2GB of memory - might work, but tight
799 2012-01-03 18:01:54 <justmoon> also tmpfs is kinda cheating :P
800 2012-01-03 18:01:56 <BlueMatt-mobile> Oh...
801 2012-01-03 18:02:06 <BlueMatt-mobile> Buy a new machine?
802 2012-01-03 18:02:23 <gmaxwell> BlueMatt-mobile: pull 740 gives me b92bbfce3fab46a79b5f06c746761a59891a3b9e which still doesn't build for me
803 2012-01-03 18:02:34 <justmoon> my desktop is faster - but... it's complicated
804 2012-01-03 18:03:36 <justmoon> although...
805 2012-01-03 18:03:51 <justmoon> if I run on tmpfs anyway I could use a livecd
806 2012-01-03 18:03:57 <justmoon> brb
807 2012-01-03 18:04:11 <BlueMatt-mobile> gmaxwell i never tested it just fixed the function that looked culprity
808 2012-01-03 18:04:55 <gmaxwell> the explody part is
809 2012-01-03 18:05:07 <gmaxwell> (and since it's a C++ ism I'm not touching it)
810 2012-01-03 18:05:13 <BlueMatt-mobile> Arg.. .
811 2012-01-03 18:05:39 <BlueMatt-mobile> Oh come on that is the simplest error ever...just remove the func in question
812 2012-01-03 18:09:05 <gmaxwell> BlueMatt-mobile: oh that was easy. I guess I should have looked.
813 2012-01-03 18:09:21 <BlueMatt-mobile> Yep
814 2012-01-03 18:15:32 <gmaxwell> BlueMatt-mobile: measurably faster, but it might end up taking longer because the keypool gen is still slow. :)
815 2012-01-03 18:15:40 <gmaxwell> Graphs when it finishes.
816 2012-01-03 18:16:58 <sipa> gmaxwell: new idea: for tried addresses, 256 buckets of 64 addresses, each based on high bits + 4 possibilities per low bits
817 2012-01-03 18:17:47 <sipa> gmaxwell: for non-tried addresses, 256 buckets of 64 addresses, where in a first step 64 buckets are selected based on high bits of source, and in a second step one of those is selected based on the address itself
818 2012-01-03 18:18:14 <sipa> this means that an attacker will have to send from many many source before he can cover all 256 buckets, as each source only covers 64 of them
819 2012-01-03 18:22:41 <luke-jr> oh ugly!
820 2012-01-03 18:22:47 <luke-jr> Devcoin uses version 0 for their addresses too
821 2012-01-03 18:22:49 <gmaxwell> I think thats quite attractive.
822 2012-01-03 18:22:57 <luke-jr> gmaxwell: then you fail
823 2012-01-03 18:22:58 <gmaxwell> oh
824 2012-01-03 18:23:09 <gmaxwell> I thought you were responding to sipa
825 2012-01-03 18:23:11 <gmaxwell> yea that sucks.
826 2012-01-03 18:23:11 <luke-jr> :p