1 2012-09-05 00:01:05 <jgarzik> quite fast strategy, if leveldb behaves like my cheap python clone, pagedb... there's just the logfile, then a bunch of read-only, mmap-able chunks.
2 2012-09-05 00:03:04 <jgarzik> fast and efficient :)
3 2012-09-05 00:03:59 <sipa> near the end of the chain, leveldb was writing at around 21MB/s here on average
4 2012-09-05 00:04:22 <sipa> now trying the same with bdb
5 2012-09-05 00:07:27 <jgarzik> since bdb's pages are smaller, it rewrites leaf nodes less often. larger pages (chunks) mean more likely to be rewritten.
6 2012-09-05 00:07:51 <denisx> util.cpp: In function 'void FileCommit(FILE*)':
7 2012-09-05 00:08:44 <jgarzik> so the question is whether bdb internal node rewriting exceeds leveldb chunk rewriting
8 2012-09-05 00:09:02 <jgarzik> and whether or not that matters, if leveldb does so much better with post-checkpoint seeking
9 2012-09-05 00:09:42 <sipa> jgarzik: very rough benchmark result: i think bdb's total amount of written data is comparable
10 2012-09-05 00:09:43 <jgarzik> gavinandresen: *OrphanTx does not need protection, but reading accounting entries etc. from BDB does. the latter is, however, much lower priority as you note in the original audit response.
11 2012-09-05 00:10:08 <jgarzik> gavinandresen: as it's just on-disk -> memory corruption being detected
12 2012-09-05 00:10:17 <sipa> denisx: you'll need to replace it with fsync, if fdatasynch doesn't exist
13 2012-09-05 00:10:32 <jgarzik> gavinandresen: I think it's nice to say "hey, your db is corrupted" rather than just crash. leads to nicer bug reports.
14 2012-09-05 00:10:41 <jgarzik> gavinandresen: no one advocates _continuing_ under such conditions, obviously
15 2012-09-05 00:10:45 <gavinandresen> jgarzik: I was playing with recovering from on-disk corruption tonight, bdb is REALLY picky....
16 2012-09-05 00:11:11 <sipa> ACTION likes leveldb's "assume process will just get killed" philosophy
17 2012-09-05 00:11:33 <gavinandresen> e.g. my db48_dump -r invariably dumps core on a one-byte-corrupted wallet.dat (-R, "aggressive recovery", works)
18 2012-09-05 00:12:46 <gavinandresen> I think the right transition strategy away from bdb might be to do what db_dump -R does, and then parse the resulting dumpfile into leveldb
19 2012-09-05 00:12:49 <sipa> jgarzik: i'll try to have better numbers soon, but seems BDB actually may do more writing (in number of written bytes)
20 2012-09-05 00:13:34 <sipa> gavinandresen: importing a possibly corrupt database sounds like a recipe for stuck nodes
21 2012-09-05 00:13:54 <gavinandresen> sipa: I'm only talking about the wallet
22 2012-09-05 00:14:05 <sipa> oh, i wouldn't use leveldb for wallets
23 2012-09-05 00:14:13 <sipa> overkill and messy
24 2012-09-05 00:14:29 <gavinandresen> ok, transitioning away from bdb to whatever the replacement is....
25 2012-09-05 00:14:47 <sipa> right
26 2012-09-05 00:14:57 <sipa> for wallets that's probably the better strategy
27 2012-09-05 00:15:02 <gavinandresen> RE: overkill and messy? if we're using leveldb anyway, why would it be either overkill or messy?
28 2012-09-05 00:15:24 <sipa> mostly the fact that it would turn wallet.dat into wallet/
29 2012-09-05 00:15:41 <gavinandresen> I don't want to reinvent leveldb's nice "assume process will just get killed" way of doing things
30 2012-09-05 00:16:00 <sipa> agree, but an entire directory per wallet...
31 2012-09-05 00:16:12 <sipa> if we're always loading it entirely into memory anyway
32 2012-09-05 00:16:40 <sipa> and we're talking about files that are almost always smaller than the smallest amount of data leveldb ever rewrites
33 2012-09-05 00:16:57 <jgarzik> ACTION already looked into that...
34 2012-09-05 00:17:21 <sipa> i started writing a "logdb" once that's really just a log of writes and erases
35 2012-09-05 00:17:25 <jgarzik> better for wallet.dat to be a flat file of our own design, separate from environment
36 2012-09-05 00:17:29 <sipa> with a checksum per write-block
37 2012-09-05 00:17:35 <jgarzik> yep, something like that
38 2012-09-05 00:18:07 <jgarzik> plenty of examples in the filesystem world
39 2012-09-05 00:18:18 <gavinandresen> okey doke. I don't really care how it is implemented, as long as it is robust.
40 2012-09-05 00:18:56 <gavinandresen> and we'll use protocol buffers! So it is extensible!
41 2012-09-05 00:19:29 <gavinandresen> (and we have more dependencies! Dependencies, yay!)
42 2012-09-05 00:19:59 <sipa> MC-Eeepc: progress? :)
43 2012-09-05 00:20:53 <sipa> jgarzik: with BDB i see around 13MB/s
44 2012-09-05 00:21:03 <sipa> but it also takes almost twice as long
45 2012-09-05 00:21:27 <jgarzik> gavinandresen: new wallet file format, no new dependencies
46 2012-09-05 00:21:48 <sipa> i know too little about BDB's implementation, but how they manage to need more writes than leveldb is beyond me :)
47 2012-09-05 00:22:26 <jgarzik> sipa: bdb page == 4096 bytes. tons more pages to manage, than leveldb's chunks.
48 2012-09-05 00:23:12 <jgarzik> (or it might be 8192 bytes... you get the picture)
49 2012-09-05 00:23:12 <sipa> yes, clearly
50 2012-09-05 00:23:37 <sipa> but i would expect more smaller pages to be more efficient in terms of required disk bandwidth
51 2012-09-05 00:23:55 <jgarzik> b+tree node insertions require a lot of rewriting (though that depends on your checkpoint interval()
52 2012-09-05 00:24:35 <denisx> sipa: remember that most disks today have 4k blocks
53 2012-09-05 00:25:41 <jgarzik> bdb's writes are 4k-aligned, so you're not looking at COW or anything
54 2012-09-05 00:25:45 <jgarzik> ditto filesystem blocks
55 2012-09-05 00:25:53 <jgarzik> er
56 2012-09-05 00:25:56 <jgarzik> RMW
57 2012-09-05 00:27:14 <sipa> RMW?
58 2012-09-05 00:27:58 <denisx> read-modify-write
59 2012-09-05 00:28:32 <jgarzik> sipa: read-modify-write. on a modern disk with 4K sectors, updating a 512-byte "sector" involves reading from hard drive (internally, in the disk firmware), updating 512 bytes of the 4K block, and then writing the 4K block.
60 2012-09-05 00:28:57 <kjj_> particularly on WD drives that still lie about their sector size
61 2012-09-05 00:29:01 <jgarzik> ATA "API" is still 512-byte blocks, even if underlying is larger.
62 2012-09-05 00:29:21 <jgarzik> kjj_: not at all. this is all documented in the standards.
63 2012-09-05 00:29:34 <jgarzik> it's just logical API vs. physical implementation
64 2012-09-05 00:29:47 <jgarzik> ATA's IDENTIFY DEVICE data gives you the full picture.
65 2012-09-05 00:30:10 <kjj_> plenty of WD drives lie about their sector size, causing problems for 4k-aware OSs, like Linux
66 2012-09-05 00:30:15 <denisx> jgarzik: no, some harddisks lie about their physical blocksize
67 2012-09-05 00:30:39 <kjj_> there are angry rants about the problem on the web
68 2012-09-05 00:31:05 <jgarzik> denisx: I've not seen any evidence of this, sorry
69 2012-09-05 00:31:31 <denisx> jgarzik: challenge accepted! ;)
70 2012-09-05 00:32:19 <jgarzik> these false claims resurface every year for past 5+ years
71 2012-09-05 00:32:21 <kjj_> http://johannes-bauer.com/linux/wdc/?menuid=3
72 2012-09-05 00:33:00 <denisx> http://svnweb.freebsd.org/base?view=revision&revision=222520
73 2012-09-05 00:33:15 <denisx> in freebsd hey hardcoded some disks to 4k
74 2012-09-05 00:33:21 <denisx> they
75 2012-09-05 00:33:51 <jgarzik> kjj_: outdated info. that is long since addressed.
76 2012-09-05 00:34:07 <jgarzik> denisx: yes, some hard drives had to be quirk'd
77 2012-09-05 00:34:21 <denisx> jgarzik: but I must admit, maybe the problem was that there was no way to give that information to the upper layers
78 2012-09-05 00:36:28 <kjj_> meh. it looks to me like it will continue to be a problem for at least 5 more years, even if the drives coming off the line today (or last year) do it right
79 2012-09-05 00:37:42 <denisx> and only microsoft did the right thing: setting the alignment of partitions to 2MB
80 2012-09-05 00:43:51 <jgarzik> fsck and parted have been lying around partitions 4k/8k-aligned for years, regardless of what the disk reports
81 2012-09-05 00:43:54 <jgarzik> anyway
82 2012-09-05 00:44:33 <jgarzik> the relevant question for bitcoin: our keys tend to include hashes... what is the most optimal storage method for key/value records with random key distribution?
83 2012-09-05 00:45:04 <jgarzik> kyotocabinet's method of one-huge-hash-table (16MB?) + leaf nodes might be more optimal than leveldb or bdb
84 2012-09-05 00:45:32 <jgarzik> of course leveldb gets close to that, for its first level
85 2012-09-05 00:45:55 <jgarzik> but leveldb's key sorting does not buy us anything
86 2012-09-05 00:46:49 <kjj_> honestly, a straight up dead simple binary tree should be more or less balanced
87 2012-09-05 00:48:00 <kjj_> build the tree down to n bits and sort each leaf. increase n any time a leaf gets too big
88 2012-09-05 00:50:36 <kjj_> you could even give each leaf two pointers, one to a sorted list, and one to a list of newly-added-not-yet-sorted keys
89 2012-09-05 00:50:48 <jgarzik> it's not a question of balance, but of minimal seeks and disk traffic. sufficient internal nodes recreates bdb's b+trees
90 2012-09-05 00:51:27 <jgarzik> and we really don't care about sorting, except as it applies to the direct-key-lookup algorithm
91 2012-09-05 00:51:48 <kjj_> right.
92 2012-09-05 00:52:50 <kjj_> if you keep n big enough, iterating an unsorted list is close enough (in practice) to iterating a sorted list
93 2012-09-05 00:54:25 <jgarzik> interestingly, with our data, I've thought about time-sorted. We often are more likely to want recent data. Therefore: append to an unsorted list, and reverse-linear search.
94 2012-09-05 00:54:26 <kjj_> still, having a tree with pointers to lists with pointers to the actual data is a reasonably efficient system, and dead simple.
95 2012-09-05 00:55:09 <jgarzik> leveldb is basically a big first-level list, with pointers to a list of sorted records
96 2012-09-05 00:55:28 <kjj_> well, is that really true though? I'd want test it. could easily turn out that recency is merely more memorable than random
97 2012-09-05 00:56:08 <jgarzik> it is generally true: TX's spend more-recent TX's
98 2012-09-05 00:56:26 <jgarzik> but that's obviously dependent on many factors, and may change
99 2012-09-05 00:56:47 <jgarzik> as long as we have HungerCoins and SD though, it is Very True ;)
100 2012-09-05 00:57:20 <kjj_> fuck 'em.
101 2012-09-05 00:58:09 <jgarzik> hehehe
102 2012-09-05 00:58:20 <jgarzik> if we want a performant implementation it matters ;)
103 2012-09-05 00:58:32 <kjj_> nothing against either of those sites, but there is no reason to bend the protocol or the client to their needs alone
104 2012-09-05 01:02:37 <kjj_> I would think that just about any index for keys should fit in RAM and be good enough
105 2012-09-05 01:03:24 <kjj_> unless we are thinking about people with HUGE key lists
106 2012-09-05 01:05:26 <jgarzik> kjj_: that is the data in the blockchain. if you don't want a slow client, you _must_ deal with it. it is unrealistic to ignore them, sadly.
107 2012-09-05 01:06:47 <kjj_> I thought that we were talking about wallets. are you talking about indexing blocks/transactions by key/address?
108 2012-09-05 01:10:48 <jgarzik> kjj_: blocks and transactions from the public blockchain must be indexed
109 2012-09-05 01:11:04 <jgarzik> kjj_: current those indices are stored in BDB
110 2012-09-05 01:11:10 <jgarzik> *currently
111 2012-09-05 01:11:18 <kjj_> right
112 2012-09-05 01:12:09 <kjj_> actually, the two level thing I said earlier might be decent for that too.
113 2012-09-05 01:12:57 <kjj_> at least for random access. the worst case shouldn't be horrible. but I don't think the best case will be much better because it'll have to head back to disk to pull the leaf list often
114 2012-09-05 01:14:01 <kjj_> but unless you store the index as a linked list, any other scheme will require a disk hit going from block n to block n+1, most of the time.
115 2012-09-05 01:14:56 <kjj_> and we already have a structure with the blocks in (nearly) chronological order, so that's pointless.
116 2012-09-05 01:19:03 <kjj_> simplicity is a beautiful thing, and a tree of lists is about as simple as it gets. robust too
117 2012-09-05 01:20:55 <kjj_> easy to check and fix. all operations are done from bottom up, so in the event of a crash, you still have the correct old structure, plus some garbage that can be collected after a reference count
118 2012-09-05 01:22:04 <kjj_> not prone to fragmentation either, since the lists can have predictable sizes and holes will be quickly re-used
119 2012-09-05 01:31:45 <jgarzik> kjj_: I think you understand the problem, yes
120 2012-09-05 01:31:55 <jgarzik> big index, keys are hashes
121 2012-09-05 01:32:11 <jgarzik> some lookups are for old data, but most are for recent data
122 2012-09-05 01:32:56 <kjj_> for recent stuff, you could keep a circular buffer in addition to the regular indexes
123 2012-09-05 01:35:14 <kjj_> the problem is that you don't know in advance which index will be faster, if you are starting from a hash
124 2012-09-05 01:36:14 <jgarzik> yep
125 2012-09-05 01:36:25 <kjj_> but that could be dynamic. check the recent stuff first, and keep track of hits/misses
126 2012-09-05 01:36:49 <jgarzik> basically a cache in front of the database, at that point
127 2012-09-05 01:36:55 <kjj_> then try to figure out how much recent stuff to keep based on expected value
128 2012-09-05 01:38:18 <kjj_> that's probably too much effort for the devs though, in exchange for too little performance for the users
129 2012-09-05 01:39:35 <jgarzik> kjj_: nothing is too much effort, for an additional 0.5% ;p
130 2012-09-05 01:40:57 <kjj_> ok, 4k blocks. 32 bit block IDs.
131 2012-09-05 01:42:38 <kjj_> a list block has 64 pointers to other blocks and covers 6 bits of index space
132 2012-09-05 01:43:43 <kjj_> the blocks pointed to are either more list blocks, with the next 6 bits, or a list of up to ~63 offsets into the block file(s)
133 2012-09-05 01:44:45 <kjj_> set a high water mark, when a list hits it, build 64 new lists and a new tree block, update pointers, mark orphaned blocks available
134 2012-09-05 01:45:55 <kjj_> maybe reserve a special value for empty. because the first N bits of a block hash are always zero, the structure will be a bit wasteful without special cases
135 2012-09-05 01:46:12 <jgarzik> yeah, and if it's a disk format that requires _updating_, you have to consider half-finished disk writes etc. that's why most formats wind up as rewrite-then-invalidate, writing [re]written data to new blocks for safe commit to media
136 2012-09-05 01:46:52 <jgarzik> updating is space efficient but not reliable
137 2012-09-05 01:47:19 <kjj_> to a large extent, this is already safe for failed IO
138 2012-09-05 01:47:48 <jgarzik> one must also be cognizant of 4K alignment issues, due to hard drive block sizes as discussed above
139 2012-09-05 01:48:12 <jgarzik> a 1K write is really RMW
140 2012-09-05 01:48:21 <kjj_> yup, that's why I'm quoting 4096 bits for everything, and why you can only get 6 bits in a tree block and 63 (or less) entries in a list block
141 2012-09-05 01:51:18 <osmosis> need link to ubuntu ppa for bitcoin dev
142 2012-09-05 01:52:13 <kjj_> https://launchpad.net/~bitcoin/+archive/bitcoin ?
143 2012-09-05 01:53:47 <dooglus> amiller: http://www.satoshidice.com/ itself lists 'biggest winners' - the biggest payout is 1070 BTC for a 100 BTC bet on 'lessthan 6000'
144 2012-09-05 01:59:52 <amiller> dooglus, the operators of the site have a "trapdoor" in the form of the secrets, right?
145 2012-09-05 02:00:10 <jgarzik> meh
146 2012-09-05 02:00:13 <kjj_> no
147 2012-09-05 02:00:19 <jgarzik> CDataStream is quite unfriendly to sendfile(2)
148 2012-09-05 02:00:46 <amiller> er, that wasn't the right word to use
149 2012-09-05 02:00:56 <amiller> someone who knows the secrets can generate winning transactions
150 2012-09-05 02:01:23 <amiller> and the winning transactions generated that way are indistinguishable from any other winning transaction placed properly
151 2012-09-05 02:03:16 <gmaxwell> amiller: or losing, yes.
152 2012-09-05 02:03:56 <gmaxwell> I suppose you could say that they are distinguishable by virtue of their increased winning/losingness.
153 2012-09-05 02:04:10 <amiller> if someone actually won at the $64k level, i would expect there to be a surprise race condition
154 2012-09-05 02:04:25 <amiller> and two transactions suddenly show up at the $64k level
155 2012-09-05 02:05:07 <gmaxwell> hm? it's not a finite pool lottery
156 2012-09-05 02:05:10 <amiller> only the first one would win, right?
157 2012-09-05 02:05:17 <gmaxwell> No.
158 2012-09-05 02:05:45 <amiller> i read that the bet is changed after each round, but maybe rounds mean 'a day'?
159 2012-09-05 02:08:13 <kjj_> I'm pretty sure that all of the secrets have been published encrypted
160 2012-09-05 02:08:59 <amiller> suppose i were able to steal their secret for the current $64k
161 2012-09-05 02:09:05 <amiller> would i be able to win once, or unlimited times?
162 2012-09-05 02:09:49 <kjj_> http://satoshidice.com/secrets.php
163 2012-09-05 02:09:57 <kjj_> read that page before you speculate any further
164 2012-09-05 02:12:43 <amiller> i've read that page several times but still don't understand what the conditions are for winning, i'm probably just making a mistake though
165 2012-09-05 02:13:38 <gmaxwell> your transaction wins if its hmac(secret,txnid)<target for the target given by the address you paid.
166 2012-09-05 02:13:46 <amiller> "Each bet transaction that comes in is assigned to the secret key of the current day"
167 2012-09-05 02:14:34 <amiller> how are the addresses published
168 2012-09-05 02:14:35 <dooglus> amiller: there's a single file listing all the secrets for many years to come, yes
169 2012-09-05 02:15:28 <dooglus> amiller: anyone with that file can test whether a transaction is a winner before transmitting it. they can't "generate winning bets", but they can avoid sending losing bets
170 2012-09-05 02:16:30 <dooglus> meh - gotta go - out of gas
171 2012-09-05 02:16:41 <gmaxwell> dooglus expertly splits hairs on that point. try-n-check is a perfectly acceptable algorithim, especially for things with such good odds. :P
172 2012-09-05 02:16:57 <kjj_> no, the secrets are encrypted
173 2012-09-05 02:17:27 <gmaxwell> kjj_: ... To _you_ they are, yes. Not to the site operator.
174 2012-09-05 02:17:38 <kjj_> right
175 2012-09-05 02:18:11 <kjj_> I'm saying that there is no way for a person to calculate their bet in advance and avoid sending losers
176 2012-09-05 02:19:11 <gmaxwell> kjj_: the site operator can happily do this.
177 2012-09-05 02:19:15 <jrmithdobbs> who cares if the site operator always wins
178 2012-09-05 02:19:24 <jrmithdobbs> which they will
179 2012-09-05 02:19:30 <amiller> because the site operator can feign foul play
180 2012-09-05 02:19:37 <gmaxwell> ("anyone with that file", that file being the preimages for their committed secrets.)
181 2012-09-05 02:19:53 <gmaxwell> amiller: foul play?
182 2012-09-05 02:20:31 <amiller> what do you think would happen if someone actually won 64k
183 2012-09-05 02:20:41 <kjj_> I think they'd be very happy
184 2012-09-05 02:20:45 <amiller> there'd probably be only one winner at a time, right? it's pretty rare
185 2012-09-05 02:20:56 <amiller> and they'd pay out 64k i guess
186 2012-09-05 02:21:07 <gmaxwell> Either the site shuts down and fails to pay (bankrupt), or they'd just pay it and life would go on.
187 2012-09-05 02:21:20 <amiller> what if there wasn't one winner at a time
188 2012-09-05 02:21:42 <gmaxwell> oh "we got hacked, see, 100 64k winning bets at once!"
189 2012-09-05 02:21:56 <amiller> yeah.
190 2012-09-05 02:23:05 <jrmithdobbs> it just amazes me that people throw money at that crap
191 2012-09-05 02:23:09 <jrmithdobbs> technical discussion aside
192 2012-09-05 02:23:15 <gmaxwell> jrmithdobbs: I like to dream that they don't.
193 2012-09-05 02:23:25 <gmaxwell> Except I know they do, not only that??? they write bots to play it.
194 2012-09-05 02:23:47 <gmaxwell> And they have elaborate ways of failing at math which make them think that some strategy or another makes it positive expectation for them.
195 2012-09-05 02:23:49 <jrmithdobbs> gmaxwell: i like to dream that the constitution is actually adhered to too
196 2012-09-05 02:23:53 <jrmithdobbs> dreams are gr8 like that
197 2012-09-05 02:24:33 <jrmithdobbs> gmaxwell: ya, the whole thing reminds me of that liquidcoin thing
198 2012-09-05 02:24:40 <jrmithdobbs> heh
199 2012-09-05 02:29:52 <jgarzik> double bleh
200 2012-09-05 02:29:59 <jgarzik> ThreadSocketHandler2 is an ugly prehistoric piece of shite
201 2012-09-05 02:30:27 <amiller> i guess the next question is why is satoshidice doing an "IPO"
202 2012-09-05 02:30:34 <jgarzik> (that's the guts of the P2P select engine)
203 2012-09-05 02:30:42 <kjj_> so they can handle bigger bets
204 2012-09-05 02:31:09 <jrmithdobbs> amiller: gotta scam as much money from as many sides as possible
205 2012-09-05 02:31:13 <jrmithdobbs> amiller: that seems pretty obvious
206 2012-09-05 02:31:21 <freewil> see: facebook
207 2012-09-05 02:32:05 <freewil> people were actually willing to pay 42.50 per share
208 2012-09-05 02:32:08 <freewil> so hey why not?
209 2012-09-05 02:33:02 <jrmithdobbs> maybe i'm just a cynical asshole tho ;p
210 2012-09-05 02:53:44 <edcba> shouldn't bitcoin exchanges requires some security certification ?
211 2012-09-05 02:55:32 <copumpkin> who would require it?
212 2012-09-05 02:56:08 <gmaxwell> their customers' of course.
213 2012-09-05 02:56:17 <copumpkin> I wish :)
214 2012-09-05 02:57:30 <edcba> just make a list of approved exchanges it may be enough to pressure existing ones to get it
215 2012-09-05 02:58:56 <Optimo> that would be nice
216 2012-09-05 02:59:44 <doublec> who would be the approver and what would happen to them if they get it wrong?
217 2012-09-05 03:00:14 <jrmithdobbs> horrible idea
218 2012-09-05 03:00:19 <Optimo> mtgox shoulda been done crowdsource ideas- that ship has sailed I think
219 2012-09-05 03:00:31 <edcba> i imagined requiring doing something verifiable
220 2012-09-05 03:01:11 <jrmithdobbs> and what certification would you require? because the honest truth is that most of them are pretty much "give us enough money and you get certified"
221 2012-09-05 03:01:25 <doublec> right. like "Mcafee security", etc.
222 2012-09-05 03:01:26 <jrmithdobbs> actually, scratch that, practically all of them
223 2012-09-05 03:01:40 <edcba> yeah i know that problem
224 2012-09-05 03:01:59 <gmaxwell> scratch that, all of them, although the enough money may also involve audits that you pay to pass but also do some good in the process.
225 2012-09-05 03:02:12 <jrmithdobbs> gmaxwell: iso 27xx is ... interesting
226 2012-09-05 03:02:36 <edcba> couldn't we find some rules that would benefit security that everybody could check ?
227 2012-09-05 03:02:40 <jrmithdobbs> it's not total bullshit but it's effectiveness depends entirely on how the processes are defined which is done by the implementing party
228 2012-09-05 03:03:13 <doublec> edcba: users would ignore them anyway. They'll still go with cheapest rate, most volume, etc.
229 2012-09-05 03:03:33 <jrmithdobbs> gmaxwell: i've been interviewing too many security/compliance people lately, can you tell? (I want to kill most of them.)
230 2012-09-05 03:03:36 <edcba> cheapest and more scammy ? :)
231 2012-09-05 03:03:46 <gmaxwell> I mean, heck people happily gave non-trivial amounts of funds to BS&T.
232 2012-09-05 03:03:59 <doublec> exactly
233 2012-09-05 03:04:05 <jrmithdobbs> bs&t?
234 2012-09-05 03:04:16 <gmaxwell> pirate's black hole for bitcoin
235 2012-09-05 03:04:26 <jrmithdobbs> oh
236 2012-09-05 03:05:20 <edcba> some rule that could be verifiable : requiring having multiple servers from different providers
237 2012-09-05 03:05:48 <edcba> to store wallets
238 2012-09-05 03:06:06 <jrmithdobbs> edcba: basically, the short version is: if anyone could actually do that effectively, noone would waste their time doing it specifically for bitcoin exchanges because we're kinda talking nobel prize type stuff ;p
239 2012-09-05 03:06:12 <edcba> and possibly not on same server than frontend :)
240 2012-09-05 03:08:30 <edcba> ok i have an easier 'certification' : a blogger surveys exchanges about their security procedures :)
241 2012-09-05 03:09:30 <edcba> then do a nice table with hearts and skulls
242 2012-09-05 03:09:30 <jrmithdobbs> edcba: they get called biased and liars for pointing out the issues
243 2012-09-05 03:09:56 <jrmithdobbs> edcba: or if involving tux, threatened with legal action
244 2012-09-05 03:10:18 <jrmithdobbs> (i'm being a bit facetious, but not entirely.)
245 2012-09-05 03:11:14 <jrmithdobbs> that also gives you the age old problem of who's policing the police/etc
246 2012-09-05 03:11:31 <edcba> no i don't mean a police
247 2012-09-05 03:11:42 <edcba> i want a disclosure/transparency
248 2012-09-05 03:11:50 <jrmithdobbs> i don't literally mean police either.
249 2012-09-05 03:12:05 <edcba> just verifiable facts
250 2012-09-05 03:12:17 <jrmithdobbs> but you still need a way to verify the integrity/good faith of the reviewers
251 2012-09-05 03:12:25 <edcba> else a scammer may just be the most secure too
252 2012-09-05 03:12:48 <jrmithdobbs> edcba: i think you're starting to see how hard this is ;p
253 2012-09-05 03:13:29 <jrmithdobbs> because that's a good point. What happens if the site appears secure, but the owner disappears with the funds? Are the reviewers liable for not vetting properly and recomending the services?
254 2012-09-05 03:13:49 <jrmithdobbs> it's complicated and people are dicks ;p
255 2012-09-05 03:14:05 <edcba> at least you know it was a scammer and not a security problem :)
256 2012-09-05 03:14:43 <kjj_> why would the reviers be liable? did anything happen to Moody's or S&P when they rubber stamped all that garbage for the last couple of decades?
257 2012-09-05 03:14:54 <jrmithdobbs> edcba: and you'll still have the lolbertarians on the cesspool known as the bitcoin forums calling for the reviewer's head.
258 2012-09-05 03:15:30 <jrmithdobbs> edcba: which brings us to another fun issue, why would anyone want to be a reviewer when they'll never get anything out of it but libel and threats?
259 2012-09-05 03:16:01 <jrmithdobbs> which leaves you with incompetent reviewers, which compromises the whole system
260 2012-09-05 03:29:13 <kjj_> jgarzik: I posted my tree/list idea in dev&tech. we'll see if anyone has comments
261 2012-09-05 03:30:11 <jrmithdobbs> kjj_: mailling list, i don't think anyone actually reads that forum except luke
262 2012-09-05 03:31:07 <kjj_> Gavin does too, from time to time.
263 2012-09-05 03:59:25 <amiller> p2pool is really great
264 2012-09-05 04:01:41 <amiller> gmaxwell, i wonder if you could cut out a step, where you use your fee to purchase shares from a pool
265 2012-09-05 04:04:19 <amiller> that way the burden of 'validation' would actually be on the people who send transactions with fees
266 2012-09-05 06:33:57 <t7> is there a tool to construct a wallet from public and private key pairs?
267 2012-09-05 06:34:32 <weex> t7: you have pairs and want to import them?
268 2012-09-05 06:35:03 <weex> with bitcoind you just --importprivkey <private_key>
269 2012-09-05 06:35:08 <t7> no i have paper backups thought, but i dont know how to actually use them if the worst came to the worst
270 2012-09-05 06:35:15 <edcba> btw if someone knows how to convert MS ecdsa keypairs to openssl...
271 2012-09-05 06:35:37 <t7> MS microsoft?
272 2012-09-05 06:35:41 <edcba> yes
273 2012-09-05 06:35:48 <weex> t7: yeah you can import the privkey to bitcoind
274 2012-09-05 06:35:49 <edcba> crypto api
275 2012-09-05 06:35:53 <t7> .net or something?
276 2012-09-05 06:35:56 <edcba> same
277 2012-09-05 06:36:07 <edcba> but yes ultimately .net
278 2012-09-05 06:36:09 <weex> or you can import it to blockchain.info/wallet if you're comfortable with web-based for the amounts on them
279 2012-09-05 06:36:39 <weex> either way you only do the private key and the client can create the address from that
280 2012-09-05 06:37:04 <t7> but there are lots of public addresses for each private key
281 2012-09-05 06:37:29 <weex> oh
282 2012-09-05 06:37:40 <weex> you're talking about a deterministic wallet?
283 2012-09-05 06:37:51 <weex> like a seed and a bunch of addresses generated from that?
284 2012-09-05 06:38:09 <weex> how did you create your paper backup? what program?
285 2012-09-05 06:38:18 <t7> pen and paper
286 2012-09-05 06:38:37 <edcba> lol
287 2012-09-05 06:39:05 <weex> i'm not aware of that program :P
288 2012-09-05 06:40:56 <edcba> but yes a deterministic wallet would be better
289 2012-09-05 06:41:07 <edcba> give a password and generates a bunch of keys
290 2012-09-05 06:41:25 <edcba> instead of storing all the keys
291 2012-09-05 06:41:29 <weex> i'm not sold on the idea
292 2012-09-05 06:41:45 <t7> edcba: yeah but then 1 password = all your keys
293 2012-09-05 06:41:48 <edcba> it's a bit weaker than what we have now
294 2012-09-05 06:42:05 <t7> pretty much everything that makes bitcoin easier makes it less secure
295 2012-09-05 06:42:12 <edcba> but it's a *lot* more convenient
296 2012-09-05 06:42:15 <weex> i would use https://github.com/weex/addrgen
297 2012-09-05 06:42:23 <weex> minimal python address generator
298 2012-09-05 06:42:26 <weex> you can read the code
299 2012-09-05 06:42:34 <edcba> especially since you can go to a friend house and use your wallet from there
300 2012-09-05 06:43:46 <t7> does that load openssl ok on windows?
301 2012-09-05 06:44:00 <weex> it should
302 2012-09-05 06:44:13 <weex> i didn't have to do anything special beyond install python for it to work
303 2012-09-05 06:44:17 <weex> python from python.org i think
304 2012-09-05 06:45:03 <t7> and openssl i presume
305 2012-09-05 06:45:13 <t7> unless python comes with...
306 2012-09-05 06:45:18 <edcba> python has ctypes by default ?
307 2012-09-05 06:45:20 <t7> in which case why dont they wrap it
308 2012-09-05 06:46:20 <t7> hows the business going weex?
309 2012-09-05 06:49:05 <weex> i can't complain
310 2012-09-05 06:49:18 <weex> and who would listen if i did :P
311 2012-09-05 06:49:55 <t7> the internet
312 2012-09-05 06:50:17 <weex> the whole thing? yikes.
313 2012-09-05 06:50:29 <t7> some of it atleast
314 2012-09-05 06:52:14 <weex> here's 24 hours of mtgoxlive in time lapse: http://www.youtube.com/watch?v=WirESL6wea4
315 2012-09-05 06:57:17 <andyrossy> hey, question, a miner uses "getwork" to find out what to do, how does a miner submit work? which RPC?
316 2012-09-05 06:57:20 <t7> what am i looking at
317 2012-09-05 06:57:24 <t7> looks pretty stable
318 2012-09-05 07:08:11 <Marf> weex
319 2012-09-05 07:08:16 <Marf> now of a hole year
320 2012-09-05 07:08:17 <Marf> ;)
321 2012-09-05 07:14:15 <weex> Marf: check back in 364 days
322 2012-09-05 07:14:31 <Marf> no
323 2012-09-05 07:14:35 <Marf> you go back in time
324 2012-09-05 07:14:36 <Marf> ;D
325 2012-09-05 07:15:05 <weex> whoever has the order book for 1 year, pm me with a link
326 2012-09-05 07:15:31 <weex> also trade data and if you can throw gold and oil prices in there please do
327 2012-09-05 07:16:02 <Marf> search the forum
328 2012-09-05 07:16:09 <Marf> someone posted the data i think
329 2012-09-05 07:16:30 <Marf> he recorded all the mtgox socket streamdata
330 2012-09-05 07:18:26 <weex> ok, i'll get right on that
331 2012-09-05 07:24:24 <Eliel> sipa: this leveldb version is noticeably faster. I'll let you know how much the whole sync took when it's done but I started it around 40 minutes ago and it's already at block 156000.
332 2012-09-05 07:25:05 <Eliel> I think bdb version took around 3 hours total.
333 2012-09-05 07:25:42 <Marf> but last blocks are the slowones
334 2012-09-05 07:25:47 <Marf> not the frst 150k
335 2012-09-05 07:26:08 <Eliel> Marf: of course. it took well over an hour to get to this point with the bdb version.
336 2012-09-05 07:27:05 <Eliel> also, it was clearly IO limited at that point already. with leveldb, it still looks CPU limited.
337 2012-09-05 07:46:24 <doublec> andyrossy: also getwork
338 2012-09-05 07:47:01 <andyrossy> ty
339 2012-09-05 07:47:36 <doublec> np. submission passes the data in the first parameter of getwork.
340 2012-09-05 08:21:03 <MC-Eeepc> could bitcoin have its own anonymity network layer instead of leeching off tor?
341 2012-09-05 08:22:13 <MC-Eeepc> or atleast work out ways that bitcoin could give back to tor
342 2012-09-05 08:22:33 <MC-Eeepc> automated exit nodes that people can send coins to for it to look after itself
343 2012-09-05 08:24:01 <freewil> why reinvent the wheel
344 2012-09-05 08:26:04 <MC-Eeepc> so that the fate of one project isnt tied to another
345 2012-09-05 08:26:36 <MC-Eeepc> tors problem is almost no one has the balls to run exit nodes
346 2012-09-05 08:27:03 <MC-Eeepc> i wonder if exits could be setup anonymously and paid anon with bitcoin
347 2012-09-05 08:29:17 <ersi> MC-Eeepc: Of course. You could buy a VPS with bitcoins and set it up there with no strings attached probably
348 2012-09-05 08:30:10 <MC-Eeepc> yes, but in a more automated manner somehow
349 2012-09-05 08:30:45 <MC-Eeepc> so like people can just throw BTC at an sddress and it just happens
350 2012-09-05 08:31:03 <MC-Eeepc> sounds like a job for those bitcoin agents things that no one ever tried
351 2012-09-05 08:32:12 <ersi> I mean, it's doable
352 2012-09-05 08:32:43 <ersi> You'd probably have to negotiate some deal with a provider for them to have an address which you deposit to that either doesn't change or get replaced on some site or whatever
353 2012-09-05 08:33:27 <MC-Eeepc> na, ideally DC hosting tor exits should not know they are hosting tor exits
354 2012-09-05 08:34:05 <ersi> who said anything about saying that it's a tor node?
355 2012-09-05 08:34:16 <ersi> durrrrrrrrrrrrrrr
356 2012-09-05 08:36:06 <MC-Eeepc> maybe someone should try and get torserver.net to take BTC
357 2012-09-05 08:36:18 <MC-Eeepc> they seem to do exactly what i describe
358 2012-09-05 08:37:05 <MC-Eeepc> i bet the forum community could sustain one node between themselves, its $20 a month apparently
359 2012-09-05 08:37:32 <MC-Eeepc> would be nice to fund one exit as restitution for taking advantage of hidden services
360 2012-09-05 08:37:37 <Eliel> MC-Eeepc: no reason to bundle anonymity layer with bitcoin by default. Anyone using Bitcoin through tor is running a tor node anyway so that means tor network gets a contribution as well.
361 2012-09-05 08:38:03 <MC-Eeepc> relays are 10 a penny
362 2012-09-05 08:38:11 <MC-Eeepc> exits are where tor is struggling
363 2012-09-05 08:38:48 <Eliel> MC-Eeepc: there's a completely within tor-mode in the works for bitcoin that doesn't need the exit nodes.
364 2012-09-05 08:38:48 <MC-Eeepc> if i had a forum account i would post a straw poll about funding a community exit node
365 2012-09-05 08:39:13 <MC-Eeepc> wat
366 2012-09-05 08:39:29 <MC-Eeepc> how does it talk to the rest of the network then
367 2012-09-05 08:40:38 <Eliel> one node running in both tor-mode and public mode is enough.
368 2012-09-05 08:41:39 <Eliel> that'd be the equivalent of a bitcoin-specific tor-exit node :P
369 2012-09-05 08:41:39 <MC-Eeepc> so such a dual mode node effectively becomes a bitcoin specific exit node?
370 2012-09-05 08:42:21 <MC-Eeepc> lol well problem fixed
371 2012-09-05 08:43:50 <MC-Eeepc> i still think torserver.net should take btc with a unique BTC address for each exit they run
372 2012-09-05 08:43:54 <MC-Eeepc> so anyone can chip in
373 2012-09-05 08:46:53 <MC-Eeepc> ok so they do already take BTC
374 2012-09-05 08:47:20 <MC-Eeepc> i just assumed they didnt
375 2012-09-05 08:47:35 <MC-Eeepc> i assumed no one has ever heard of bitcoin, that seems to be changing
376 2012-09-05 08:48:08 <MC-Eeepc> Total Received\t1,498.7201101 BTC
377 2012-09-05 08:48:10 <MC-Eeepc> gaddamn
378 2012-09-05 08:56:27 <Eliel> :D
379 2012-09-05 09:39:42 <Eliel> sipa: hmm... interesting. the sync with this leveldb version looked to be going faster at first but in the end it seems it hasn't really been faster. Although, I'd best check how long it took with the bdb version to be sure my memory is working right.
380 2012-09-05 09:44:46 <Eliel> sipa: ok, scratch that, it took 5 hours with the bdb version.
381 2012-09-05 10:45:38 <MC-Eeepc> this seems to have slowed down rather a lot now
382 2012-09-05 10:49:20 <justinm001> hi
383 2012-09-05 10:49:55 <justinm001> would it be possible.. to have a feature in the normal client were it locks BTC in the wallet for a set time ?
384 2012-09-05 10:50:20 <justinm001> to prevent ppl with no will power.. like myself.. to stop spending it
385 2012-09-05 10:50:38 <justinm001> I know it sounds kind of stupid... but i am serious :P
386 2012-09-05 10:53:10 <CluckCreek> Yes, I think scripts would make that possible.
387 2012-09-05 10:53:37 <TD> lol
388 2012-09-05 10:54:29 <TD> i think it's possible to implement using nLockTime and a multi-sig transaction with a third party, but it'd be kind of annoying to set up
389 2012-09-05 10:54:55 <TD> a better approach is probably to have a friend or partner sign up for a notification service so they get informed if you spend the money
390 2012-09-05 10:55:01 <TD> then they can help you work through your problems / guilt trip you
391 2012-09-05 10:55:14 <justinm001> its not so much money problems
392 2012-09-05 10:55:17 <justinm001> more the other way
393 2012-09-05 10:55:26 <justinm001> were i got lots of btc and tend to gamble...
394 2012-09-05 10:55:31 <justinm001> sometimes it works out realy well
395 2012-09-05 10:55:38 <justinm001> othertimes its not.. thus :)
396 2012-09-05 10:56:03 <TD> ah
397 2012-09-05 10:56:17 <justinm001> anyways if anyone has any suggestions.. would be interested to here.. and how many it would cost to implement for me
398 2012-09-05 10:56:20 <TD> i still think multi-sig is the solution
399 2012-09-05 10:56:24 <justinm001> how many BTC i mean
400 2012-09-05 10:56:30 <TD> so you'd have coins that require more than one person to spend
401 2012-09-05 10:56:38 <justinm001> hmmmm
402 2012-09-05 10:56:40 <justinm001> not bad
403 2012-09-05 10:56:42 <TD> if you really need to spend the money / want to spend it on something legitimate, they'd just sign for you
404 2012-09-05 10:56:55 <TD> if you are trying to spend it on gambling they wouldn't sign and you couldn't move the money
405 2012-09-05 10:56:56 <justinm001> is that available at the moment ?
406 2012-09-05 10:57:04 <TD> sort of. it's partly implemented. there's no way to do it in the gui
407 2012-09-05 10:57:27 <justinm001> so how would it work
408 2012-09-05 10:57:37 <justinm001> i wana spend 100btc
409 2012-09-05 10:57:40 <justinm001> 1 sign it
410 2012-09-05 10:57:45 <justinm001> then how does the other person sign it ?
411 2012-09-05 10:57:46 <TD> that's what still has to be figured out. at the protocol level it's easy enough. it's more the gui that needs design
412 2012-09-05 10:58:03 <TD> probably the way it'd work is you'd click spend, fill out the payment as normal
413 2012-09-05 10:58:10 <TD> now instead of it immediately going through (being broadcast), it'd generate a file
414 2012-09-05 10:58:19 <TD> you can then get that file to the other person (your trusted partner)
415 2012-09-05 10:58:23 <TD> like, via email
416 2012-09-05 10:58:43 <TD> they open it and review what the spend does. then if they agree to it, they click "confirm", it gets signed and then they broadcast it for you
417 2012-09-05 10:58:46 <TD> there's a problem with this though
418 2012-09-05 10:58:54 <TD> which is that the second person only sees a set of addresses
419 2012-09-05 10:59:05 <justinm001> i do like the idea of that
420 2012-09-05 10:59:09 <justinm001> but might be overkill for this
421 2012-09-05 10:59:11 <TD> they don't actually know who/what will receive the money. they'd have to trust you, and that defeats the point - presumably you can't be trusted
422 2012-09-05 10:59:20 <justinm001> that would be good for things were 2 ppl the btcs
423 2012-09-05 10:59:20 <TD> (btw this is all needed for security reasons anyway)
424 2012-09-05 10:59:32 <justinm001> in my case tho
425 2012-09-05 10:59:42 <TD> it's going to be implemented no matter what to handle cases like multiple business partners who have a business account, 2-factor signing of coins, other security systems
426 2012-09-05 10:59:43 <justinm001> i am protecting myself.... from myself :P
427 2012-09-05 10:59:57 <TD> we need to solve identity in the bitcoin network first
428 2012-09-05 10:59:59 <justinm001> so some type of "lock" would be good
429 2012-09-05 11:00:12 <TD> so people can see/handle some kind of useful identifier instead of an address.
430 2012-09-05 11:00:22 <justinm001> locks the btc or transactiosn until X time passerf
431 2012-09-05 11:00:26 <TD> if you just want to tie up money for a time period, a bit of custom software can do that for you, yes
432 2012-09-05 11:00:38 <TD> you still need a trusted partner but they'd only have to get involved if you want to undo the time-locking early
433 2012-09-05 11:01:00 <TD> but i don't see why that'd help you stop gambling. once your money becomes available again, you'd still have the temptation to gamble
434 2012-09-05 11:01:13 <TD> honestly i don't think this has a technical fix. you may just need to get professional help. a gambling addiction can be quite dangerous
435 2012-09-05 11:01:25 <justinm001> hehe nah nah its nothing like that
436 2012-09-05 11:01:43 <kjj_> justinm001: just set up an offline wallet
437 2012-09-05 11:02:02 <TD> yeah if you have a way to time-lock a physical object that can work
438 2012-09-05 11:02:16 <justinm001> hmm
439 2012-09-05 11:02:21 <kjj_> doesn't have to be a time lock, it just needs to be annoying to get to
440 2012-09-05 11:02:24 <justinm001> not a bad idea at all
441 2012-09-05 11:02:41 <kjj_> generate a key pair, print the private key. send money to the address of that key
442 2012-09-05 11:02:57 <kjj_> then store the paper key in an inconvenient, but safe, place
443 2012-09-05 11:03:01 <TD> that's true
444 2012-09-05 11:03:06 <justinm001> i was just hopinh to have it more intergrated with btc wallet
445 2012-09-05 11:03:06 <kjj_> actually, several copies, several safe places
446 2012-09-05 11:03:11 <justinm001> fotr educational purposes
447 2012-09-05 11:03:23 <justinm001> so i can learn basic cofing
448 2012-09-05 11:03:27 <justinm001> codinh
449 2012-09-05 11:03:32 <justinm001> gggr
450 2012-09-05 11:03:38 <justinm001> coding
451 2012-09-05 11:04:08 <kjj_> like c++ hacking the client coding, or scripting?
452 2012-09-05 11:05:24 <justinm001> anyways long story short.... anyone want to try setup a lock system using 0.6.3 client where you put specify X BTC gets locked for Y time
453 2012-09-05 11:05:34 <justinm001> ACTION will pay BTC
454 2012-09-05 11:05:48 <justinm001> but also... i'd like to learn how you are doing it aswell
455 2012-09-05 11:06:24 <TD> how much coding do you already know?
456 2012-09-05 11:06:32 <TD> bitcoin is a complex system. it's not really a beginners project
457 2012-09-05 11:06:43 <kjj_> heh. that's for sure
458 2012-09-05 11:06:46 <justinm001> pretty good at perl, php
459 2012-09-05 11:06:47 <TD> you can do it with java and bitcoinj BUT it requires you to know what you're doing, and using presently unused features of the protocol
460 2012-09-05 11:06:57 <justinm001> if the gui was in perl.. i'd be able to do it myselgf
461 2012-09-05 11:07:06 <justinm001> but i dont no any c/c++
462 2012-09-05 11:07:08 <TD> transactions have a field called nLockTime
463 2012-09-05 11:07:25 <TD> you can set it to a timestamp or block number. the transaction won't confirm until that time has passed.
464 2012-09-05 11:07:36 <TD> however, it will stay in the memory pool and block double spends
465 2012-09-05 11:07:48 <TD> so - the money can still be "spent" but only to recipients who accept zero-confirmation transactions
466 2012-09-05 11:08:08 <freewil> TD, is that a possible DoS vector? having a really far-in-future nLockTime?
467 2012-09-05 11:08:10 <kjj_> TD: is that field currently enforced?
468 2012-09-05 11:08:17 <TD> the field should work
469 2012-09-05 11:08:52 <justinm001> hmm
470 2012-09-05 11:09:27 <justinm001> so when i recieve a trans i can set the field >?
471 2012-09-05 11:09:45 <TD> freewil: it might be. i don't remember if there are any limits on mempool size. sipa?
472 2012-09-05 11:09:58 <TD> freewil: i mean, the mempool implementation needs a lot of love in general to allow it to scale
473 2012-09-05 11:10:09 <TD> justinm001: no, when you create a transaction
474 2012-09-05 11:10:12 <kjj_> justinm001: no, you'd need to send to yourself
475 2012-09-05 11:10:13 <TD> justinm001: you'd spend the money back to yourself
476 2012-09-05 11:10:19 <justinm001> ah ok
477 2012-09-05 11:10:33 <TD> freewil: looking at the code i don't see anywhere that mempool size is limited
478 2012-09-05 11:10:34 <kjj_> or, convince whoever is sending to you to set it
479 2012-09-05 11:10:46 <freewil> is the mempool ever written to disk
480 2012-09-05 11:10:49 <justinm001> so i'd send 10btc and flag it to only confirm in 6 days or something ?
481 2012-09-05 11:10:58 <kjj_> right
482 2012-09-05 11:11:26 <justinm001> and then i recieve it but it will have 0 confrms until the 6 day ?
483 2012-09-05 11:11:36 <TD> freewil: no
484 2012-09-05 11:11:38 <justinm001> that sounds perferct
485 2012-09-05 11:11:45 <TD> justinm001: correct
486 2012-09-05 11:11:51 <sipa> TD: orphan transactions are limited, but the mempool isn't, afaics
487 2012-09-05 11:11:52 <TD> justinm001: if you want to play with this, do it all on the testnet first
488 2012-09-05 11:12:06 <TD> sipa: we should probably fix that. i agree, it seems mempool could just bloat up until the nodes run out of memory
489 2012-09-05 11:12:09 <justinm001> cool thanks :)
490 2012-09-05 11:12:44 <TD> justinm001: php/perl won't be the best languages to try this project in. it really needs to be either java or c++, i think
491 2012-09-05 11:12:51 <justinm001> any sites u guys recommend that have info about thr flag stuff ?
492 2012-09-05 11:13:23 <freewil> wiki/bitcointalk/bitcoin.stackoverflow.com
493 2012-09-05 11:13:41 <justinm001> TD ya :( also why i wanted to do it with client so i can play with c/c++
494 2012-09-05 11:13:46 <justinm001> thanks
495 2012-09-05 11:14:28 <TD> justinm001: as far as i know, nobody has ever used this field
496 2012-09-05 11:14:34 <TD> justinm001: so don't expect to find any documentation about it
497 2012-09-05 11:14:46 <TD> ... another reason why it'd be tricky if you don't have a good knowledge of java or c++
498 2012-09-05 11:15:00 <TD> justinm001: look in the bitcoin c++ for nLockTime
499 2012-09-05 11:15:40 <justinm001> so nlocktime is the field ? or something else
500 2012-09-05 11:16:04 <TD> nLockTime is the field in the CTransaction class that controls when a transaction becomes confirmable
501 2012-09-05 11:16:27 <justinm001> ah cool
502 2012-09-05 11:16:38 <justinm001> so ultimately i'd want in the gui
503 2012-09-05 11:16:38 <TD> look in main.h function CTransaction::IsFinal
504 2012-09-05 11:16:44 <TD> one step at a time
505 2012-09-05 11:16:50 <justinm001> true true :P
506 2012-09-05 11:16:50 <TD> try locking up some coins on the testnet first
507 2012-09-05 11:16:57 <TD> with a hacked up c++ codebase
508 2012-09-05 11:17:05 <TD> then look at the gui code and you should be able to "cargo cult" most of it
509 2012-09-05 11:20:13 <justinm001> ok found the nlocktime
510 2012-09-05 11:24:31 <justinm001> oh im getting it i think.. slowly..
511 2012-09-05 11:43:03 <justinm001> hmm
512 2012-09-05 11:44:33 <justinm001> so im trying to compile now.. :P
513 2012-09-05 11:44:34 <justinm001> main.h:40:2: error: invalid preprocessing directive #static
514 2012-09-05 11:44:44 <justinm001> ive installed all the stuff the doc mentions
515 2012-09-05 11:45:20 <sipa> no idea where that comes from, but "#static" is nowhere in the source code
516 2012-09-05 11:45:34 <edcba> i'd say line 40 from main.h
517 2012-09-05 11:45:49 <sipa> there's a "static" there, but no "#static"
518 2012-09-05 11:45:54 <Diablo-D3> erm
519 2012-09-05 11:45:57 <thermoman> is 0.6.3 downwards-compatible with 0.3.24? i want to upgrade 0.3.24 to latest stable release. i'm now using GUI at all but only RPC and i need to know if i have to change my code. is there a document describing incompatible modofications to the RPC protocol?
520 2012-09-05 11:45:59 <Diablo-D3> it looks like someone failed to comment it out
521 2012-09-05 11:46:22 <thermoman> s/i'm now using GUI/i'm not using GUI/
522 2012-09-05 11:46:31 <Diablo-D3> thermoman: there isnt any changes to the existing rpc api that'd effect existing code afaik
523 2012-09-05 11:46:35 <gmaxwell> my guess is that justinm001 set USE_UPNP to something weird?
524 2012-09-05 11:46:50 <sipa> gmaxwell: i think he made a typo
525 2012-09-05 11:47:08 <thermoman> Diablo-D3: i would like to have this written down somewhere on the official bitcoin.org website to read :)
526 2012-09-05 11:47:36 <Diablo-D3> thermoman: sure, but thats not going to happen.
527 2012-09-05 11:48:23 <justinm001> hmmm
528 2012-09-05 11:49:03 <Diablo-D3> thermoman: the only rpc stuff that were broke of that Im aware of were all private patch sets used by large sites
529 2012-09-05 11:49:05 <thermoman> why is there no changelog included in the tar.gz files?
530 2012-09-05 11:49:32 <Luke-Jr> thermoman: I'm not aware of any breaking changes from 0.3.24 to 0.6.3
531 2012-09-05 11:49:32 <sipa> thermoman: i can't think about any incompatible changes between 0.3.24 and 0.6.3, and we're generally very conservative with changes to RP
532 2012-09-05 11:49:43 <sipa> RPC
533 2012-09-05 11:49:49 <thermoman> do i really have to read all changelogs of all interim releases in chronological order on sourceforce.net because no changelog in included in the tar file?
534 2012-09-05 11:49:50 <gmaxwell> thermoman: because the git repository provides the complete log of changes.
535 2012-09-05 11:49:57 <Luke-Jr> though breaking changes aren't especially avoided either
536 2012-09-05 11:50:09 <thermoman> gmaxwell: ah ok
537 2012-09-05 11:50:11 <jgarzik> kjj_: can you whip up some demo code in C++ or python?
538 2012-09-05 11:51:33 <thermoman> gmaxwell: nevertheless it would be nice to include a CHANGES file in the tar file - it's common practice
539 2012-09-05 11:51:37 <justinm001> i am trying to play with the nlocktime thing
540 2012-09-05 11:51:45 <justinm001> just changed the 500000 to todays date + a few mins
541 2012-09-05 11:51:51 <justinm001> then recompiled
542 2012-09-05 11:51:58 <sipa> justinm001: why change the 500000?
543 2012-09-05 11:52:06 <Luke-Jr> thermoman: you might note that 0.7 is right around the corner though, and does have a few minor RPC changes
544 2012-09-05 11:52:06 <sipa> justinm001: that means an incompatible protocol change
545 2012-09-05 11:52:23 <thermoman> Luke-Jr: few changes as in incompatible changes?
546 2012-09-05 11:52:28 <gmaxwell> Luke-Jr: not relative to 0.3.24 as far as I can recall.
547 2012-09-05 11:52:53 <TD> justinm001: that's not really the right thing to change
548 2012-09-05 11:52:54 <Luke-Jr> gmaxwell: hmm, probably true
549 2012-09-05 11:52:57 <sipa> justinm001: you mean the 500000000, right?
550 2012-09-05 11:52:59 <TD> justinm001: you need to change the code that creates transactions
551 2012-09-05 11:53:10 <Luke-Jr> 0.3.24 is ancient :p
552 2012-09-05 11:53:18 <thermoman> that's why we want to update :)
553 2012-09-05 11:53:20 <justinm001> oh..
554 2012-09-05 11:53:34 <justinm001> i thought i could perhaps just change that to be a current time.. and send a transaction
555 2012-09-05 11:53:36 <justinm001> see what happens..
556 2012-09-05 11:53:39 <justinm001> and go from there:P
557 2012-09-05 11:53:40 <TD> no
558 2012-09-05 11:53:46 <justinm001> this is why i was happy to pay one of you ppls to do it heh
559 2012-09-05 11:53:50 <TD> you need to read the code much more carefully before making such changes
560 2012-09-05 11:54:24 <edcba> you need to understand bitcoin paper too
561 2012-09-05 11:54:25 <sipa> actually, thermoman's suggestion is a good one I think; just concatenate the list of changes of every release in a CHANGES file
562 2012-09-05 11:55:44 <sipa> justinm001: that 500000000 number is the boundary between which an nLockTime is considered a timestamp or a block number
563 2012-09-05 11:56:04 <sipa> and it only occurs in the transaction checking code, which is not what you want to change
564 2012-09-05 11:56:29 <sipa> justinm001: i'd say first experiment with building the code, make sure you can build a functioning non-modified client
565 2012-09-05 11:56:31 <justinm001> perhaps ill go through it.. abit slower :P
566 2012-09-05 11:56:39 <justinm001> ya thinking that to
567 2012-09-05 11:56:39 <sipa> and then go look at the wallet code that creates transactions
568 2012-09-05 11:58:04 <thermoman> sipa: the CHANGES file should then contain changes of *every* release, not only the changes from last release. like the debian_changes file of debian packages
569 2012-09-05 11:58:13 <Luke-Jr> git log --pretty=tformat:'- %s' --no-merges
570 2012-09-05 11:58:27 <sipa> thermoman: that's why i said "concatenation"
571 2012-09-05 11:59:13 <sipa> git commit list is certainly interesting but too detailed for most users, i think
572 2012-09-05 11:59:22 <thermoman> agreed
573 2012-09-05 12:00:14 <thermoman> are there any pitfalls or is upgrading straigth forward as in 1) stop 0.3.24 client 2) backup everything 3) start 0.6.3 ?
574 2012-09-05 12:00:27 <thermoman> 4) pray
575 2012-09-05 12:00:50 <sipa> it should work, if your database files are clean
576 2012-09-05 12:01:07 <sipa> recent versions do have stronger integrity checks
577 2012-09-05 12:01:25 <sipa> so there may be slumbering corruptions which cause a recent version to fail loading
578 2012-09-05 12:01:26 <thermoman> can i check for corruption before starting 0.6.3?
579 2012-09-05 12:01:34 <sipa> not really
580 2012-09-05 12:01:45 <sipa> but make a backup, and you're safe
581 2012-09-05 12:01:51 <thermoman> should i do -rescan with 0.3.24 first?
582 2012-09-05 12:02:03 <sipa> no
583 2012-09-05 12:02:17 <sipa> -rescan has almost never been needed since 0.3.21 as it's done automatically at startup when necessary
584 2012-09-05 12:02:26 <Luke-Jr> sipa: err, not quite
585 2012-09-05 12:02:50 <sipa> the only case where it's needed is if you manually modify the wallet.dat file with bdb tools
586 2012-09-05 12:02:54 <Luke-Jr> sipa: in theory maybe, but in practice I've seen transactions get missed
587 2012-09-05 12:03:07 <sipa> Luke-Jr: yes, that was a GUI problem that got solved by restarting without -rescan too
588 2012-09-05 12:03:07 <thermoman> me too with 0.3.24
589 2012-09-05 12:03:28 <ErnestoJuarell> Is there any good open source stuff for getting blockchain stats?
590 2012-09-05 12:03:42 <thermoman> i hate berkeley DB files ... not only because they perform so badly
591 2012-09-05 12:03:52 <sipa> thermoman: we're moving away from BDB
592 2012-09-05 12:03:58 <sipa> and you're not alone :)
593 2012-09-05 12:04:03 <thermoman> haha, nice
594 2012-09-05 12:04:06 <thermoman> move to what?
595 2012-09-05 12:04:24 <Luke-Jr> LevelDB for indexs, and some custom format for the wallet
596 2012-09-05 12:04:25 <sipa> LevelDB for blockchain indexes, something custom for wallets probably
597 2012-09-05 12:04:30 <Luke-Jr> :p
598 2012-09-05 12:04:41 <thermoman> ;)
599 2012-09-05 12:05:01 <thermoman> but not in 0.7 i guess
600 2012-09-05 12:05:07 <Luke-Jr> 0.8
601 2012-09-05 12:05:07 <sipa> 0.8 probably
602 2012-09-05 12:05:17 <gmaxwell> 0.8 (AM I TOO LATE AGAIN GUYS?)
603 2012-09-05 12:05:17 <TD> consider the bitcoinj protocol buffer format?
604 2012-09-05 12:05:23 <Luke-Jr> lol
605 2012-09-05 12:05:54 <sipa> TD: i prefer something append-only, which is probably hard with protobuf, no?
606 2012-09-05 12:06:08 <Luke-Jr> sipa: should be hard
607 2012-09-05 12:06:14 <Luke-Jr> shouldn't*
608 2012-09-05 12:06:14 <TD> right. it describes the current state of the wallet, not a log of changes
609 2012-09-05 12:06:15 <thermoman> starting the 0.3.24 daemon takes ages ... when checking file system stats i see reads of 3 MByte/s ...
610 2012-09-05 12:06:21 <TD> (in the format we used)
611 2012-09-05 12:06:39 <Luke-Jr> TD: does the format you used support HD wallets?
612 2012-09-05 12:06:50 <thermoman> using bitcoin on an external usb hdd is a pain in the ass and not usable ... BDB is so damn slow
613 2012-09-05 12:06:52 <sipa> nothing supports those yet, Luke-Jr
614 2012-09-05 12:06:57 <TD> not yet. jim is working on adding encrypted keys and is nearly done, i just need to do another review and merge of his code
615 2012-09-05 12:07:22 <TD> it is not a perfect fit for bitcoind because it was designed for SPV clients, so transactions have "pools" and stuff like that
616 2012-09-05 12:07:41 <TD> but it could maybe be evolved in a compatible direction. that said, if you want a log-structured format ...
617 2012-09-05 12:07:50 <sipa> yeah, i'm more of a fan of a nice standard interchange format for wallets
618 2012-09-05 12:07:53 <TD> what's the rationale for that btw? you'd have to replay the log on startup?
619 2012-09-05 12:08:15 <sipa> not needing to rewrite the entire wallet file for every change, little chance for corruption, ...
620 2012-09-05 12:08:18 <TD> yes, well, it could work quite well as an interchange format
621 2012-09-05 12:08:45 <TD> yeah, but at the cost of huge wallets taking ages to replay into memory
622 2012-09-05 12:08:57 <Luke-Jr> TD: they have to be loaded anyway.
623 2012-09-05 12:08:58 <gmaxwell> and replaying should be little more expensive than a non-append-only-format because for the most part the wallet content doesn't change.
624 2012-09-05 12:09:29 <TD> i'm thinking about high-traffic websites here like satoshidice or whatever. where you have potentially high tx throughput on a single wallet
625 2012-09-05 12:09:44 <sipa> well all that happens is adding new transactions to it
626 2012-09-05 12:09:51 <sipa> and marking some outputs spent
627 2012-09-05 12:09:56 <TD> yes
628 2012-09-05 12:10:57 <Luke-Jr> maybe a separate wallet (keys only) and metadata file (comments/txns) would make sense
629 2012-09-05 12:11:03 <TD> we went for a protobuf format in bitcoinj because we need ~instant startup (people don't really tolerate loading screens on mobile apps). the format reflects that requirement - for instance it allows you to store the public parts of keys alongside the private parts. it's obviously redundant but means there's no need to do the point multiplies at startup
630 2012-09-05 12:11:23 <thermoman> last question: 0.3.24 and testnet - is this buggy? because blockexplorer says current block count is about 23150 but my 0.3.24 testnet client is at 46722 blocks
631 2012-09-05 12:11:36 <sipa> thermoman: 0.3.24 was testnet1 or testnet2
632 2012-09-05 12:11:40 <sipa> 0.6.3 uses testnet3
633 2012-09-05 12:11:42 <Luke-Jr> thermoman: testnet has been restarted a few times, the current version is only supported by 0.7
634 2012-09-05 12:11:54 <gmaxwell> sipa: no, 0.7.0 uses testnet3. 0.6.3 uses testnet2
635 2012-09-05 12:12:07 <gmaxwell> blockexplorer is on testnet3.
636 2012-09-05 12:12:12 <sipa> gmaxwell: oh really? :o
637 2012-09-05 12:12:21 <sipa> has 0.6.0 been that long ago...
638 2012-09-05 12:12:21 <thermoman> so i can't exchange testnet bitcoins between 0.3.24 and 0.6.3, right?
639 2012-09-05 12:12:40 <gmaxwell> I think 0.3.24 and 0.6.3 are both testnet2.
640 2012-09-05 12:12:46 <sipa> indeed
641 2012-09-05 12:12:47 <sipa> just checked
642 2012-09-05 12:12:48 <gmaxwell> though testnet2 is mostly broken.
643 2012-09-05 12:12:57 <sipa> right, 0.6.3 has testnet2.2
644 2012-09-05 12:14:28 <justinm001> ok getting over this now..
645 2012-09-05 12:14:45 <justinm001> ACTION will pay someone make btc client so i can use nlocktime for only certain payments
646 2012-09-05 12:16:12 <sipa> Luke-Jr: meh about splitting keys and metadata; sounds like a hell to keep synchronized
647 2012-09-05 12:16:34 <Luke-Jr> sipa: maybe, I didn't give it much thought
648 2012-09-05 12:17:32 <TD> backups would be more annoying if nothing else
649 2012-09-05 12:17:55 <justinm001> surely one of you wants some beeteec's
650 2012-09-05 12:18:00 <TD> my code just does a serialize to protobuf, save to temp file in same directory, rename over previous
651 2012-09-05 12:18:12 <TD> justinm001: heh, sorry dude, maybe ask on the forum? most people have their own project
652 2012-09-05 12:18:22 <justinm001> ok thanks anyways heaps for your help :)
653 2012-09-05 12:28:22 <thermoman> what do you mean when you say "testnet was reset"? am i right that different daemon versions use different testnets and all these testnets exist concurrently?
654 2012-09-05 12:29:14 <jgarzik> thermoman: new genesis block, new pchMessageStart ("magic number" identifying the network, and required for all testnet P2P messages to be understood)
655 2012-09-05 12:29:57 <jgarzik> older testnet block chain transactions are not recognized as valid
656 2012-09-05 12:31:14 <jgarzik> TD: where is the *.proto file? (I'm unfamiliar with the bitcoinj source dir)
657 2012-09-05 12:32:04 <jgarzik> is this it? http://code.google.com/p/bitcoinj/source/browse/core/src/bitcoin.proto
658 2012-09-05 12:32:14 <jgarzik> trying to look at the wallet file format
659 2012-09-05 12:40:44 <MC-Eeepc> bonus round
660 2012-09-05 12:40:57 <MC-Eeepc> why did satoshi go with obscure ecdsa
661 2012-09-05 12:41:12 <MC-Eeepc> because its Q resistant?
662 2012-09-05 12:41:15 <Luke-Jr> MC-Eeepc: small signatures
663 2012-09-05 12:41:23 <sipa> and it is not quantum resistant
664 2012-09-05 12:41:42 <sipa> MC-Eeepc: how many blocks done?
665 2012-09-05 12:41:45 <MC-Eeepc> so it was just the sze thing
666 2012-09-05 12:41:56 <sipa> i believe so; TD corresponded with him about that
667 2012-09-05 12:42:21 <Luke-Jr> size is pretty important right now
668 2012-09-05 12:42:31 <MC-Eeepc> sipa 165k
669 2012-09-05 12:42:39 <sipa> is that good?
670 2012-09-05 12:42:57 <MC-Eeepc> i think so
671 2012-09-05 12:43:04 <MC-Eeepc> its stopping and starting all the time
672 2012-09-05 12:43:27 <Eliel> sipa: strange. this leveldb version appears to be slower in processing the later blocks.
673 2012-09-05 12:43:51 <sipa> Eliel: what are you syncing from?
674 2012-09-05 12:43:52 <jgarzik> later blocks are bigger, with more transactions to spend
675 2012-09-05 12:44:02 <MC-Eeepc> im getting the same vibe as Eliel tbh
676 2012-09-05 12:44:03 <sipa> if you happen to be connected to a slow peer, IBD will be slow
677 2012-09-05 12:44:05 <Eliel> sipa: my local bitcoin 0.6.3 node.
678 2012-09-05 12:44:09 <sipa> ah
679 2012-09-05 12:44:16 <Eliel> as with the bdb version
680 2012-09-05 12:44:21 <sipa> really?
681 2012-09-05 12:44:42 <sipa> that's unexpected
682 2012-09-05 12:45:12 <MC-Eeepc> maybe windows is handling the writes in a stupid way
683 2012-09-05 12:45:19 <Eliel> regardless, the bottleneck seems to be CPU somehow
684 2012-09-05 12:45:21 <Eliel> not IO
685 2012-09-05 12:45:21 <Luke-Jr> maybe the reads are interfering?
686 2012-09-05 12:45:24 <Luke-Jr> oh
687 2012-09-05 12:45:46 <Eliel> it's been nearly fully utilizing one of the cores
688 2012-09-05 12:45:57 <Eliel> CPU time measure is at 5:40 at the moment.
689 2012-09-05 12:46:00 <sipa> that's normal, after the last checkpoint
690 2012-09-05 12:46:12 <Eliel> it's not reached the last checkpoint yet
691 2012-09-05 12:46:20 <sipa> ah, good!
692 2012-09-05 12:46:36 <Eliel> 186923 is the block it was processing a moment ago
693 2012-09-05 12:46:37 <MC-Eeepc> my cpu goes between 30% and 0
694 2012-09-05 12:47:34 <Eliel> the I/O write bytes and I/O read bytes are at 26.4 GB and 23.5 GB respectively.
695 2012-09-05 12:47:45 <TD> sipa: did you bring across my change to ProcessBlock to print tx/sec?
696 2012-09-05 12:47:46 <TD> jgarzik: yes
697 2012-09-05 12:47:51 <sipa> TD: oh, no
698 2012-09-05 12:47:56 <sipa> Eliel: well, feel free to benchmark BDB as well
699 2012-09-05 12:48:05 <sipa> Eliel: s/BDB/git head/
700 2012-09-05 12:48:06 <TD> sipa: i found it useful (in the later stages of the chain)
701 2012-09-05 12:48:19 <TD> MC-Eeepc: yes it was just a size thing
702 2012-09-05 12:48:34 <TD> MC-Eeepc: though it turned out to be a prescient choice for other reasons. ECC is basically the "next gen" crypto system
703 2012-09-05 12:48:37 <TD> or current gen, i guess by now
704 2012-09-05 12:48:49 <Eliel> I guess I can't just switch this version to use BDB with a command line parameter, can I? :)
705 2012-09-05 12:48:56 <sipa> Eliel: no
706 2012-09-05 12:49:20 <sipa> i can do a build of git head, or of ultraprune without BDB if you want, though
707 2012-09-05 12:49:29 <sipa> without leveldb, i mean
708 2012-09-05 12:50:00 <MC-Eeepc> it seems like it can still chew thru 10blox/sec or so when is not stalled
709 2012-09-05 12:50:12 <MC-Eeepc> not smooth at all
710 2012-09-05 12:50:23 <sipa> define stalled?
711 2012-09-05 12:50:28 <TD> this is your eeepc right?
712 2012-09-05 12:50:39 <TD> if you aren't on a multi-core machine leveldb may not be any faster or could even be slower
713 2012-09-05 12:50:43 <MC-Eeepc> not dling blox, 0% cpu
714 2012-09-05 12:50:55 <sipa> MC-Eeepc: how long do those stalls last?
715 2012-09-05 12:51:25 <MC-Eeepc> maybe 30 seconds
716 2012-09-05 12:51:44 <sipa> that feels like the OS blocking to flush to your USB stick
717 2012-09-05 12:52:30 <MC-Eeepc> yes
718 2012-09-05 12:52:42 <sipa> well, very little to do about that
719 2012-09-05 12:53:10 <sipa> Eliel: what hardware are you on?
720 2012-09-05 12:54:02 <MC-Eeepc> 165k blox 3:45 cpu time 11.5gb i/o write
721 2012-09-05 12:56:40 <Eliel> sipa: AMD Athlon 64 X2 4200+ for processor and Samsung HD501LJ for HDD
722 2012-09-05 12:56:46 <MC-Eeepc> does hyperthreading count
723 2012-09-05 12:56:51 <Eliel> also, Windows XP
724 2012-09-05 12:58:00 <Eliel> I have a dual boot ubuntu in this system (I think, I haven't booted it up in a long while) so I could test the linux version with the same hardware to see what kind of a difference there is too.
725 2012-09-05 12:58:43 <MC-Eeepc> there was something greg did a whle back which made it lots faster on linux but not windows
726 2012-09-05 12:59:20 <sipa> that was mostly because it was buggy on linux before that
727 2012-09-05 12:59:32 <MC-Eeepc> i wonder if it would be optimised for the way certain common filesystems work
728 2012-09-05 12:59:41 <sipa> leveldb?
729 2012-09-05 12:59:56 <MC-Eeepc> dunno, suppose
730 2012-09-05 13:00:03 <MC-Eeepc> in whatever way
731 2012-09-05 13:00:05 <Eliel> I'm kind of curious though, since it looks like it's CPU bound, not IO bound.
732 2012-09-05 13:00:40 <Eliel> I still have the BDB version from earlier and it did a complete sync in 5 hours.
733 2012-09-05 13:00:51 <sipa> when was earlier?
734 2012-09-05 13:00:55 <Eliel> a week ago
735 2012-09-05 13:01:03 <sipa> and how long is it been running now?
736 2012-09-05 13:01:15 <Eliel> 6 hours 15 minutes
737 2012-09-05 13:01:20 <sipa> very strange
738 2012-09-05 13:01:22 <MC-Eeepc> wat
739 2012-09-05 13:02:03 <sipa> Eliel: can you try with a larger chache? -dbcache=150
740 2012-09-05 13:02:08 <Eliel> BDB version was clearly IO bound.
741 2012-09-05 13:02:10 <Eliel> I'll try
742 2012-09-05 13:02:34 <sipa> i think the cache is very important for leveldb, but since my OS cache is large enough, i don't notice anything here
743 2012-09-05 13:03:28 <Eliel> system cache is listed as around 2GB in task manager
744 2012-09-05 13:04:03 <Eliel> I did not touch the settings, I just restarted it. It's blazing through the blocks now.
745 2012-09-05 13:04:22 <MC-Eeepc> ok im gonna pull the stick again and see what happens
746 2012-09-05 13:04:47 <Eliel> sipa: I think there's some kind of a leak somewhere perhaps.
747 2012-09-05 13:04:53 <MC-Eeepc> fuck this usb is hot
748 2012-09-05 13:05:03 <sipa> Eliel: why?
749 2012-09-05 13:05:13 <Eliel> it's doing 5 blocks per second after restart. I didn't try the dbcache setting yet
750 2012-09-05 13:06:05 <MC-Eeepc> ok stick pulled 1 min ago and bitcoin hasnt crashed yet lol
751 2012-09-05 13:06:21 <Eliel> before restart it was doing maybe a block every few seconds or so
752 2012-09-05 13:07:40 <Eliel> I think there's some subtask that takes more and more CPU the longer it syncs.
753 2012-09-05 13:08:20 <sipa> later blocks are just a lot larger
754 2012-09-05 13:08:27 <sipa> and the database is larger too at that point
755 2012-09-05 13:09:02 <sipa> oh, you just quit and restarted?
756 2012-09-05 13:09:22 <Eliel> yes
757 2012-09-05 13:09:31 <Eliel> an it sped up by a factor of 20x or more
758 2012-09-05 13:09:54 <sipa> weird
759 2012-09-05 13:13:36 <MC-Eeepc> ok its rescanning that means it knows it crashed right
760 2012-09-05 13:13:43 <MC-Eeepc> i think this is fucked up now
761 2012-09-05 13:13:55 <MC-Eeepc> it said importing blocks from prev database then crashed
762 2012-09-05 13:14:03 <sipa> hmm, ok
763 2012-09-05 13:14:10 <sipa> anything in debug.log?
764 2012-09-05 13:15:49 <MC-Eeepc> Loading block index...
765 2012-09-05 13:15:49 <MC-Eeepc> Opening LevelDB in E:\\blockchain\\blktree
766 2012-09-05 13:15:50 <MC-Eeepc> LoadBlockIndex(): last block file = 7
767 2012-09-05 13:15:52 <MC-Eeepc> LoadBlockIndex(): last block file: CBlockFileInfo(blocks=2964, size=63427819, heights=163725..166688, time=2012-01-25..2012-02-13)
768 2012-09-05 13:15:55 <MC-Eeepc> LoadBlockIndex(): hashBestChain=0000000000000823caf9 height=166669 date=02/13/12 18:50:55
769 2012-09-05 13:15:57 <MC-Eeepc> Verifying last 2500 blocks at level 1
770 2012-09-05 13:16:00 <MC-Eeepc> block index 41902ms
771 2012-09-05 13:16:01 <MC-Eeepc> Loading wallet...
772 2012-09-05 13:16:03 <MC-Eeepc> dbenv.open LogDir=E:\\blockchain\\database ErrorFile=E:\\blockchain\\db.log
773 2012-09-05 13:16:05 <MC-Eeepc> nFileVersion = 70000
774 2012-09-05 13:16:07 <MC-Eeepc> wallet 2355ms
775 2012-09-05 13:16:08 <BlueMatt> please dont paste that much
776 2012-09-05 13:16:09 <MC-Eeepc> Rescanning last 2244 blocks (from block 164425)...
777 2012-09-05 13:16:13 <MC-Eeepc> rescan 27503ms
778 2012-09-05 13:19:15 <sipa> MC-Eeepc: anything over 3 lines, use a paste site
779 2012-09-05 13:21:32 <sipa> can you paste it again?
780 2012-09-05 13:23:23 <MC-Eeepc> assertion failed
781 2012-09-05 13:23:47 <MC-Eeepc> src/main.cpp
782 2012-09-05 13:23:58 <MC-Eeepc> line 1177
783 2012-09-05 13:24:10 <MC-Eeepc> i dont even
784 2012-09-05 13:24:29 <MC-Eeepc> http://pastebin.com/PmixiKaY
785 2012-09-05 13:26:30 <MC-Eeepc> ok i think i trashed the blockchain that time
786 2012-09-05 13:31:31 <Eliel> sipa: another strangeness found. now that it hit the last checkpoint, it's still syncing but not showing a progress meter this time. the block count keeps increasing if I use the getinfo command
787 2012-09-05 13:32:09 <sipa> MC-Eeepc: which assertion?
788 2012-09-05 13:32:20 <sipa> Eliel: hmm ok, thanks; i'll look into that
789 2012-09-05 13:55:32 <gavinandresen> wumpus sipa jgarzik gmaxwell : any pull requests or issues that need attention before tagging a 0.7rc2 ?
790 2012-09-05 13:56:04 <gavinandresen> Only worrisome thing on my list is the valgrind errors on shutdown
791 2012-09-05 13:57:08 <jgarzik> gavinandresen: go for it
792 2012-09-05 13:57:40 <gmaxwell> gavinandresen: I'd rather cut the RC sooner rather than later, even with the valgrind errors on shutdown.
793 2012-09-05 13:59:10 <gavinandresen> ACK. I'll bump version numbers, tag and push
794 2012-09-05 14:00:18 <gavinandresen> ummm... right after I edit the release notes a bit
795 2012-09-05 14:01:17 <sipa> let's see
796 2012-09-05 14:02:25 <jgarzik> gavinandresen: uh oh :)
797 2012-09-05 14:02:29 <sipa> jgarzik: no need for 1786? not sure how serious it is
798 2012-09-05 14:03:07 <gavinandresen> jgarzik: I'm just adding an Incompatible Changes section at the top with the getmemorypool change, so people are slightly less likely to be surprised
799 2012-09-05 14:03:18 <jgarzik> sipa: it's needed, but it looks like the first arg should be the only conditional, not the entire call
800 2012-09-05 14:03:47 <jgarzik> sipa: Windows wants !NULL, contra what some cygwin code implies
801 2012-09-05 14:03:48 <sipa> doc/Tor.txt still mentions -dnsseed, which was removed
802 2012-09-05 14:04:15 <jgarzik> I'll respin today, but don't wait on rc2 for that
803 2012-09-05 14:04:53 <jgarzik> gavinandresen: note some other RPCs were removed as well (these are mentioned in the JSON-RPC section)
804 2012-09-05 14:05:42 <gavinandresen> sipa: I'll fix doc/Tor.txt
805 2012-09-05 14:05:47 <jgarzik> ACTION -> Mexican food
806 2012-09-05 14:06:50 <sipa> why is the new checkpoint mentioned under RPC changes?
807 2012-09-05 14:07:40 <gavinandresen> sipa: ACK, moving that to core section
808 2012-09-05 14:14:00 <Luke-Jr> gavinandresen: is there a formal announcement for CVE-2012-4682/4683 yet?
809 2012-09-05 14:15:18 <gavinandresen> Luke-Jr: no
810 2012-09-05 14:16:52 <Eliel> these CVEs are popping up with somewhat worrying frequency :P
811 2012-09-05 14:17:20 <Luke-Jr> Eliel: meh, as long as the easy-major ones are somewhat rare it's ok :p
812 2012-09-05 14:17:40 <Luke-Jr> on the bright side, they're not *new* vulns, just old ones being discovered
813 2012-09-05 14:18:20 <Eliel> so, how severe is this one?
814 2012-09-05 14:18:21 <gavinandresen> i don't think these latest ones qualify as 'major'
815 2012-09-05 14:18:51 <gavinandresen> run-of-the-mill denial-of-service attack... send a node a few gigabytes of data and you might be able to get it to fill up memory and crash is the most serious
816 2012-09-05 14:19:08 <Luke-Jr> ACTION suspects 4682/4683 are just "not worth the time" rather than intentionally withheld for security
817 2012-09-05 14:19:09 <gavinandresen> (so it crashes and restarts... meh)
818 2012-09-05 14:19:33 <freewil> yeah as far as bitcoin is concerned i would never qualify a DOS as major
819 2012-09-05 14:20:06 <sipa> gavinandresen: i feel we should mention the BDB upgrade problem
820 2012-09-05 14:20:11 <Luke-Jr> freewil: well, if someone exploits it to shut a merchant down, it might be a pain
821 2012-09-05 14:20:24 <gavinandresen> sipa: there's a BDB upgrade problem? did we switch versions somewhere?
822 2012-09-05 14:20:37 <freewil> Luke-Jr, yes but relatively minor issue
823 2012-09-05 14:20:58 <sipa> gavinandresen: apparently, -detachdb is even necessary when just the minor version of BDB changed
824 2012-09-05 14:21:11 <Luke-Jr> ACTION didn't know we changed minor version even <.<
825 2012-09-05 14:21:29 <sipa> gavinandresen: and for example official linux builds and the PPA use different ones
826 2012-09-05 14:22:19 <sipa> so we saw a lot of people who used the PPA and wanted to try RC builds, and got errors
827 2012-09-05 14:22:30 <Luke-Jr> git tells me Windows binaries use the same exact bdb - so if it's occuring there, something else is going on
828 2012-09-05 14:23:53 <TD> ppa
829 2012-09-05 14:24:55 <gavinandresen> TD: ppa is personal package anarchy, a debian thing....
830 2012-09-05 14:25:18 <TD> we should make the code exit(1) at startup if it detects that it was built by debian :)
831 2012-09-05 14:26:19 <TD> or more realistically have the binary check its own signature and print a warning at startup about not being supported if the signature doesn't match
832 2012-09-05 14:26:21 <gavinandresen> sipa: I'll add a "how to upgrade" section that recommends shutdown first on every platform. And run -detachdb, shutdown on Linux
833 2012-09-05 14:26:32 <Luke-Jr> lol @ anarchy
834 2012-09-05 14:26:45 <Luke-Jr> and afaik PPA only works on Ubuntu :p
835 2012-09-05 14:26:59 <TD> debian/ubuntu/whatever
836 2012-09-05 14:27:08 <gavinandresen> yeah, they all look the same to me
837 2012-09-05 14:27:20 <Luke-Jr> TD: you're anti-packaging?
838 2012-09-05 14:27:28 <TD> yes
839 2012-09-05 14:27:39 <Luke-Jr> explains a bit.
840 2012-09-05 14:28:37 <BlueMatt> TD: the ppa is packaged by me
841 2012-09-05 14:28:58 <BlueMatt> TD: though I agree that we should complain if we find we are running the "official" debian/ubuntu packages
842 2012-09-05 14:29:25 <Luke-Jr> so BlueMatt is just anti-distro :P
843 2012-09-05 14:30:01 <BlueMatt> depends on the specific one and the specific program
844 2012-09-05 14:31:10 <Luke-Jr> ACTION hopes Debian pain is over once they get over 0.3
845 2012-09-05 14:35:06 <gavinandresen> Can y'all proofread this How to Upgrade for me: https://gist.github.com/3639513
846 2012-09-05 14:35:43 <BlueMatt> gavinandresen: ACK
847 2012-09-05 14:35:49 <Luke-Jr> gavinandresen: looks ok to me
848 2012-09-05 14:36:01 <BlueMatt> gavinandresen: or maybe s/How to Upgrade/How to run this RC Version/
849 2012-09-05 14:36:16 <BlueMatt> if you are on the ppa "How to upgrade" is just apt-get update && apt-get upgrade
850 2012-09-05 14:37:13 <gavinandresen> BlueMatt: ppa users will know that already, though, right?
851 2012-09-05 14:38:14 <BlueMatt> they should, but maybe "Remember: when switching from a bitcoin compiled with one version of BDB to one compiled with another (this RC is built with a different version than the PPA), you must..."
852 2012-09-05 14:40:04 <gavinandresen> BlueMatt: I'll put something like that in the release announcement, in the "A PPA is available at...." secion
853 2012-09-05 14:40:20 <BlueMatt> sure
854 2012-09-05 14:41:54 <gavinandresen> all righty, pushed tag v0.7.0rc2
855 2012-09-05 14:43:04 <gavinandresen> My gitian-building machine is still broken (dying hard disk, should have a replacement in a few days) so I can't gitian build right now
856 2012-09-05 14:43:48 <Joric> puik's testing tx comments on me https://bitcointalk.org/index.php?topic=40264.msg1159289#msg1159289
857 2012-09-05 14:43:57 <Joric> are zero output transactions still legit
858 2012-09-05 14:44:01 <Joric> i hate this thing
859 2012-09-05 14:45:05 <gavinandresen> zero-value outputs are non-standard in the 0.7 release
860 2012-09-05 14:45:22 <gavinandresen> ... to discourage using them for stuff like messages
861 2012-09-05 14:45:58 <Joric> he said 'Should be a while before miners upgrade.'
862 2012-09-05 14:46:27 <gmaxwell> FWIW, I had no idea he was planning on doing that before suggesting it. It's insane in any case??? an unspendable pubkey zero value output is the worst case scalablity wise.
863 2012-09-05 14:47:36 <gmaxwell> _all_ embedded message are bad, but an OP_DROP in a scriptsig is at least instantly prunable.
864 2012-09-05 14:48:51 <gmaxwell> This also highlights that fee rules are misaligned because they don't penalize txout set bloat relative to e.g. txn size created by consuming junk inputs.
865 2012-09-05 14:50:31 <gavinandresen> gmaxwell: making it non-standard is a bigger stick than a fee rule change... but I agree, transaction priority ought to take into account number of txin versus txout
866 2012-09-05 14:51:02 <gavinandresen> so... how do we figure out what the 'right' formula is ?
867 2012-09-05 14:51:21 <gavinandresen> or do we just give miners a knob they can twiddle ?