1 2012-05-19 00:01:24 <luke-jr> jgarzik: I don't have that - line&
  2 2012-05-19 00:03:07 <jgarzik> bleh
  3 2012-05-19 00:03:15 <jgarzik> strace shows fdatasync every other system call
  4 2012-05-19 00:04:22 <gmaxwell> gettimeofday();fdatasync(); .. you've described bitcoin!
  5 2012-05-19 00:04:58 <gmaxwell> causes fantastic write inflation in the FS too.
  6 2012-05-19 00:05:43 <luke-jr> jgarzik: -    fDetachDB = GetBoolArg("-detachdb", false); <-- not in master
  7 2012-05-19 00:06:08 <jgarzik> luke-jr: it was approximate
  8 2012-05-19 00:06:26 <jgarzik> luke-jr: look for fDetachDb, and call bitdb.SetDetach() instead
  9 2012-05-19 00:06:42 <jgarzik> gmaxwell: that's what I'm hitting
 10 2012-05-19 00:06:57 <jgarzik> gmaxwell: we use DB_TXN_NOSYNC, so I really want to know where the fdatasync() is coming from
 11 2012-05-19 00:07:44 <luke-jr> jgarzik: and for reading?
 12 2012-05-19 00:08:31 <luke-jr> actually, maybe not needed
 13 2012-05-19 00:09:15 <jgarzik> luke-jr: reading would require creating CDBEnv::GetDetach()
 14 2012-05-19 00:09:36 <sipa> jgarzik: closedb does some bdb cleanup calls
 15 2012-05-19 00:09:49 <sipa> maybe those cause the fdatasync
 16 2012-05-19 00:09:52 <gmaxwell> aww, no DB_TXN_BULK in whatever version of BDB I have.
 17 2012-05-19 00:10:00 <luke-jr> bleh, it does need to read
 18 2012-05-19 00:10:45 <sipa> luke-jr: what's the problem?
 19 2012-05-19 00:11:12 <luke-jr> sipa: jgarzik's dbenv pullreq trilogy doesn't build
 20 2012-05-19 00:11:55 <sipa> ok, and where do you need to read fDetach?
 21 2012-05-19 00:11:56 <luke-jr> jgarzik: this fixes it: http://codepad.org/nNatwauU
 22 2012-05-19 00:12:03 <jgarzik> sipa: we don't forcibly CloseDb for each ProcessBlock(), do we?
 23 2012-05-19 00:12:15 <luke-jr> sipa: src/qt/optionsmodel.cpp
 24 2012-05-19 00:12:30 <jgarzik> luke-jr: thanks
 25 2012-05-19 00:12:46 <jgarzik> gmaxwell: db 5.x
 26 2012-05-19 00:13:01 <jgarzik> maybe we should be setting DB_TXN_NOSYNC on the environment, rather than each txn
 27 2012-05-19 00:13:30 <jgarzik> also found a comment from Oracle that indicates repeated put's of the same record can trigger unintentional fdatasync's
 28 2012-05-19 00:13:55 <jgarzik> gmaxwell: Fedora 16 here
 29 2012-05-19 00:14:00 <gmaxwell> 17 here.
 30 2012-05-19 00:14:08 <gmaxwell> But linking to 4.8.
 31 2012-05-19 00:14:50 <gmaxwell> what the heck is the difference between  DB_TXN_WRITE_NOSYNC  and  DB_TXN_NOSYNC  ?
 32 2012-05-19 00:16:26 <jgarzik> gmaxwell: fwrite(3) versus write(2)
 33 2012-05-19 00:16:26 <sipa> jgarzik: only a txn_checkpoint call between processblocks
 34 2012-05-19 00:16:33 <jgarzik> gmaxwell: nosync, it's just buffered
 35 2012-05-19 00:16:46 <jgarzik> gmaxwell: write_nosync it's written to the fd, but not sync'd
 36 2012-05-19 00:17:06 <jgarzik> in theory
 37 2012-05-19 00:19:14 <luke-jr> jgarzik: does blockindex break backward compat?
 38 2012-05-19 00:19:24 <luke-jr> ie, do I need to redownload the blockchain if I downgrade?
 39 2012-05-19 00:19:28 <gmaxwell> seems about equally fats on my lapop.
 40 2012-05-19 00:19:32 <gmaxwell> fast
 41 2012-05-19 00:27:51 <jgarzik> wooo!!!!
 42 2012-05-19 00:28:11 <jgarzik> adding DB_TXN_NOSYNC to dbenv flags killed all fdatasyncs
 43 2012-05-19 00:28:44 <jgarzik> -loadblock is easily 100x faster now
 44 2012-05-19 00:28:57 <jgarzik> (of course, this is storage and filesystem dependent)
 45 2012-05-19 00:30:02 <jgarzik> luke-jr: well
 46 2012-05-19 00:30:12 <jgarzik> luke-jr: new db is in blockchain/
 47 2012-05-19 00:30:29 <jgarzik> luke-jr: if you don't touch the old block chain files, they will still be there when you run an older version
 48 2012-05-19 00:30:58 <sipa> jgarzik: is the dbenv also in blockchain/ ?
 49 2012-05-19 00:31:14 <sipa> or is it shared with wallet's dbenv?
 50 2012-05-19 00:31:27 <jgarzik> sipa: shared with wallet
 51 2012-05-19 00:31:52 <jgarzik> sipa: I didn't bother a split, since you seemed to be working on a !BDB wallet solution anyway
 52 2012-05-19 00:32:00 <sipa> yeah
 53 2012-05-19 00:32:46 <luke-jr> so "should be safe as long as it shuts down cleanly"?
 54 2012-05-19 00:33:48 <gmaxwell> jgarzik: woot!
 55 2012-05-19 00:33:50 <jgarzik> luke-jr: old bitcoin only writes blockchain data to $datadir.  new bitcoin only writes blockchain data to $datadir/blockchain
 56 2012-05-19 00:34:22 <gmaxwell> FWIW, DB_TXN_WRITE_NOSYNC on the txn was exactly as fast.. down the the second. Probably because both do nothing there.
 57 2012-05-19 00:34:28 <jgarzik> here is the secret recipe:
 58 2012-05-19 00:34:29 <jgarzik> dbenv.set_flags(DB_AUTO_COMMIT, 1);
 59 2012-05-19 00:35:02 <jgarzik> provided we can be sure that our existing "hard flush" techniques are safe, the above code should be safe as well
 60 2012-05-19 00:35:15 <jgarzik> ie. checkpoint/lsn reset
 61 2012-05-19 00:37:15 <sipa> we basically give up durability betweeb checkpoints
 62 2012-05-19 00:38:07 <sipa> for the blockchain... unimportant
 63 2012-05-19 00:38:16 <sipa> for the wallet... unsure
 64 2012-05-19 00:38:55 <sipa> it should be safe as long as there is checkpoint max every 100 new addresses used
 65 2012-05-19 00:39:37 <jgarzik> gmaxwell: if there is little perf diff...  DB_TXN_WRITE_NOSYNC is safer
 66 2012-05-19 00:39:42 <gmaxwell> or if we split the env.
 67 2012-05-19 00:39:47 <jgarzik> I would rather have data in OS buffers than app buffers!
 68 2012-05-19 00:39:49 <gmaxwell> jgarzik: well, I'll try it on the env in a minute.
 69 2012-05-19 00:39:51 <jgarzik> app is most likely to crash
 70 2012-05-19 00:40:07 <gmaxwell> I was hoping write might actually be faster keep the disk saturated.
 71 2012-05-19 00:41:08 <jgarzik> gmaxwell: the bdb log buffer is pretty small, 4-8k I think.  OS waits for many more [4|8]k dirty pages than that, before initiating writeback
 72 2012-05-19 00:41:34 <sipa> jgarzik: our log buffer is a lot larger i think
 73 2012-05-19 00:41:47 <jgarzik> so there wouldn't be much performance difference, as you're waiting on filesystem writeback either way
 74 2012-05-19 00:42:27 <gmaxwell> jgarzik: odd, doesn't appear to be any faster here with your env flag.
 75 2012-05-19 00:42:27 <jgarzik> dbenv.set_flags(DB_TXN_NOSYNC, 1) is awesome.  this has literally saved me hours.
 76 2012-05-19 00:42:48 <gmaxwell> I wonder if this is a behaviorial difference in the bdb versions we're running.
 77 2012-05-19 00:42:54 <jgarzik> gmaxwell: are you on ssd?  !ext[34]?
 78 2012-05-19 00:43:02 <gmaxwell> SSD on ext4.
 79 2012-05-19 00:43:02 <jgarzik> or nfs?
 80 2012-05-19 00:43:09 <jgarzik> gmaxwell: yeah, you will see no difference
 81 2012-05-19 00:43:24 <sipa> jgarzik: full db import using loadblock takes half an hour for me (up to last checkpoint)
 82 2012-05-19 00:43:31 <jgarzik> gmaxwell: writes are instant for you, not for me
 83 2012-05-19 00:43:45 <gmaxwell> IIRC is wasn't _that_ much slower on the spinning disk system. I'll go test it there.
 84 2012-05-19 00:43:48 <jgarzik> sipa: takes hours here, without the above patch
 85 2012-05-19 00:43:57 <sipa> i'll try that enc flag tomorrow
 86 2012-05-19 00:44:04 <sipa> env
 87 2012-05-19 00:45:01 <sipa> jgarzik: that's on an encrypted fs on ext4 on a laptop harddisk, by the way
 88 2012-05-19 00:45:56 <gmaxwell> hours?
 89 2012-05-19 00:46:01 <jgarzik> hah!
 90 2012-05-19 00:46:05 <gmaxwell> no way. something is weird with your libdb.
 91 2012-05-19 00:46:21 <gmaxwell> Even on a 5400 rpm disk it wasn't taking hours for me.
 92 2012-05-19 00:46:51 <gmaxwell> (fwiw, I do most testing with bitcoin in tmpfs)
 93 2012-05-19 00:46:54 <jgarzik> eliminating fdatasync()'s sped things up on the ailing disk
 94 2012-05-19 00:50:56 <gmaxwell> jgarzik: I don't get the fdatasync's without your change.
 95 2012-05-19 00:51:46 <sipa> conclusion: failing disks cause fdatasync calls
 96 2012-05-19 00:51:52 <gmaxwell> libdb_cxx-4.8.so => /lib64/libdb_cxx-4.8.so (0x00007f5198cde000)
 97 2012-05-19 00:51:55 <sipa> ;)
 98 2012-05-19 00:52:01 <gmaxwell> libdb 4.8 ftw \0/
 99 2012-05-19 01:00:29 <jgarzik> yeah, this is libdb 5.x
100 2012-05-19 01:01:01 <jgarzik> amusing that a fried disk led me to discover the source of fdatasync() showing up as _every other system call_ in the strace
101 2012-05-19 01:01:14 <jgarzik> there were more fdatasync's than gettimeofday's in there :)
102 2012-05-19 01:02:59 <jgarzik> interesting.  while I wasn't looking, the block chain load got stuck at height 173928
103 2012-05-19 01:04:28 <gmaxwell> network connected node or a loadblock nod?
104 2012-05-19 01:04:51 <jgarzik> loadblock
105 2012-05-19 01:05:01 <gmaxwell> er. did it reject anything?
106 2012-05-19 01:05:57 <jgarzik> it's failing a bunch of ConnectBlocks
107 2012-05-19 01:06:10 <jgarzik> 05/19/12 03:05:25 ERROR: Reorganize() : ConnectBlock 00000000000008b193b9 failed
108 2012-05-19 01:06:11 <jgarzik> 05/19/12 03:05:25 ERROR: SetBestChain() : Reorganize failed
109 2012-05-19 01:06:13 <jgarzik> 05/19/12 03:05:25 ERROR: AcceptBlock() : AddToBlockIndex failed
110 2012-05-19 01:06:15 <jgarzik> 05/19/12 03:05:25 ERROR: ProcessBlock() : AcceptBlock FAILED
111 2012-05-19 01:06:17 <jgarzik> 05/19/12 03:05:25 received block 00000000000008f9db8a
112 2012-05-19 01:06:19 <jgarzik> 05/19/12 03:05:25 Postponing 1353 reconnects
113 2012-05-19 01:06:47 <jgarzik> but the disk is suspect, so.....
114 2012-05-19 01:07:24 <jgarzik> time to fsck and shuffle disks and fun.  the above dbenv flag definitely removes many thousands of fdatasync calls for me, on db 5.x
115 2012-05-19 01:07:43 <gmaxwell> Thats block 175281
116 2012-05-19 01:07:51 <gmaxwell> (the correct one)
117 2012-05-19 01:08:39 <gmaxwell> sounds like jgarzik and this person https://bitcointalk.org/index.php?topic=82426.msg908655 are kindred spirits
118 2012-05-19 01:09:28 <sipa> how t f can connectblock fail, without the block failing POW checks?
119 2012-05-19 01:09:52 <gmaxwell> sipa: index corruption?
120 2012-05-19 01:10:15 <sipa> right, if the block index being created is already corrupted
121 2012-05-19 01:11:23 <sipa> does a butterfs contain FAT?
122 2012-05-19 02:29:15 <jgarzik> well
123 2012-05-19 02:30:31 <jgarzik> no disk errors found.  symptom was:
124 2012-05-19 02:30:33 <jgarzik> b_state=0x00000029, b_size=4096
125 2012-05-19 02:30:51 <jgarzik> written to kernel log many times, while fdatasync-heavy block import was occurring
126 2012-05-19 02:31:26 <jgarzik> if hardware seems ok, and 'e2fsck -fvy' turns up nothing, maybe ext4 was having problems
127 2012-05-19 02:31:36 <jgarzik> now testing with DB_TXN_WRITE_NOSYNC
128 2012-05-19 02:31:48 <jgarzik> as that is IMO the preferred mode for dbenv and txn_begin
129 2012-05-19 02:32:37 <jgarzik> W-O-W that is fast
130 2012-05-19 02:32:47 <jgarzik> 50,000 blocks in a few seconds
131 2012-05-19 02:32:58 <jgarzik> 80,000
132 2012-05-19 02:33:18 <jgarzik> 103,000
133 2012-05-19 02:33:49 <jgarzik> 115,000.  now slowing down just a tad.
134 2012-05-19 02:35:03 <jgarzik> 130,000
135 2012-05-19 02:35:05 <gmaxwell> jgarzik: thats how it's been for sipa and I for some time..
136 2012-05-19 02:35:06 <gmaxwell> e.g.
137 2012-05-19 02:35:16 <gmaxwell> 05/19/12 01:58:53 SetBestChain: new best=000000001aeae195809d  height=50000  work=363124009197915
138 2012-05-19 02:35:30 <gmaxwell> 05/19/12 02:00:04 SetBestChain: new best=000000000000e58392f3  height=115000  work=2701115064210100617
139 2012-05-19 02:35:47 <gmaxwell> and thats on a fairly slow 1.6ghz core 2.
140 2012-05-19 02:36:17 <gmaxwell> it gets slow later..
141 2012-05-19 02:36:20 <jgarzik> gmaxwell: is there any penalty for DB_TXN_WRITE_NOSYNC on db4.8?
142 2012-05-19 02:36:47 <gmaxwell> I didn't see one. But I should try it on a system with a spinning disk.
143 2012-05-19 02:44:45 <luke-jr> 3.3 MB   cb29bccd6bfcb21f726628b08166d12ea94682c42face7dfd3977069bd75ce15+ bin-osx-bdb-4.8.30.tbz2
144 2012-05-19 02:44:48 <Diablo-D3> [12:36:19] <jgarzik> gmaxwell: is there any penalty for DB_TXN_WRITE_NOSYNC on db4.8?
145 2012-05-19 02:44:48 <luke-jr> ^ deterministic
146 2012-05-19 02:44:51 <Diablo-D3> jgarzik: it sounds bad
147 2012-05-19 02:45:16 <jgarzik> Diablo-D3: huh?
148 2012-05-19 02:45:45 <Diablo-D3> well, wouldnt that basically mean DAMN THE TORPEDOS, RAMMING SPEED and everybody dies at the end of the movie?
149 2012-05-19 02:48:44 <Diablo-D3> but I like database integrity :<
150 2012-05-19 02:49:10 <jgarzik> on the topic of transaction pruning...  I wonder what would be the impact of simply removing spent transactions from the tx index
151 2012-05-19 02:49:24 <Diablo-D3> do not want :<
152 2012-05-19 02:49:28 <jgarzik> that seems the easiest to do, from bitcoin's point of view
153 2012-05-19 02:50:30 <jgarzik> Diablo-D3: we already use DB_TXN_NOSYNC, in case you haven't been reading scrollback
154 2012-05-19 02:51:02 <Diablo-D3> but doesnt that lead to the aformentioned database integrity lacking of?
155 2012-05-19 02:51:33 <jgarzik> Diablo-D3: the details would take too long to explain to a java programmer
156 2012-05-19 02:51:55 <luke-jr> jgarzik: you missed Diablo-D3's defection?
157 2012-05-19 02:51:56 <Diablo-D3> okay, good, Im a C programmer, go ahead.
158 2012-05-19 02:51:59 <luke-jr> jgarzik: he hates Java now
159 2012-05-19 02:52:04 <Diablo-D3> luke-jr: its not a defection really
160 2012-05-19 02:52:11 <Diablo-D3> Ive done C longer than any language I know
161 2012-05-19 02:52:25 <Diablo-D3> its sorta like one of those marriages where they split up and sleep with other people but come back together in the end
162 2012-05-19 02:52:36 <neofutur> +1 for hating java ;)
163 2012-05-19 02:52:43 <Diablo-D3> I dont hate java _the language_
164 2012-05-19 02:52:47 <Diablo-D3> I hate oracle.
165 2012-05-19 02:52:48 <jgarzik> Diablo-D3: going DB_TXN_NOSYNC -> DB_TXN_WRITE_NOSYNC increases durability and integrity
166 2012-05-19 02:52:54 <luke-jr> Diablo-D3: Oracle made bdb too
167 2012-05-19 02:52:55 <neofutur> I hate java the stack exec :p
168 2012-05-19 02:53:02 <Diablo-D3> oracle did not make bdb you fucktard.
169 2012-05-19 02:53:08 <luke-jr> fine, owns
170 2012-05-19 02:53:18 <luke-jr> Oracle made bdb just as much as they made java :p
171 2012-05-19 02:53:19 <Diablo-D3> they only own 5.x
172 2012-05-19 02:53:25 <Diablo-D3> I do not recognize 5.x as a legitimate version of bdb
173 2012-05-19 02:53:28 <luke-jr> then why does 4.8 come from their site?
174 2012-05-19 02:53:36 <Diablo-D3> they only host old versions
175 2012-05-19 02:53:39 <Diablo-D3> they predate oracle.
176 2012-05-19 02:54:12 <Diablo-D3> luke-jr: btw, I suspect bdb is older than you
177 2012-05-19 02:54:15 <Diablo-D3> and its almost as old as me
178 2012-05-19 02:54:57 <luke-jr> Diablo-D3: I'd have to be young for bdb to be older than me.
179 2012-05-19 02:55:14 <Diablo-D3> you're like in your early 20s
180 2012-05-19 02:55:15 <luke-jr> Google says it only goes back to 1986
181 2012-05-19 02:56:14 <jgarzik> we should look at kyotocabinet
182 2012-05-19 02:56:21 <jgarzik> if it has recovery tools
183 2012-05-19 02:57:06 <Diablo-D3> jgarzik: I looked at that guys code
184 2012-05-19 02:57:10 <Diablo-D3> I was impressed actually
185 2012-05-19 02:57:39 <jgarzik> tokyocabinet was nice, lean and fast
186 2012-05-19 02:57:58 <jgarzik> but you were screwed, if any corruption or such happened
187 2012-05-19 02:58:36 <Diablo-D3> kyotocabinet is tc 2.0 isnt it?
188 2012-05-19 02:58:52 <jgarzik> yes
189 2012-05-19 02:59:06 <jgarzik> it went C++, too
190 2012-05-19 02:59:10 <jgarzik> tc was in C
191 2012-05-19 02:59:15 <Diablo-D3> oh eww
192 2012-05-19 02:59:17 <Diablo-D3> do not want
193 2012-05-19 02:59:25 <Diablo-D3> Ill just have to write my own
194 2012-05-19 03:02:26 <jgarzik> <shrug> kyotocabinet provides a C API
195 2012-05-19 03:02:48 <Diablo-D3> yeah but I want a lib that does everything I want it to do
196 2012-05-19 03:02:53 <Diablo-D3> so if I want a kv write backend
197 2012-05-19 03:02:57 <Diablo-D3> Ill have to write my own
198 2012-05-19 03:05:36 <jgarzik> 05/19/12 05:05:20 ERROR: Reorganize() : ConnectBlock 00000000000008b193b9 failed
199 2012-05-19 03:05:42 <jgarzik> again we get stuck
200 2012-05-19 03:11:37 <gmaxwell> 20:06 <@jgarzik> 05/19/12 03:05:25 InvalidChainFound: invalid block=000000000000045f40a3  height=175281  work=291024964310457300938
201 2012-05-19 03:11:44 <gmaxwell> remarkable.. a bit earlier that time.
202 2012-05-19 03:12:09 <gmaxwell> you should probably try adding a bunch of debugging to figure out exactly why it's failing?
203 2012-05-19 03:14:01 <jgarzik> gmaxwell: the 'invalid block' keeps incrementing, so that doesn't say much
204 2012-05-19 03:14:24 <jgarzik> gmaxwell: the 'current best' is frozen at 173928 this time... which is same as last time
205 2012-05-19 03:15:27 <jgarzik> 'invalid block' continues until 180,000 or whatever
206 2012-05-19 03:15:56 <jgarzik> I -think- this stuck problem is independent of my stuff
207 2012-05-19 03:18:25 <jgarzik> so the symptom is that every block after 173928 failed to ConnectBlock, and is therefore considered invalid.
208 2012-05-19 03:18:32 <jgarzik> same :)
209 2012-05-19 03:19:29 <jgarzik> 122,466
210 2012-05-19 03:20:55 <jgarzik> 130,000
211 2012-05-19 03:21:34 <gmaxwell> hm was moa7 basically complaining about this earlier?
212 2012-05-19 03:22:35 <ThomasV> sup?
213 2012-05-19 03:22:57 <gmaxwell> hard to tell. he was complaining sort of generically.
214 2012-05-19 03:23:27 <gmaxwell> jgarzik: 150000 .. started later than you though.
215 2012-05-19 03:23:32 <jgarzik> :)
216 2012-05-19 03:23:35 <jgarzik> 135,000
217 2012-05-19 03:23:44 <jgarzik> I think your SSD is winning the race
218 2012-05-19 03:24:10 <gmaxwell> well my laptop is height=136981  .. the faster machine (started even later than the laptop is at 160)
219 2012-05-19 03:24:13 <gmaxwell> (now)
220 2012-05-19 03:24:23 <jgarzik> 138,000
221 2012-05-19 03:24:32 <gmaxwell> The faster machine is also using tmpfs.
222 2012-05-19 03:25:52 <gmaxwell> 05/19/12 05:25:50 SetBestChain: new best=00000000000009ca28af  height=169087  work=250260744989263374336
223 2012-05-19 03:26:12 <gmaxwell> laptop is still back at height=143806
224 2012-05-19 03:28:26 <jgarzik> 150,000
225 2012-05-19 03:28:44 <jgarzik> I have enough memory to use ramfs... that would be fastest :)
226 2012-05-19 03:28:46 <gribble> New news from bitcoinrss: jgarzik opened pull request 1360 on bitcoin/bitcoin <https://github.com/bitcoin/bitcoin/pull/1360>
227 2012-05-19 03:30:30 <gmaxwell> moment of truth...05/19/12 05:30:28 SetBestChain: new best=00000000000003a29a70  height=173922  work=281530853585628171623
228 2012-05-19 03:31:12 <jgarzik> 157,000
229 2012-05-19 03:31:23 <gmaxwell> just crossed 174300 on the fast machine, but it's also a bit behind master because it's been testing gavin's caching anti-ddos patches.
230 2012-05-19 03:31:38 <gmaxwell> 174800 actually.
231 2012-05-19 03:32:00 <gmaxwell> laptop at height=161672.
232 2012-05-19 03:32:10 <jgarzik> 158,000
233 2012-05-19 03:32:38 <jgarzik> who knows... maybe it was my removal of thought-to-be-unused nested transactions that borked something
234 2012-05-19 03:33:23 <gmaxwell> Will be good to figure out what.
235 2012-05-19 03:35:09 <jgarzik> eu1.bitcoincharts.com, that is
236 2012-05-19 03:35:24 <gmaxwell> oh interesting though. Know which file you grabbed?
237 2012-05-19 03:35:54 <jgarzik> yes, blockchain-2012-05-10.tar, but I think it has rotated off by now
238 2012-05-19 03:36:18 <gmaxwell> ls
239 2012-05-19 03:36:32 <gmaxwell> yea, 404. :(
240 2012-05-19 03:36:44 <jgarzik> it is an interesting dataset -- I can watch REORGANIZE appear periodically, as bitcoin explores and drops side chains.  it's a well known site, so it probably is targeted by miners for quick block propagation and such
241 2012-05-19 03:36:59 <jgarzik> and as such gets more orphans
242 2012-05-19 03:37:08 <jgarzik> 168,000
243 2012-05-19 03:37:21 <gmaxwell> that chain is probably also written by an outdated version which is happy to follow crazy impossible invalid block forks.
244 2012-05-19 03:37:22 <jgarzik> gmaxwell: I would go for the next day.  I think 05-11 is still up there.
245 2012-05-19 03:37:45 <jgarzik> gmaxwell: quite possibly
246 2012-05-19 03:37:49 <gmaxwell> -12 is the oldest up. I'll have it in about 13 minutes.
247 2012-05-19 03:39:09 <jgarzik> eu1.bitcoincharts.com is publicly mentioned in various places -- definitely the wiki, maybe even bitcoin.org -- as a place to download the block chain database.  if tcatm is producing that with outdated software, we should poke him
248 2012-05-19 03:39:45 <gmaxwell> I've been discouraging people for some time, it's very risky esp to have miners on it.
249 2012-05-19 03:40:04 <gmaxwell> I was planning on nagging him to drop the index out of it once we have code with loadblock released.
250 2012-05-19 03:40:21 <gmaxwell> then at least it will get validated.
251 2012-05-19 03:40:24 <jgarzik> indeed
252 2012-05-19 03:42:31 <jgarzik> 170,000
253 2012-05-19 03:46:24 <jgarzik> 171,000
254 2012-05-19 03:48:07 <gmaxwell> "Paint?" "It's what we had before smart pigments" "Oh! why'd we switch to smart pigments? they can't simulate a wet surface. Sounds like paint was better!"
255 2012-05-19 03:49:07 <jgarzik> 172,000
256 2012-05-19 03:50:45 <gmaxwell> k. have blockchain-2012-05-12.tar now
257 2012-05-19 03:52:45 <gmaxwell> my laptop surived.
258 2012-05-19 03:54:05 <jgarzik> 173,600
259 2012-05-19 03:54:39 <gmaxwell> at 100k on the eu1-12 chain.
260 2012-05-19 03:55:49 <sturles> A friend of mine had a nice summer job last summer.  Her job was to sit and watch the grass grow.  It's not even a joke.  She did some measurements, but that was all.  Must be the most relaxed job ever.
261 2012-05-19 03:56:02 <jgarzik> 174,000
262 2012-05-19 03:56:05 <luke-jr> boring*
263 2012-05-19 03:56:08 <jgarzik> seeing a lot of
264 2012-05-19 03:56:10 <jgarzik> though
265 2012-05-19 03:56:28 <jgarzik> nevertheless, master survived
266 2012-05-19 03:56:41 <luke-jr> jgarzik: I sure hope that node making the snapshots is P2SH-enforcing
267 2012-05-19 03:56:50 <gmaxwell> luke-jr: ^ appears that its not.
268 2012-05-19 03:56:53 <gmaxwell> or wasn't.
269 2012-05-19 03:57:09 <gmaxwell> this is not a shock to me in the slighest.
270 2012-05-19 03:57:14 <luke-jr> if it's not, that means people who just put it in .bitcoin get tricked possibly?
271 2012-05-19 03:58:06 <gmaxwell> no  because by the time you come up that chain would have been orphaned.
272 2012-05-19 03:58:15 <jgarzik> definitely a lot of "P2SH VerifySignature failed" blocks in here
273 2012-05-19 03:58:23 <jgarzik> yep
274 2012-05-19 03:58:55 <luke-jr> gmaxwell: also, VerifySignature only runs after the block is in blk0001.dat anyway, right?
275 2012-05-19 03:59:05 <gmaxwell> I'm going to keep this chain though, it's a better reorg test than any of my chains.
276 2012-05-19 03:59:11 <luke-jr> so we really can't tell if that node is enforcing just from loadblock log
277 2012-05-19 03:59:29 <gmaxwell> I don't get those on my nodes.
278 2012-05-19 03:59:55 <gmaxwell> (I mean I got them off the live network, but not while loadblocking my own chains)
279 2012-05-19 04:00:04 <luke-jr> hmm
280 2012-05-19 04:00:44 <gmaxwell> I wonder if anyone has been collecting all the orphans.. it would be fun to make a blockfile that lists all the orphans for a block first to maximize the reorgs.
281 2012-05-19 04:02:19 <gmaxwell> 05/19/12 06:01:54 SetBestChain: new best=0000000000000acbc021  height=171073  work=262897188360575878842
282 2012-05-19 04:02:22 <gmaxwell> 05/19/12 06:01:54 ProcessBlock: ACCEPTED
283 2012-05-19 04:02:24 <gmaxwell> 05/19/12 06:01:54 ProcessBlock: ACCEPTED
284 2012-05-19 04:02:32 <gmaxwell> ^ thats odd... wonder where the repetition came from?
285 2012-05-19 04:02:50 <luke-jr> I saw that when loadblocking too
286 2012-05-19 04:04:54 <jgarzik> gmaxwell: I see that too
287 2012-05-19 04:05:20 <gmaxwell> I'm awash in 05/19/12 06:05:09 ERROR: ConnectInputs() : 4005d6bea3 P2SH VerifySignature failed
288 2012-05-19 04:05:24 <gmaxwell> but it's still going.
289 2012-05-19 04:05:36 <jgarzik> this SSD loadblock is fast :)
290 2012-05-19 04:05:54 <gmaxwell> at 175k and still going.
291 2012-05-19 04:06:07 <gmaxwell> 05/19/12 06:04:42 REORGANIZE: Disconnect 0 blocks; 0000000000000500d6fc..0000000000000500d6fc
292 2012-05-19 04:06:14 <gmaxwell> kinda .. odd disconnect count there.
293 2012-05-19 04:06:15 <luke-jr> O.o
294 2012-05-19 04:06:19 <jgarzik> 05/19/12 06:04:19 SetBestChain: new best=00000000839a8e6886ab  height=1
295 2012-05-19 04:07:27 <jgarzik> gmaxwell: definitely a useful dataset for testing
296 2012-05-19 04:08:11 <gmaxwell> yea, anyways.. syncs up for me.
297 2012-05-19 04:08:34 <jgarzik> sounds like one for my bug pile :)
298 2012-05-19 04:08:46 <gmaxwell> should try to get some more chains from old long running nodes in order to make the ultimate orphanhouse.
299 2012-05-19 04:09:28 <jgarzik> 'git bisect' in paralle
300 2012-05-19 04:10:22 <gmaxwell> I have a couple 32 way machines which I do that for codec testing. But sadly bitcoin is so iolimited I can't really test it that way.
301 2012-05-19 04:20:15 <luke-jr> jgarzik: except 'git bisect' is inherently serial& :p
302 2012-05-19 04:20:40 <luke-jr> https://bitcointalk.org/?topic=82480
303 2012-05-19 04:22:30 <gmaxwell> luke-jr: parallel bisection isn't rocket science... you just preexpand out the tree until you get enough leaves to fill your processors. It's less efficient but still much lower latency than serial probing.
304 2012-05-19 04:22:57 <luke-jr> gmaxwell: sure, but bisecting code is serial :p
305 2012-05-19 04:23:13 <luke-jr> until you check commit 1/2, you don't know whether the next step is 1/4 or 3/4
306 2012-05-19 04:23:32 <gmaxwell> even better would just be to have a big suite of tests including things like "loadblocks this chain and keep the debug.log" that run on every commit. Then instead of bisecting you just look through the test logs for the first sign of trouble.
307 2012-05-19 04:23:50 <gmaxwell> luke-jr: check them both.
308 2012-05-19 04:24:07 <luke-jr> gmaxwell: we have a test suite, but it often seems Gavin and I are the only ones who run it :p
309 2012-05-19 04:25:34 <gmaxwell> http://wuriddles.com/hard.shtml (see CRIMINAL CUPBEARERS)
310 2012-05-19 04:26:12 <gmaxwell> oh great db on my laptop is nice and corrupt.
311 2012-05-19 04:26:41 <gmaxwell>  "why is my criticial battery low light on, I'm plugged in arn't I"  <unplugs> <laptop instantly shuts off> "oh thats why"
312 2012-05-19 04:26:54 <luke-jr> >_<
313 2012-05-19 04:28:42 <gmaxwell> Okay, help me with my C++ cluelessness. How do you get a useful backtrace when an exception has already unwound the stack away from where the baddness happened?
314 2012-05-19 04:29:13 <luke-jr> I don't think you can
315 2012-05-19 04:29:39 <luke-jr> other than breaking at the spot where it occurs or removing the try{}
316 2012-05-19 04:29:44 <gmaxwell> is there some magical thing I can do so that anything throwing an exception instantly crashes?
317 2012-05-19 04:30:01 <luke-jr> catch throw <-- breaks on any exception, I think
318 2012-05-19 04:30:18 <gmaxwell> yea I don't know where it's happening...
319 2012-05-19 04:30:20 <gmaxwell> I'm getting
320 2012-05-19 04:30:21 <gmaxwell> EXCEPTION: NSt8ios_base7failureE
321 2012-05-19 04:33:21 <jgarzik> gmaxwell: during loadblock?
322 2012-05-19 04:33:40 <gmaxwell> jgarzik: I lost power during loadblock.. now I'm bombing out right no startup
323 2012-05-19 04:33:42 <jgarzik> gmaxwell: running in gdb sometimes results in a useful backtrace, when an exception is thrown
324 2012-05-19 04:33:47 <gmaxwell> luke-jr: thanks killing the try was helpful.
325 2012-05-19 04:33:51 <jgarzik> gmaxwell: ah!
326 2012-05-19 04:33:59 <gmaxwell> jgarzik: it wasnt, but I got one now.
327 2012-05-19 04:34:42 <gmaxwell> http://pastebin.mozilla.org/1645971
328 2012-05-19 04:34:53 <jgarzik> in CTxDB::LoadBlockIndex(), the de-serialize needs a try{} around it, FWIW.  though that's not a CAutoFile, so not your problem
329 2012-05-19 04:36:24 <jgarzik> luke-jr: and no bisection is not inherently serial
330 2012-05-19 04:36:46 <jgarzik> luke-jr: it's easy to pick N points between A and B, even in a tree structure
331 2012-05-19 04:37:36 <gmaxwell> I guess what happend is that index updates made it to disk ahead of the blockchain updates.
332 2012-05-19 04:37:46 <gmaxwell> so the index points past the end of the file and it just fails.
333 2012-05-19 04:38:13 <gmaxwell> too sleepy now to think, goodnight.
334 2012-05-19 04:38:15 <jgarzik> gmaxwell: IBD only fsync's every 500 blocks
335 2012-05-19 04:38:23 <jgarzik> night!
336 2012-05-19 04:53:49 <moa7> sipa : reverify blockchain from scratch (-loadblock) using git head worked
337 2012-05-19 04:54:06 <moa7> sipa : thnx muchly ... you must have patience of a saint.
338 2012-05-19 05:00:49 <moa7> sipa : gonna run with git head for a while so just ask if you need anything tested
339 2012-05-19 07:44:59 <gribble> New news from bitcoinrss: laanwj opened issue 1361 on bitcoin/bitcoin <https://github.com/bitcoin/bitcoin/issues/1361>
340 2012-05-19 07:49:04 <tcatm> jgarzik: bitcoind on eu1 updated to 0.6.2
341 2012-05-19 07:50:08 <coingenuity> tcatm: my memory is failing me, which service do you run...bitcoincharts?
342 2012-05-19 07:50:25 <coingenuity> or was it something else
343 2012-05-19 08:53:32 <gribble> New news from bitcoinrss: laanwj opened pull request 1362 on bitcoin/bitcoin <https://github.com/bitcoin/bitcoin/pull/1362>
344 2012-05-19 10:03:19 <PK> I noticed my server still runs bitcoin 5.2. If my pool finds a block, is it still considered valid? What the worst bad thing that can happen if i don't upgrade right now.
345 2012-05-19 10:18:21 <sturles> PK: Probably not valid.  It may include an invalid P2SH transaction.
346 2012-05-19 10:19:24 <sturles> This may be the worst thing.  Another bad scenario is that it may lock up on a block and not get you any further.
347 2012-05-19 10:20:01 <PK> I checked that already, my block chain looks normal. My last block is the same as on blockexplorer
348 2012-05-19 10:20:43 <luke-jr> another bad scenario is that you're vulnerable to known exploits
349 2012-05-19 10:21:08 <luke-jr> upgrade to 0.5.5 at least
350 2012-05-19 10:21:30 <luke-jr> http://luke.dashjr.org/programs/bitcoin/files/bitcoind/0.5.5/
351 2012-05-19 10:21:53 <PK> that's a bit of a problem. Forgot root pw :)
352 2012-05-19 10:22:01 <luke-jr> &
353 2012-05-19 10:22:19 <PK> but at least it's linux. Easy enough to edit the shadow file.
354 2012-05-19 10:22:21 <wachtwoord> :(
355 2012-05-19 11:16:07 <PK> ok, I reset my root pw and upgraded to 0.6.2 now.
356 2012-05-19 11:41:59 <JFK911> ik vergaat mijn wachtwoord!
357 2012-05-19 11:43:53 <sipa> (vergat)
358 2012-05-19 11:44:37 <JFK911> oh thanks
359 2012-05-19 11:45:52 <gavinandresen> anybody else getting failing unit tests with git HEAD?
360 2012-05-19 12:00:14 <da2ce7> anyone good at autotools here? I could appreaceate a small amount of help.
361 2012-05-19 12:11:31 <sipa> gavinandresen: let's see
362 2012-05-19 12:12:16 <gavinandresen> sipa: I think something is wrong with my build environment
363 2012-05-19 12:13:39 <sipa> mental note: do not use make -j55 instead of -j5
364 2012-05-19 12:18:51 <sipa> gavinandresen: all fine here
365 2012-05-19 12:19:10 <gavinandresen> yeah, fine here in a VM.  grr.
366 2012-05-19 13:41:52 <jgarzik> tcatm: thanks!
367 2012-05-19 14:03:00 <gmaxwell> tcatm: at some point you should nuke the chain on that node and resync it from the network all the ophans in that database are good for testing but not helpful to the people downloading it. :)
368 2012-05-19 14:33:00 <JFK911> poons
369 2012-05-19 14:56:32 <luke-jr> so anyhow
370 2012-05-19 14:56:43 <luke-jr> Mac binaries for bdb, miniupnpc, openssl, and zlib can be built deterministically now
371 2012-05-19 14:57:12 <luke-jr> potentially this should be enough for bitcoind& but we don't seem to ship a Mac bitcoind >_<K
372 2012-05-19 15:03:14 <sipa> luke-jr: and Qt would be... ambitious?
373 2012-05-19 15:05:36 <luke-jr> sipa: scary
374 2012-05-19 15:05:38 <luke-jr> daunting
375 2012-05-19 15:06:09 <sipa> luke-jr: and do they builds work?
376 2012-05-19 15:14:01 <BlueMatt> luke-jr: nice! Im assuming you are compiling your own gcc?
377 2012-05-19 16:37:19 <luke-jr> sipa: no idea, got a Mac?
378 2012-05-19 16:37:24 <sipa> nope
379 2012-05-19 16:37:27 <luke-jr> BlueMatt: yes
380 2012-05-19 17:34:59 <luke-jr> jgarzik: am I supposed to get this for every block during upgrade? ProcessBlock: ORPHAN BLOCK, prev=*
381 2012-05-19 17:35:10 <luke-jr> my client crashed after it was done, so I'm guessing not
382 2012-05-19 17:39:44 <luke-jr> http://luke.dashjr.org/tmp/code/debug.log.bz2
383 2012-05-19 17:45:39 <luke-jr> hmm, so I can't get my clinet to start now at all :D
384 2012-05-19 17:45:51 <luke-jr> guess I should test that before I post it on the forum <.<
385 2012-05-19 17:47:18 <ronaz> i had the same with bitcoin qt
386 2012-05-19 17:47:49 <ronaz> nono wait.. no, sorry
387 2012-05-19 19:44:38 <BlueMatt> luke-jr: osx works great in virtualbox (with a few .conf additions)
388 2012-05-19 19:48:25 <luke-jr> BlueMatt: not legally.
389 2012-05-19 19:48:35 <BlueMatt> if you are just testing bitcoin...
390 2012-05-19 19:48:41 <luke-jr> the law doesn't care why
391 2012-05-19 19:48:48 <BlueMatt> (and it is legal on apple hardware)
392 2012-05-19 19:48:54 <luke-jr> true
393 2012-05-19 19:49:08 <luke-jr> I guess I have a Mac laptop from one of the companies I work for
394 2012-05-19 19:49:17 <luke-jr> and hard for them to claim IP ownership of testing
395 2012-05-19 19:49:37 <BlueMatt> s
396 2012-05-19 19:51:55 <luke-jr> jgarzik: bisected the crash to aa79af8..jgarzik/blockindex
397 2012-05-19 21:07:26 <gribble> New news from bitcoinrss: Diapolo opened pull request 1363 on bitcoin/bitcoin <https://github.com/bitcoin/bitcoin/pull/1363>
398 2012-05-19 21:17:28 <TuxBlackEdo> /msg chainmon #next 3
399 2012-05-19 21:17:41 <TuxBlackEdo> sorry
400 2012-05-19 21:17:43 <TuxBlackEdo> ignore that
401 2012-05-19 21:59:15 <jgarzik> luke-jr:  did you have a crash?  or rename blkindex.dat -> txhash.dat or anything strange like that?  any manual database file manipulation?
402 2012-05-19 21:59:45 <jgarzik> luke-jr: er, by that I mean, did you have a crash _before_ the crash you reported?  something that might have caused the database to give a short read?
403 2012-05-19 22:00:10 <luke-jr> jgarzik: not that I'm aware of
404 2012-05-19 22:00:20 <luke-jr> (and I would have probably noticed)
405 2012-05-19 22:01:23 <luke-jr> moreover, I recently (about a week ago) rebuilt my block db with -loadblock
406 2012-05-19 22:07:38 <luke-jr> jgarzik: getting the ORPHAN BLOCK flood on another HOME dir too - is that expected?
407 2012-05-19 22:09:53 <luke-jr> crashed
408 2012-05-19 22:12:35 <jgarzik> luke-jr: the crash is easy to fix:  wrap a try {} around the CDataStream usage
409 2012-05-19 22:12:51 <jgarzik> luke-jr: but #blockindex should spend a long time upgrading your database -- does that happen?
410 2012-05-19 22:12:56 <luke-jr> jgarzik: yes
411 2012-05-19 22:13:36 <luke-jr> jgarzik: will wrapping a try{} fix it, or just hide it?
412 2012-05-19 22:18:31 <sipa> the problem is when your old blk0001.dat has a corrupted block on it, a corruption that happened after downloading
413 2012-05-19 22:19:09 <sipa> in that case every block after it will result in an orphan, and potentially exhaust your memory soon afterwards
414 2012-05-19 22:19:31 <sipa> maybe LoadExternalBlockFile should just stop at the point where it encounters a corrupt block
415 2012-05-19 22:19:55 <BlueMatt> or not load orphans?
416 2012-05-19 22:20:12 <BlueMatt> or limit orphan chain size?
417 2012-05-19 22:20:18 <sipa> just not storing orphans is maybe even beter
418 2012-05-19 22:23:42 <sipa> but, to be more specific to luke-jr's problem... i'don understand how that can occur without a corrupt block
419 2012-05-19 22:29:20 <luke-jr> sipa: it's reporting orphans from the first block
420 2012-05-19 22:30:36 <sipa> hmmm... the first block should be the genesis block no matter what happened
421 2012-05-19 22:30:41 <luke-jr> ProcessBlock: ORPHAN BLOCK, prev=00000000000000000000
422 2012-05-19 22:30:48 <sipa> and the genesis block shouldn't ever be an orphan
423 2012-05-19 22:31:14 <sipa> wait... jgarzik: did you put the genesis block in the new block chain being established?
424 2012-05-19 22:36:11 <jgarzik> sipa: The code calls LoadBlockIndex() to initialize the [empty] database, then LoadExternalBlockFile() just like -loadblock=
425 2012-05-19 22:36:24 <jgarzik> sipa: there is nothing strange or new
426 2012-05-19 22:36:27 <sipa> hmmm
427 2012-05-19 22:37:55 <jgarzik> sipa: luke-jr "on the top" (i.e. not the root cause) crash is caused by a problem that we should fix anyway:  several CDataStream uses for deserialization inside db.{h,cpp} are not enclosed with try{}, and will therefore throw an uncaught exception if there is any problem during de-ser.
428 2012-05-19 22:38:34 <jgarzik> but why this is happening, even so?  good question...
429 2012-05-19 22:40:19 <sipa> luke-jr: any chance your merging/rebasing caused this, or are you just running jgarzik's patch?
430 2012-05-19 22:40:40 <luke-jr> sipa: I don't see how it would be related, but I can try without it
431 2012-05-19 22:41:03 <luke-jr> to clarify, should I try jgarzik's branch directly, or merge it on top of master?
432 2012-05-19 22:42:09 <luke-jr> (also, git bisect did go into the branch itself, IIRC)
433 2012-05-19 22:42:27 <sipa> probably doesn't matter, but as it doesn't merge cleanly with master anymore, just use his branch?
434 2012-05-19 22:43:37 <luke-jr> k
435 2012-05-19 22:44:09 <BlueMatt> jgarzik: while Im rebasing cblockstore, are there any non-major things that would make it more palatable to you?
436 2012-05-19 22:46:12 <BlueMatt> ooo, I could go for some pizza
437 2012-05-19 22:46:53 <sipa> BlueMatt: i have one - don't call it CBlockStore; it's not a blockstore at all at this point, and if i understand things correctly neither will it
438 2012-05-19 22:47:05 <luke-jr> O.o
439 2012-05-19 22:47:34 <sipa> CHub, CMessager, ...
440 2012-05-19 22:47:45 <BlueMatt> sipa: fair enough
441 2012-05-19 22:48:13 <BlueMatt> wumpus: re: #1336/commit f1e757, mind adding the new stuff from doc/assets-attribution.txt to contrib/debian/copyright?
442 2012-05-19 22:48:29 <sipa> i expect network and blockchain management to be turned into classes as a next step, and making them clients to what-is-currently-called-CBlockStore
443 2012-05-19 22:49:29 <sipa> BlueMatt: also, semaphores known to you?
444 2012-05-19 22:49:47 <BlueMatt> sipa: elaborate?
445 2012-05-19 22:51:24 <jgarzik> BlueMatt: rename would be nice, as sipa said
446 2012-05-19 22:51:27 <BlueMatt> yea, yea...noob and all...
447 2012-05-19 22:51:44 <sipa> BlueMatt: it's basically a mutex that allows more than one process to take a lock
448 2012-05-19 22:52:00 <sipa> or a mutex can be seen as a semaphore that can only have value 1 or 0
449 2012-05-19 22:52:15 <BlueMatt> yea, so I guessed(ish)
450 2012-05-19 22:52:49 <sipa> it's currently implemented using a boost condition variable (though god knows how many times it will still need to be changed...)
451 2012-05-19 22:53:11 <sipa> but what you need actually sounds like a semaphore as well
452 2012-05-19 22:53:33 <sipa> it's just one maintaining the counter of queued messages
453 2012-05-19 22:53:55 <jgarzik> smashing everything together into one big kitchen sink, only to plan to pull it apart later, seems like more work
454 2012-05-19 22:54:06 <jgarzik> bitcoin is already one big kitchen sink
455 2012-05-19 22:54:11 <sipa> jgarzik: i don't think we see it the same way
456 2012-05-19 22:54:39 <sipa> as i said, i like your idea, but it requires a lot more changes
457 2012-05-19 22:55:00 <jgarzik> it really doesn't
458 2012-05-19 22:55:18 <BlueMatt> sipa: have you seen the updated version, using CSemaphore? https://github.com/TheBlueMatt/bitcoin/commit/e99b0bc4355c63eeccd6d7cf825e61245b672287
459 2012-05-19 22:55:19 <jgarzik> I'll create a few patches next week
460 2012-05-19 22:56:07 <BlueMatt> as in, you'll create a few patches which are your version of cblockstore, or you'll write a few patches that fix cblockstore?
461 2012-05-19 22:56:44 <jgarzik> I don't think CBlockStore is the right design at all, so it would go in a different direction
462 2012-05-19 22:56:47 <luke-jr> branch itself works; testing with merged-to-master now
463 2012-05-19 22:56:54 <BlueMatt> jgarzik: fair enough
464 2012-05-19 22:57:04 <jgarzik> anyway, time to put the kid to sleep, back in 60 minutes
465 2012-05-19 22:58:24 <jgarzik> design criticisms:  1) network should be hidden behind blockchain service, not exposed.  2) don't put blockchain service and wallet service together, as they are not only separate from have a 1-to-N relationship.  3) putting everything together in one big class is only marginally better than the current one-big-heap mess.
466 2012-05-19 22:58:45 <luke-jr> merged-to-master is working too, but I don't recall making these merges before O.o
467 2012-05-19 22:59:10 <sipa> jgarzik: wallet service has nothing to do with cblockstore at all
468 2012-05-19 22:59:52 <sipa> cblockstore manages communication between net/chain/wallets; but not between wallet/ui/rpc
469 2012-05-19 23:01:13 <gmaxwell> There is sort of an interest question about what we'd do with the RPC after splitting the wallet and blockchain.
470 2012-05-19 23:01:25 <gmaxwell> Half the the rpc commands are blockchain ones, half are wallet ones.
471 2012-05-19 23:01:34 <BlueMatt> jgarzik: 1) I suppose we see it opposite ways here, then...network should, imho, be entirely separate from blockchain stuff 2) they arent, nor are they very well split yet...currently they are still coupled where there used to be coupling (to keep cblockstore as close to exactly the same as master, for now, as I thought that would make it more palatable for merge/review) 3) true, but the goal is to open up room for improvement, not actually
472 2012-05-19 23:01:55 <luke-jr> gmaxwell: bitcoind can talk to 2 separate "upstreams"?
473 2012-05-19 23:02:06 <luke-jr> gmaxwell: or better yet, make bitcoind modular&
474 2012-05-19 23:02:14 <luke-jr> gmaxwell: the rpcdump.c seems to be a step toward that
475 2012-05-19 23:02:22 <gmaxwell> or make the wallet just proxy some rpc calls to the chainserver.
476 2012-05-19 23:02:51 <luke-jr> gmaxwell: seems uglier to proxy
477 2012-05-19 23:02:59 <luke-jr> but maybe not
478 2012-05-19 23:03:51 <sipa> jgarzik: i guess putting network behind the blockchain manager is meaningful; but remember that we may one day want to swap the blockchain component with another (SPV mode, to be specific)
479 2012-05-19 23:03:52 <luke-jr> if it's a C++ API, proxying is IMO ugly; but when we go to a network stack, the protocol can be designed in a way that there's different directories
480 2012-05-19 23:04:22 <luke-jr> aha, figured out the blockindex thing
481 2012-05-19 23:04:34 <luke-jr> latest blockindex branch is OK, but the one I merged before was buggy
482 2012-05-19 23:04:39 <gmaxwell> well, if the wallet itself wands to make those calls for the UI they have to cross the link ... and I think it would be annoying to figure out which one to ask. (Also from a security management perspective I think it's imporant to now have a bunch of ports which must be managed/secured)
483 2012-05-19 23:04:44 <luke-jr> so I guess jgarzik fixed it
484 2012-05-19 23:05:03 <sipa> jgarzik: so it feels easier to me to connect wallet/net/chain to one hub - you could drop the chain one, and things should keep functioning at least marginally
485 2012-05-19 23:05:18 <BlueMatt> gmaxwell: I would assume gui would sit on top of/control wallet directly, then point it to+connect to a blockstore backend
486 2012-05-19 23:05:58 <gmaxwell> BlueMatt: right, and RPC? I assume it would talk to a wallet (it must) but then there are a bunch of rpccalls like getblock which are purely backend calls.
487 2012-05-19 23:06:10 <BlueMatt> well rpc is an entirely other story...
488 2012-05-19 23:06:11 <luke-jr> 35280ac <-- ORPHAN BLOCK + crash
489 2012-05-19 23:10:41 <sipa> how about one can configure different rpc users, each linked to one wallet?
490 2012-05-19 23:11:24 <BlueMatt> or...if we assume wallets are gonna be run in another process from backend anyway (I hope they will, someday), the wallet can handle the rpc and just pass things on to the backend when needed
491 2012-05-19 23:11:36 <BlueMatt> ugly, but...
492 2012-05-19 23:11:48 <c_k> BlueMatt: Version messages aren't displayed correctly in bitcoind from stable ppa
493 2012-05-19 23:11:55 <BlueMatt> hm?
494 2012-05-19 23:12:01 <gmaxwell> sipa: one usecase for wallet backed split is that the wallet talks to a hosted backed run by someone trusted to not defraud you.  I don't think you'd want to proxy all your RPC through them, since they shouldn't have access to spend your funds.
495 2012-05-19 23:12:10 <BlueMatt> oh, that might make sense...
496 2012-05-19 23:12:22 <BlueMatt> c_k: what do they show, exactly?
497 2012-05-19 23:12:26 <gmaxwell> (I expect that that would be very popular for VPS services the vps could 0wn you anyways, why not use their blockchain server?)
498 2012-05-19 23:12:27 <BlueMatt> and isnt it broken for bitcoin-qt too?
499 2012-05-19 23:12:40 <sipa> BlueMatt: in that case the wallet rpcs must move to those other processes as well, imho
500 2012-05-19 23:12:46 <c_k> I only use cli, I'll go look again
501 2012-05-19 23:12:51 <BlueMatt> gmaxwell: mmm, yea...
502 2012-05-19 23:13:06 <sipa> BlueMatt, c_k: known problem in 0.6.2
503 2012-05-19 23:13:15 <BlueMatt> sipa: yea, just wanna make sure its the same one
504 2012-05-19 23:13:21 <sipa> it's fixed for 0.7.0
505 2012-05-19 23:13:28 <gmaxwell> c_k: does it call itself CLIENT_VERSION ?
506 2012-05-19 23:15:47 <c_k> Yeah Bitcoin version vCLIENT_VERSION_MAJOR.CLIENT_VERSION_MINOR.CLIENT_VERSION_REVISION.CLIENT_VERSION_BUILD-g8ff1873-beta
507 2012-05-19 23:16:03 <c_k> At top when doing bitcoind -help
508 2012-05-19 23:16:18 <sipa> yup, known problem
509 2012-05-19 23:16:26 <c_k> k, cool
510 2012-05-19 23:17:21 <c_k> Another thing - is there a plan to get bitcoind included in ubuntu replaces with your stable releases? It's still at 0.3.24 -_-
511 2012-05-19 23:17:57 <BlueMatt> c_k: not really anything I can do about that, have to talk to jonas (the debian packager of bitcoind)
512 2012-05-19 23:17:57 <luke-jr> c_k: that's Debian's fault
513 2012-05-19 23:18:14 <c_k> Ah
514 2012-05-19 23:18:27 <luke-jr> 0.3.24 was the last version before we added the test suite
515 2012-05-19 23:18:45 <luke-jr> now Debian is "omg 0.4.0+ breaks with big endian oh noes!"
516 2012-05-19 23:18:58 <c_k> Of course, damn debian and it's glacial speed
517 2012-05-19 23:19:08 <c_k> Oh I see
518 2012-05-19 23:19:12 <luke-jr> (even though 0.3.24 and earlier were still broken - we just didn't test it)
519 2012-05-19 23:19:17 <BlueMatt> its in unstable, just not testing...
520 2012-05-19 23:19:20 <BlueMatt> which is quite fail
521 2012-05-19 23:19:40 <luke-jr> BlueMatt: I reported it, and explained that there's known security issues with 0.3.24 etc
522 2012-05-19 23:19:54 <luke-jr> and got told off basically
523 2012-05-19 23:20:02 <luke-jr> "old versions aren't bugs"
524 2012-05-19 23:20:04 <BlueMatt> as did I
525 2012-05-19 23:20:21 <BlueMatt> though less strongly worded on both sides, I think
526 2012-05-19 23:20:48 <luke-jr> better to have nothing at all IMO
527 2012-05-19 23:20:50 <BlueMatt> agreed
528 2012-05-19 23:21:11 <gmaxwell> luke-jr: .. they had older versions working with big endian!?! bullshit
529 2012-05-19 23:21:23 <BlueMatt> gmaxwell: no, but they ship them...
530 2012-05-19 23:21:25 <gmaxwell> they might have had it successfully building (Because it has...) but not actually working.
531 2012-05-19 23:21:25 <luke-jr> gmaxwell: no, but the older versions built and didn't-test without error
532 2012-05-19 23:22:04 <BlueMatt> also, I wish jonas would give up and just use the debian folder I wrote, though he prefers using some specific building wrappers, or atleast rewrite his scripts to build bitcoin-qt...
533 2012-05-19 23:22:11 <gmaxwell> luke-jr: who is the involved debian people?
534 2012-05-19 23:22:45 <BlueMatt> http://packages.debian.org/sid/bitcoind
535 2012-05-19 23:22:47 <gmaxwell> and was this on a public list?
536 2012-05-19 23:22:50 <BlueMatt> under maintainers on the right side
537 2012-05-19 23:22:59 <luke-jr> gmaxwell: their bug tracker
538 2012-05-19 23:23:43 <BlueMatt> http://bugs.debian.org/cgi-bin/bugreport.cgi?bug=665922
539 2012-05-19 23:24:43 <luke-jr> for 0.4.x, I could do that because I maintain it& but not 0.3
540 2012-05-19 23:26:27 <c_k> I'm glad I stirred some discussion then :)
541 2012-05-19 23:26:58 <c_k> BlueMatt: Many thanks for the ppa btw :)
542 2012-05-19 23:27:01 <gmaxwell> I wish debian would stop packaging things that they won't maintain and won't let people who will maintain them maintain them.
543 2012-05-19 23:28:24 <BlueMatt> even if they refuse to upgrade the version on stable, upgrading testing would make sense...atleast then ubuntu would get upgraded
544 2012-05-19 23:29:00 <luke-jr> BlueMatt: well, testing is about to become new stable too
545 2012-05-19 23:29:11 <BlueMatt> that too
546 2012-05-19 23:29:17 <luke-jr> if they upgrade testing to 0.6.2 now, I can probably maintain 0.6.x for their 3 years
547 2012-05-19 23:29:20 <gmaxwell> yea, so the screwed up thing is that it'll also be stuck with a broken version.
548 2012-05-19 23:30:51 <jgarzik> back, though the baby might explode again :)
549 2012-05-19 23:31:26 <luke-jr> o.O
550 2012-05-19 23:31:29 <jgarzik> it's not difficult to have two kitchen sinks, instead of one:  label kitchen sink A "wallets" and kitchen sink B "block chain and network"
551 2012-05-19 23:31:41 <jgarzik> RPC and UI talk to those two entities
552 2012-05-19 23:31:55 <jgarzik> yes, the entities sometimes talk to one another
553 2012-05-19 23:32:06 <luke-jr> jgarzik: my orphan/crash issue was somehow fixed in your changed (blockindex) FWIW
554 2012-05-19 23:32:33 <jgarzik> but putting them all together into one big kitchen sink is not the right direction WRT useful security and responsibility separation
555 2012-05-19 23:32:49 <jgarzik> two kitchen sinks should be sufficient
556 2012-05-19 23:33:12 <BlueMatt> I dont see how wallets are in the cblockstore kitchen sink?
557 2012-05-19 23:33:13 <jgarzik> luke-jr: I dunno if you saw, but I just rebased extra-abort / dbenv / blockindex
558 2012-05-19 23:33:46 <BlueMatt> also, digging a bit, it looks like testing may get updated soon, the non-supported big-edian archs have been removed from testing, see: http://bugs.debian.org/cgi-bin/bugreport.cgi?bug=672802
559 2012-05-19 23:34:03 <sipa> jgarzik: wallets send transactions to network and blockchain, network sends messages to blockchain and wallets, wallet requests data from blockchain; those are for me enough to state that wallet/net/blockchain ought to communicate through one interface
560 2012-05-19 23:34:04 <luke-jr> jgarzik: yep
561 2012-05-19 23:34:17 <luke-jr> jgarzik: finishing up a new next-test build with those rebased versions
562 2012-05-19 23:34:47 <sipa> jgarzik: it's possible to separate network from that and put it behind the blockchain; i'm not sure which is best, but that is certainly reasonable
563 2012-05-19 23:35:08 <sipa> jgarzik: but i do not consider that idea a kitchen sink
564 2012-05-19 23:35:25 <sipa> just a fixed interface through which those services communicate
565 2012-05-19 23:36:11 <luke-jr> BlueMatt: any idea why they forgot to remove sparc64 hppa powerpcspe ? or what problem kfreebsd-i386 has with 0.6.1+?
566 2012-05-19 23:36:17 <jgarzik> sipa: network sends messages to blockchain server, which may be providing services for one or more wallets
567 2012-05-19 23:36:25 <luke-jr> http://packages.debian.org/sid/bitcoind
568 2012-05-19 23:36:34 <BlueMatt> luke-jr: nfc
569 2012-05-19 23:36:42 <sipa> jgarzik: as i said, that is a reasonable alternative
570 2012-05-19 23:36:43 <jgarzik> sipa: network may be entirely encapsulated within "blockchain server" concept, and need not be a separate entity (gmaxwell convinced me of this yesterday on IRC)
571 2012-05-19 23:36:52 <BlueMatt> luke-jr: arent their buillogs public?
572 2012-05-19 23:37:20 <luke-jr> hmm, not sure
573 2012-05-19 23:37:33 <gmaxwell> luke-jr: also, do our unit tests fail on BE?  e.g. do any try to validate the genesis block (I know that fails on BE)
574 2012-05-19 23:37:51 <luke-jr> gmaxwell: they do fail, that's why 0.4.0 wasn't allowed into testing
575 2012-05-19 23:37:55 <gmaxwell> If so we should try to get debian to run the tests after build so they'll stop thinking that BE works.
576 2012-05-19 23:37:59 <gmaxwell> ah!
577 2012-05-19 23:38:16 <gmaxwell> hilarious.
578 2012-05-19 23:38:58 <jgarzik> serialize stuff is already pretty regimented
579 2012-05-19 23:39:14 <sipa> BlueMatt: yes, that would be very nice, but that wouldn't be P2P protocol in between them
580 2012-05-19 23:40:14 <jgarzik> the conceptual model of RPC and UI being clients of the internal chain server and wallet server(s) seems effective and powerful
581 2012-05-19 23:40:38 <sipa> jgarzik: absolutely, no discussion about that
582 2012-05-19 23:40:39 <luke-jr> https://buildd.debian.org/status/fetch.php?pkg=bitcoin&arch=kfreebsd-i386&ver=0.6.2.2-1&stamp=1337043580
583 2012-05-19 23:40:50 <BlueMatt> sipa: doesnt really matter much to me what protocol would be between them, Im thinking of cblockstore as where one would split bitcoin if it were split into separate processes.  If I were running a huge bitcoin exchange, and it were an option, I would prefer to have p2p client outside my firewall, and then just a pinhole to the chain storage manager, to multiple p2p clients, ie one with a publicly released ip, others without
584 2012-05-19 23:40:53 <luke-jr> it seems somehow the test suite is failing to find the script json files on bsd
585 2012-05-19 23:44:52 <sipa> gmaxwell: sounds like you would want to comment on BlueMatt's idea, then
586 2012-05-19 23:52:09 <sipa> one argument in favor of not hiding network behind blockchain, is if someday memory pool management gets split from blockchain management
587 2012-05-19 23:52:51 <sipa> one could run a blockchain somewhere in the network, and have several wallet-handler machines connected to it, but each wallet handler has its own memory pool
588 2012-05-19 23:54:14 <sipa> apart from that, i'm not sure, and putting network behind chainmgmt feels easier
589 2012-05-19 23:55:21 <sipa> that said, cblockstore could evolve to just be the message dispatcher between wallets and blockchain
590 2012-05-19 23:55:45 <sipa> and that's certainly easier to implement
591 2012-05-19 23:58:00 <BlueMatt> Im not actually so sure putting network behind chain mgmt is easier, its quite trivial to split the two
592 2012-05-19 23:59:12 <sipa> not that part, but if network is behind the chain, the chain doesn't need to be a normal client to hub - it could be just the hub itself