1 2013-08-27 00:05:00 <Diablo-D3> http://intercaloninterstates.org/
  2 2013-08-27 00:07:10 <phantomcircuit> Diablo-D3, wat
  3 2013-08-27 00:07:21 <phantomcircuit> please tell me someone actually implemented that
  4 2013-08-27 00:07:22 <phantomcircuit> lol
  5 2013-08-27 01:58:45 <sipa> maaku: getnewaddress will get an address from the pool and mark it used
  6 2013-08-27 03:31:44 <maaku> sipa: when i call getnewaddress, the new address shows up in the address book, which is annoying
  7 2013-08-27 03:31:49 <maaku> is there a way around that?
  8 2013-08-27 03:32:15 <warren> Is the 0.8.2+ getwork problem filed or described anywhere?  I think doublec and a few others have mentioned parts of it.
  9 2013-08-27 03:33:04 <maaku> i want the address to act like a change address - marked as used, part of the wallet, but otherwise ignored
 10 2013-08-27 05:36:07 <sipa> maaku: there is no wqy to remove addresses in any case
 11 2013-08-27 05:36:46 <sipa> maaku: removing them fdrom just the address book shouldn't hurt, but i don't think anyone bothered
 12 2013-08-27 06:32:01 <Freebitcoinslist> is anyone here awake?
 13 2013-08-27 06:33:06 <Freebitcoinslist> I need some help with the BTC source
 14 2013-08-27 06:35:58 <Freebitcoinslist> anyone here
 15 2013-08-27 06:40:57 <fanquake> Freebitcoinslist Just ask your question
 16 2013-08-27 06:43:33 <Freebitcoinslist> I am looking into releasing a new coin and would like a lil help into where in the source code where to change the Algo used for the Coin as I am wanting to try this coin with a total different algo one that is not used on any coin and i am looking for any papers or sites that might help me reprogram the source to release this coin on the testnet
 17 2013-08-27 06:48:13 <gavinandresen> ???. maybe we should change the channel topic to "No altcoin support"
 18 2013-08-27 06:50:12 <Freebitcoinslist> Sorry I am wanting to do this on a private test net just to better understand BTC and have some fun
 19 2013-08-27 06:50:48 <Freebitcoinslist> and Nice to meet you Gavin I have read alot of articles about you and some of the comments you have made about BTC
 20 2013-08-27 06:51:49 <gavinandresen> Freebitcoinslist: the bits of the source code that are if (TestNet()) ??? are a good place to start.
 21 2013-08-27 06:52:27 <gavinandresen> I believe the bitcoinj source code is easier to follow, too. The reference code wasn't written to be easy to modify.
 22 2013-08-27 06:53:52 <Freebitcoinslist> so there it will let me change the algo used instead of Sha-256?
 23 2013-08-27 06:53:59 <gavinandresen> nope
 24 2013-08-27 06:55:46 <sipa> gavinandresen: opinion about leveldb 1.13 in 0.8.4?
 25 2013-08-27 06:56:23 <gavinandresen> sipa: no opinion yet-- what do you think?
 26 2013-08-27 06:57:15 <sipa> gavinandresen: i'd wait on more reports of corruption on osx, but if there is no clear "problem is gone!", i think we should try whatever can help to reduce the problem
 27 2013-08-27 06:57:22 <sipa> (1.13 has some extra sync() calls)
 28 2013-08-27 06:57:59 <gavinandresen> I should run a stress-test overnight; something like kill -9 bitcoind, then restart, then sleep 600???..
 29 2013-08-27 06:58:01 <phantomcircuit> sipa, what change was made to fix that?
 30 2013-08-27 06:58:24 <sipa> it obviously cannot fix the mid-runtime corruptions that we sometimes see
 31 2013-08-27 06:58:40 <sipa> but perhaps the ones from a system going in standby or unclean shutdown
 32 2013-08-27 06:59:10 <gavinandresen> mmm.  I'd really like to get an even-somewhat-reproduceable test case
 33 2013-08-27 06:59:16 <sipa> yes, me too
 34 2013-08-27 07:00:29 <gavinandresen> ok, I'll run stress tests on 0.8.3 and git HEAD tonight, see if I can get one or both to corrupt
 35 2013-08-27 07:00:41 <sipa> 0.8.4 or git head?
 36 2013-08-27 07:00:56 <gavinandresen> shouldn't be much difference, yes?
 37 2013-08-27 07:00:57 <phantomcircuit> what filesystems has that corruption been seen on?
 38 2013-08-27 07:01:07 <Diablo-D3> which corruption?
 39 2013-08-27 07:01:07 <gavinandresen> phantomcircuit: OSX
 40 2013-08-27 07:01:10 <Diablo-D3> the sleep corruption?
 41 2013-08-27 07:01:11 <Diablo-D3> osx
 42 2013-08-27 07:01:22 <sipa> phantomcircuit: i don't think there is a "that corruption" - i've heard it reported on debian, windows and OSX
 43 2013-08-27 07:01:23 <phantomcircuit> gavinandresen, right but are we 100% sure they were all UDF?
 44 2013-08-27 07:01:32 <sipa> but there may be several issues here
 45 2013-08-27 07:01:48 <phantomcircuit> sipa, well the cursory review of leveldb consistency schema scared me quite a bit
 46 2013-08-27 07:01:48 <sipa> ... UDF?
 47 2013-08-27 07:01:51 <Diablo-D3> phantomcircuit: you mean hfs+ I think
 48 2013-08-27 07:01:53 <gavinandresen> UDF??? unidentified different files?
 49 2013-08-27 07:02:02 <sipa> UDF is the filesystem used on DVD's
 50 2013-08-27 07:02:08 <phantomcircuit> universal disk format it's dvd/cds and stuff
 51 2013-08-27 07:02:10 <Diablo-D3> phantomcircuit: UDF is the new file system that replaced iso9660
 52 2013-08-27 07:02:15 <phantomcircuit> Diablo-D3, i know
 53 2013-08-27 07:02:18 <phantomcircuit> also
 54 2013-08-27 07:02:20 <phantomcircuit> i meant HFS
 55 2013-08-27 07:02:22 <Diablo-D3> hfs+ is what osx uses
 56 2013-08-27 07:02:34 <Diablo-D3> did you know hsf+ supports volume sizes up to 8 exabytes?
 57 2013-08-27 07:02:42 <gavinandresen> there are almost certainly multiple bug/problems/hardware issues here
 58 2013-08-27 07:02:43 <phantomcircuit> i did not
 59 2013-08-27 07:02:52 <Diablo-D3> I dont get the point, its sure as hell no zfs
 60 2013-08-27 07:02:53 <phantomcircuit> example of things leveldb doesn't protect against
 61 2013-08-27 07:03:06 <gavinandresen> ??? but we've seen lots of reports of leveldb corruption on OSX, and I've seen it twice on otherwise reliable machines
 62 2013-08-27 07:03:10 <phantomcircuit> phantom writes, shorn writes, relocated writes, serializable write failures
 63 2013-08-27 07:03:17 <Diablo-D3> gavinandresen: btw, I said this the other day but you wernt here
 64 2013-08-27 07:03:24 <Diablo-D3> gavinandresen: osx has apis to deal with power management
 65 2013-08-27 07:03:36 <Diablo-D3> gavinandresen: for the short term, the fix is to close the files and then reopen them on wake
 66 2013-08-27 07:03:36 <sipa> phantomcircuit: its writes are synchronized
 67 2013-08-27 07:03:38 <phantomcircuit> most of which could be solved by adding a sequentially incrementing id to journal entries
 68 2013-08-27 07:03:45 <Diablo-D3> sipa: that may not be true
 69 2013-08-27 07:03:51 <Diablo-D3> sipa: its REQUESTING sync writes
 70 2013-08-27 07:03:52 <phantomcircuit> sipa, yeah except actual disks are stupid and make mistakes all the time
 71 2013-08-27 07:03:55 <Diablo-D3> doesnt mean osx doesnt lie
 72 2013-08-27 07:03:57 <sipa> Diablo-D3: that's not what i mean
 73 2013-08-27 07:04:07 <sipa> Diablo-D3: i mean it never writes data that depends on yet unwritten data
 74 2013-08-27 07:04:12 <Diablo-D3> ahh
 75 2013-08-27 07:04:15 <Diablo-D3> sipa: but it cant tell
 76 2013-08-27 07:04:26 <phantomcircuit> sipa, yeah i know i went through the journal schema a while ago
 77 2013-08-27 07:04:28 <Diablo-D3> since there is no hardware atomic transactional layer the whole way into the drive
 78 2013-08-27 07:04:43 <phantomcircuit> the problem is relying 100% on writes to happen in the correct order is a bad idea
 79 2013-08-27 07:04:43 <sipa> still
 80 2013-08-27 07:04:53 <Diablo-D3> which I dont know why we dont have that yet
 81 2013-08-27 07:04:56 <Diablo-D3> thats just asking for trouble
 82 2013-08-27 07:05:00 <sipa> all that cannot explain why databases corrupt right in the middle of execution
 83 2013-08-27 07:05:03 <sipa> during a reindex
 84 2013-08-27 07:05:12 <sipa> unless hardware error
 85 2013-08-27 07:05:16 <Diablo-D3> sipa: maybe a legitimate software bug?
 86 2013-08-27 07:05:24 <sipa> Diablo-D3: that's what i want to know
 87 2013-08-27 07:05:31 <phantomcircuit> sipa, i was going to say that actually it could explain a lot of things
 88 2013-08-27 07:05:34 <Diablo-D3> I mean, if osx really had this corrupt on sleep bug
 89 2013-08-27 07:05:40 <Diablo-D3> other people with other software would have noticed it
 90 2013-08-27 07:05:57 <sipa> well bitcoin does put some unusual load on desktop systems
 91 2013-08-27 07:05:59 <Diablo-D3> and I know a few osx devs, they've never mentioned such an oddity
 92 2013-08-27 07:06:08 <Diablo-D3> and Ive heard some pretty scary stories from them on other osx aspects
 93 2013-08-27 07:06:46 <phantomcircuit> sipa, adding a sequential id to journal entries would be a fairly simple way to identify if this is being caused by some bizarre error outside bitcoin or if it's an internal logic issue
 94 2013-08-27 07:07:12 <sipa> phantomcircuit: agree, that's something i don't understand
 95 2013-08-27 07:07:17 <phantomcircuit> Diablo-D3, that fdatasync exists but does nothing says pretty much all i need to know about how os x actually works :/
 96 2013-08-27 07:07:20 <Diablo-D3> 
ACTION honestly has no clue how osx _exists_ sometimes from the stories hes heard

 97 2013-08-27 07:07:35 <Diablo-D3> phantomcircuit: it does nothing on linux too
 98 2013-08-27 07:07:38 <sipa> phantomcircuit: also, the fact that it tries to continue reading after seeing a corrupted sub-record is weird to me
 99 2013-08-27 07:07:52 <Diablo-D3> phantomcircuit: and, as a side note, you cannot use fdatasync on /dev/null.
100 2013-08-27 07:08:04 <sipa> define "cannot"
101 2013-08-27 07:08:08 <sipa> toffoo: present?
102 2013-08-27 07:08:20 <Diablo-D3> sipa: on linux it bitches that you cant, but otherwise works
103 2013-08-27 07:08:27 <Diablo-D3> sipa: btw, you know what this almost sounds like?
104 2013-08-27 07:08:32 <Diablo-D3> somebody isnt trapping an error
105 2013-08-27 07:08:32 <sipa> define "bitches"
106 2013-08-27 07:08:48 <Diablo-D3> like, previous op errored, error is ignored, leveldb keeps going
107 2013-08-27 07:08:58 <phantomcircuit> Diablo-D3, you sure about that, im pretty sure fdatasync does exactly what fsync does except doesn't update the metadata unless the filesize has grown larger
108 2013-08-27 07:09:19 <phantomcircuit> unless you're talking about the disk write cache
109 2013-08-27 07:09:25 <phantomcircuit> in which case all bets are off
110 2013-08-27 07:10:07 <phantomcircuit> sipa, leveldb doesn't really seem all that complicated... i wonder if it would be easier to just write something new with the same basic design but a ton more consistency checks
111 2013-08-27 07:10:58 <Diablo-D3> sipa: do something like dd if=/dev/zero of=/dev/null bs=16k count=256k conv=fdatasync
112 2013-08-27 07:11:12 <Diablo-D3> says its an invalid argument for /dev/null
113 2013-08-27 07:11:54 <phantomcircuit> Diablo-D3, which sort of makes sense
114 2013-08-27 07:12:01 <phantomcircuit> since you cant flush /dev/null
115 2013-08-27 07:12:04 <Diablo-D3> yeah /dev/null is beyond magic
116 2013-08-27 07:12:17 <Diablo-D3> where /dev/zero is a file of infinite zeros
117 2013-08-27 07:12:25 <Diablo-D3> /dev/null is a file of a single EOF.
118 2013-08-27 07:14:15 <Diablo-D3> sipa: I dunno, I really think Im right given the details
119 2013-08-27 07:14:25 <Diablo-D3> sipa: a file op is erroring and not being trapped
120 2013-08-27 07:14:44 <Diablo-D3> sipa: its the only way that I can think of to fit all the evidence
121 2013-08-27 07:14:44 <phantomcircuit> hmm
122 2013-08-27 07:14:54 <Diablo-D3> especially the whole corrupts and keeps going part
123 2013-08-27 07:15:06 <phantomcircuit> maybe we should just go through the leveldb tree and add as much additional error handling logic as possible
124 2013-08-27 07:15:13 <phantomcircuit> and when i say handling i mean panic and close
125 2013-08-27 07:15:21 <Diablo-D3> phantomcircuit: this is why I refuse to do C and C++ anymore
126 2013-08-27 07:15:35 <Diablo-D3> I mean, Ive always refused to do C++, but you know what I mean
127 2013-08-27 07:15:39 <phantomcircuit> Diablo-D3, well afaict leveldb wasn't intended to be a "serious" durable database
128 2013-08-27 07:15:51 <phantomcircuit> the design just isn't right for that
129 2013-08-27 07:15:56 <Diablo-D3> yes, and C wasnt intended to be a serious durable language
130 2013-08-27 07:16:00 <phantomcircuit> it is however spot on for being ludicrous speed
131 2013-08-27 07:16:02 <Diablo-D3> so why do we keep using it
132 2013-08-27 07:16:06 <sipa> phantomcircuit: not sure
133 2013-08-27 07:16:22 <sipa> phantomcircuit: the whole write-all-data-twice seems a clear design choice for consistency in favor of speed
134 2013-08-27 07:16:24 <phantomcircuit> sipa, the lack of sequence id's is baffling
135 2013-08-27 07:16:37 <sipa> it's in the details that things seem to be missing
136 2013-08-27 07:16:42 <Diablo-D3> sipa: okay so wait
137 2013-08-27 07:16:43 <Diablo-D3> rewind a bit
138 2013-08-27 07:16:50 <Diablo-D3> who provides the journal
139 2013-08-27 07:16:51 <Diablo-D3> us or them
140 2013-08-27 07:16:53 <phantomcircuit> sipa, the basic overall design seems to be pretty solid
141 2013-08-27 07:16:56 <sipa> Diablo-D3: them
142 2013-08-27 07:17:03 <Diablo-D3> how many files does leveldb use?
143 2013-08-27 07:17:08 <sipa> many
144 2013-08-27 07:17:12 <phantomcircuit> but honestly i wrote a journaling class in java that is way more reliable in a much shorter amount of code
145 2013-08-27 07:17:16 <Diablo-D3> when does it create a new file?
146 2013-08-27 07:17:24 <sipa> on compactions
147 2013-08-27 07:17:32 <Diablo-D3> when does reindexing happen?
148 2013-08-27 07:17:39 <phantomcircuit> sequence_id/payload_length/checksum(s)/payload
149 2013-08-27 07:17:46 <sipa> when you start bitcoin with -reindex
150 2013-08-27 07:18:03 <phantomcircuit> there is effectively no way for corruption to occur there beyond the sequence id being corrupted such that two records are perfectly swapped
151 2013-08-27 07:18:05 <Diablo-D3> so the corruption is only being detected during reindex?
152 2013-08-27 07:18:09 <phantomcircuit> which is just ludicrously unlikely to happen
153 2013-08-27 07:18:12 <Diablo-D3> or is it being caused there?
154 2013-08-27 07:18:12 <sipa> Diablo-D3: ???
155 2013-08-27 07:18:26 <Diablo-D3> did I misunderstand the question?
156 2013-08-27 07:18:29 <phantomcircuit> Diablo-D3, it's more likely to happen but isn't the only time it happens
157 2013-08-27 07:18:35 <Diablo-D3> phantomcircuit: k
158 2013-08-27 07:18:39 <phantomcircuit> which is why he's ??? you
159 2013-08-27 07:18:52 <phantomcircuit> which is why im thinking it's a journaling bug in leveldb
160 2013-08-27 07:18:54 <Diablo-D3> I heard reindexing mentioned earlier
161 2013-08-27 07:18:55 <sipa> i'm just saying that we have evidence of people doing a reindex (=wipe the database as start from scratch), and seeing a corruption later on... without any power cycling or standby or even exiting the program in between
162 2013-08-27 07:19:06 <Diablo-D3> sipa: yes which will be hard
163 2013-08-27 07:19:06 <phantomcircuit> since reindexing is where there is the absolute highest turn over for the journal
164 2013-08-27 07:19:16 <Diablo-D3> I think the entire file IO code needs to be ripped out
165 2013-08-27 07:19:32 <Diablo-D3> and then replaced with code that is bullet proof
166 2013-08-27 07:19:36 <sipa> where?
167 2013-08-27 07:19:37 <phantomcircuit> sipa, what's the format of the table files? (high level overview)
168 2013-08-27 07:19:42 <Diablo-D3> all over leveldb
169 2013-08-27 07:19:51 <Diablo-D3> everywhere it opens a file, closes a file, reads a file, writes a file
170 2013-08-27 07:19:55 <sipa> all IO code in leveldb is abstracted into an environment class
171 2013-08-27 07:20:00 <sipa> you can write your own if you want
172 2013-08-27 07:20:03 <Diablo-D3> sipa: so...
173 2013-08-27 07:20:07 <Diablo-D3> how the hell is it not error checking?
174 2013-08-27 07:20:13 <sipa> it is
175 2013-08-27 07:20:15 <Diablo-D3> does that code have no error checking at all?
176 2013-08-27 07:20:31 <Diablo-D3> because for me to be right, it has to have not trapped and error and then kept going
177 2013-08-27 07:20:48 <sipa> it has error detection all over the place
178 2013-08-27 07:20:57 <sipa> look at src/leveldb/util/env_posix.cc
179 2013-08-27 07:21:09 <Diablo-D3> sipa: then I dont think Im right
180 2013-08-27 07:21:31 <sipa> phantomcircuit: sorted list of key-value pairs, with an index at the end
181 2013-08-27 07:21:33 <Diablo-D3> either that, or an error is happening and not returning an error
182 2013-08-27 07:21:36 <sipa> and a bloom filter
183 2013-08-27 07:21:39 <Diablo-D3> which is highly unlikely
184 2013-08-27 07:21:59 <phantomcircuit> Diablo-D3, https://leveldb.googlecode.com/svn/trunk/helpers/memenv/memenv.cc + https://leveldb.googlecode.com/svn/trunk/util/env_posix.cc
185 2013-08-27 07:22:04 <Diablo-D3> sipa: does leveldb mmap everything or not?
186 2013-08-27 07:22:10 <sipa> some things
187 2013-08-27 07:22:17 <sipa> on 64-bit systems
188 2013-08-27 07:22:26 <sipa> maybe we should try disabling that on OSX
189 2013-08-27 07:22:30 <Diablo-D3> does it ever mmap then not mmap a f ile later?
190 2013-08-27 07:22:50 <sipa> Diablo-D3: IIRC yes, bu it closes and syncs in between
191 2013-08-27 07:22:56 <Diablo-D3> sipa: thats still bad vooodoo
192 2013-08-27 07:23:06 <sipa> phantomcircuit: sstable files are immutable, so if they are corrupted, it means they are corrupted at write time
193 2013-08-27 07:23:25 <Diablo-D3> sipa: try disabling mmap on OSX and see if it goes away
194 2013-08-27 07:23:52 <phantomcircuit> sipa, so i figured out why it's skipping invalid log entries
195 2013-08-27 07:24:10 <phantomcircuit> sipa, check env_posix.cc class PosixMmapFile : public WritableFile {
196 2013-08-27 07:24:13 <phantomcircuit> the comment right above it
197 2013-08-27 07:24:46 <sipa> phantomcircuit: zero _suffixes_
198 2013-08-27 07:24:59 <sipa> which means that if you encounter one, you can stop reading, afaik
199 2013-08-27 07:25:01 <phantomcircuit> sipa, which are invalid records i believe
200 2013-08-27 07:25:12 <phantomcircuit> so it's being lazy and just ignoring them
201 2013-08-27 07:25:24 <sipa> so ok, skip zero records
202 2013-08-27 07:25:25 <phantomcircuit> instead of verifying that the end of the journal is all 0
203 2013-08-27 07:25:37 <sipa> still no point in skipping things with an invalid checksum (that are nonzero)
204 2013-08-27 07:26:21 <Diablo-D3> I agree with phantomcircuit
205 2013-08-27 07:26:29 <Diablo-D3> that code isnt robust enough
206 2013-08-27 07:26:46 <sipa> still none of this explains mid-runtime corruptions