1 2013-07-13 02:35:03 <TradeFortress> bitcoind STILL keeps deadlocking even after I moved it to another server
2 2013-07-13 03:07:40 <helo> TradeFortress: your use case must be fairly unique, as i haven't seen anyone else mention deadlocking problems
3 2013-07-13 03:08:12 <TradeFortress> helo, I don't think it's very unique. It looks like it's caused by getbalance after a sendtoaddress.
4 2013-07-13 03:08:18 <TradeFortress> Or a getbalance during a sendtoaddress?
5 2013-07-13 03:08:31 <helo> TradeFortress: but i'm sure that if you can give enough details for others to reproduce it, it will get fixed
6 2013-07-13 03:09:09 <TradeFortress> The thing is this happens on a prod enviroment, and I cannot wait 30 minutes for someone to tell me what to do with gdb
7 2013-07-13 03:09:24 <helo> so you're following a sendtoaddress immediately with a getbalance?
8 2013-07-13 03:11:02 <helo> yeah i've been there... no fun... i'll load up a debugging version on testnet and bang on it with some scripts. how many addresses do you have?
9 2013-07-13 03:11:22 <helo> and what version?
10 2013-07-13 03:12:00 <helo> did you build it?
11 2013-07-13 03:15:02 <helo> all of the devs are out rolling down the strip in their rolls-royces, but i'll give it a shot
12 2013-07-13 03:16:08 <gmaxwell> helo: pft, nah, we're just eating popcoin and waiting for the y'all to step up and produce a reproducable test case. :P
13 2013-07-13 03:17:43 <TradeFortress> helo, I'm not following that, it's a web app and there often are getbalance calls
14 2013-07-13 03:17:54 <TradeFortress> I've tried v0.8.3 and master
15 2013-07-13 03:18:05 <TradeFortress> 1k addresses
16 2013-07-13 03:20:24 <helo> did you compile both?
17 2013-07-13 03:21:21 <TradeFortress> yes, I compile mine
18 2013-07-13 03:21:27 <helo> what platform?
19 2013-07-13 03:21:47 <helo> can you pastebin the output of "ldd bitcoind"
20 2013-07-13 03:26:38 <helo> you may want to try using the released pre-compiled binary, as it is statically linked against the "right" lib versions
21 2013-07-13 03:27:33 <TradeFortress> ubuntu 12.04 LTS
22 2013-07-13 03:27:38 <TradeFortress> let me try that
23 2013-07-13 03:28:12 <TradeFortress> helo: pastebin.com/7Y3nBcGE
24 2013-07-13 03:30:10 <helo> the wallet's still using libdb, and that's linked to a different version that all of the precompiled (i.e. most well tested) releases
25 2013-07-13 03:32:01 <gmaxwell> good spotting, but its still worth trying to get a reproduction under both to see if that makes a difference, unfortunately TradeFortress can't run the precompiled binaries due to him using the bdb5.
26 2013-07-13 03:32:08 <gmaxwell> (since his wallet will be incompatible)
27 2013-07-13 03:32:21 <gmaxwell> (5 can read 4 but not vice versa, it only goes one way)
28 2013-07-13 03:32:23 <helo> yeah, that is a problem...
29 2013-07-13 03:32:29 <TradeFortress> So my wallet.dat is incompatible?
30 2013-07-13 03:32:42 <helo> if you use 5, it upgrades your wallet
31 2013-07-13 03:32:57 <TradeFortress> OK, so I download a precompiled v0.8.3 ?
32 2013-07-13 03:33:50 <helo> gmaxwell: he could start up a v0.8.3 new wallet, and import all of his addresses to have a db4 wallet?
33 2013-07-13 03:34:08 <gmaxwell> helo: there is no easy and reliable way to "import all of his addresses to have a db4 wallet"
34 2013-07-13 03:35:58 <TradeFortress> grweat
35 2013-07-13 03:37:41 <TradeFortress> I'm guessing I'm fairly screwed then?
36 2013-07-13 03:39:09 <helo> gmaxwell: listreceivedbyaddress to get the addresses, dumpprivkey to get they keys, and then importprivkey into an unsynched 0.8.3 wallet, and then sync?
37 2013-07-13 03:43:18 <TradeFortress> ACTION will start a 10 btc bounty: fix my bitcoind's troubles :P
38 2013-07-13 03:44:18 <helo> 10btc bounty sounds pretty contradictory to what gmaxwell just said ;)
39 2013-07-13 03:45:48 <helo> this is all just to see if it is reproducable on the production build, too...
40 2013-07-13 03:46:29 <TradeFortress> I don't think this is omething that will take 5 days :P
41 2013-07-13 03:51:18 <helo> afaik (cue gmaxwell explaining otherwise) getting your owned addresses into a v4 wallet seems feasible
42 2013-07-13 03:53:20 <helo> ACTION adds 10k addresses to his testnet v5 wallet
43 2013-07-13 03:56:12 <TradeFortress> helo, try doing a sendtoaddress and calling getbalance at the same time?
44 2013-07-13 04:03:43 <TradeFortress> yeah, corrupt wallet when I try the ppa builds
45 2013-07-13 04:04:57 <helo> i know how to get your addresses that have received balances
46 2013-07-13 04:06:02 <TradeFortress> helo, I need *all* my addresses.
47 2013-07-13 04:08:05 <helo> are you using the accounts feature?
48 2013-07-13 04:08:37 <TradeFortress> yes
49 2013-07-13 04:09:25 <helo> so your ~1k address are distributed among a bunch of accounts?
50 2013-07-13 04:10:24 <helo> how many accounts?
51 2013-07-13 04:10:33 <helo> how many addresses per account etc
52 2013-07-13 04:24:26 <helo> are you using accounts in bitcoin to correspond to individual customer accounts?
53 2013-07-13 04:37:46 <TradeFortress> helo, yes
54 2013-07-13 04:38:22 <TradeFortress> 1300 accounts
55 2013-07-13 04:38:29 <TradeFortress> each account in bitcoin == 1 user account
56 2013-07-13 04:40:00 <helo> TradeFortress: you can try installing the db5.1-util, and then (after shutting down bitcoind so the db file is closed) doing db5.1_dump wallet.dat.db5 > wallet.dat.dump. then if you can install db4.8-util, you may be able to cat wallet.dat.dump | db4.8_load wallet.dat.db4 to get a db4 wallet. your service would have to be down though. and then you'd need to start up the statically linked bitcoind. it would then do a (probably long) rescan, and would hopef
57 2013-07-13 04:40:41 <TradeFortress> well not really I can just shut down bitcoind for like less than a minute, get a wallet and do operations on that
58 2013-07-13 04:41:20 <helo> but then how do you get what happens to the db5 wallet while you're getting the db4 instance up into the db4 wallet?
59 2013-07-13 04:41:37 <helo> do you have one address per user (and bitcoin) account?
60 2013-07-13 04:41:42 <TradeFortress> a large keypool?
61 2013-07-13 04:41:49 <TradeFortress> block generation of new addresses for a while?
62 2013-07-13 04:42:33 <TradeFortress> is it even likely that it is db5 causing the problem ??
63 2013-07-13 04:43:00 <helo> it's possible... those are wallet operations, so they're using libdb
64 2013-07-13 04:43:16 <helo> just a guess though. as i said, nobody has had this problem that i've heard
65 2013-07-13 04:48:13 <TradeFortress> what should I do when it happens again with gdb?
66 2013-07-13 05:01:54 <helo> you'd need to be running a debug build to have anything to work with
67 2013-07-13 05:02:15 <helo> and then you could look through the threads to see where they are deadlocked
68 2013-07-13 05:02:35 <helo> hopefully it won't happen with db4
69 2013-07-13 05:03:39 <helo> if users can change the address associated with their account, the activity that happens on the db5 wallet while you're getting the db4 wallet up could cause a mess
70 2013-07-13 05:04:13 <TradeFortress> helo, do you mind testing to see if it's possible to go back to db4? I'm happy to tip you for your time
71 2013-07-13 05:04:16 <helo> or if new accounts are created, etc... or other things i don't know about how your service works
72 2013-07-13 05:04:34 <helo> yeah, i just generated a db5 wallet. going to try converting to db4
73 2013-07-13 05:04:50 <TradeFortress> ok
74 2013-07-13 05:05:14 <fanquake> Is anyone building with Boost 1.54.0? Upgraded this morning to see Undefined symbols for architecture x86_64:
75 2013-07-13 05:05:15 <fanquake> boost::asio::ssl::context::context(boost::asio::ssl::context_base::method) in bitcoinrpc.o
76 2013-07-13 05:05:15 <fanquake> "_TLSv1_1_client_method", referenced from:
77 2013-07-13 05:09:34 <helo> TradeFortress: do you use sendfrom <account> <tobitcoinaddress>, or just sendtoaddress ...?
78 2013-07-13 05:09:55 <TradeFortress> sendtoaddress
79 2013-07-13 05:11:05 <TradeFortress> I know this happens after someone sends coins
80 2013-07-13 05:11:19 <TradeFortress> json-rpc becomes unresponsive and times out, then deadlock
81 2013-07-13 05:14:53 <helo> when i do sendtoaddress and then a getbalance <account>, the balance of that account doesn't change... the result of getbalance "" goes down by the amount sent
82 2013-07-13 05:15:58 <helo> or are you just doing a "getbalance" without listing the account?
83 2013-07-13 05:16:06 <helo> without *specifying
84 2013-07-13 05:16:17 <TradeFortress> I keep track of user balances in my db
85 2013-07-13 05:16:20 <TradeFortress> I don't check the balance of accounts
86 2013-07-13 05:16:22 <willg> ^^
87 2013-07-13 05:16:23 <helo> ahh ok
88 2013-07-13 05:16:41 <helo> so the getbalance is more of a sanity check
89 2013-07-13 05:17:15 <TradeFortress> nah, the getbalance is from other users opening up their wallet
90 2013-07-13 05:17:29 <TradeFortress> here is the txid that caused it to deadlock: http://blockchain.info/tx/79542ad431294752eed0f80b407e6683e874d7c2e8daa672fcb6eb582f0f384b
91 2013-07-13 05:17:35 <TradeFortress> don't see anything strange
92 2013-07-13 05:19:11 <willg> i keep track and report out of the db and give every user a separate account to use to getbalance to double check
93 2013-07-13 05:21:38 <TradeFortress> willg, the more rpc you use, the less your app scales
94 2013-07-13 05:22:08 <willg> oh for sure .. i mean on a cron timer kinda thing
95 2013-07-13 05:22:24 <willg> to look for reasons the db should update
96 2013-07-13 05:26:10 <TradeFortress> I think the problem is with sendtoaddress actaully..
97 2013-07-13 05:26:16 <TradeFortress> getbalance may or may not be related
98 2013-07-13 05:26:37 <TradeFortress> after all it fails at sendtoaddress
99 2013-07-13 05:26:41 <helo> TradeFortress: the db5.1_dump wallet.dat.5.1 > wallet.dump followed by cat wallet.dump | db4.8_load > wallet.dat.4.1 seems to have worked
100 2013-07-13 05:26:53 <helo> it loaded without corruption complaints, and the accounts are there
101 2013-07-13 05:27:05 <helo> and ~10k addresses
102 2013-07-13 05:28:03 <TradeFortress> ok, so I should use the bitcoind in the ppa?
103 2013-07-13 05:28:36 <helo> the transacitons are there
104 2013-07-13 05:28:52 <helo> well, you'll need to install db4.8_load to create the db4.8 wallet
105 2013-07-13 05:29:06 <helo> does ubuntu 12.04 have the db4.8-util package?
106 2013-07-13 05:29:33 <helo> i happen to have a debian install which includes libdb4.8
107 2013-07-13 05:30:25 <TradeFortress> yup ubuntu has both
108 2013-07-13 05:30:32 <TradeFortress> so I need the bitcoind from ppa?
109 2013-07-13 05:30:44 <helo> you can use the ppa, or you can download it directly from bitcoin.org
110 2013-07-13 05:31:12 <TradeFortress> bitcoin.org links to the ppa
111 2013-07-13 05:31:12 <TradeFortress> lol
112 2013-07-13 05:32:12 <helo> you can download the tarball from bitcoin.org
113 2013-07-13 05:32:49 <helo> be sure you use the same architecture binary (the tarball includes 32 and 64)
114 2013-07-13 05:33:25 <helo> in a production environment, i'd probably hand-install the bitcoind from the tarball so the package manager doesn't upgrade to a version that i haven't tested
115 2013-07-13 05:35:54 <TradeFortress> wallet.dat.db5 == wallet.dat, right?
116 2013-07-13 05:36:18 <helo> yeah... i'm assuming you copied the v5 wallet.dat to wallet.dat.db5 for safe keeping
117 2013-07-13 05:36:52 <helo> since you have both on the same system, you can do db5.1_dump wallet.dat.db5 | db4.8_load > wallet.dat.db4
118 2013-07-13 05:37:31 <helo> and then copy the wallet.dat.db4 to ~/.bitcoin/wallet.dat and start up the statically linked bitcoind
119 2013-07-13 05:37:55 <TradeFortress> helo, that command doesn't wrok, gives me db4.8_load usage
120 2013-07-13 05:38:23 <TradeFortress> db5.1_dump wallet.dat > db4.8_load ?
121 2013-07-13 05:38:45 <sipa> TradeFortress: sendtoaddress always deducta from the "" account
122 2013-07-13 05:38:54 <sipa> TradeFortress: so what you see is normal
123 2013-07-13 05:39:02 <TradeFortress> sipa, what I'm seeing is a deadlock
124 2013-07-13 05:39:09 <TradeFortress> when I do sendtoaddress
125 2013-07-13 05:39:10 <sipa> yes, i know that
126 2013-07-13 05:39:14 <TradeFortress> not after getbalance or whatever.
127 2013-07-13 05:39:20 <sipa> but that's a separate problem
128 2013-07-13 05:39:21 <helo> db5.1_dump wallet.dat.db5 > wallet.db.dump ; db4.8_load wallet.db.dump > wallet.dat.db4
129 2013-07-13 05:39:58 <TradeFortress> that took like a second
130 2013-07-13 05:40:18 <helo> i did cat wallet.testnet.dump.5.1 | db4.8_load wallet.dat.4.1.load
131 2013-07-13 05:40:43 <helo> and then copied wallet.dat.4.1.load to ~/.bitcoin/[testnet3]/wallet.dat
132 2013-07-13 05:41:53 <TradeFortress> ok, db4.8_load is taking a while and I think terminal stopped responding
133 2013-07-13 05:42:00 <TradeFortress> no longer blinking
134 2013-07-13 05:42:12 <helo> ignore that "db4.8_load wallet.db.dump > wallet.dat.db4" nonsense
135 2013-07-13 05:42:13 <TradeFortress> surprising since dumping it took less than a second
136 2013-07-13 05:42:19 <TradeFortress> wut
137 2013-07-13 05:42:41 <TradeFortress> Now I have wallet.dat.dump, what do I do with that
138 2013-07-13 05:43:53 <helo> sorry, i haven't used the dump/load much... what you could have done originally is simply: db5.1_dump wallet.dat.db5 | db4.8_load wallet.dat.db4
139 2013-07-13 05:44:24 <TradeFortress> unexpected file type or format
140 2013-07-13 05:45:25 <TradeFortress> nvm now I got db4. I'll put that in .bitcoin
141 2013-07-13 05:45:41 <TradeFortress> do I need to delete any directories?
142 2013-07-13 05:46:27 <helo> i'm not sure if the 0.8.3 will need to resync
143 2013-07-13 05:47:03 <TradeFortress> nope
144 2013-07-13 05:47:08 <TradeFortress> Error: Error loading wallet.dat: Wallet corrupted
145 2013-07-13 05:47:43 <helo> what does "file wallet.dat" show?
146 2013-07-13 05:48:08 <TradeFortress> version 9
147 2013-07-13 05:49:36 <helo> you're sure you shut down the bitcoind and let it sync to disk before you copied wallet.dat to wallet.dat.db5?
148 2013-07-13 05:50:00 <TradeFortress> yep, I did ./bitcoind stop and waited
149 2013-07-13 05:50:09 <TradeFortress> version 9 is the v5?
150 2013-07-13 05:50:10 <TradeFortress> or v4?
151 2013-07-13 05:50:18 <helo> my v4 says version 9
152 2013-07-13 05:50:39 <TradeFortress> well the thing is all my wallet dats are version 9.
153 2013-07-13 05:51:00 <helo> yeah, i think that's as much as the "file" util can tell
154 2013-07-13 05:51:29 <helo> how big is wallet.dat.db4 compared to wallet.dat.db5?
155 2013-07-13 05:52:49 <TradeFortress> smaller in size
156 2013-07-13 05:53:51 <TradeFortress> by like 4%
157 2013-07-13 05:54:36 <helo> mine is smaller too.
158 2013-07-13 05:55:01 <helo> but it didn't give me a corruption error, loaded up the addresses, and seems to have the right balances
159 2013-07-13 05:55:19 <TradeFortress> could it be that I'm using diff versions
160 2013-07-13 05:55:33 <TradeFortress> my 5.1 is master, I'm now trying to use the 0.8.3
161 2013-07-13 05:55:59 <helo> my 5.1 is master too
162 2013-07-13 05:56:19 <TradeFortress> well that is marvellous.
163 2013-07-13 05:57:06 <helo> your wallet is a lot more complicated than mine... i guess the dump/load could have resulted in some changes that bitcoin didn't like
164 2013-07-13 05:57:14 <TradeFortress> I can load this wallet.dat from db5.1 fine
165 2013-07-13 05:57:35 <TradeFortress> helo, maybe because your wallet didn't include any transactions
166 2013-07-13 05:57:51 <sipa> seems unlikely
167 2013-07-13 05:57:59 <TradeFortress> grr
168 2013-07-13 05:58:00 <sipa> dump/load should always work
169 2013-07-13 05:58:03 <helo> when you try to load your wallet.dat.db5 in the statically linked bitcoind, does it give the same wallet corruption error as you're getting with the converted wallet?
170 2013-07-13 05:58:12 <helo> i did a few transactions
171 2013-07-13 05:58:13 <TradeFortress> helo, let me try that
172 2013-07-13 05:58:47 <sipa> ACTION wants bdb to die a most painful death 2 years ago
173 2013-07-13 05:58:48 <helo> hopefully you made a mistake with the dump/load procedure
174 2013-07-13 05:58:58 <TradeFortress> helo, no, I didn't, I tried multiple times. got same file size
175 2013-07-13 05:59:02 <sipa> you have no idea how much issues it has caused us already
176 2013-07-13 05:59:03 <TradeFortress> and "Wallet corrupted"
177 2013-07-13 05:59:19 <TradeFortress> sipa, april fork
178 2013-07-13 05:59:32 <helo> so loading the converted db4 wallet gives the same error as the db5 wallet?
179 2013-07-13 05:59:36 <TradeFortress> yes
180 2013-07-13 05:59:57 <helo> are you sure you didn't db5.1_dump wallet.dat.db5 | db5.1_load wallet.dat.db4?
181 2013-07-13 06:00:04 <sipa> the march 11 fork, you mean?
182 2013-07-13 06:00:49 <TradeFortress> well, I might be off with the exact wording. helo I did db4.8
183 2013-07-13 06:00:53 <TradeFortress> db4.8_load
184 2013-07-13 06:01:04 <TradeFortress> with the exact timing*
185 2013-07-13 06:01:22 <sipa> helo: what version is your self compiled?
186 2013-07-13 06:01:38 <helo> sipa: HEAD
187 2013-07-13 06:01:59 <sipa> there may just be an incompatibility between master and 0.8.3
188 2013-07-13 06:02:09 <sipa> as it's not released
189 2013-07-13 06:02:49 <helo> booo
190 2013-07-13 06:03:17 <sipa> TradeFortress: try wiping your database/ subdir before load
191 2013-07-13 06:03:39 <TradeFortress> sipa, I don't have a database
192 2013-07-13 06:03:48 <sipa> ok, good
193 2013-07-13 06:04:07 <sipa> right, we clean that up automatically now
194 2013-07-13 06:04:54 <TradeFortress> I guess salvagewallet or whatever won't work?
195 2013-07-13 06:05:02 <sipa> it should
196 2013-07-13 06:05:23 <sipa> but you will lose account information
197 2013-07-13 06:05:31 <sipa> it only salvages private keys
198 2013-07-13 06:05:32 <helo> no-go :/
199 2013-07-13 06:05:36 <sipa> no transactions
200 2013-07-13 06:06:03 <TradeFortress> that is ok
201 2013-07-13 06:06:10 <TradeFortress> because my accounts have a balance of 0 anyway
202 2013-07-13 06:06:14 <TradeFortress> I move them to "sinkhole"
203 2013-07-13 06:06:25 <TradeFortress> but I will lose which private key belongs to which account?
204 2013-07-13 06:06:33 <sipa> then what do you use accounts for anyway?
205 2013-07-13 06:06:37 <sipa> yes
206 2013-07-13 06:08:18 <sipa> maybe your db4.8 load is not the same version as the bdb version the bitcoin ppa version was compiled with?
207 2013-07-13 06:08:36 <sipa> even though both are 4.8, this would not surproise me
208 2013-07-13 06:08:49 <TradeFortress> I got it from sudo apt-get install db4.8_load
209 2013-07-13 06:09:18 <helo> you could listaccounts, parse through it for addresses, dumpprivkey to get the privkey, and then importprivkey and use setaccount :(
210 2013-07-13 06:09:38 <sipa> git head does have dumpwallet and loadwallet
211 2013-07-13 06:09:48 <sipa> which does include that information
212 2013-07-13 06:10:01 <sipa> but 0.8.3 does not have that
213 2013-07-13 06:10:18 <TradeFortress> but I can just build v0.8.3 withv4.8?
214 2013-07-13 06:10:24 <sipa> yes
215 2013-07-13 06:10:39 <TradeFortress> doesn't dumpwallet just give you the same wallet.dat
216 2013-07-13 06:11:00 <sipa> no, it dumps in a human-readable format
217 2013-07-13 06:11:20 <TradeFortress> but still works as wallet.dat?
218 2013-07-13 06:11:24 <sipa> no
219 2013-07-13 06:11:31 <sipa> you need to import it
220 2013-07-13 06:11:33 <TradeFortress> odd
221 2013-07-13 06:11:42 <TradeFortress> I remember I got my current wallet.dat from a dumpwallet wallet.dat
222 2013-07-13 06:11:49 <sipa> you mean backupwallet
223 2013-07-13 06:11:54 <sipa> not dumpwallet
224 2013-07-13 06:11:56 <TradeFortress> ah backupwallet
225 2013-07-13 06:13:06 <TradeFortress> but should I be running head on prod anyway
226 2013-07-13 06:13:16 <helo> well i gotta get up in a few hours... hopefully you can get static 0.8.3 up and stop having deadlocks :/
227 2013-07-13 06:13:27 <sipa> but what are you trying? whether 0.8.3 from the ppa has the same deadlocks?
228 2013-07-13 06:13:52 <TradeFortress> sipa, I do not think I have used v0.8.3 from ppa. I always used leveldb 5
229 2013-07-13 06:14:06 <sipa> bdb you mean
230 2013-07-13 06:14:14 <helo> sipa: whether the db5.1 is causing the deadlocks, vs statically linked to 4.8
231 2013-07-13 06:14:22 <TradeFortress> bdb yup
232 2013-07-13 06:14:26 <sipa> that seems very unlikely
233 2013-07-13 06:14:42 <sipa> i'm pretty sure a deadlock would be our fdault
234 2013-07-13 06:14:50 <sipa> not bdb's
235 2013-07-13 06:14:54 <TradeFortress> well I get a deadlock with arbintary sendtoaddress faults
236 2013-07-13 06:15:03 <TradeFortress> as in unexpected
237 2013-07-13 06:15:22 <sipa> well deadlocks are always unexpected, i hope
238 2013-07-13 06:15:41 <sipa> why do you need to run head, btw?
239 2013-07-13 06:15:58 <TradeFortress> sipa, I don't need to but I can't seem to get back to 0.8.3
240 2013-07-13 06:16:05 <TradeFortress> I thought maybe head fixed the problem
241 2013-07-13 06:16:16 <sipa> i see
242 2013-07-13 06:16:36 <TradeFortress> plus I think you will be very mad at me if it turns out it was already fixed
243 2013-07-13 06:16:59 <sipa> does a self-compiled 0.8.3 work?
244 2013-07-13 06:17:18 <TradeFortress> no that had deadlocks
245 2013-07-13 06:17:27 <sipa> well so does head
246 2013-07-13 06:17:48 <TradeFortress> maybe I should go back to a self compiled 0.8.3?
247 2013-07-13 06:17:56 <TradeFortress> :/
248 2013-07-13 06:18:02 <sipa> that should be the first thing to try
249 2013-07-13 06:18:24 <TradeFortress> I started with self compiled 0.8.3
250 2013-07-13 06:18:25 <sipa> to assess whether the corruotion is because of the ppa or because of a difference between head and 0.8 e
251 2013-07-13 06:18:29 <sipa> i know
252 2013-07-13 06:18:36 <TradeFortress> ok, so how would I properly go back with git?
253 2013-07-13 06:18:51 <sipa> git checkout v0.8.3
254 2013-07-13 06:19:00 <sipa> and then rebuild
255 2013-07-13 06:20:05 <sipa> how reproducible are these deadlocks?
256 2013-07-13 06:20:26 <TradeFortress> funnily enough they only happen when I am away / asleep
257 2013-07-13 06:20:30 <sipa> as in, how frequently do they occue
258 2013-07-13 06:20:37 <TradeFortress> once every 1-2 days
259 2013-07-13 06:20:41 <sipa> ok
260 2013-07-13 06:20:54 <sipa> is your wallet encrypted?
261 2013-07-13 06:20:57 <TradeFortress> every time that happens someone gets coins sent but their balance not deducted. so I'm outta ~10 btc alreadyn
262 2013-07-13 06:20:58 <TradeFortress> no
263 2013-07-13 06:21:25 <helo> if you haven't made any progress by tomorrow, i'll try to write some scripts to hammer on my debug build and get a deadlock
264 2013-07-13 06:21:25 <sipa> ow
265 2013-07-13 06:21:26 <TradeFortress> not bitcoind's fault, my fault for assuming rpc error = coins not sent
266 2013-07-13 06:21:47 <TradeFortress> it's ok I can cover that, I spent more on other things anyway
267 2013-07-13 06:22:05 <sipa> but there is no rpc error, only a timeout, i guess?
268 2013-07-13 06:22:11 <TradeFortress> yeah, just a timeout
269 2013-07-13 06:22:20 <sipa> well that's an error too of course
270 2013-07-13 06:22:39 <TradeFortress> anyway this isn't machine specific, I've already changed
271 2013-07-13 06:23:45 <sipa> does 0.8.1 have the problem too?
272 2013-07-13 06:23:51 <TradeFortress> haven't tried v0.8.1
273 2013-07-13 06:23:54 <sipa> the deadlocks i mean
274 2013-07-13 06:23:58 <TradeFortress> and it'd be a bad idea to run that anyway
275 2013-07-13 06:24:49 <sipa> well... testing in production :)
276 2013-07-13 06:25:31 <TradeFortress> not like I have a choice haha
277 2013-07-13 06:26:42 <TradeFortress> I think it may have something to do with coin selection
278 2013-07-13 06:26:48 <TradeFortress> v0.8.3 built, will try
279 2013-07-13 06:27:34 <TradeFortress> v0.8.3: wallet corrupted
280 2013-07-13 06:29:05 <sipa> with the exact same file as one that works in head?
281 2013-07-13 06:29:11 <TradeFortress> uh huh
282 2013-07-13 06:29:24 <TradeFortress> yeah it doesn't load any wallet.dats
283 2013-07-13 06:29:32 <sipa> ok, so we have a backward compatibility problem
284 2013-07-13 06:29:38 <sipa> interesting
285 2013-07-13 06:29:51 <TradeFortress> which is probably something that should be solved before another versoin :P
286 2013-07-13 06:30:02 <sipa> of course
287 2013-07-13 06:30:08 <TradeFortress> I could build the debug version and help track it down?
288 2013-07-13 06:31:31 <TradeFortress> i probably need to go back to master. and keep running that for the time being
289 2013-07-13 06:33:22 <TradeFortress> out of the box "fix": if sendtoaddress takes more than 5 seconds, kill bitcoind, assume send went through, and restart bitcoind
290 2013-07-13 06:36:11 <TradeFortress> sipa: you don't have a "if beta throw random deadlock" function to discourage using it for mining / merchant applications anywhere in the code, right?
291 2013-07-13 06:39:12 <sipa> no
292 2013-07-13 06:39:38 <sipa> TradeFortress: is anythimg written in debug.log when this wallet corrupted appears in 0.8.3?
293 2013-07-13 06:42:40 <TradeFortress> Commiting 29k changed transactions to coin database.. FLush(true)
294 2013-07-13 06:42:54 <TradeFortress> wallet.dat refcount=0; wallet.dat checpoint; wallet.dat detach; wallet.dat closed; DBFlush(true) ended
295 2013-07-13 06:43:25 <sipa> that's too far
296 2013-07-13 06:43:25 <TradeFortress> OH WAIT
297 2013-07-13 06:43:31 <TradeFortress> sipa, CPrivKey pubkey inconsistency
298 2013-07-13 06:43:45 <TradeFortress> I literally see this like 1000+ times
299 2013-07-13 06:43:58 <TradeFortress> my debug.log has more than 60,000 lines
300 2013-07-13 06:44:03 <sipa> and head doesn't complain about that?
301 2013-07-13 06:45:30 <TradeFortress> no, it is happy accepting transactions
302 2013-07-13 06:45:35 <TradeFortress> happily*
303 2013-07-13 06:47:25 <TradeFortress> would seeing parts of the wallet.dat help?
304 2013-07-13 06:51:34 <sipa> i know enough
305 2013-07-13 06:52:09 <sipa> there was a refactor of the private-key handling code in head
306 2013-07-13 06:52:53 <warren> hmm... I included that refactor in litecoin-0.8.3.4
307 2013-07-13 06:53:08 <warren> for no particular reason other than to make it easier to drop-in secp256k1
308 2013-07-13 06:53:29 <TradeFortress> sipa, well at least that refactor didn't eat up my private keys
309 2013-07-13 06:53:51 <TradeFortress> have you ever had something that did that?
310 2013-07-13 06:54:23 <warren> TradeFortress: I missed the earlier part of this conversation, what is the issue?
311 2013-07-13 06:54:48 <warren> TradeFortress: I'm alarmed if the private-key handling refactor lead to a bug
312 2013-07-13 06:56:51 <TradeFortress> warren, I experience unpredictable deadlocks when calling sendtoaddress
313 2013-07-13 06:57:11 <TradeFortress> the rpc times out, bitcoind deadlocks, and I have to restart.
314 2013-07-13 06:57:12 <warren> TradeFortress: with the refactor, or only after downgrade?
315 2013-07-13 06:57:28 <TradeFortress> happened on 0.8.3, happened on head.
316 2013-07-13 06:57:54 <warren> ok, so the privkey refactor didn't make it worse?
317 2013-07-13 06:58:00 <TradeFortress> or.. that's what I think. I'm not sure if I *actually* ran 0.8.3
318 2013-07-13 06:58:12 <TradeFortress> I'm a noobie when it comes to git.. I don't think git checkout was what I typed.
319 2013-07-13 06:58:14 <warren> TradeFortress: i'm very interested in what you find
320 2013-07-13 06:58:27 <TradeFortress> so am I because I lose coins everytime Iit deadlocks
321 2013-07-13 06:58:33 <TradeFortress> :P
322 2013-07-13 07:02:50 <TradeFortress> what is the least resource intensive rpc function that I can check query to see if there's a deadlock?
323 2013-07-13 07:03:26 <warren> can you reproduce it on testnet?
324 2013-07-13 07:04:07 <warren> TradeFortress: make a backup of the entire .bitcoin and try a -reindex, problems continue after?
325 2013-07-13 07:04:18 <Scrat> TradeFortress: what HDD are you using, also whats your iowait
326 2013-07-13 07:05:21 <TradeFortress> oh and I have txindex=1 if that matters
327 2013-07-13 07:05:25 <warren> OH
328 2013-07-13 07:05:28 <TradeFortress> ?
329 2013-07-13 07:05:40 <warren> just saying OH, that shouldn't matter
330 2013-07-13 07:06:17 <TradeFortress> Scrat, not the hdd's issue, it's not confined to a machine. I believe it's a problem with the wallet.dat
331 2013-07-13 07:07:00 <Scrat> TradeFortress: had the same problem and it disappeared when I moved the thing to an SSD (literally copied the dir along with the wallet)
332 2013-07-13 07:08:28 <Scrat> I was convinced that bitcoind did something funky with fsync which frequently locked up for ~10 seconds on a 5400 rpm drive (high load server)
333 2013-07-13 07:13:09 <TradeFortress> Scrat, literally the same problem?
334 2013-07-13 07:14:20 <Scrat> for whatever reason it is very sensitive to IO latency when sending
335 2013-07-13 07:15:22 <TradeFortress> so it'd time out and then bitcoind deadlocks?
336 2013-07-13 07:15:37 <sipa> slowdowns are not deadlocks
337 2013-07-13 07:15:37 <warren> when I combine 67 inputs to send bitcoind/litecoind seems to lockup for a minute before sending
338 2013-07-13 07:15:46 <warren> but it isn't a deadlock
339 2013-07-13 07:15:59 <sipa> a MINUTE?
340 2013-07-13 07:16:02 <sipa> :o
341 2013-07-13 07:16:03 <warren> yeah
342 2013-07-13 07:16:24 <Scrat> I'd have to kill it once in a while after waiting for like a minute, but on average it was 10 seconds under high server load
343 2013-07-13 07:16:54 <TradeFortress> but this literally deadlocks for hours
344 2013-07-13 07:17:10 <sipa> a deadlock is infinitre
345 2013-07-13 07:17:36 <TradeFortress> yeah
346 2013-07-13 07:17:37 <Scrat> oh, then I doubt it's the same issue
347 2013-07-13 07:23:58 <TradeFortress> so there's nothing I could help debug when a deadlock happens again?
348 2013-07-13 07:24:13 <warren> TradeFortress: are you SURE the other databases aren't corrupt?
349 2013-07-13 07:24:40 <TradeFortress> warren, other databases? I have tried clearing out everything but wallet.dat
350 2013-07-13 07:24:49 <warren> ok
351 2013-07-13 07:25:25 <warren> TradeFortress: and you aren't sure if the privkey refactor is an issue here or not?
352 2013-07-13 07:25:42 <TradeFortress> warren, I do not think it is, however I cannot be sure.
353 2013-07-13 07:26:36 <TradeFortress> but
354 2013-07-13 07:26:42 <TradeFortress> it is AFTER the tx has been sent to the network
355 2013-07-13 07:27:03 <sipa> nothing wrt other databases happens then
356 2013-07-13 07:27:27 <TradeFortress> does it store the tx locally then?
357 2013-07-13 07:27:33 <sipa> in the wallet
358 2013-07-13 07:27:35 <sipa> and it's not coin selection, as your transaction was already sent
359 2013-07-13 07:27:44 <TradeFortress> it may not have been pushed out to the network, just stored
360 2013-07-13 07:27:44 <warren> txindex=1 doesn't seem relevant here
361 2013-07-13 07:27:57 <TradeFortress> as the actual sending time may be when I restart bitcoind and it finds a tx not broadcast
362 2013-07-13 07:28:04 <sipa> right
363 2013-07-13 07:28:11 <sipa> but still, coin selection was complete
364 2013-07-13 07:28:46 <sipa> some debug.log lines from around the time of a deadlock would be useful
365 2013-07-13 07:28:55 <warren> TradeFortress: was the wallet.dat always db4, or db5?
366 2013-07-13 07:29:00 <TradeFortress> warren, db5
367 2013-07-13 07:29:02 <warren> TradeFortress: what version of db5?
368 2013-07-13 07:29:06 <TradeFortress> db5.1
369 2013-07-13 07:29:35 <TradeFortress> I have been saving all my debug.log. but the total is 60k lines long. do you think there's anything I can narrow it down to deadlocks
370 2013-07-13 07:29:50 <TradeFortress> like something that only happens when it starts up
371 2013-07-13 07:30:05 <warren> sipa: would debuginfo, gdb thread apply all bt full help at his deadlock?
372 2013-07-13 07:54:04 <sipa> TradeFortress: the deadlock doesn't happen at startup, right?
373 2013-07-13 07:54:17 <sipa> warren: yes
374 2013-07-13 07:54:36 <TradeFortress> sipa, no
375 2013-07-13 07:54:44 <TradeFortress> so when it deadlocks and this room is empty, what do I do with gdb
376 2013-07-13 07:55:08 <sipa> attach to the deadlocked bitcoind
377 2013-07-13 07:55:10 <warren> TradeFortress: make a bitcoind build with all debuginfo available
378 2013-07-13 07:55:19 <sipa> oh yes, run a debug version from the start
379 2013-07-13 07:55:27 <warren> TradeFortress: after the deadlock, in another terminal use: gdb attach <pid>
380 2013-07-13 07:55:34 <warren> TradeFortress: then thread apply all bt full
381 2013-07-13 07:55:55 <TradeFortress> "thread apply all bt full"
382 2013-07-13 07:55:57 <TradeFortress> ?
383 2013-07-13 07:56:02 <warren> I think...
384 2013-07-13 07:56:20 <TradeFortress> and how do I do that?
385 2013-07-13 07:56:24 <TradeFortress> (debug build)
386 2013-07-13 07:56:43 <warren> TradeFortress: run gdb ./bitcoind, it might tell you what symbols in library deps are missing. on fedora it even tells you the command to install all the debuginfo packages. Not sure if ubuntu has that.
387 2013-07-13 07:57:01 <warren> TradeFortress: a local build has the debuginfo by default, but your library deps probably don't
388 2013-07-13 07:57:26 <TradeFortress> well I made using dev libraries
389 2013-07-13 07:57:34 <warren> those dev libraries are stripped
390 2013-07-13 07:57:45 <warren> I don't know if/how Ubuntu distributes debuginfo
391 2013-07-13 07:57:58 <sipa> install libdb5.1-dbg
392 2013-07-13 07:58:00 <warren> they're available in side-packages on fedora
393 2013-07-13 07:58:02 <warren> ah
394 2013-07-13 07:58:03 <warren> ok
395 2013-07-13 08:00:26 <TradeFortress> k so i installed libdb5.1-dbg and remade it. now I do gdb ./bitcoind after stopping?
396 2013-07-13 08:00:44 <warren> TradeFortress: no
397 2013-07-13 08:00:56 <warren> TradeFortress: gdb ./bitcoind is just to test if debug symbols are available
398 2013-07-13 08:01:02 <TradeFortress> ok
399 2013-07-13 08:01:12 <warren> TradeFortress: run bitcoind normally and use "gdb attach" after the deadlock
400 2013-07-13 08:01:26 <TradeFortress> when it actually happens, I type gdb attach "pid" and "thread apply all bt full" ?
401 2013-07-13 08:01:48 <warren> I vaguely recall pid, I haven't done it in a month or two.
402 2013-07-13 08:01:59 <sipa> with pid the pid of the bitcoind, not the string "pid"
403 2013-07-13 08:02:10 <TradeFortress> lol I know what a pid is
404 2013-07-13 08:02:28 <sipa> ok!
405 2013-07-13 08:19:43 <sipa> TradeFortress: i have a theory about the 0.8.3 incompatibility
406 2013-07-13 08:19:56 <TradeFortress> sipa, wasn't that the wallet.dat refactor?
407 2013-07-13 08:19:59 <TradeFortress> private key
408 2013-07-13 08:20:35 <sipa> well, i suspected that that refactor introduced the incompatibility yes, but i didn't know what the exact problem was
409 2013-07-13 08:20:38 <sipa> just where to look :)
410 2013-07-13 08:24:41 <sipa> no, i'm wrong
411 2013-07-13 08:32:36 <sipa> TradeFortress: can you make this change to 0.8.3, and try again: http://pastebin.com/bbA5AiF2 ?
412 2013-07-13 08:33:02 <warren> sipa: if the refactor did introduce incompatibility is that a bug that must be fixed? or downgrades are not supported?
413 2013-07-13 08:33:23 <TradeFortress> ok sipa
414 2013-07-13 08:33:45 <TradeFortress> I will test that when it deadlocks.. this is a prod enviroment I'm talking about anyway
415 2013-07-13 08:34:28 <warren> sipa: I included the refactor in litecoin about a month ago. we didn't release this version yet.
416 2013-07-13 08:34:51 <sipa> warren: yes, if it's what i think it is, it's a bug
417 2013-07-13 08:35:16 <warren> sipa: hmm, so I should pull it in our 0.8.3.x release, but doing so might screw up some early testers?
418 2013-07-13 08:35:30 <warren> s/pull it/remove it from/
419 2013-07-13 08:35:33 <sipa> that's always a risk with pre-releases, yes
420 2013-07-13 08:35:57 <sipa> warren: wait, just to be clear: the incompatibility is in head, not in 0.8.3
421 2013-07-13 08:36:10 <sipa> so it's a bug that must be fixed before head is released
422 2013-07-13 08:36:16 <warren> sipa: https://github.com/litecoin-project/litecoin-0.8/commits/awesomecoin?page=2
423 2013-07-13 08:36:37 <warren> sipa: we didn't release the litecoin with cprivkey refactor, but testers have been using it for a month now
424 2013-07-13 08:36:49 <TradeFortress> hehe love the " No LTC support" :P
425 2013-07-13 08:37:07 <warren> TradeFortress: Litecoin secp256k1 has already found a bug in secp256k1
426 2013-07-13 08:37:23 <sipa> yeah, i have absolutely no problem with litecoin being a guinea pig :)
427 2013-07-13 08:38:17 <warren> sipa: I've done thousands of sendtoaddress and haven't seen his problem though
428 2013-07-13 08:38:30 <sipa> warren: i'm not talking about the deadlock at all
429 2013-07-13 08:38:33 <warren> oh
430 2013-07-13 08:38:34 <sipa> it's unrelated
431 2013-07-13 08:38:45 <warren> sipa: what is the suspected bug?
432 2013-07-13 08:38:58 <warren> sipa: we're close to release, I need to know if I should remove the refactor
433 2013-07-13 08:39:16 <sipa> warren: git head always writes private keys with uncompressed pubkeys in them
434 2013-07-13 08:39:53 <sipa> and i expected that previous versions used the length of the (separately stored) public key to determine whether it's supposed to be compressed or not
435 2013-07-13 08:40:41 <sipa> TradeFortress: when was this wallet created?
436 2013-07-13 08:41:13 <TradeFortress> v0.8.3 days
437 2013-07-13 08:41:15 <sipa> TradeFortress: specifially, with which version?
438 2013-07-13 08:41:17 <sipa> ok
439 2013-07-13 08:41:27 <sipa> can you do a validateaddress <addr> for a random address in your wallet?
440 2013-07-13 08:41:40 <sipa> and tell me the public key? (in PM in you prefer)
441 2013-07-13 08:42:14 <warren> sipa: in our case we never had an official release of 0.8.x so we don't need backward compatibility
442 2013-07-13 08:42:27 <sipa> warren: but 0.8.* is unaffected!
443 2013-07-13 08:42:32 <sipa> the problem is only in head
444 2013-07-13 08:42:49 <sipa> whether you have a 0.8.x release is not relevant
445 2013-07-13 08:42:50 <warren> sipa: we have 0.8.x + your privkey refactor
446 2013-07-13 08:42:53 <sipa> ah
447 2013-07-13 08:43:03 <warren> sipa: https://github.com/litecoin-project/litecoin-0.8/commits/awesomecoin?page=2 specifically those commits
448 2013-07-13 08:43:21 <warren> sipa: as I was subjecting litecoin testers to secp256k1
449 2013-07-13 08:45:08 <TradeFortress> sipa, iscompressed is yes
450 2013-07-13 08:45:19 <sipa> ok
451 2013-07-13 08:45:23 <warren> ACTION checks here
452 2013-07-13 08:46:21 <warren> sipa: So ... we're not concerned about backward compatibility, but if the refactor has a bug, I need to decide what to do about it before we release.
453 2013-07-13 08:46:50 <sipa> if it is what i think it is, it's trivial to fix
454 2013-07-13 08:47:59 <warren> The fixed refactor will be compatible with keys written by the unfixed refactor?
455 2013-07-13 08:48:17 <warren> as we aren't concerned with backward compat (no 0.8.x releases) that would be fine
456 2013-07-13 08:48:47 <sipa> yes
457 2013-07-13 08:49:11 <warren> ok, delaying release for this and LOTS of testing
458 2013-07-13 08:50:56 <warren> well hmm, the only drawback of not fixing the refactor is inability to downgrade to pre-refactor 0.8?
459 2013-07-13 08:51:01 <sipa> yes
460 2013-07-13 08:51:03 <warren> which we don't have
461 2013-07-13 08:51:12 <sipa> any older litecoin?
462 2013-07-13 08:51:16 <warren> 0.6.x
463 2013-07-13 08:51:20 <sipa> yes
464 2013-07-13 08:51:29 <warren> 0.8.x is a full rebase
465 2013-07-13 08:51:40 <sipa> yes, so?
466 2013-07-13 08:52:01 <sipa> it's generally very hard already to downgrade to pre-0.8
467 2013-07-13 08:52:07 <sipa> because of the database changes
468 2013-07-13 08:52:08 <warren> yeah, so no issue here
469 2013-07-13 08:52:13 <sipa> but it's still possible
470 2013-07-13 08:52:19 <sipa> wallets are intended to be backward compatible
471 2013-07-13 08:52:34 <warren> between major versions?
472 2013-07-13 08:52:39 <warren> didn't know that...
473 2013-07-13 08:53:32 <sipa> you can actually create a wallet still that should be readable by 0.3.x or something
474 2013-07-13 08:53:38 <sipa> (but by default it won't)
475 2013-07-13 08:53:52 <warren> we expect 0.6.x will die quickly because Litecoin has no vendors integrated yet, no reason not to upgrade to 0.8.x when we release it.
476 2013-07-13 08:53:53 <sipa> there's a walletversion in getinfo
477 2013-07-13 08:54:01 <warren> ah
478 2013-07-13 08:54:09 <sipa> which will tell you the earlier version that can read your wallet
479 2013-07-13 08:54:33 <warren> 0.8.x will be the first version that vendors will likely use.
480 2013-07-13 08:54:41 <sipa> 'vendors' ?
481 2013-07-13 08:55:07 <warren> err... people who modify the client for whatever purpose and are resistant to upgrades in the future
482 2013-07-13 08:55:40 <sipa> ok, bug confirmed
483 2013-07-13 09:00:07 <sipa> and fix confirmed too
484 2013-07-13 09:00:18 <warren> cool
485 2013-07-13 09:01:46 <warren> sipa: validateaddress should be able to see the bug?
486 2013-07-13 09:02:00 <sipa> no
487 2013-07-13 09:02:17 <sipa> to reproduce: create a new wallet using head, and try to load it in 0.8.3
488 2013-07-13 09:02:34 <warren> in my case, rebuild with refactor reverted
489 2013-07-13 09:03:16 <warren> anyway, if you confrmed the bug, then we have it too.
490 2013-07-13 09:04:43 <sipa> #2825 fixes it
491 2013-07-13 09:04:58 <warren> trying without, with, and #2825
492 2013-07-13 09:05:26 <sipa> a 'broken' wallet will not be fixed by running with #2825, however
493 2013-07-13 09:07:14 <warren> we're not concerned, we have no client releases that are expected to read those "broken" wallets
494 2013-07-13 09:07:21 <sipa> yeah
495 2013-07-13 09:20:26 <warren> sipa: thank you!
496 2013-07-13 09:55:43 <warren> grrr, accidentally blew away my origin-pull config again
497 2013-07-13 09:55:56 <warren> I should write down how to configure it...
498 2013-07-13 11:25:35 <warren> sipa: your secp256k1 patches need a rebase after that
499 2013-07-13 11:51:13 <sipa> warren: i know; will do after it's merged
500 2013-07-13 13:33:05 <rfish> hey, I was looking at BTC-E's website, and it looks like it is vulnerable to a CSFR attack
501 2013-07-13 13:56:56 <TradeFortress> help
502 2013-07-13 13:57:09 <TradeFortress> bitcoind deadlock again
503 2013-07-13 13:59:28 <sipa> TradeFortress: ping
504 2013-07-13 13:59:37 <sipa> can you gdb attach?
505 2013-07-13 14:00:32 <TradeFortress> sipa, yes
506 2013-07-13 14:00:41 <TradeFortress> I have debug.log. it was after selectcoins, committransaction
507 2013-07-13 14:00:53 <TradeFortress> and then addtowallet
508 2013-07-13 14:01:10 <TradeFortress> Flush wallet.dat, getbalance twice and then socket.closed and a bunch of disconnects
509 2013-07-13 14:02:12 <TradeFortress> sipa, I attached what do I do
510 2013-07-13 14:03:58 <sipa> thread apply all bt all
511 2013-07-13 14:04:12 <sipa> and pastebin the output or something
512 2013-07-13 14:04:20 <sipa> eh
513 2013-07-13 14:04:24 <sipa> thread apply all bt full
514 2013-07-13 14:05:26 <TradeFortress> sipa, pastein.com/0dHYn1zW
515 2013-07-13 14:05:33 <TradeFortress> pastebin.com/0dHYn1zW
516 2013-07-13 14:06:29 <sipa> that's only a small piece
517 2013-07-13 14:06:44 <sipa> it'll likely be several pages of output
518 2013-07-13 14:07:07 <TradeFortress> yeah goes type return to continue
519 2013-07-13 14:07:41 <sipa> yeah, i'd like to see the rest of the output too :)
520 2013-07-13 14:08:00 <TradeFortress> this like never ends
521 2013-07-13 14:08:02 <TradeFortress> can I cat it to a file
522 2013-07-13 14:08:06 <TradeFortress> I feel like my buffer is going to be overwritten
523 2013-07-13 14:08:36 <sipa> ?
524 2013-07-13 14:08:54 <TradeFortress> I'm accessing through terminal
525 2013-07-13 14:08:58 <TradeFortress> I will only see the last x lines or whatever
526 2013-07-13 14:09:30 <sipa> it pauses after every page written
527 2013-07-13 14:09:53 <TradeFortress> You expect me to copy that 30 times or so? Can't I dump it to a txt file
528 2013-07-13 14:10:32 <sipa> sec
529 2013-07-13 14:10:37 <sipa> exit gdb
530 2013-07-13 14:10:42 <sipa> and do this:
531 2013-07-13 14:10:55 <TradeFortress> ok
532 2013-07-13 14:10:56 <sipa> gdb -batch -ex 'attach 5736' -ex 'thread apply all bt full' | tee stacktrace.txt
533 2013-07-13 14:11:02 <sipa> with 5736 the pid
534 2013-07-13 14:12:17 <TradeFortress> GOT IT
535 2013-07-13 14:14:17 <sipa> can you put it somewhere?
536 2013-07-13 14:15:39 <TradeFortress> sipa: http://pastebin.com/3uZua1A5
537 2013-07-13 14:18:41 <TradeFortress> find the cause and I will tip 20 BTC
538 2013-07-13 14:24:38 <TradeFortress> out of curiosirty, how do you freakng debug that?
539 2013-07-13 14:28:26 <sipa> are you sure this deadlock occurred in 0.8.3 as well?
540 2013-07-13 14:28:47 <sipa> one of the locked threads is blocked on something that was added in head only
541 2013-07-13 14:31:27 <sipa> TradeFortress: i found a deadlock
542 2013-07-13 14:31:37 <sipa> TradeFortress: though it shouldn't have been present in 0.8.3
543 2013-07-13 14:31:37 <TradeFortress> sipa, you did?
544 2013-07-13 14:31:43 <TradeFortress> holy shit
545 2013-07-13 14:33:15 <TradeFortress> sipa, i probably never used v0.8.3 in the first place. I'm bad with git, you know
546 2013-07-13 14:33:47 <sipa> i'll try to find a fix later today
547 2013-07-13 14:33:49 <sipa> gotta go now
548 2013-07-13 14:36:46 <TradeFortress> okay..
549 2013-07-13 14:36:49 <CodeShark> TradeFortress: what are the steps to reproduce this issue?
550 2013-07-13 14:36:56 <TradeFortress> I hope I can pull it tomorrow when I wake up. CodeShark sendtoaddress
551 2013-07-13 14:36:58 <TradeFortress> occurs randomly
552 2013-07-13 14:37:01 <gmaxwell> TradeFortress: do try to take care when reporting bugs, I spent a fair amount of time w/ 0.8.3 on testnet trying to reproduce your deadlock. :P
553 2013-07-13 14:37:30 <sipa> you need a payment being received simultaneously with sending one
554 2013-07-13 14:37:56 <gmaxwell> How about sending to yourself?
555 2013-07-13 14:38:09 <sipa> one first locks setpwalletregistered first, the other locks wallet first
556 2013-07-13 14:38:29 <sipa> setpwalletregistered needs a shared lock, i think
557 2013-07-13 14:38:43 <sipa> though i'll need to think whether there are no edge cases
558 2013-07-13 14:38:45 <CodeShark> I remember we had discussed something along those lines
559 2013-07-13 14:39:02 <CodeShark> the locks in the wallet registration functions are too strict
560 2013-07-13 14:39:30 <CodeShark> really, it's only when registering and unregistering that we need a mutex
561 2013-07-13 14:39:45 <sipa> well, you still don't want to modify them while it's being used
562 2013-07-13 14:40:16 <sipa> amyway, afk
563 2013-07-13 14:40:27 <CodeShark> it's ok if say, SyncWithWallets and EraseFromWallets are called concurrently
564 2013-07-13 14:40:37 <TradeFortress> gmaxwell, I am sorry
565 2013-07-13 14:40:41 <TradeFortress> I'll share 20 BTC with the core dev team anyway
566 2013-07-13 14:40:46 <CodeShark> since the wallet locks should handle those situations
567 2013-07-13 14:41:20 <CodeShark> but it's not ok if UnregisterWallet and EraseFromWallets are called concurrently
568 2013-07-13 14:43:10 <CodeShark> so we need something a little bit more than just a simple lock
569 2013-07-13 14:43:27 <gmaxwell> TradeFortress: no biggie, it happens, don't worry about it??? and thanks for actually testing git and reporting the deadlock.
570 2013-07-13 14:43:50 <TradeFortress> gmaxwell, i loes coins every time it happens
571 2013-07-13 14:43:59 <CodeShark> sipa is far more an expert than I am on sync stuff, though :)
572 2013-07-13 14:44:55 <gmaxwell> TradeFortress: hm? presumably they'll get picked up in a rescan.
573 2013-07-13 14:45:13 <TradeFortress> gmaxwell, no. basically I treat errors as coins not sent
574 2013-07-13 14:45:15 <TradeFortress> time outs are errors.
575 2013-07-13 14:45:23 <TradeFortress> not sent means the balance goes back and isn't sent. but the coins are sent.
576 2013-07-13 14:55:57 <gmaxwell> TradeFortress: you should perhapt not be running untested snapshots from git in production then...
577 2013-07-13 14:56:22 <TradeFortress> gmaxwell, definitely, i thought I was through
578 2013-07-13 14:56:28 <gmaxwell> (although, if you hadn't found this its possible it would have made it into the next release too??? we simply need more testing)
579 2013-07-13 14:57:00 <CodeShark> I think I have a fix for that, TradeFortress
580 2013-07-13 14:57:13 <CodeShark> sipa is right - we need a shared lock
581 2013-07-13 14:57:27 <CodeShark> testing the fix - will post shortly
582 2013-07-13 14:57:31 <TradeFortress> ok, cool
583 2013-07-13 15:13:17 <CodeShark> TradeFortress: https://github.com/bitcoin/bitcoin/pull/2828
584 2013-07-13 15:16:12 <CodeShark> we might want to add some more macros to sync.h for shared locks
585 2013-07-13 15:18:44 <TradeFortress> codeshark and how'd I test that?
586 2013-07-13 15:19:03 <CodeShark> try to reproduce your earlier deadlock
587 2013-07-13 15:20:03 <CodeShark> attempt multiple simultaneous operations on the wallet, I suppose
588 2013-07-13 15:21:16 <TradeFortress> has that code been tested? I am doing it on prod after all
589 2013-07-13 15:22:04 <CodeShark> I definitely would like other devs to look at it before using it for prod.
590 2013-07-13 15:22:19 <CodeShark> sipa in particular
591 2013-07-13 15:22:56 <CodeShark> the code builds and seems to run correctly for me - but I haven't done very rigorous testing on it
592 2013-07-13 15:24:01 <CodeShark> very rigorous testing would require deliberately creating circumstances where multiple wallet operations are performed concurrently
593 2013-07-13 15:26:10 <CodeShark> it is my understanding that none of the functions I modified should lock at all except for the registration and unregistration functions, which currently are only called when the program starts and stops
594 2013-07-13 15:27:16 <CodeShark> and if I understood sipa correctly (I haven't looked at all your output), the problem was that the RPC was locking the wallet before locking the registration functions while main was doing the opposite
595 2013-07-13 16:53:33 <sipa> CodeShark: a shared lock will work, but there is still a (much smaller) chance for deadlock when you modify the registered wallet set while a callback is happening
596 2013-07-13 16:53:39 <sipa> i think
597 2013-07-13 17:05:53 <CodeShark> in principle each wallet should be responsible for protecting its own integrity when used reentrantly
598 2013-07-13 17:06:30 <CodeShark> how would a deadlock occur?
599 2013-07-13 17:06:43 <CodeShark> oh, you mean if you modify the container
600 2013-07-13 17:07:46 <CodeShark> if you add or remove a wallet to or from the registered wallet set then yes, you could get a deadlock
601 2013-07-13 17:08:33 <CodeShark> the RPC code would have to be surrounded by a shared lock
602 2013-07-13 17:09:18 <CodeShark> I'm not a big fan of externing that mutex, though
603 2013-07-13 17:11:50 <warren> sipa: which post-0.8.3 commit introduced the deadlock?
604 2013-07-13 17:13:42 <sipa> warren: only in head
605 2013-07-13 17:14:09 <warren> sipa: I know, just wondering which commit it was
606 2013-07-13 17:14:48 <warren> oh, I see CodeShark's PR, i'll figure it out
607 2013-07-13 17:25:43 <warren> sipa: CodeShark: is there a reproduce case for this deadlock?
608 2013-07-13 18:49:47 <serp> If I'm using the rpc api and I use getblock to get info on a block it lists a bunch of transaction id/keys. How can I use that id/key to get information about a transaction then? I've tried using it with gettransaction, getrawtransaction, and decoderawtransaction without luck.
609 2013-07-13 18:51:40 <gmaxwell> serp: you need txindex enabled to get info on spent transactions via getrawtransaction.
610 2013-07-13 18:52:17 <serp> i'm assuming that's going to increase disk usage a good bit
611 2013-07-13 18:52:52 <warren> serp: a little slower too
612 2013-07-13 18:53:05 <serp> ok... thank you all for the answer
613 2013-07-13 18:54:19 <gmaxwell> serp: yup.
614 2013-07-13 18:54:40 <gmaxwell> warren: It didn't seem to really matter much for anything I've tested.
615 2013-07-13 18:54:53 <gmaxwell> (speed wise)
616 2013-07-13 18:55:20 <gmaxwell> looks like it adds about 1.1GBytes right now.
617 2013-07-13 18:55:25 <warren> ok
618 2013-07-13 18:55:40 <serp> that's not too bad
619 2013-07-13 18:55:43 <gmaxwell> maybe if you didn't have enough ram to get all the relevant data in ram.
620 2013-07-13 18:55:50 <gmaxwell> serp: it'll grow more in the future, obviously.
621 2013-07-13 18:55:55 <serp> right
622 2013-07-13 18:56:11 <serp> all the latest blocks seem to have a lot more tx in them
623 2013-07-13 18:56:22 <gmaxwell> Sure.
624 2013-07-13 19:09:52 <Luke-Jr> serp: depending on what you need to do, getrawtransaction might be sufficient without txindex
625 2013-07-13 19:10:17 <Luke-Jr> serp: basically, without txindex it will just error if the coins have been spent already; but if you know that isn't the case, you can ignore it
626 2013-07-13 19:10:32 <Luke-Jr> so for example, post-processing a new block
627 2013-07-13 19:10:57 <Luke-Jr> while coins might be spent in the same block, that shouldn't happen if you're controlling the receiving wallet and wait for confirms
628 2013-07-13 19:35:40 <serp> Yeah, getrawtransaction was passing back a "No information available about transaction" error
629 2013-07-13 19:36:31 <serp> it was for transactions in a different wallet
630 2013-07-13 19:39:48 <Luke-Jr> yeah, depends on what you need it for
631 2013-07-13 21:01:58 <TradeFortress> How do I check which account an address belongs to
632 2013-07-13 21:02:22 <sipa> what do you need that for?
633 2013-07-13 21:03:00 <sipa> validateaddress will tell you
634 2013-07-13 21:03:08 <TradeFortress> sipa, someone is complaining that their deposit was not credited. I looked at listtransactions for that users acc and nothing
635 2013-07-13 21:03:46 <TradeFortress> sipa, OK, this account is correct however listtransactions isn't working
636 2013-07-13 21:04:30 <sipa> that's remarkable
637 2013-07-13 21:04:42 <sipa> is the transaction in the wallet at all
638 2013-07-13 21:04:43 <sipa> ?
639 2013-07-13 21:05:03 <TradeFortress> no. error: {"code":-5,"message":"Invalid or non-wallet transaction id"}
640 2013-07-13 21:05:16 <TradeFortress> db8f80d2311c3b3e46a3205f898268e365208726bc8557690a0140678fe71718
641 2013-07-13 21:06:03 <sipa> and 1JWLk4mGBwVuKbDb7UAnjr1zbd6fkvUmMu or 1CD73oJmamZSdkqVowPupxPknurmaLuqDT is the relevant address?
642 2013-07-13 21:06:29 <TradeFortress> 1JWLk4mGBwVuKbDb7UAnjr1zbd6fkvUmMu
643 2013-07-13 21:06:52 <sipa> and validateaddress 1JWLk4mGBwVuKbDb7UAnjr1zbd6fkvUmMu says ismine?
644 2013-07-13 21:07:00 <TradeFortress> true
645 2013-07-13 21:07:04 <TradeFortress> and I have txindex=1 anyway
646 2013-07-13 21:07:10 <sipa> that's not relevant
647 2013-07-13 21:08:08 <sipa> i wonder, was that transaction perhaps received exactly when the deadlock occurred?
648 2013-07-13 21:08:13 <sipa> perhaps even triggered it?
649 2013-07-13 21:08:16 <TradeFortress> hmmm
650 2013-07-13 21:08:20 <TradeFortress> I can getrawtransaction.
651 2013-07-13 21:08:26 <sipa> yes, not relevant
652 2013-07-13 21:08:38 <sipa> blockchain engine and wallet are almost entirely separate
653 2013-07-13 21:09:01 <sipa> your deadlock involved a call to the wallet to inform it of a new transaction
654 2013-07-13 21:09:26 <TradeFortress> hmm
655 2013-07-13 21:09:39 <TradeFortress> Was there a patch to fix the deadlock btw?
656 2013-07-13 21:09:43 <sipa> yes
657 2013-07-13 21:09:55 <sipa> CodeShark wrote one, which will work in practice
658 2013-07-13 21:10:03 <TradeFortress> work in practice .. ?
659 2013-07-13 21:10:16 <sipa> in theory there is still a risk for deadlocks, but that won't be exposed until multiwallet
660 2013-07-13 21:10:33 <TradeFortress> ok, so how do I apply this patch
661 2013-07-13 21:11:07 <sipa> sorry, i'm tired now - i'm sure others can help you with that :)
662 2013-07-13 21:11:21 <CodeShark> clone the repo, checkout the branch, build
663 2013-07-13 21:11:21 <TradeFortress> ok. mornining here lol
664 2013-07-13 21:11:28 <sipa> 1am here
665 2013-07-13 21:11:28 <TradeFortress> ah so it's in head ok
666 2013-07-13 21:11:36 <sipa> it's in head of that branch
667 2013-07-13 21:11:40 <sipa> it's not in master head
668 2013-07-13 21:11:41 <TradeFortress> which branch?
669 2013-07-13 21:11:46 <TradeFortress> which branch do I checkout?
670 2013-07-13 21:12:03 <sipa> afk :)
671 2013-07-13 21:12:04 <CodeShark> one sec, TradeFortress
672 2013-07-13 21:13:00 <CodeShark> TradeFortress: do you already have a local clone of the repo?
673 2013-07-13 21:13:09 <TradeFortress> yeah!
674 2013-07-13 21:13:20 <CodeShark> ok, then git remote add codeshark https://github.com/CodeShark/bitcoin.git
675 2013-07-13 21:13:50 <CodeShark> then git fetch codeshark wallet_registration_shared_lock
676 2013-07-13 21:14:34 <TradeFortress> done
677 2013-07-13 21:14:39 <TradeFortress> git pull > build?
678 2013-07-13 21:14:43 <CodeShark> then git branch --track wallet_registration_shared_lock codeshark/wallet_registration_shared_lock
679 2013-07-13 21:14:58 <TradeFortress> not a valid obj name
680 2013-07-13 21:15:13 <CodeShark> what happens if you do git branch -r?
681 2013-07-13 21:15:38 <TradeFortress> I don't see anything like wallet_registration
682 2013-07-13 21:16:26 <CodeShark> but the git fetch didn't cause any errors?
683 2013-07-13 21:16:59 <TradeFortress> bope
684 2013-07-13 21:17:09 <TradeFortress> * branch wallet_registration_shared_lock -> FETCH_HEAD
685 2013-07-13 21:18:17 <CodeShark> try git config remote.codeshark.fetch +refs/heads/*:refs/remotes/codeshark/*
686 2013-07-13 21:19:12 <TradeFortress> Still not a valid object name.
687 2013-07-13 21:20:26 <CodeShark> try git remote update
688 2013-07-13 21:21:21 <TradeFortress> Branch wallet_registration_shared_lock set up to track remote branch wallet_registration_shared_lock from codeshark.
689 2013-07-13 21:21:27 <CodeShark> ok, excellent
690 2013-07-13 21:21:41 <CodeShark> so now if you do git branch, does this branch have the asterisk?
691 2013-07-13 21:21:48 <TradeFortress> master has
692 2013-07-13 21:21:57 <TradeFortress> I switched to the lock
693 2013-07-13 21:23:06 <CodeShark> so main.cpp should now be using a shared_mutex for cs_setpwalletRegister
694 2013-07-13 21:23:20 <CodeShark> if that's the case you're on the correct branch
695 2013-07-13 21:23:31 <TradeFortress> yes
696 2013-07-13 21:23:54 <TradeFortress> building
697 2013-07-13 21:30:43 <TradeFortress> ok. lets hope no more deadlocks
698 2013-07-13 21:30:49 <TradeFortress> CodeShark, what do I do in the future when I want to upgrade?
699 2013-07-13 21:32:06 <CodeShark> hopefully we'll get this fix merged into master soon
700 2013-07-13 21:32:14 <CodeShark> then you won't have to worry about it anymore :)
701 2013-07-13 21:32:56 <CodeShark> if you do run into any problems, please let me know
702 2013-07-13 21:33:46 <TradeFortress> ok
703 2013-07-13 21:34:09 <CodeShark> if it does fix it, are you still offering rewards? ;)
704 2013-07-13 21:34:48 <CodeShark> make sure to tip sipa
705 2013-07-13 21:35:07 <CodeShark> because he was the one who discovered it
706 2013-07-13 21:35:58 <TradeFortress> CodeShark, yes, if I don't get a deadlock in a couple days I will split 20 btc with people who have helped me :)
707 2013-07-13 22:09:28 <saulimus> the malware-ridden multibit clone is still online... 25 downloads so far
708 2013-07-13 22:09:52 <saulimus> the author name just changed