1 2017-09-15 00:09:57 <lrvick> looks like I was confusing rpc calls from 0.10 docs. it is happy now
  2 2017-09-15 00:11:37 <CryptAxe> yay!
  3 2017-09-15 07:44:52 <Cocodude> Hey all. Just upgrading to v0.15 (good work btw), but there's one change you've put in that I don't like and I think it's not far off being a bug.
  4 2017-09-15 07:45:15 <Cocodude> You now require that the wallet.dat file is a regular file, and don't allow it to be a symlink (Error: Error loading wallet wallet.dat. -wallet filename must be a regular file.)
  5 2017-09-15 07:45:43 <Cocodude> And if I specify the real wallet in the -wallet command line, the client complains with "-wallet parameter must only specify a filename (not a path)"
  6 2017-09-15 07:46:19 <Cocodude> This is a bit of a pain - I store the wallet on an encrypted partition so this now stops me from being able to do this effectively. Why was this change put in (especially the lack of love to symlinks)?
  7 2017-09-15 07:50:40 <esotericnonsense> i believe that it's to enforce locking semantics on the wallet file (it has always been necessary that the wallet file exist in the datadir and symlinking is a hack around that)
  8 2017-09-15 07:50:59 <esotericnonsense> e.g. two bitcoin instances operating on the same wallet file would be bad news
  9 2017-09-15 07:51:39 <Cocodude> Ah, yes, that would be bad
 10 2017-09-15 07:52:36 <Cocodude> I wish there was a way to override this check then I guess, but that sounds like a lot of work which only benefits people doing things in a special way, like me
 11 2017-09-15 07:53:30 <Cocodude> I don't imagine I'm the only one doing this though - I don't want to keep the entire blockchain encrypted (it's not exactly a secret) but I want the wallet to be. It provides some security as everything important as I have encryption at rest.
 12 2017-09-15 07:53:47 <Cocodude> I can't think of a neat way to do this any other way off the top of my head
 13 2017-09-15 07:53:58 <arubi> how about having the datadir on the encrypted filesystem and symlink the big directories out?
 14 2017-09-15 07:54:12 <esotericnonsense> i'm not sure why you wouldn't keep the blockchain encrypted, overhead is ~zero on modern CPUs
 15 2017-09-15 07:54:14 <Cocodude> "neat way" :)
 16 2017-09-15 07:54:25 <esotericnonsense> aesni is 3GB/sec per core
 17 2017-09-15 07:54:32 <Cocodude> esotericnonsense: Encrypted for me is also on a SSD as the wallet has a lot of I/O on it
 18 2017-09-15 07:54:36 <Cocodude> And SSD space is limited
 19 2017-09-15 07:55:01 <esotericnonsense> you could do it backwards and symlink blocks/chainstate ? :P
 20 2017-09-15 07:55:10 <arubi> that's what I was suggesting :)
 21 2017-09-15 07:55:14 <Cocodude> Yeah, it's possible, but even more ugly than just symlinking the wallet
 22 2017-09-15 07:55:38 <esotericnonsense> arubi: sorry, didn't see that :P
 23 2017-09-15 07:56:10 <Cocodude> arubi/esotericnonsense: It would work, I agree, but what happens when Bitcoin Core adds another special folder/file there that I don't really want on the SSD?
 24 2017-09-15 07:56:37 <arubi> well like you said, it's a specialized setup.  you'll have to adapt
 25 2017-09-15 07:56:56 <Cocodude> Thinking about it again, do you think it really is that specialised?
 26 2017-09-15 07:57:05 <Cocodude> Wallets do have a totally different security profile to the rest of the directory
 27 2017-09-15 07:57:05 <esotericnonsense> i'm confused by 'wallet has a lot of IO on it' really, unless you're using it as a merchant and chucking many tx per second at it
 28 2017-09-15 07:57:13 <Cocodude> esotericnonsense: Guess what, I am!
 29 2017-09-15 07:57:17 <esotericnonsense> :P
 30 2017-09-15 07:57:18 <arubi> it does have a bunch of io
 31 2017-09-15 07:57:29 <Cocodude> OK, not many per second, but often a few a minute
 32 2017-09-15 07:57:43 <Cocodude> When a new block comes in, there's lots of fun that happens with wallet IO too
 33 2017-09-15 07:57:53 <esotericnonsense> my understanding might be outdated but I believe that work done on wallet.dat can leak outside of the file anyway
 34 2017-09-15 07:58:01 <esotericnonsense> only private keys are treated with care
 35 2017-09-15 07:58:29 <Cocodude> That's the only security-critical thing though. If metadata or whatever spills out, then I can live with that (although I thought all that was in the wallet too)
 36 2017-09-15 07:58:42 <arubi> it was, but only keys are encrypted
 37 2017-09-15 07:58:46 <esotericnonsense> can you not use the built in encryption?
 38 2017-09-15 07:58:48 <arubi> it is*
 39 2017-09-15 07:59:13 <Cocodude> It's possible, but it's a lot neater all round to just have block encryption on a partition for me
 40 2017-09-15 07:59:41 <arubi> wait, so it's unencrypted by itself?
 41 2017-09-15 07:59:44 <Cocodude> Yes
 42 2017-09-15 07:59:54 <arubi> and core is running, and it is loaded?
 43 2017-09-15 08:00:02 <Cocodude> Yes
 44 2017-09-15 08:00:08 <Cocodude> Now you're going to tell me that's a stupid thing to do... :/
 45 2017-09-15 08:00:13 <arubi> not sure how encrypted filesystem helps?
 46 2017-09-15 08:00:16 <esotericnonsense> Yes
 47 2017-09-15 08:00:17 <Cocodude> Ah, phew :)
 48 2017-09-15 08:00:18 <esotericnonsense> ;)
 49 2017-09-15 08:00:26 <Cocodude> It helps as I have encryption at rest
 50 2017-09-15 08:00:35 <Cocodude> Someone steals the server, I should be safe
 51 2017-09-15 08:00:39 <arubi> yea, but the wallet is wide open all the time
 52 2017-09-15 08:00:50 <Cocodude> Not if they've stolen the server and it's powered off or rebooted!
 53 2017-09-15 08:00:57 <esotericnonsense> do you have encrypted swap? (now i'm just being an arse :P)
 54 2017-09-15 08:01:00 <Cocodude> Or just taken out the HDDs or whatever
 55 2017-09-15 08:01:02 <Cocodude> esotericnonsense: Yes, I do
 56 2017-09-15 08:01:05 <esotericnonsense> :D
 57 2017-09-15 08:01:22 <arubi> yea if the pc turns off then it's encrypted, but most times it's not off
 58 2017-09-15 08:01:45 <esotericnonsense> it might be better to ask this in #bitcoin-core-dev. the specific commit seems to be here: https://github.com/bitcoin/bitcoin/commit/a6da027d83e48f05c933149ff89c9b9ad5ced915#diff-b2bb174788c7409b671c46ccc86034bd
 59 2017-09-15 08:02:03 <arubi> Cocodude, is that actually a hot wallet then?
 60 2017-09-15 08:02:13 <Cocodude> arubi: Yes
 61 2017-09-15 08:02:32 <Cocodude> esotericnonsense: Thanks for pointing out the commit
 62 2017-09-15 08:02:47 <arubi> I mean, hopefully larger funds are stored fully encrypted and offline right?
 63 2017-09-15 08:02:57 <Cocodude> Yes, indeed
 64 2017-09-15 08:02:59 <esotericnonsense> i suppose the easiest option if you're fully committed to doing this
 65 2017-09-15 08:03:01 <Cocodude> I'm only on about the hot wallet
 66 2017-09-15 08:03:03 <esotericnonsense> is to rip that out and recompile
 67 2017-09-15 08:03:10 <arubi> alright, understood
 68 2017-09-15 08:03:10 <esotericnonsense> but i'd try and work out if there's a good reason for it
 69 2017-09-15 08:03:25 <arubi> yea I guess that's the best you can do in this case if the wallet is used for sending a lot
 70 2017-09-15 08:03:30 <Cocodude> esotericnonsense: Yeah, it would be trivial to just comment out a few lines, but it's only a very short term solution
 71 2017-09-15 08:04:01 <Cocodude> arubi: Indeed. There can't be negligable funds in the hot wallet either as the amount needed in there fluctuates quite a bit).
 72 2017-09-15 08:04:04 <esotericnonsense> if you're using rpc you can also just use the built in encryption and unlock it with walletpassphrase for an absurd amount of time, i'm not sure if there's an upper bound
 73 2017-09-15 08:04:21 <Cocodude> esotericnonsense: I need a new wallet as well for that
 74 2017-09-15 08:04:32 <Cocodude> My wallet is, ehm, quite big in terms of size
 75 2017-09-15 08:04:57 <arubi> I think multiwallet is in 0.15.0 ?
 76 2017-09-15 08:05:11 <Cocodude> Yeah I think that would be helpful in migrating.
 77 2017-09-15 08:05:24 <esotericnonsense> it should be doable, it'll just take a while, no? i'd be surprised if the enc/dec wasn't using aesni
 78 2017-09-15 08:05:36 <Cocodude> There is some slowdown with my large wallet (takes a few seconds to send a tx) so some code core which is O(n) or similar
 79 2017-09-15 08:06:11 <Cocodude> esotericnonsense: Doesn't stop the requirement to have the wallet on a SSD - the wallet really does appear to use a lot of I/O.
 80 2017-09-15 08:06:23 <esotericnonsense> ah yeah, that's true
 81 2017-09-15 08:06:59 <esotericnonsense> well, wrt 'another special file/folder you don't really want on the SSD' and the symlinking blocks/chainstate approach
 82 2017-09-15 08:07:22 <esotericnonsense> it seems like it would be obvious ahead of time if another large set of data is produced
 83 2017-09-15 08:07:51 <esotericnonsense> e.g. chucking a few extra megs on the SSD doesn't seem to be that much of a problem, only if it's actually say, >1GB
 84 2017-09-15 08:08:09 <Cocodude> That's true
 85 2017-09-15 08:08:35 <Cocodude> It just feels a lot more messy. There's only one thing I want moved, so it seems nicer to move that one thing (rather than everything *but* that one thing)
 86 2017-09-15 08:09:35 <arubi> well having peers.dat , mempool.dat, debug.log .. etc on the encrypted ssd will help with io and privacy if the pc is stolen
 87 2017-09-15 08:09:37 <esotericnonsense> the entire contents of my .bitcoin folder aside from chainstate and blocks is about 15MB, you might have a bigger debug.log and mempool.dat could be a few meg i suppose
 88 2017-09-15 08:09:44 <esotericnonsense> yes, that too
 89 2017-09-15 08:09:47 <esotericnonsense> debug.log especially
 90 2017-09-15 08:10:53 <esotericnonsense> having chainstate on SSD would probably improve performance too if you can spare the 3GB
 91 2017-09-15 08:11:00 <esotericnonsense> depends on your dbcache setting
 92 2017-09-15 08:11:03 <Cocodude> Yeah, just the chainstate, blocks and debug.log really shouldn't be on the SSD
 93 2017-09-15 08:11:11 <Cocodude> Hmm, good point about chainstate
 94 2017-09-15 08:11:41 <Cocodude> Actually, I just use the default dbcache values. With the wallet on the SSD, I think I'm more CPU bound because the wallet is so huge.
 95 2017-09-15 08:15:40 <Cocodude> Just thinking about this, all this commit does is to stop the wallet from being specified twice on the -wallet command line (one regular, one symlink to that file). I think that sounds like a more niche "security improvement" than my setup.
 96 2017-09-15 08:15:50 <Cocodude> It wouldn't stop, for instance, two instances of Core running, both pointing to the same wallet
 97 2017-09-15 08:16:13 <esotericnonsense> Cocodude: you can't run two copies of bitcoin from the same datadir without hacking
 98 2017-09-15 08:16:44 <Cocodude> Oh yes, the -wallet has to be in the same datadir as you can't specify a path. Got it.
 99 2017-09-15 08:16:51 <esotericnonsense> yes :P
100 2017-09-15 08:22:19 <Cocodude> I've added my comment to https://github.com/bitcoin/bitcoin/pull/10885 so we'll see if it gets a response
101 2017-09-15 08:22:38 <Cocodude> This is going to be especially fun as other alts pick up on this and I need to shuffle everything around for Litecoin, Doge etc.
102 2017-09-15 08:26:40 <esotericnonsense> thinking about it, if that's the rationale (two wallet files symlinked to each other in the same instance), then it should be possible to determine that at startup, if a bit of a faff
103 2017-09-15 08:27:25 <esotericnonsense> it seems like there must be a way to determine if a file and a symlink point to the same file, if not, you could hash them both and compare (though i suppose with a large wallet that might take a while :P)
104 2017-09-15 08:29:21 <Cocodude> Youch, that seems like a lot of work
105 2017-09-15 08:29:33 <Cocodude> Honestly, I think what's been done here is too much handholding anyway (at the expensive of people like me).
106 2017-09-15 08:29:50 <Cocodude> If you're symlinking, you're doing something special anyway and should know what you're doing
107 2017-09-15 08:30:16 <Cocodude> Also, hashing wouldn't work - it would be reasonable to have the same, precise wallet on there twice
108 2017-09-15 08:30:26 <Cocodude> (just two physically different files, but the same contents)
109 2017-09-15 08:32:44 <Cocodude> Got a nice response at https://github.com/bitcoin/bitcoin/pull/10885
110 2017-09-15 08:33:51 <esotericnonsense> yeah, that's about in line with my expectation, that wallet.dat is not self contained
111 2017-09-15 08:34:07 <Cocodude> I'm a little confused though - what does it rely on that, say, a -rescan can't solve?
112 2017-09-15 08:34:08 <esotericnonsense> it's been a while since I looked at it
113 2017-09-15 08:38:22 <esotericnonsense> another 'supported' way unless you have very very limited storage would be to move everything and prune to the minimum, then you'd have about 4GB on SSD (plus wallet.dat) in total, though that breaks rescan
114 2017-09-15 08:38:34 <esotericnonsense> it's a tricky one
115 2017-09-15 08:38:44 <Cocodude> Yeah, I don't want to do that really, and like running a full node for altruistic reasons
116 2017-09-15 08:50:34 <Cocodude> Now sipa is saying, "There are no dependencies between the wallet and the rest of the datadir, in either direction."
117 2017-09-15 08:50:49 <Cocodude> That's what I thought, but didn't sipa just say the opposite before?
118 2017-09-15 08:51:42 <esotericnonsense> wallet, not wallet.dat
119 2017-09-15 08:52:49 <esotericnonsense> bitcoin wiki is pretty out of date, but seems to claim that there's a database/ folder that stores bdb state
120 2017-09-15 08:53:19 <esotericnonsense> my nodes are running with disablewallet so don't have it here
121 2017-09-15 08:53:43 <Cocodude> Yikes, does that mean I should be backing that up too?
122 2017-09-15 08:54:11 <esotericnonsense> it only exists during runtime, so i expect that if there's a hard shutdown, that it'll be used for recovery
123 2017-09-15 08:54:39 <Cocodude> Indeed, which means it's probably best to store it in the same partition as the wallet.dat
124 2017-09-15 08:56:32 <arubi> BDB2549 [4][397219] Checkpoint record, ckp_lsn: [4][397088], timestamp: Wed Sep 13 20:51:20 2017
125 2017-09-15 08:56:32 <arubi> that file has data like BDB2565 [4][397132] Non-transactional update, log type: 2, fileid: 1.
126 2017-09-15 08:56:32 <arubi> . Total checkpoint: 4
127 2017-09-15 08:56:44 <esotericnonsense> by the way, it's great to see you guys are still running, feels like an eternity since I first saw you pop up :)
128 2017-09-15 08:57:16 <arubi> (yes looks like it's related to bdb and wallet.dat)
129 2017-09-15 08:57:24 <Cocodude> arubi: Sounds like it'll be quite I/O intensive too
130 2017-09-15 08:57:39 <arubi> that's right
131 2017-09-15 08:57:58 <arubi> I mean, I'm doing a couple of addnewaddress now and it's writing a lot of that ^^ each time
132 2017-09-15 08:58:22 <Cocodude> Is this the database folder?
133 2017-09-15 08:58:30 <arubi> yea, the 000000..log file
134 2017-09-15 08:59:03 <arubi> run `db_log_verify -c | less` (if you have db-utils)
135 2017-09-15 08:59:04 <Cocodude> Yeah. I guess a flush is when they's all put into the actual wallet.dat?
136 2017-09-15 08:59:17 <Cocodude> Nah, I just have the library here
137 2017-09-15 08:59:18 <wumpus> yes, the database/ folder has bdb state, it's compacted into wallet.dat extremely regularly so it will usually be empty
138 2017-09-15 08:59:38 <Cocodude> Cheers wumpus
139 2017-09-15 08:59:52 <Cocodude> I guess then, technically, it's a little safer to store that in the same place I'm storing the wallet.dat
140 2017-09-15 09:00:01 <wumpus> after a clean shutdown, there is no need to backup anything but wallet.dat
141 2017-09-15 09:00:06 <Cocodude> As if I want to recover from a hard crash, they're more likely to be in sync on the same parittion
142 2017-09-15 09:00:16 <wumpus> yes
143 2017-09-15 09:00:19 <Cocodude> Yeah, only talking about hard crash, and I'd probably consider using a backup anyway
144 2017-09-15 09:01:26 <Cocodude> wumpus: What's your take on my rant?
145 2017-09-15 09:01:29 <wumpus> what I usually do is place the "blocks" folder on another partition, sometimes "chainstate" too, and keep the rest together
146 2017-09-15 09:01:43 <Cocodude> Ooooh, so I'm not the only one doing crazy symlink stuff
147 2017-09-15 09:01:47 <wumpus> no
148 2017-09-15 09:02:13 <Cocodude> wumpus: Do you think the symlink check is a good thing for wallet.dat or a bad thing?
149 2017-09-15 09:03:03 <wumpus> by far not, I do crazy hardlink stuff too, like https://gist.github.com/laanwj/3c4614a23e072cbb3d39090da1834a68
150 2017-09-15 09:03:37 <wumpus> what should it do if the wallet is a symlink?
151 2017-09-15 09:03:54 <Cocodude> It won't start, quite simply. There's a new check in there.
152 2017-09-15 09:03:55 <wumpus> probably just reject it, as all database state is expected to be in one directory by bdb
153 2017-09-15 09:04:00 <wumpus> oh that's good
154 2017-09-15 09:04:06 <Cocodude> Oh, I don't think it's good
155 2017-09-15 09:04:30 <Cocodude> If I symlinked both the database folder and wallet to a different partition, it would still error out
156 2017-09-15 09:04:40 <wumpus> bdb works with a database *directory*, where all databases need to be
157 2017-09-15 09:05:18 <esotericnonsense> so you'd basically want wallets/database and wallets/wallet.dat
158 2017-09-15 09:05:25 <Cocodude> Yes, just clicked
159 2017-09-15 09:05:35 <esotericnonsense> within the datadir or otherwise, and then this wallets directory could be symlinked off
160 2017-09-15 09:05:38 <wumpus> allowing symlinking would allow crazy things like linking multiple wallets from multiple partitions w/ multiwallet
161 2017-09-15 09:05:47 <wumpus> don't symlink bdb files
162 2017-09-15 09:05:58 <wumpus> could add a walletdir option that sets a different wallet database directory for the data directory, if you really need that
163 2017-09-15 09:06:05 <wumpus> s/for/from
164 2017-09-15 09:06:16 <Cocodude> Yes, that's what sipa is suggesting
165 2017-09-15 09:06:17 <wumpus> but all bdb files need to be together and not be symlinks
166 2017-09-15 09:07:01 <wumpus> ok, that's the way forward then
167 2017-09-15 09:07:33 <Cocodude> Sounds like the neatest way, but not a quick solution.
168 2017-09-15 09:17:35 <wumpus> doesn't sound particularly difficult or much work, but maybe I'm missing some subtlety
169 2017-09-15 09:18:01 <wumpus> the other option would be to symlink the *non-wallet* things as I do
170 2017-09-15 09:18:47 <Cocodude> Yeah, agreed it's an option, but a bit yucky.
171 2017-09-15 09:19:16 <wumpus> symlinking the wallet is yucky and brittle too
172 2017-09-15 09:19:25 <Cocodude> Agreed it doesn't sound too difficult really, but anything code-wise is likely to take some time to go into a stable release
173 2017-09-15 09:19:56 <Cocodude> It's a bit yucky, but it makes mores sense in terms of "I want this file secured, so I'll move it" rather than "I want this file secured, so I'll move everything but this file"
174 2017-09-15 09:20:38 <wumpus> that's all good in theory, but in practice bdb works with a database context directory, not a file, so it's really a directory that you need to secure
175 2017-09-15 09:21:01 <wumpus> e.g. your scheme falls apart if you have multiple wallets
176 2017-09-15 09:21:25 <Cocodude> Yes, understood there. We're both agreeing that the wallet really should be its own directory.
177 2017-09-15 09:21:41 <wumpus> yes
178 2017-09-15 09:35:40 <Cocodude> Is there an enhancements tracker for Core?
179 2017-09-15 10:40:18 <wumpus> github issues?
180 2017-09-15 10:40:45 <wumpus> or the release notes, for a summary per release
181 2017-09-15 10:43:16 <Cocodude> No, I mean to potentially raise the wallet directory as a new enhancement.
182 2017-09-15 10:43:19 <Cocodude> Not sure if github issues is the right place as the default contents talks about everything to do with bugs
183 2017-09-15 10:59:06 <wumpus> well you can certainly create an issue for an feature request, though the chance of someone picking it up is slim, unless another developer happens to have exactly the same need
184 2017-09-15 12:01:35 <Shaun3811> Hello
185 2017-09-15 12:12:48 <Shaun3811> chatty bunch then
186 2017-09-15 14:53:03 <cedenday> Does anyone have more info on how SegWit is going to be supported in QT?
187 2017-09-15 14:55:18 <cedenday> IMO they should be default with some sort of flag disabling that behaviour.
188 2017-09-15 15:00:57 <wumpus> I don't think that's decided yet, there is nothing implementing that at least
189 2017-09-15 15:01:10 <wumpus> probably will be some setting
190 2017-09-15 18:26:46 <esotericnonsense> Cocodude: see https://github.com/bitcoin/bitcoin/pull/11343
191 2017-09-15 18:27:40 <esotericnonsense> Cocodude: confirmation there that the database folder can store sensitive information if the wallet is unencrypted