1 2014-06-08 01:05:46 <andytoshi> hey, fyi mdev has an assertion failure in #bitcoin. (i don't want to send him here because he doesn't play well with others)
2 2014-06-08 01:13:25 <jcorgan> i got him to run with -reindex
3 2014-06-08 01:13:47 <jcorgan> he was kbanned from #bitcoin a couple days ago, but seems to have learned his lesson
4 2014-06-08 02:23:02 <metacoin> I am wondering, is it wrong to create a blockchain bootstrap file by simply zipping blkindex.dat and blk000*.dat files? Why are the blk000*.dat files generally concatenated rather than compressed in this process?
5 2014-06-08 02:23:43 <metacoin> I feel like the answer may be obvious, but I'm fried from writing HTML and javascript all day, which are not my favorite things to do.
6 2014-06-08 02:30:37 <deego> metacoin: just copy the subdirectories such as chainstate, etc., to bootstrap.
7 2014-06-08 02:31:55 <metacoin> So I can create a .zip of those files/directories and that would be ok? What is the purpose of bootstrap.dat?
8 2014-06-08 02:33:23 <deego> metacoin: afaik, they were used by older bitcoin versions.
9 2014-06-08 02:33:32 <metacoin> deego: Thanks. :)
10 2014-06-08 02:56:16 <LightningFury> hi
11 2014-06-08 02:58:06 <LightningFury> little question what mean server mininput
12 2014-06-08 03:17:40 <PRab> metacoin: O
13 2014-06-08 03:17:54 <PRab> metacoin: I'm a bit late, but bootsrap.dat is untrusted.
14 2014-06-08 03:18:23 <metacoin> PRab: I'm still here! Yes, I'm aware it's not a good 'decentralized' solution, and bootstrap.zip wouldn't be any better right?
15 2014-06-08 03:18:55 <PRab> metacoin: Even if a malicious person provided bootstrap.dat, it wouldn't harm your node. If you let a malicious party modify the blk...dat files, that would cause your node to behave incorrectly.
16 2014-06-08 03:19:59 <metacoin> There is a functional difference between concatenating the blk..dat files and compressing them?
17 2014-06-08 03:20:36 <PRab> They will be bigger if you just concat them...
18 2014-06-08 03:20:55 <PRab> Other than that, they must come from a trusted party or be created by your own node.
19 2014-06-08 03:21:35 <metacoin> Yeah, I'm just creating a bootstrap for my own purposes. I was wondering if there was a functional difference between using bootstrap.dat and simply compressing everything into a zip and unzipping it elsewhere.
20 2014-06-08 03:21:55 <Luke-Jr> metacoin: a huge difference
21 2014-06-08 03:22:04 <Luke-Jr> metacoin: bootstrap.dat is verified when it gets loaded
22 2014-06-08 03:22:16 <Luke-Jr> ZIPping the dbs may be a security issue
23 2014-06-08 03:22:33 <Luke-Jr> that's why bootstrap.dat is still decentralised
24 2014-06-08 03:22:33 <metacoin> I see. Thank you, Luke-Jr.
25 2014-06-08 03:22:48 <metacoin> That is a big difference.
26 2014-06-08 03:23:03 <PRab> metacoin: what Luke-Jr said. If you trust the zip file, then that there is no practical difference, but there is a huge functional difference.
27 2014-06-08 03:23:15 <metacoin> Understood. Thanks everyone. That answers my question.
28 2014-06-08 03:23:18 <Luke-Jr> metacoin: btw, blkindex.dat is dead. you can delete it if you use 0.8.x or newer
29 2014-06-08 03:23:54 <metacoin> Deleting things in the Bitcoin directory makes me uneasy, but ok! I'll keep that in mind :P
30 2014-06-08 03:40:46 <bassguitarman> Anyone around?
31 2014-06-08 03:41:53 <PRab> bassguitarman: Yep, lots of people. If you have a question, just ask it.
32 2014-06-08 07:55:41 <michagogo> metacoin: if you've upgraded from pre-0.8 to 0.8 or later, blkindex.dat is probably just a hard link to a file in blocks/
33 2014-06-08 07:56:38 <michagogo> metacoin: also, bootstrap.dat is effectively just the blk#####.dat files concatenated
34 2014-06-08 07:56:57 <michagogo> (Except that it's cleaned up, and is only the main chain)
35 2014-06-08 07:57:20 <michagogo> Sure, you could compress it if you wanted
36 2014-06-08 07:58:00 <michagogo> If you're bootstrapping a new node from a current trusted one, you can copy blocks/ and chainstate/ and skip having to reindex
37 2014-06-08 08:08:53 <gdm85> michagogo: is there a sensible advantage in compression?
38 2014-06-08 08:10:22 <michagogo> gdm85: well, I don't know how well it compresses
39 2014-06-08 08:10:39 <gdm85> I am going to give it a try later
40 2014-06-08 08:10:46 <michagogo> I know there's a lot of random data in there (hashes of blocks and transactions)
41 2014-06-08 08:11:05 <michagogo> (And pubkeys, and signatures)
42 2014-06-08 08:12:18 <michagogo> I mean, there are some series of 0 bytes and 255 bytes
43 2014-06-08 08:13:02 <michagogo> And a few sequences in the single-digit number of bytes that recur for most transactions
44 2014-06-08 08:14:02 <michagogo> But the bulk of data is random... Each block header is mostly (64/80 bytes) random
45 2014-06-08 08:14:23 <michagogo> (32-bit prevblockhash and merkle root)
46 2014-06-08 08:15:23 <sipa> with per-block zipping ypu get 10-20% maybe
47 2014-06-08 08:15:33 <michagogo> And then each transaction has 32 bytes prevtx, 33/65 bytes pubkey, and a bunch of bytes (32? Idk) signature per input
48 2014-06-08 08:15:34 <sipa> and that break per-transaction indexing
49 2014-06-08 08:15:49 <dsnrk> I got ~30% by compressing block files.
50 2014-06-08 08:16:04 <michagogo> And 20 bytes per output
51 2014-06-08 08:16:18 <sipa> entire block files, with heavier algorithms than zip, yes
52 2014-06-08 08:16:40 <sipa> bootstrap.dat could be zipped, true
53 2014-06-08 08:16:45 <sipa> but not the on-disk files
54 2014-06-08 08:17:00 <michagogo> Can a zipped file be append-only?
55 2014-06-08 08:17:28 <sipa> unsure whether the import algorithm for bootstrap.dat works on nonseekable data though
56 2014-06-08 08:17:37 <dsnrk> I suppose that's how gzip works
57 2014-06-08 08:17:57 <sipa> zip is append only yes, but you need the conpression state to append
58 2014-06-08 08:18:10 <sipa> which implies decompressing what goes before
59 2014-06-08 08:19:01 <michagogo> sipa: well, that's not a total dealbreaker
60 2014-06-08 08:19:14 <wumpus> dsnrk: probably on the initial files only?
61 2014-06-08 08:19:32 <michagogo> The major benefit of append-only is in the ability to update a torret
62 2014-06-08 08:19:36 <dsnrk> if you were curious, compressing a block file with xz gets them down to 90MB from 134MB
63 2014-06-08 08:19:38 <michagogo> Torrent*
64 2014-06-08 08:19:42 <wumpus> I've found that latter block files compress way worse
65 2014-06-08 08:19:49 <gdm85> dsnrk: you mean -70%?
66 2014-06-08 08:20:00 <michagogo> dsnrk: wumpus makes a good point. Which files did you test on?
67 2014-06-08 08:20:05 <wumpus> no he means -30%
68 2014-06-08 08:20:06 <dsnrk> wumpus: no, I compressed a block 3 from the head
69 2014-06-08 08:20:15 <dsnrk> *block file
70 2014-06-08 08:20:39 <gdm85> btw, bitcoin.conf is shared with bitcoin-qt right? but only some options do matter for bitcoin-qt..
71 2014-06-08 08:20:44 <michagogo> gdm85: yes
72 2014-06-08 08:20:49 <gdm85> michagogo: ok, thanks
73 2014-06-08 08:20:52 <wumpus> gdm85: all options that matter for bitcoind also matter for bitcoin-qt
74 2014-06-08 08:21:01 <gdm85> but not the other way round
75 2014-06-08 08:21:11 <michagogo> Right
76 2014-06-08 08:21:13 <wumpus> there are a few bitcoin-qt only options, but not much
77 2014-06-08 08:21:56 <michagogo> Even some things that you'd think were -Qt only aren't
78 2014-06-08 08:22:04 <michagogo> -server, for example
79 2014-06-08 08:22:07 <wumpus> the bitcoin-qt-specific configuration is in the GUI itself, not command line options
80 2014-06-08 08:22:17 <gdm85> I have to admit I haven't played with the daemon for a while now, and I do not understand rpcconnect option: you could run a node only to be a 'proxy' of RPC calls?
81 2014-06-08 08:22:26 <michagogo> (How do you stop such a bitcoind? Does it take SIGTERM well?)
82 2014-06-08 08:22:37 <wumpus> michagogo: yes
83 2014-06-08 08:22:39 <michagogo> gdm85: nah
84 2014-06-08 08:22:49 <wumpus> gdm85: rpcconnect is for the command line interface
85 2014-06-08 08:22:54 <michagogo> I think that's for when you use it for sending rpc
86 2014-06-08 08:22:56 <wumpus> gdm85: it doesn't do anything in server mode
87 2014-06-08 08:22:59 <gdm85> michagogo: what does that do? -server for bitcoind I mean
88 2014-06-08 08:23:03 <michagogo> (Note that this is now deprecated)
89 2014-06-08 08:23:18 <gdm85> wumpus: ah. so bitcoin-cli is reading options also from bitcoin.conf?
90 2014-06-08 08:23:18 <michagogo> gdm85: if server=0 bitcoind doesn't listen for rpc
91 2014-06-08 08:23:22 <wumpus> michagogo: right, the bitcoin-cli functionality will be removed from bitcoind
92 2014-06-08 08:23:26 <michagogo> gdm85: indeed
93 2014-06-08 08:23:34 <wumpus> gdm85: yes, that makes it easy to use it with the same password/user/port
94 2014-06-08 08:23:45 <gdm85> a bit like mysql, I see
95 2014-06-08 08:23:46 <michagogo> wumpus: hm, does SIGTERM exist on Wondows?
96 2014-06-08 08:23:50 <michagogo> Windows*
97 2014-06-08 08:23:58 <wumpus> michagogo: I have no clue
98 2014-06-08 08:24:13 <gdm85> sorry to say the obvious, but having [sections] would be nicer there
99 2014-06-08 08:24:15 <michagogo> Also, I like how my phone knows to capitalize SIGTERM
100 2014-06-08 08:24:31 <wumpus> gdm85: would have been, but we're stuck with what we have now
101 2014-06-08 08:24:47 <wumpus> gdm85: you can already divide it into sections with comments if you want
102 2014-06-08 08:24:50 <gdm85> wumpus: in lack of that, we could still make some comment blocks
103 2014-06-08 08:24:51 <gdm85> yep
104 2014-06-08 08:24:59 <michagogo> Well, you could have a new format
105 2014-06-08 08:25:16 <michagogo> With auto conversion or backwards compatibility or something
106 2014-06-08 08:25:20 <gdm85> because I just submitted a PR to update the default debian bitcoin.conf, but now I am thinking such comment-made sections would be nice too
107 2014-06-08 08:25:23 <wumpus> you could, but there's not much chance it would be accepted, just too much bother
108 2014-06-08 08:25:27 <dsnrk> just for the hell of it I compressed a sample of block files from my directory. blocks 0, 40, 80, 120, 146 went from 134MB to 88MB, 74MB, 82MB, 90MB, 90MB in turn. worked a lot better than I expected.
109 2014-06-08 08:25:45 <michagogo> dsnrk: I guess you mean block files
110 2014-06-08 08:26:09 <dsnrk> mm, said that in the first bit but missed it in the second.
111 2014-06-08 08:26:58 <wumpus> I suppose you could change the config file format if you combine it with an overall options system redesign, see discussion in https://github.com/bitcoin/bitcoin/issues/1044 and https://github.com/bitcoin/bitcoin/pull/4194
112 2014-06-08 08:27:00 <gdm85> I am going to specify all the bitcoind command line options in the config and ignore the others. dsnrk as you said those will be picked up without issues
113 2014-06-08 08:27:11 <wumpus> but tacking it on the current system... please no
114 2014-06-08 08:27:41 <gdm85> wumpus: eh, for me it would be already good enough that next Debian package ships with a better documented config
115 2014-06-08 08:27:42 <michagogo> wumpus: yeah, I think that's what I meant
116 2014-06-08 08:28:17 <michagogo> It wouldn't make sense to just make it something more restrictive/strict than what we have now with the same options
117 2014-06-08 08:28:19 <gdm85> but first I'd have to learn better about all the currently available options to make a comprehensive bitcoin.conf with good comments on it
118 2014-06-08 08:28:43 <wumpus> gdm85: yes, that'd be nice
119 2014-06-08 08:28:54 <wumpus> michagogo: at least a start was made with catgorizing the options with the --help output
120 2014-06-08 08:29:05 <michagogo> Yep
121 2014-06-08 08:29:25 <dsnrk> gdm85: the generate, 4way and IP receive stuff all need to go. the rpcssl stuff is of questionable value, nobody should be exposing it externally anyway.
122 2014-06-08 08:29:59 <michagogo> Is the contrib/debian dir actually what gets used in the ppa?
123 2014-06-08 08:30:15 <gdm85> michagogo: good question
124 2014-06-08 08:30:45 <gdm85> michagogo: Debian has ppas too? I thought it was Ubuntu-specific :s
125 2014-06-08 08:30:54 <wumpus> michagogo: yes, it is what is used for the bluematt's ppa
126 2014-06-08 08:30:55 <michagogo> (And maybe the Debian package? (a bit less likely))
127 2014-06-08 08:31:05 <michagogo> gdm85: no, PPAs are Ubuntu
128 2014-06-08 08:31:17 <michagogo> But Ubuntu uses Debiany packaging
129 2014-06-08 08:31:34 <michagogo> wumpus: yeah, guessed it probably was
130 2014-06-08 08:32:23 <michagogo> Also, good news: there's a SRU pending to remove bitcoind from Precise (12.04, supported for 5 years, ships 0.3.24)
131 2014-06-08 08:32:24 <gdm85> dsnrk: ok, I will update my PR. but what do you mean with exposing rpcssl externally?
132 2014-06-08 08:33:05 <michagogo> an* SRU, I guess
133 2014-06-08 08:33:34 <gdm85> michagogo: because the version they ship is terribly broken?
134 2014-06-08 08:33:47 <michagogo> gdm85: yep
135 2014-06-08 08:33:51 <wumpus> yes, they ship 0.3.x
136 2014-06-08 08:33:55 <dsnrk> gdm85: well the example debian config includes bits for allowing RPC connections to things outside of localhost, and then goes on to talk about rpcssl. it's sort of something nobody should ever be doing, and if they are doing it they're probably doing something wrong.
137 2014-06-08 08:34:00 <wumpus> that's truly, desperately awful
138 2014-06-08 08:34:17 <michagogo> At some point, the Debian maintainers did the smart thing and pulled it out of the stable versions
139 2014-06-08 08:34:34 <michagogo> It's in unstable only, where it can be freely upgraded
140 2014-06-08 08:34:53 <wumpus> wow, that sounds like an actutally sensible thing to do
141 2014-06-08 08:34:55 <gdm85> dsnrk: eheh..I felt like I was dealing with coprolitis while reading it :D
142 2014-06-08 08:35:13 <michagogo> wumpus: yeah
143 2014-06-08 08:35:31 <michagogo> Unfortunately, Ubuntu syncs packages from Debian
144 2014-06-08 08:35:35 <gdm85> yeah, better patch it or drop it
145 2014-06-08 08:35:47 <dsnrk> gdm85: for that check out the receive by IP thing, that hasn't been in bitcoind forever. nasty insecure.
146 2014-06-08 08:35:49 <michagogo> Either testing or unstable
147 2014-06-08 08:36:30 <michagogo> I don't know if it got pulled from unstable, or if it got pulled before Debian got responsible about it
148 2014-06-08 08:36:48 <michagogo> But they pulled it in to their stable releases
149 2014-06-08 08:37:10 <sipa> debian had itnin a stable release too, no?
150 2014-06-08 08:37:24 <michagogo> sipa: at one point, I think they did
151 2014-06-08 08:37:50 <michagogo> IIRC when the maintainers changed the new ones were smart enough to realize it was stupid
152 2014-06-08 08:37:54 <sipa> dsnrk: anyway, we can't conpress the block files, as that means losing the ability to read just one block from it
153 2014-06-08 08:38:03 <sipa> dsnrk: we could comoress bootstrap.dat
154 2014-06-08 08:38:07 <gdm85> dsnrk: already removed that
155 2014-06-08 08:38:26 <michagogo> So versions in various stages of oldness got into Ubuntu's precise, quantal, raring, and saucy releases
156 2014-06-08 08:38:28 <sipa> dsnrk: compressing per-block in the files could work, but has lower compression rate
157 2014-06-08 08:38:45 <gdm85> right now what I am trying to accomplish (via config) is an isolated regtest node, that connects only to other specified regtest nodes or nothing if they are not online
158 2014-06-08 08:38:58 <sipa> -connect
159 2014-06-08 08:39:05 <dsnrk> sipa: I don't think I ever thought it was a sane suggestion, I just found it interesting. I'm actually going to see how it works on bootstrap.dat in a minute when linearize.py finishes doing it's thing.
160 2014-06-08 08:39:15 <michagogo> (I wrote to them in (iirc) October and got the package removed from Trusty, and blacklisted from syncing in the future)
161 2014-06-08 08:39:17 <sipa> dsnrk: ok
162 2014-06-08 08:40:29 <michagogo> At the moment, they have 0.3.24 in precise and 0.8.5 (I think) in saucy