1 2012-08-25 02:45:12 <Tril> [A
2 2012-08-25 04:32:53 <BlueMattBot> Yippie, build fixed!
3 2012-08-25 04:47:35 <Varan> lol
4 2012-08-25 11:48:27 <Joric> Luke-Jr, do you drop satoshidice transactions?
5 2012-08-25 14:25:51 <MC-Eeepc> hot damn cve 20122459 was a doozy
6 2012-08-25 14:26:35 <MC-Eeepc> is that the first real protocol break or what
7 2012-08-25 14:26:51 <MC-Eeepc> apart from the time someone gave themselves 70 gorrilion coins
8 2012-08-25 14:31:05 <Eliel> I don't think it's of a level where you could call it a protocol break. Especially since it was fixable without a protocol change.
9 2012-08-25 14:31:44 <Eliel> granted, in hindsight, pretty crappy design decision there.
10 2012-08-25 14:32:32 <lianj> Eliel: is the same tx included in a block twice allowed anyway?
11 2012-08-25 14:32:33 <kjj> took 3 years to notice the problem.
12 2012-08-25 14:33:01 <Eliel> lianj: no, not anymore.
13 2012-08-25 14:39:40 <TD> it was never allowed
14 2012-08-25 14:39:49 <TD> the only issue was the point at which the checks occurred
15 2012-08-25 14:39:57 <TD> it's a bug, not a design or protocol issue
16 2012-08-25 14:39:59 <MC-Eeepc> from what i read, i wonder if someone coul deliberately do this attack now to make lazy assholes on 0.3 still upgrade
17 2012-08-25 14:40:02 <TD> (was)
18 2012-08-25 14:40:13 <MC-Eeepc> now that its not so dangerous
19 2012-08-25 14:40:57 <MC-Eeepc> it just mongs all unpatched nodes so i read
20 2012-08-25 14:47:16 <kjj> MC-Eeepc: the network won't rebroadcast the attack, so you'd need to specifically target the old client you want to mess with
21 2012-08-25 14:48:43 <lianj> thanks easy :D
22 2012-08-25 16:24:05 <BlueMattBot> Project Bitcoin build #38: ABORTED in 9 hr 52 min: http://jenkins.bluematt.me/job/Bitcoin/38/
23 2012-08-25 17:20:27 <grondilu> On https://en.bitcoin.it/wiki/Running_bitcoind I can read:
24 2012-08-25 17:20:42 <grondilu> ?? By default, Bitcoin (or bitcoind) will look for a file named 'bitcoin.conf' in the bitcoin data directory, but both the data directory and the configuration file path may be changed using the -datadir and -conf command-line arguments. ??
25 2012-08-25 17:20:52 <grondilu> Isn't that a bit circular?
26 2012-08-25 17:21:21 <grondilu> How can it find bitcoin.conf if its location is defined in bitcoin.conf?
27 2012-08-25 17:22:02 <gmaxwell> 'command-line arguments'
28 2012-08-25 17:22:07 <grondilu> Oh sorry.
29 2012-08-25 17:23:02 <grondilu> I mean, is there a way to specify the datadir option for good, without having to use it as command-line argument each time?
30 2012-08-25 17:23:38 <gmaxwell> No, as you point out??? that would be circular.
31 2012-08-25 17:23:51 <gmaxwell> Unless you feel like editing the source to change the default. :)
32 2012-08-25 17:24:01 <grondilu> Well, configuration and datadir should be two different things. There should be a .bitcoinrc file.
33 2012-08-25 17:24:53 <gmaxwell> So, you want a configuration file to store the location of the configuration file. Am I following you?
34 2012-08-25 17:25:13 <grondilu> No, I want a configuration file to store the location of the data directory.
35 2012-08-25 17:25:50 <grondilu> This is quite common on *nix.
36 2012-08-25 17:26:58 <gmaxwell> grondilu: Er, you already have that.
37 2012-08-25 17:27:20 <gmaxwell> You can put the datadir setting in the configuration file in the default datadir, and it will do the right thing.
38 2012-08-25 17:27:56 <grondilu> will it? I'll try.
39 2012-08-25 17:28:18 <gmaxwell> It will.
40 2012-08-25 17:30:09 <grondilu> Oh yes indeed. Sweet.
41 2012-08-25 17:30:38 <grondilu> I thought I had tried it already. I guess not.
42 2012-08-25 18:22:14 <BlueMattBot> Project Bitcoin build #39: ABORTED in 1 hr 57 min: http://jenkins.bluematt.me/job/Bitcoin/39/
43 2012-08-25 18:27:29 <Raccoon> Btw. I think there's something wrong with the systray icon double-click. ie, it doesn't work.
44 2012-08-25 18:27:48 <Raccoon> must right-click to select Show/Hide Bitcoin
45 2012-08-25 18:28:10 <Raccoon> WinXP
46 2012-08-25 18:30:13 <Luke-Jr> Raccoon: it's a single click.
47 2012-08-25 18:30:54 <Raccoon> Single click doesn't work either.
48 2012-08-25 18:31:26 <Raccoon> Double-click should be inherent with single click, too, when fixed
49 2012-08-25 19:09:37 <MC-Eeepc> bit bitcoin moves to the new database
50 2012-08-25 19:09:53 <MC-Eeepc> will it convert the ld format to the new
51 2012-08-25 19:10:05 <MC-Eeepc> or will people have to sync again
52 2012-08-25 19:12:31 <gmaxwell> MC-Eeepc: we will have a migration at startup. It will make your first startup with the new version take a long time as it copies from one to the other.
53 2012-08-25 19:13:31 <MC-Eeepc> so its like syncing but from a local database
54 2012-08-25 19:14:02 <gmaxwell> Right.
55 2012-08-25 19:14:24 <gmaxwell> And faster, simply because the new stuff is faster.
56 2012-08-25 19:14:36 <MC-Eeepc> umm, wont that be even worse than syncing from the network, due to two databases thrashing at once
57 2012-08-25 19:14:59 <MC-Eeepc> and create an incentive for everyone to delete thier blockchain at once to speed it up..........
58 2012-08-25 19:15:21 <gmaxwell> No.
59 2012-08-25 19:15:28 <gmaxwell> I like you, you ask easy questions.
60 2012-08-25 19:15:42 <MC-Eeepc> :)
61 2012-08-25 19:16:15 <MC-Eeepc> so your saying olddb + newdb < olddb alone?
62 2012-08-25 19:17:12 <MC-Eeepc> two DBs thrashin from the same storage medium, i mean
63 2012-08-25 19:17:34 <gmaxwell> MC-Eeepc: reading is not the same as writing, and technically its not even using the old database, it's just reindexing the blocks.
64 2012-08-25 19:18:03 <gmaxwell> and the blocks are just a simple sequential append only file, super cheap to both read and write.
65 2012-08-25 19:18:03 <MC-Eeepc> okie dokie
66 2012-08-25 19:47:33 <Raccoon> Is there a graph of block data size over time?
67 2012-08-25 19:48:58 <Raccoon> And would it be possible to extrapolate some prediction of download size at a future block number
68 2012-08-25 19:49:16 <Raccoon> then adjust the % complete based on that predicted size, and not based on block number
69 2012-08-25 19:49:17 <sipa> we can give an upper bound
70 2012-08-25 19:49:22 <sipa> 144 MB per day
71 2012-08-25 19:49:31 <Raccoon> hmm?
72 2012-08-25 19:49:40 <Raccoon> is that a hard bound?
73 2012-08-25 19:50:03 <sipa> 1440 minutes per day, 1 block per 10 minutes, 144 blocks per day, max 1 MB per block
74 2012-08-25 19:50:21 <Raccoon> i didn't know there was a 1 MB max
75 2012-08-25 19:50:34 <Raccoon> Have we ever maxed out yet?
76 2012-08-25 19:51:07 <Eliel> Raccoon: I think there are some blocks that hit the max.
77 2012-08-25 19:51:16 <Raccoon> that's not good.
78 2012-08-25 19:51:27 <Raccoon> especially if we're calling Bitcoin "young"
79 2012-08-25 19:51:37 <Raccoon> hasn't even reached Point Of Sale
80 2012-08-25 19:51:44 <Eliel> it's a temporary measure
81 2012-08-25 19:51:49 <Eliel> will be lifted eventually.
82 2012-08-25 19:51:56 <sipa> that will be a hard fork
83 2012-08-25 19:51:58 <Raccoon> how can it be lifted
84 2012-08-25 19:52:10 <Raccoon> old clients would start complaining
85 2012-08-25 19:52:17 <sipa> it will have to be planned a long time in advance
86 2012-08-25 19:52:18 <Raccoon> rejecting blocks
87 2012-08-25 19:52:20 <sipa> probably years
88 2012-08-25 19:52:56 <Eliel> Raccoon: look here for an idea of how big the blocks usually are https://blockchain.info/blocks
89 2012-08-25 19:54:17 <Raccoon> so there's no easy way to trend the growing size of blocks into a simple equation
90 2012-08-25 19:54:33 <Raccoon> that would help the progress bar look more realistic
91 2012-08-25 19:54:40 <Raccoon> estimate time to completoin sync
92 2012-08-25 19:55:31 <Raccoon> what about a protocol extension that reports total size to a requesting client?
93 2012-08-25 19:55:34 <sipa> well if satoshidice stops tomorrow, blocks will be significantly smaller again (i suspect_
94 2012-08-25 19:55:44 <Raccoon> heh
95 2012-08-25 19:56:02 <Eliel> Raccoon: if the 1MB limit starts being hit often, you can still get high priority transactions through by paying a bit more fee.
96 2012-08-25 19:56:03 <yellowhat> guess what will happen after the IPO is finished :)
97 2012-08-25 19:56:15 <Eliel> IPO?
98 2012-08-25 19:56:26 <yellowhat> satoshidice ipo
99 2012-08-25 19:56:37 <Eliel> more tx volume?
100 2012-08-25 19:57:06 <Raccoon> how hard would a protocol extension be?
101 2012-08-25 19:57:33 <sipa> Raccoon: to get a better progressbar?
102 2012-08-25 19:57:34 <sipa> nothing
103 2012-08-25 19:57:45 <sipa> we can already do that, but it's not implemented
104 2012-08-25 19:58:46 <Raccoon> eg: ClientA -> ClientB: "TotalBlockSize?", ClientB -> ClientA: "1234567890"
105 2012-08-25 19:58:55 <sipa> Raccoon: clients already do that
106 2012-08-25 19:59:13 <sipa> but there is no guarantee that your peers have all blocks
107 2012-08-25 19:59:16 <sipa> and they can lie
108 2012-08-25 19:59:27 <Raccoon> So the progress bar can already be set to display by download left?
109 2012-08-25 19:59:56 <Raccoon> just go by the most common number reported
110 2012-08-25 19:59:56 <sipa> wait, you're talking about block sizes
111 2012-08-25 20:00:01 <sipa> not block numbers
112 2012-08-25 20:00:06 <Raccoon> i'm talking about the total database size
113 2012-08-25 20:00:09 <Raccoon> how many gigs
114 2012-08-25 20:00:26 <sipa> but the size is not the problem
115 2012-08-25 20:00:31 <Raccoon> then ClientA looks at his total size, subtracted from the reported size
116 2012-08-25 20:00:36 <Raccoon> knows what size is left
117 2012-08-25 20:00:39 <Raccoon> size is the proble
118 2012-08-25 20:00:41 <Raccoon> m
119 2012-08-25 20:00:42 <sipa> it's the number of modifications to the database
120 2012-08-25 20:00:54 <sipa> larger blocks need more changes written
121 2012-08-25 20:01:11 <Raccoon> larger blocks are physically larger in bytes
122 2012-08-25 20:01:18 <sipa> yes, but that's not the problem
123 2012-08-25 20:01:24 <Raccoon> so what is
124 2012-08-25 20:01:26 <sipa> it's the fact that they are harder to process
125 2012-08-25 20:01:31 <Raccoon> yes
126 2012-08-25 20:01:37 <sipa> and that depends on the number of txins, mostly
127 2012-08-25 20:01:42 <Raccoon> so difficulty is representable in bytes
128 2012-08-25 20:01:49 <sipa> no
129 2012-08-25 20:01:53 <jaxtr> anyone built bitcoin-qt on win8
130 2012-08-25 20:02:03 <sipa> but bytes i a already a better estimator than just the constant we have now
131 2012-08-25 20:02:08 <Raccoon> larger it is, more changes
132 2012-08-25 20:02:17 <Raccoon> right
133 2012-08-25 20:02:26 <sipa> yes, but later blocks are still slower than earlier blocks, even if they'd be the same size
134 2012-08-25 20:02:37 <Raccoon> they wouldn't be the same size
135 2012-08-25 20:02:41 <sipa> *sigh*
136 2012-08-25 20:02:56 <Raccoon> size matters :)
137 2012-08-25 20:02:56 <sipa> we still have small blocks now, sometimes
138 2012-08-25 20:03:08 <sipa> even those are slower than older blocks of the same size
139 2012-08-25 20:03:14 <Raccoon> really? why?
140 2012-08-25 20:03:22 <sipa> because the database is larger now
141 2012-08-25 20:03:40 <sipa> and maintaining a larger database is slower, as fewer parts of it fit in cache
142 2012-08-25 20:04:00 <Raccoon> so back to consolidating the database every so often.
143 2012-08-25 20:04:08 <Raccoon> my so-called "super blocks" idea.
144 2012-08-25 20:04:11 <sipa> that already happens
145 2012-08-25 20:04:16 <sipa> super blocks gain you nothing
146 2012-08-25 20:04:57 <Raccoon> if everyone consolidates the database exactly the same, say each year or each 100,000 blocks
147 2012-08-25 20:05:05 <sipa> how can you consolidate it?
148 2012-08-25 20:05:09 <Raccoon> they can safely "forget" empty blocks
149 2012-08-25 20:05:28 <Raccoon> er empty transaction codes and empty addresses
150 2012-08-25 20:05:40 <sipa> bitcoin internally doesn't know addresses
151 2012-08-25 20:05:43 <Raccoon> they could still be accessed in the "archive"
152 2012-08-25 20:06:01 <Raccoon> but wouldn't sit at the top of the database
153 2012-08-25 20:06:27 <sipa> i'm working on changing the block validation engine to use an "unspent coin set database", plus un-indexed block files
154 2012-08-25 20:06:45 <sipa> instead of using an index block database for everything
155 2012-08-25 20:06:46 <Raccoon> that'd be awesome
156 2012-08-25 20:07:08 <sipa> i can import 185000 blocks into it, in 9 minutes
157 2012-08-25 20:07:47 <sipa> plus, it supports block pruning (but that's not implemented yet)
158 2012-08-25 20:08:06 <Raccoon> trick is, could we ever mark a fixed point in time where everything before than can be easily validated by a simple file hash?
159 2012-08-25 20:08:17 <Raccoon> instead of hashing every transaction?
160 2012-08-25 20:08:26 <Raccoon> *before then
161 2012-08-25 20:08:28 <sipa> sure, that hash is called the block hash
162 2012-08-25 20:08:35 <sipa> it depends on the entire history of everything before it
163 2012-08-25 20:08:38 <sipa> and we checkpoint it
164 2012-08-25 20:08:46 <Raccoon> but a super-block hash
165 2012-08-25 20:08:58 <Raccoon> right, a checkpoint every month or something
166 2012-08-25 20:09:04 <MC-Eeepc> thats onlylike 3000 txn a block
167 2012-08-25 20:10:00 <sipa> Raccoon: the real solution is keeping the validation database in a huge merkle tree, and storing the merkle root of it in the coinbase of blocks
168 2012-08-25 20:10:21 <sipa> Raccoon: if that happens, you can just download a database from someone, and validate it by looking at the blockchain
169 2012-08-25 20:10:31 <sipa> instead of needing the working through all history
170 2012-08-25 20:11:06 <sipa> the only disadvantage is that you lose the zero-trust property that bitcoin has, as you're going to trust the longest chain without validation that that longest chain is actually valid
171 2012-08-25 20:11:56 <amiller> Raccoon, i think you asked your question backwards (but the answer is merkle trees)
172 2012-08-25 20:12:22 <MC-Eeepc> how feasible will it be for how long for a person with fairly average hardware to run a full node
173 2012-08-25 20:12:33 <amiller> if you mark a fixed point in time as a checkpoint, the question isn't how you validate eveyrthing before it - you have to assume that everything before it is already valid
174 2012-08-25 20:12:45 <amiller> the question is, how long does it take you, relative to how far back the checkpoint is, to validate everything else forwards
175 2012-08-25 20:12:47 <MC-Eeepc> if they really wanted to, convieniece aside
176 2012-08-25 20:13:06 <amiller> this comes up if you keep backups for yourself of just the head node every so often (once a month for example)
177 2012-08-25 20:13:16 <amiller> but you have a data catastrophe and lose all your hard drives in a fire
178 2012-08-25 20:14:41 <amiller> even if you made the checkpoint yourself (and therefore it still counts as zero trust, maybe you signed it too), you would need the merkle tree commitments in order to download a new database and validate it, without having to revalidate everything (including the stuff before your checkpoint)
179 2012-08-25 20:20:44 <Eliel> sipa: did the person who requested this http://piratepad.net/e931wLGDJE get back to you yet?
180 2012-08-25 20:20:58 <sipa> no
181 2012-08-25 20:21:07 <Eliel> (there were a couple of questions from him added in the document, I answered them just now)
182 2012-08-25 20:21:42 <Eliel> I think this document is worth finishing, whether or not he shows up again :)
183 2012-08-25 20:24:25 <Luke-Jr> so what happen with 0.7rc1?
184 2012-08-25 20:25:15 <sipa> i hope we'll have 0.7rc1 tomorrow
185 2012-08-25 20:25:37 <Luke-Jr> I was expecting it yesterday. o.o;
186 2012-08-25 20:27:29 <Mephisto> c'? qualche italiano?
187 2012-08-25 20:27:50 <sipa> Raccoon: actually, your comments about estimating block download more precisely made me thing: as soon as we have initial headers-only mode, we can *know* (and not just guess) what the total number of transactions in the chain is
188 2012-08-25 20:28:34 <Raccoon> there we go :)
189 2012-08-25 20:28:57 <Luke-Jr> sipa: we can? headers don't tell you???
190 2012-08-25 20:29:02 <sipa> Luke-Jr: they do!
191 2012-08-25 20:29:07 <Luke-Jr> where?
192 2012-08-25 20:29:16 <sipa> the answer to getheaders includes block headers + tx counts
193 2012-08-25 20:29:20 <Luke-Jr> o
194 2012-08-25 20:29:26 <Raccoon> now another thing
195 2012-08-25 20:29:30 <Eliel> is the patch that allows checking for txs related to any address in the blockchain going to be in 0.7rc1?
196 2012-08-25 20:29:37 <Raccoon> I know that zero trust is important
197 2012-08-25 20:29:47 <Luke-Jr> Eliel: no such thing exists. perhaps you mean any transaction
198 2012-08-25 20:29:48 <sipa> Eliel: not to any address, but you can query any transaction
199 2012-08-25 20:29:52 <Raccoon> but not all users can stay online for very long, especially with mobile devices
200 2012-08-25 20:29:53 <Raccoon> so
201 2012-08-25 20:30:08 <Raccoon> can we offer a rush download, and offline validation?
202 2012-08-25 20:30:17 <sipa> Raccoon: you mean Electrum?
203 2012-08-25 20:30:25 <Raccoon> unfamiliar with that
204 2012-08-25 20:30:33 <sipa> oh, you mean client-side validation?
205 2012-08-25 20:30:36 <Raccoon> right
206 2012-08-25 20:31:10 <Raccoon> download at full throttle, go offline, validate on battery power w/o internet
207 2012-08-25 20:31:16 <Eliel> sipa: how about the patch that allows launching arbitrary shell script on every block or tx received?
208 2012-08-25 20:31:28 <sipa> Eliel: we already have that (for blocks)
209 2012-08-25 20:31:31 <sipa> since 0.6 or so
210 2012-08-25 20:31:35 <sipa> -blocknotify
211 2012-08-25 20:32:12 <sipa> Raccoon: if you don't mind doing it a bit manually, you can download the blk000* files yourself, and import them while offline
212 2012-08-25 20:32:27 <Raccoon> right now it's not uncommon to take 16 hours for an average laptop to validate after a few months between using bitcoin
213 2012-08-25 20:32:37 <Eliel> sipa: but not for new transactions?
214 2012-08-25 20:32:53 <sipa> Eliel: not for new mempool transactions, no
215 2012-08-25 20:32:54 <Raccoon> sipa: couldn't the client grab-all?
216 2012-08-25 20:33:21 <Raccoon> I'm also afraid that people might be able to track users by rating their download speed
217 2012-08-25 20:33:49 <Raccoon> collecting metrics based on user's system profile (hdd speed)
218 2012-08-25 20:33:49 <sipa> Raccoon: in theory
219 2012-08-25 20:34:00 <sipa> Raccoon: such things will be easier with initial header-only, i suspect
220 2012-08-25 20:34:18 <Raccoon> so all clients should be downloading at full throttle
221 2012-08-25 20:34:23 <sipa> as that makes block processing a much more pipelined thing with several independent steps
222 2012-08-25 21:35:06 <Luke-Jr> evoorhees owner of SatoshiDice is in #bitcoin-assets if anyone wants to flame him..
223 2012-08-25 21:35:37 <sipa> evoorhees is owner of SD? :o
224 2012-08-25 21:36:35 <Luke-Jr> apparently
225 2012-08-25 21:43:38 <Luke-Jr> fwiw, I suggested that if he isn't actually trying to kill Bitcoin, he might consider hiring some Bitcoin developer to fix the problem (that is, the impact transactions have on block relaying), so he might join the channel later looking
226 2012-08-25 21:44:09 <Luke-Jr> likely fixing it requires reworking the p2p networking code completely
227 2012-08-25 21:44:19 <Luke-Jr> since it's all too synchronous right now
228 2012-08-25 21:46:17 <Matt_von_Mises> Luke-Jr: What is the problem exactly? Bigger block sizes?
229 2012-08-25 21:46:43 <sipa> the problem is just the time it takes for a large block to be validated, imho?
230 2012-08-25 21:46:54 <Luke-Jr> Matt_von_Mises: blocks do not begin upload from peerB to peerC, until peerB has finished his own download and verification
231 2012-08-25 21:46:54 <sipa> not sure how rewriting the p2p code would help too much there
232 2012-08-25 21:47:11 <Luke-Jr> sipa: right now, it's impossible to be involved in relaying a block to other peers so long as you're downloading it
233 2012-08-25 21:47:16 <Luke-Jr> or verifying it
234 2012-08-25 21:47:22 <Luke-Jr> the p2p code is just frozen during that
235 2012-08-25 21:47:43 <Luke-Jr> an asynchronous model could begin relaying as soon as the header is validated
236 2012-08-25 21:47:55 <Luke-Jr> so the entire network downloads the block in parallel, and verifies in paralle
237 2012-08-25 21:48:03 <Matt_von_Mises> "since it's all too synchronous right now" Well I've made my cbitcoin network code asynchronous. Asynchronous networking is no doubt better.
238 2012-08-25 21:48:55 <sipa> Luke-Jr: well, all that'd be required is instead of doing a ProcessBlock call, is adding it to some queue and have that processed by a separate thread
239 2012-08-25 21:49:09 <sipa> no need to touch the p2p code there at all
240 2012-08-25 21:49:18 <Luke-Jr> sipa: that won't begin upload before download completes
241 2012-08-25 21:49:18 <Matt_von_Mises> Oh right, sounds like you meant something different in how the blocks are relayed. Well, I'll probably use a separate thread for block validation.
242 2012-08-25 21:49:26 <sipa> no, but is that the problem?
243 2012-08-25 21:49:28 <Luke-Jr> sipa: most of the delay is likely from limited upload bandwidth
244 2012-08-25 21:49:31 <Matt_von_Mises> One single thread for networking and another for validation.
245 2012-08-25 21:50:01 <Mephisto> what sites are kptn e btcn?
246 2012-08-25 21:50:29 <Mephisto> what sites are kptn e btcn?
247 2012-08-25 21:50:33 <Luke-Jr> or maybe not, I haven't really timed the upload/verify barrier
248 2012-08-25 21:50:42 <sipa> it would be interesting to know
249 2012-08-25 21:51:00 <sipa> my guess would be that at least on well-connected nodes, verification is the problem mostly
250 2012-08-25 21:51:05 <Luke-Jr> but in any case, the problem is only fixed by parallel distribution; parallel verification only reduces the problem a bit
251 2012-08-25 21:51:26 <Luke-Jr> that is, there's still an impact from larger blocks with the latter
252 2012-08-25 21:51:45 <sipa> right, preview-submitting blocks is one solution
253 2012-08-25 21:51:47 <Matt_von_Mises> Luke-Jr: The way you would do it is start relaying blocks even when they have not been validated and stop if they are invalid later on?
254 2012-08-25 21:52:12 <sipa> but preview-submitting also increases exposure to DOS
255 2012-08-25 21:52:17 <sipa> s/DOS/DoS/
256 2012-08-25 21:52:27 <Luke-Jr> Matt_von_Mises: right, but you really need to start relaying blocks *even before your own download of it is done*
257 2012-08-25 21:52:35 <Luke-Jr> sipa: not so long as the header is checked first
258 2012-08-25 21:52:40 <sipa> agree
259 2012-08-25 21:52:41 <Luke-Jr> and of sufficient difficulty
260 2012-08-25 21:53:28 <Mephisto> what sites are kptn e btcn?
261 2012-08-25 21:53:39 <sipa> Mephisto: i have no idea what you're asking
262 2012-08-25 21:53:50 <sipa> but you're more likely to find an answer in #bitcoin
263 2012-08-25 21:53:53 <Mephisto> it is an abbreviation of the real name
264 2012-08-25 21:54:06 <Luke-Jr> sipa: the header download + check is basically constant time; more or less txns in the block is irrelevant to it ???
265 2012-08-25 21:54:06 <Matt_von_Mises> Luke-Jr: Doesn't bitcoin validate headers after downloading the payload? I thought I saw that. If so the headers need to validated when they are downloaded before the payload.
266 2012-08-25 21:54:11 <Mephisto> yes, i see :))
267 2012-08-25 21:54:15 <Mephisto> sorry
268 2012-08-25 21:54:22 <Luke-Jr> Matt_von_Mises: correct
269 2012-08-25 21:54:53 <sipa> we *could* switch to a push of headers only, initially
270 2012-08-25 21:55:15 <Luke-Jr> sipa: that doesn't help really
271 2012-08-25 21:55:28 <Luke-Jr> you can't do anything with just the headers
272 2012-08-25 21:55:34 <sipa> you can validate them
273 2012-08-25 21:55:38 <sipa> and store them in the block tree
274 2012-08-25 21:55:54 <sipa> the block data follows immediately afterwards in a separate message
275 2012-08-25 21:56:05 <sipa> and at that time, you know it's good enough to start relaying immediately
276 2012-08-25 21:56:28 <Matt_von_Mises> sipa: You just receive the data with non-blocking sockets and when you get the data you look for the header before the payload.
277 2012-08-25 21:56:39 <Matt_von_Mises> It's the way cbitcoin works. No need to change the messages.
278 2012-08-25 21:56:50 <Luke-Jr> sipa: ah, so split the block-receiving connection to another thread?
279 2012-08-25 21:57:08 <Eliel> would also make sense to change the block download into push headers, the answer tx requests for the txs that don't exist in mempool.
280 2012-08-25 21:57:23 <gmaxwell> sipa: sync transmission means you get head of line blocking??? a single tarpit (very slow) peer can block your relaying to fast peers.
281 2012-08-25 21:57:42 <Eliel> perhaps push headers + merkle tree + coinbase tx
282 2012-08-25 21:57:52 <gmaxwell> A simple improvement would be to time relays and relay in order of fastest peer first.
283 2012-08-25 21:57:58 <Luke-Jr> Eliel: that has overhead based on transaction count
284 2012-08-25 21:58:17 <Eliel> Luke-Jr: ok, just the merkle path for coinbase then?
285 2012-08-25 21:58:33 <sipa> indeed, that'd reduce bandwidth at the cost of slower full validation
286 2012-08-25 21:58:35 <Luke-Jr> Eliel: not useful? :P
287 2012-08-25 21:59:26 <Eliel> Luke-Jr: coinbase tx is the only tx that's guaranteed to be missing from mempool.
288 2012-08-25 21:59:35 <Eliel> so would make sense to push that too.
289 2012-08-25 21:59:54 <Luke-Jr> Eliel: the point is to parallelize everything that involves transactions
290 2012-08-25 22:00:04 <Luke-Jr> so the cost to including them is once again the intended zero
291 2012-08-25 22:00:25 <Luke-Jr> (transactions in a block, that is)
292 2012-08-25 22:00:37 <gmaxwell> beyond the block real issues, there is the whole bloating the txout set issue.
293 2012-08-25 22:01:21 <Luke-Jr> gmaxwell: but that's much smaller in theory?
294 2012-08-25 22:01:32 <Eliel> Luke-Jr: well, have the headers and coinbase pushed in separate messages then :P
295 2012-08-25 22:02:02 <Luke-Jr> IMO SatoshiDice can justify their (ab)use by financing a dev fixing the relay issue :p
296 2012-08-25 22:02:18 <Matt_von_Mises> So you get the headers, the transaction hashes and the coinbase (or not) and then ask for the transactions separately, if you do not already have them? That would certainly be a better idea.
297 2012-08-25 22:02:29 <sipa> Luke-Jr: send and receive are already separate threads, no>
298 2012-08-25 22:02:38 <Luke-Jr> sipa: no
299 2012-08-25 22:02:59 <Luke-Jr> that's why my block preview patch did nothing in practice
300 2012-08-25 22:03:14 <sipa> oh
301 2012-08-25 22:03:30 <sipa> there's a message handler thread that continuously switches between receive and send
302 2012-08-25 22:03:40 <sipa> shouldn't be hard to split that into two threads?
303 2012-08-25 22:04:14 <Luke-Jr> sipa: as gmaxwell mentioned, there's the tarpit risk with that
304 2012-08-25 22:04:18 <Matt_von_Mises> Mew messages? gettxhashes and txhashes? Something like that. You can use getheaders and headers and then use gettxhashes for each block header?
305 2012-08-25 22:06:41 <Luke-Jr> actually, we probably already have the tarpit problem
306 2012-08-25 22:07:00 <sipa> yes
307 2012-08-25 22:07:14 <sipa> if the first node to send you a block is slow to upload it to you, you can't do anything
308 2012-08-25 22:08:28 <Matt_von_Mises> I was going to integrate a timing system into cbitcoin that times nodes so you can rank them based upon speed.
309 2012-08-25 22:08:37 <Matt_von_Mises> But I skipped over that for now.
310 2012-08-25 22:10:59 <Matt_von_Mises> If you were downloading from multiple nodes in parrellel you can find abnormally slow ones and stop downloading from them, so you keep the fastest for download.
311 2012-08-25 22:11:46 <Matt_von_Mises> I'll mention this in bitcointalk.org to hopefully get more discussion on it.
312 2012-08-25 22:13:31 <sipa> Luke-Jr: well, the trivial solution is multiple receive/send threads
313 2012-08-25 22:14:19 <Matt_von_Mises> You should only have one network thread for the entire program and then when work needs to be done that would hang the thread, the network thread can create worker threads.
314 2012-08-25 22:14:22 <sipa> and some logic to prevent trying to download a block from everyone at once
315 2012-08-25 22:14:52 <sipa> anyway, i think i'm close to adding initial headers-only, actually...
316 2012-08-25 22:15:36 <sipa> for every block index entry, keep a state variable that defines how far the block got in the verification process
317 2012-08-25 22:16:14 <sipa> the first stages are only for the header, which must be connected to the block tree before txn can be accepted for it
318 2012-08-25 22:17:09 <sipa> the only problem is: what to do in case of a post-acceptance failed check (e.g. signature validation being done in a separate thread)
319 2012-08-25 22:17:29 <Luke-Jr> sipa: same thing we do now, minus the kick/ban stuff
320 2012-08-25 22:18:03 <sipa> it's a bit more complex, as you need to reorg back
321 2012-08-25 22:18:42 <sipa> but that shouldn't be a problem, really; i keep block indices ordered by work in a set, so it's easy to find the next best tip
322 2012-08-25 22:22:42 <gmaxwell> initial headers only would be fantastic because it could allow multiple @#$@ peer fetches.
323 2012-08-25 22:22:50 <sipa> indeed
324 2012-08-25 22:23:21 <sipa> not really "initial headers only", as there is no actual "headers only mode"
325 2012-08-25 22:24:13 <sipa> it's just headers and block data being fetched independently
326 2012-08-25 22:25:00 <sipa> and you never accept block data that isn't already known to be part of the best chain
327 2012-08-25 22:26:04 <gmaxwell> Right, I sketched out a particular sequence which I think gives high DOS resistance. https://en.bitcoin.it/wiki/User:Gmaxwell/Reverse_header-fetching_sync
328 2012-08-25 22:27:06 <helo> where's the testing forum thread?
329 2012-08-25 22:27:52 <helo> nm
330 2012-08-25 22:29:11 <sipa> gmaxwell: https://github.com/sipa/bitcoin/commit/30ec157606da9bdc56751d73ff1a64ad0fa605d1#L3R1362
331 2012-08-25 22:31:10 <helo> is http://luke.dashjr.org/programs/bitcoin/w/bitcoind/luke-jr.git/shortlog/refs/tags/next_20120813 still the best thing to test?
332 2012-08-25 22:31:27 <sipa> depends what you want to test :)
333 2012-08-25 22:31:49 <Luke-Jr> helo: if you want to test 0.7, rc1 is due any day now; if you want to test a bunch of crap that might or might not be in 0.8, next-test
334 2012-08-25 22:33:14 <helo> i have a couple hours to play around; testing either would be fine by me...
335 2012-08-25 22:33:35 <sipa> helo: you can also test ultraprune if you like
336 2012-08-25 22:33:58 <helo> yeah, i'll do that
337 2012-08-25 22:36:09 <sipa> gmaxwell: idea would be: arriving header: { check level 1, find parent, check level 2, store in block tree }, separate thread requests blocks from best non-failed branch in tree, arriving block: { check merkle root; if false, drop and exit; if true { check transactions; if ok, mark level 3 valid, if nok mark block failed }; every time block data is modified, try to connect best chain of all valid-level3 blocks
338 2012-08-25 22:36:10 <helo> git://github.com/sipa/bitcoin.git ultraprune-dev?
339 2012-08-25 22:36:17 <sipa> helo: yes, try _dev
340 2012-08-25 22:36:23 <helo> ahh right
341 2012-08-25 22:36:48 <sipa> though it's database format is likely to change in the future in incompatible ways still
342 2012-08-25 22:37:33 <helo> this round of testing brought to you by Mickey's Fine Malt Liquor
343 2012-08-25 22:37:48 <sipa> :D
344 2012-08-25 22:38:40 <sipa> oh, despite the name, don't expect any pruning to actually be implemented
345 2012-08-25 22:39:08 <helo> its ok, i have 80GB free
346 2012-08-25 22:41:13 <sipa> helo: i just pushed a new version to ultraprune_dev; the former was buggy
347 2012-08-25 22:43:05 <sipa> also, -loadblock=<old blk0001.dat file> is advised, network will most likely be the bottleneck before the last checkpoint
348 2012-08-25 22:44:09 <helo> great, thank you
349 2012-08-25 22:48:57 <helo> down the road, ultraprune will be the default initial state, transitioning to a full-node as time and resources allow?
350 2012-08-25 22:49:38 <sipa> ultraprune is just a different validation engine and associated database layout, that permits pruning
351 2012-08-25 22:49:55 <sipa> as long as nothing is effectively deleted, it's a full node
352 2012-08-25 22:50:12 <sipa> and even when deleting blocks, it remains a fully validating node
353 2012-08-25 22:51:17 <sipa> ACTION sleeps
354 2012-08-25 22:51:25 <helo> ahh right... i was thinking of the "unspent output tree hash in block" thing
355 2012-08-25 22:53:28 <Joric> pruning the blockchain? hows results?
356 2012-08-25 22:54:17 <helo> littler
357 2012-08-25 22:55:42 <Joric> like, 50%?
358 2012-08-25 22:59:11 <sipa> helo: yes, it keeps a utxo database next to the block database, and uses that for almost everything
359 2012-08-25 22:59:41 <sipa> except reorgs, rescans, and obviously serving blocks to others
360 2012-08-25 23:00:21 <sipa> Joric: more like up to 15x now
361 2012-08-25 23:01:07 <sipa> Joric: not that 14/15 is pruned, but just keeping the utxo set only requires 200 MB or ak
362 2012-08-25 23:01:48 <sipa> you need some recent blocks and undo data as well, though
363 2012-08-25 23:10:52 <Matt_von_Mises> Here I suggested a way in which blocks can be downloaded and relayed in segements: https://bitcointalk.org/index.php?topic=103295.0
364 2012-08-25 23:11:28 <Joric> sipa, how does it work? what about block hashes are they remain the same?
365 2012-08-25 23:14:16 <Luke-Jr> sipa: ping
366 2012-08-25 23:34:51 <sipa> Joric: blocks remain blocks, nothing changes
367 2012-08-25 23:35:10 <sipa> Joric: it's just that you don't need the blocks anymore
368 2012-08-25 23:35:20 <sipa> Luke-Jr: yes?
369 2012-08-25 23:39:29 <sipa> Joric: right now, you have block files, and an index for all blocks, transactions, and transaction spendings
370 2012-08-25 23:40:09 <sipa> Joric: that means the data set you need fast access to is several gigabytes already
371 2012-08-25 23:40:37 <Joric> yeah ok what changed
372 2012-08-25 23:40:57 <sipa> ultraprune only needs block files and an index for all blocks, and additionally just a set of unspent transaction outputs
373 2012-08-25 23:41:34 <sipa> that means the txout data is effectively stored twice, but only the txout set is necessary for validation
374 2012-08-25 23:41:47 <sipa> and this is so small it is easily kept in cache
375 2012-08-25 23:43:02 <sipa> and if you don't care about the ability to serve, reorg or rescan, you can just delete the block files
376 2012-08-25 23:53:18 <Luke-Jr> sipa: beyond the documented split, 7f3ccb5 also replaces a boost::interprocess::interprocess_mutex with a CCriticalSection - what was the purpose of that?