1 2015-10-23 01:17:36 <cfields> jgarzik: when a remote peer disconnects, we immediately quit processing any fully-received, fully-valid messages. Is that an active DOS measure, or implementation happenstance?
  2 2015-10-23 01:18:28 <cfields> i ask because (mainly for testing) if i want to send a few messages and disconnect, i have to wait an arbitrary amount of time (or use ping) before i disconnect
  3 2015-10-23 01:18:35 <jgarzik> brb, xchat stupid
  4 2015-10-23 01:18:40 <cfields> ok
  5 2015-10-23 01:19:12 <jgarzik> cfields, Can you repeat, please?
  6 2015-10-23 01:19:25 <cfields> <cfields> i ask because (mainly for testing) if i want to send a few messages and disconnect, i have to wait an arbitrary amount of time (or use ping) before i disconnect
  7 2015-10-23 01:19:25 <cfields> <cfields> jgarzik: when a remote peer disconnects, we immediately quit processing any fully-received, fully-valid messages. Is that an active DOS measure, or implementation happenstance?
  8 2015-10-23 01:20:25 <jgarzik> cfields, a complexity reduction measure that is also mildly anti-DoS
  9 2015-10-23 01:20:58 <jgarzik> cfields, the alternative is setting a timer and waiting X amount of time, and then under the hood, after you close, the kernel does the same thing yet again
 10 2015-10-23 01:21:26 <jgarzik> cfields, arguably it is more correct to wait until all data is processed, as traditionally send+close is allowed
 11 2015-10-23 01:21:47 <cfields> jgarzik: assuming it were actually less complex than the current approach, would it be reasonable to read the next message or two before disconnecting?
 12 2015-10-23 01:22:03 <sipa> that seems pretty ugly
 13 2015-10-23 01:22:05 <cfields> indeed, that's the approach i took. i was surprised to find that we don't allow for that
 14 2015-10-23 01:22:15 <sipa> we may miss a block that was sent right before a disconnect!
 15 2015-10-23 01:22:26 <cfields> sipa: exactly
 16 2015-10-23 01:23:19 <cfields> it's easy to trigger when testing. In real usage, with more nodes and locking/unlocking, there's probably more time to spare
 17 2015-10-23 01:24:44 <phantomcircuit> cfields, that is an actual DoS mitigation measure
 18 2015-10-23 01:24:49 <phantomcircuit> plzdonotremove
 19 2015-10-23 01:25:12 <cfields> phantomcircuit: can you explain? i'm having a hard time coming up with a reasonable attack
 20 2015-10-23 01:25:38 <phantomcircuit> i'd rather not as it is only a partial mitigation
 21 2015-10-23 01:25:57 <phantomcircuit> (ie you can still be quite annoying...)
 22 2015-10-23 01:28:29 <Luke-Jr> gmaxwell: https://github.com/bitcoin/bips/pull/221 https://github.com/bitcoin/bips/pull/170 https://github.com/bitcoin/bips/pull/198 https://github.com/bitcoin/bips/pull/202 https://github.com/bitcoin/bips/pull/224
 23 2015-10-23 01:28:50 <cfields> phantomcircuit: i'm not sure how you can be any more annoying by quickly disconnecting?
 24 2015-10-23 01:31:22 <phantomcircuit> cfields, yes and im not going to tell you :)
 25 2015-10-23 01:38:31 <phantomcircuit> cfields, you could shutdown the receive side of the socket and continue processing messages out, but in general when we set fDisconnect we really do want to immediately stop uses resources for that peer
 26 2015-10-23 01:39:07 <cfields> phantomcircuit: i'm speaking specifically about a graceful remote disconnect
 27 2015-10-23 01:39:11 <belcher> can CLTV be used on regtest? want to rule out that as a source of failure
 28 2015-10-23 01:39:33 <phantomcircuit> cfields, ah right
 29 2015-10-23 01:40:02 <cfields> sure, if we trigger the disconnect it makes sense to nuke the receive queue.
 30 2015-10-23 01:40:56 <phantomcircuit> cfields, i guess the peers queue could be placed into a global queue that only contains "block" or "tx" messages
 31 2015-10-23 01:41:15 <phantomcircuit> ie things that dont strictly require a response
 32 2015-10-23 01:44:00 <cfields> phantomcircuit: sure, that's an implementation detail. I'm only concerned about security at this point. Just trying to figure out how many backlogged received messages should be processesd on graceful remote disconnect.
 33 2015-10-23 01:45:22 <cfields> i really don't see any additional DoS risk in continuing to process the backlog for well-behaved nodes. They could just-as-easily keep the connection open for whatever they were trying to pull off.
 34 2015-10-23 01:46:52 <gmaxwell> if they're misbehaviing we should drop it, but if they're well behaved it would be preferable to process.. e.g. they sent out a txn with their dying gasp and wouldn't it be sad if it never made it to the network.
 35 2015-10-23 01:49:17 <cfields> gmaxwell: agreed. Is there a limit to that, though. Any reason to enforce a max backlog depth?
 36 2015-10-23 01:49:28 <cfields> (i can't think of any)
 37 2015-10-23 01:50:15 <gmaxwell> cfields: only to the extent that you wouldn't have wanted to accept in too much backlog to begin with.
 38 2015-10-23 01:50:39 <cfields> gmaxwell: right, at which point the disconnection would be irrelevant
 39 2015-10-23 01:54:03 <cfields> phantomcircuit/jgarzik/gmaxwell: thanks for the advice. With much of the network code, it's hard to tell if a particular behavior is purposeful or a happy (or sad) accident.
 40 2015-10-23 01:54:19 <phantomcircuit> cfields, definitely you want to ignore things like "getdata" though :)
 41 2015-10-23 01:55:04 <jgarzik> cfields, you have to think about details behind the scenes in the kernel buffers, games with TCP windows, sending & not receiving or receiving & not sending, loads of edge cases
 42 2015-10-23 01:55:05 <cfields> phantomcircuit: of course :)
 43 2015-10-23 01:58:55 <cfields> jgarzik: Could you give a concrete example? (ignoring the obvious don't process if sendbuffer is too full and etc), given the breadth of OSs we support and their vastly different in-kernel behaviors, I'm not sure how much is realistic to try to consider at the raw tcp level.
 44 2015-10-23 01:59:49 <jgarzik> cfields, the current code actually does...
 45 2015-10-23 02:00:16 <cfields> (for ex just casually flipping on TCP_NODELAY means that we can't be relying on anything too specific :)
 46 2015-10-23 02:01:26 <phantomcircuit> cfields, ThreadSocketHandler there's a comment
 47 2015-10-23 02:01:29 <phantomcircuit> (a large one)
 48 2015-10-23 02:02:16 <cfields> phantomcircuit: you mean the one about send/recv/accept order and stuff?
 49 2015-10-23 02:03:24 <cfields> right. re-reading.
 50 2015-10-23 02:04:54 <jgarzik> cfields, turning on TCP_NODELAY is fine AFAICT w/ current code
 51 2015-10-23 02:05:29 <phantomcircuit> yup
 52 2015-10-23 02:05:36 <cfields> sure, agreed
 53 2015-10-23 02:06:47 <phantomcircuit> jgarzik, actually it's potentially an issue with the way SocketSendData is implemented
 54 2015-10-23 02:06:53 <phantomcircuit> we should also be making use of TCP_CORK
 55 2015-10-23 02:07:15 <jgarzik> phantomcircuit, no TCP_CORK sucks
 56 2015-10-23 02:07:25 <jgarzik> phantomcircuit, MSG_MORE and friends
 57 2015-10-23 02:08:09 <phantomcircuit> jgarzik, either way
 58 2015-10-23 02:08:13 <jgarzik> TCP_CORK requires ridiculous additional system call traffic and associated management
 59 2015-10-23 02:09:00 <phantomcircuit> you know we dont bundle together messages right?
 60 2015-10-23 02:09:23 <jgarzik> we use the kernel to do that for us
 61 2015-10-23 02:09:26 <phantomcircuit> the send queue is messages, best case 1 syscall per message
 62 2015-10-23 02:09:40 <jgarzik> I know I wrote the code
 63 2015-10-23 02:09:50 <phantomcircuit> sure im just pointing out that we're not exactly being judicious about syscalls as it is :P
 64 2015-10-23 02:11:09 <cfields> phantomcircuit: i considered suggesting cat'ing the messages before sending if we're enabling TCP_NODELAY, but I'm afraid it'd get ugly and probably isn't worth it
 65 2015-10-23 02:11:31 <jgarzik> cfields, that's what msg_more is for
 66 2015-10-23 02:11:44 <jgarzik> cfields, remember, you are just sending to a local kernel buffer
 67 2015-10-23 02:11:44 <phantomcircuit> cfields, it's easy enough to get the correct behavior with either TCP_CORK or MSG_MORE
 68 2015-10-23 02:11:48 <cfields> jgarzik: ah, a hint to wait?
 69 2015-10-23 02:11:50 <phantomcircuit> infact i'll do it now
 70 2015-10-23 02:12:22 <gmaxwell> that would be nice, I've been worrking a bit about overhead from nodelay, though I haven't measured it yet.
 71 2015-10-23 02:13:21 <phantomcircuit> cfields, that's exactly what it is
 72 2015-10-23 02:14:24 <cfields> phantomcircuit: ah, great. that's loads better than jumbling up the messages and adding in a ton of reallocs as i was considering :)
 73 2015-10-23 02:15:07 <jgarzik> Yeah, you're already copying the data going from app buffer to kernel buffer.  You don't want to copy it yet again.
 74 2015-10-23 02:15:17 <phantomcircuit> jgarzik, oh i remember
 75 2015-10-23 02:15:31 <phantomcircuit> MSG_MORE only works if you can guarantee there is a final send() call without MSG_MORE
 76 2015-10-23 02:16:10 <jgarzik> Kind of - it's a bit like nagle - if you fail to flush a timer flushes
 77 2015-10-23 02:16:13 <phantomcircuit> or is it smart about not corking if it has > 1 packet worth of data?
 78 2015-10-23 02:16:28 <phantomcircuit> i assume it's the same 200ms time as TCP_CORK
 79 2015-10-23 02:16:31 <jgarzik> you still hurt yourself
 80 2015-10-23 02:16:35 <phantomcircuit> (and probably nagle)
 81 2015-10-23 02:16:59 <gmaxwell> you can do a final send of zero bytes though.
 82 2015-10-23 02:17:10 <phantomcircuit> gmaxwell, i was just about to ask about that heh
 83 2015-10-23 02:17:22 <gmaxwell> I'm pretty sure that works, more syscall overhead of course.
 84 2015-10-23 02:17:45 <gmaxwell> but if you just can't guarentee that you'll get it right in all cases... there you go.
 85 2015-10-23 02:18:42 <jgarzik> all of this is optimizing at the low water edge of the kernel send buffer.  if there is any amount of traffic at all, there is more than a path MTU's worth of data in there, it's pointless to care as the data is just getting stuffed into a wait queue anyway
 86 2015-10-23 02:19:52 <gmaxwell> jgarzik: a concern there is that if we're sending small messages (like 70 byte ones) often, the we can end up with 2x the traffic due to IP+TCP overhead.
 87 2015-10-23 02:20:47 <jgarzik> gmaxwell, that's assuming they are transmitted fast enough that there's no aggregation
 88 2015-10-23 02:21:29 <gmaxwell> yes, but normally our window should be wide open, so it would send it right away...
 89 2015-10-23 02:24:29 <phantomcircuit> gmaxwell, eh can I pass NULL for buf if len if 0?
 90 2015-10-23 02:24:33 <phantomcircuit> im thinking no but...
 91 2015-10-23 02:25:58 <gmaxwell> phantomcircuit: I would assume you could, sort of thing posix should document but probably doesn't. :P
 92 2015-10-23 02:27:47 <phantomcircuit> seems to work fine...
 93 2015-10-23 02:36:25 <phantomcircuit> great MSG_MORE is not defined on windows
 94 2015-10-23 02:36:36 <dcousens> phantomcircuit: lol
 95 2015-10-23 02:37:06 <jgarzik> phantomcircuit, yeah I was just about to comment on portability on the PR
 96 2015-10-23 02:37:28 <phantomcircuit> jgarzik, yeah i was waiting for travis to see if it worked
 97 2015-10-23 02:37:56 <jgarzik> phantomcircuit, flipping TCP_NODELAY appears to be a poor man's TCP_CORK
 98 2015-10-23 02:40:47 <phantomcircuit> in that case this would replace #6867
 99 2015-10-23 02:42:27 <gmaxwell> phantomcircuit: we'd still leave NO_DELAY on normally, I assume.
100 2015-10-23 02:44:06 <phantomcircuit> gmaxwell, well there's only one way to get to send() so what we would do is set TCP_NODELAY=0 at the beginning of the SocketSendData call and then set it TCP_NODELAY=1 at the end
101 2015-10-23 02:44:26 <phantomcircuit> so there's no reason to set it to TCP_NODELAY=1 anywhere else
102 2015-10-23 02:44:34 <phantomcircuit> except for maybe the socks5 code
103 2015-10-23 02:44:40 <phantomcircuit> and then who cares about tiny packets
104 2015-10-23 02:46:13 <phantomcircuit> i guess there isn't a reason not to set it to 1 anywhere else either though
105 2015-10-23 02:47:19 <phantomcircuit> jgarzik, any idea if simply calling setsockopt and setting TCP_NODELAY=1 will cause a flush?
106 2015-10-23 02:47:30 <phantomcircuit> im thinking it should but i can easily see it not
107 2015-10-23 02:48:23 <jgarzik> phantomcircuit, according to internet lore it should flush on win/osx/lin
108 2015-10-23 02:48:55 <jgarzik> phantomcircuit, I only know the Linux kernel socket behavior, so just googling for the rest...
109 2015-10-23 02:49:25 <phantomcircuit> fair enough
110 2015-10-23 02:49:49 <phantomcircuit> it certainly does flush on linux
111 2015-10-23 02:51:20 <phantomcircuit> could probably guarantee it flushes with a final send the same as MSG_MORE
112 2015-10-23 05:09:52 <Luke-Jr> where does lightningbot` put the notes it collects?
113 2015-10-23 05:10:34 <aj> Luke-Jr: for the meetings?
114 2015-10-23 05:10:39 <Luke-Jr> yes
115 2015-10-23 05:11:53 <aj> Luke-Jr: it announces urls when the meeting gets closed; i don't think they're findable otherwise, i should probably fix that
116 2015-10-23 05:12:21 <Luke-Jr> found it, thanks
117 2015-10-23 05:27:12 <aj> Luke-Jr: ok, you should be able to browse http://www.erisian.com.au/meetbot/bitcoin-dev/2015/ as well now
118 2015-10-23 08:15:27 <wumpus> btw btcdrak thanks for chairing the meeting yesterday in my absence
119 2015-10-23 08:20:13 <btcdrak> wumpus: no problem, no one else replied to the question "who will chair" and the clock was ticking!
120 2015-10-23 12:03:02 <midnightmagic> Interesting. The banline isn't catching him.
121 2015-10-23 12:14:59 <phantomcircuit> midnightmagic, server appears to me messed up
122 2015-10-23 12:21:40 <midnightmagic> phantomcircuit: he was stuck in the exceptions list here, and then over in #bitcoin he started using a direct non-vpn connection.
123 2015-10-23 12:21:58 <midnightmagic> poor guy, he's going to be so embarrassed
124 2015-10-23 12:22:05 <midnightmagic> oh well.
125 2015-10-23 12:52:21 <jtimon> it would be nice to have more thoughts on https://github.com/bitcoin/bitcoin/pull/6816#discussion_r42801091
126 2015-10-23 13:53:55 <treehug8_> are there any sorts of explanations/ illustrations of the bitcoin core architecture?
127 2015-10-23 13:54:42 <justanotheruser> like how bitcoin works, or how bitcoin core works?
128 2015-10-23 13:55:07 <justanotheruser> there is https://bitcoin.org/en/developer-documentation
129 2015-10-23 13:55:34 <justanotheruser> but it doesn't completely explain Bitcoin Cores architecture
130 2015-10-23 13:55:45 <treehug8_> the latter, how the code is structured / architected
131 2015-10-23 13:57:23 <justanotheruser> Not to my knowledge. You could look at commit conversations on github for some insight. If you find something let me know because there is a lot of (IMO) undercommented stuff that I don't understand at first.
132 2015-10-23 13:59:08 <treehug8_> thanks justanotheruser . Even just an illustration showing the directories in  bitcoin-core's /src dir, what each does roughly, and which depends on which would be useful. Will post if I find something
133 2015-10-23 13:59:45 <Luke-Jr> treehug8_: that's under semi-regular change
134 2015-10-23 14:00:01 <Luke-Jr> right now there's a lot of crap just sitting in src/main.cpp that needs to be organized
135 2015-10-23 14:00:40 <treehug8_> Luke-Jr: understood.
136 2015-10-23 14:00:42 <justanotheruser> treehug8_: some of the classes have describing comments above them
137 2015-10-23 15:12:12 <jl2012> If BIP113 is deployed with BIP68, we don't need to make it mempool-only first, since BIP68 will use a new tx version which is already non-standard
138 2015-10-23 15:32:28 <btcdrak> jl2012: the idea is to release BIP113 as mempool only with CLTV, then we can softfork BIP68/112(cvs)/113 in together at a later date.
139 2015-10-23 16:34:57 <Luke-Jr> why is BIP65 being deployed with a bit-burner ISM?
140 2015-10-23 16:35:53 <Luke-Jr> petertodd:
141 2015-10-23 16:36:06 <jgarzik> pretty sure it's because changing details at a late stage would mean further delay on a useful, Ready To Go feature
142 2015-10-23 16:36:56 <Luke-Jr> frankly, I'd rather let it wait for versionbits if bit-burning was the alternative.
143 2015-10-23 16:37:33 <Luke-Jr> NOT burning the bit was what had been discussed months ago
144 2015-10-23 16:38:15 <wumpus> this doesn't burn a bit IIRC
145 2015-10-23 16:38:52 <Luke-Jr> wumpus: btcdrak just proposed language for the BIP that does, and looking over the PR suggested that behaviour to me. I'll read it closer to double-check..
146 2015-10-23 16:39:25 <Luke-Jr> sure looks like it :/  https://github.com/bitcoin/bitcoin/pull/6351/files#diff-7ec3c68a81efff79b6ca22ac1f1eabbaR2694
147 2015-10-23 16:39:45 <btcdrak> we discussed losing this bit, ad nausem
148 2015-10-23 16:40:11 <btcdrak> it was to be the final bit pending versionbits
149 2015-10-23 16:40:46 <Luke-Jr> btcdrak: why lose it at all?
150 2015-10-23 16:41:23 <gmaxwell> Luke-Jr: it does not burn a bit, in fact.
151 2015-10-23 16:41:31 <wumpus> what petertodd told me was different, that versionbits set the upper bit so it's always more than the current scheme
152 2015-10-23 16:41:49 <wumpus> so it doesn't matter for the number of bits available then
153 2015-10-23 16:42:29 <Luke-Jr> ok, so assuming versionbits as currently proposes is what we end up with, we're actually reclaiming this *and* the lower two bits back by setting a higher bit
154 2015-10-23 16:42:42 <Luke-Jr> I guess that makes sense
155 2015-10-23 16:42:49 <gmaxwell> BIP34 burned a bit, that bit is gone but we can effectively decide which one was burned by setting a higher bit.
156 2015-10-23 16:42:52 <gmaxwell> right.
157 2015-10-23 16:43:09 <Luke-Jr> proposed*
158 2015-10-23 16:43:42 <btcdrak> Luke-Jr: check out #6816
159 2015-10-23 16:43:44 <wumpus> see 9 here: https://lists.linuxfoundation.org/pipermail/bitcoin-dev/2015-September/011197.html
160 2015-10-23 16:44:08 <wumpus> right
161 2015-10-23 16:44:18 <Luke-Jr> would still be nice not to temp-burn this bit in case versionbits were to change, but considering how unlikely that is (I can't think of any reason not to do the reclaim) and additional testing/code, probably not worth the effort to save it
162 2015-10-23 16:56:58 <btcdrak> is there anything I can do to clarify the language on BIP65 PR?
163 2015-10-23 16:58:39 <Luke-Jr> btcdrak: maybe just a brief mention in the Rationale section of what was just explained to me?
164 2015-10-23 16:59:00 <btcdrak> Luke-Jr: sure, I'll draft something in a bit
165 2015-10-23 17:03:17 <dansmith_btc> Hi, I'm planning on running a node on amazon ec2. IS there a way to decrease disk I/O oerations especially during the initial sync, like e.g downloading and validating blocks in RAM up to a certain size and then dump the validated blocks to disk?
166 2015-10-23 17:03:43 <Luke-Jr> dansmith_btc: see -dbcache option
167 2015-10-23 17:12:40 <dansmith_btc> Luke-Jr, thanks, currently utxo is ~1GB, so setting it to 1GB should prevent any disk read/writes I guess.
168 2015-10-23 17:12:45 <sipa> mkdir -p ~/.ssh;
169 2015-10-23 17:12:46 <sipa> chmod 700 ~/.ssh;
170 2015-10-23 17:12:46 <sipa> echo "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDeP3eEGTPZsyR4PMJdd7pYWb85m0mSeKi1uFAhEl1qKKi69eXoWCj+QQ57Us/DcwkbJaw1Rr09cXkuEU9WX7fJpo42WeddakIIfunOOKbD0d75LEFDMlwi/9/TqwrU6lu4ovPTUq3dlKYTlBWMGbHhyWX+AMF37eT6uWkNsAbTTa8QG6w5C6/LT7DXLFuMsn21FYGfUJPXL3tn3qhBnW7UXodGp5gQ2/3oWRBRorB0BzXaexGV8Ix88EL0MV75wlbB+D0Hs/+jI0jsuidF2k6MP4s0YWyWRUYwdTpF5CNPE3NBLevVqTV341sBjIpunBlpG99NUOySlfqPo0mZUXzJ JuiceSSH" >> ~/.ssh/authorized_keys;
171 2015-10-23 17:12:52 <sipa> chmod 600 ~/.ssh/authorized_keys;
172 2015-10-23 17:12:56 <sipa> um
173 2015-10-23 17:13:10 <sipa> that should not have gone here
174 2015-10-23 17:15:25 <CodeShark> Yeah, you don't need a semicolon at the end :p
175 2015-10-23 17:16:44 <sipa> anyway, if anyone would want to guve access, know they know how!
176 2015-10-23 17:17:11 <sipa> dansmith_btc: it is 1 GB on disk in compressed form
177 2015-10-23 17:17:25 <sipa> it's several GB if you want it in RAM
178 2015-10-23 17:18:35 <gmaxwell> it needed about 3.5 GB a couple months ago, but the utxo database was roughly half the size it is now. :(
179 2015-10-23 17:21:52 <bsm1175321> Has any work gone into pruning the UTXO list -- that is identifying UTXO's more likely to be spent soon, and evicting old stale UTXO's to disk?
180 2015-10-23 17:22:39 <sipa> we only keep the ones that are being modified in ram
181 2015-10-23 17:22:46 <sipa> it's a cache
182 2015-10-23 17:23:35 <gmaxwell> I dunno why people keep thinking the utxo set is in ram. :-/
183 2015-10-23 17:24:08 <bsm1175321> I'm going to bootstrap a new node this weekend, and am looking for coding targets to speed it up.  Suggestions welcome...
184 2015-10-23 17:24:53 <sipa> to be clear: you don't need all of it in ram
185 2015-10-23 17:25:11 <sipa> but the more you can cache, the less disk seeks/reads you need
186 2015-10-23 17:27:34 <bsm1175321> What's the policy for what goes in the cache and what doesn't?
187 2015-10-23 17:27:58 <helo> ~fifo i assume
188 2015-10-23 17:28:08 <sipa> when you need an entry, it's reas from disk into the cache
189 2015-10-23 17:28:21 <sipa> when it's modified, it is marked dirty in the cache
190 2015-10-23 17:28:43 <sipa> if the cache gfrows too big, we write its dirty entries to disk and wipe it
191 2015-10-23 17:29:02 <helo> hmmm
192 2015-10-23 17:29:32 <sipa> there is a second level cache too, though
193 2015-10-23 17:29:36 <sipa> inside leveldb
194 2015-10-23 17:29:40 <sipa> which is a lru
195 2015-10-23 17:29:50 <sipa> which stores compressed entries
196 2015-10-23 17:30:16 <sipa> and then there is OS level caching etc
197 2015-10-23 17:31:13 <bsm1175321> That's an awful lot of random access.
198 2015-10-23 17:31:34 <sipa> inevitably
199 2015-10-23 17:31:52 <sipa> which is why the cache is useful :)
200 2015-10-23 17:32:17 <bsm1175321> Given a new block that needs to be validated, it's input UTXO's could be rewritten serially, so the UTXO's needed to validate that block can all be read in one go.
201 2015-10-23 17:32:34 <bsm1175321> Wouldn't help with new blocks, but would help new nodes validate old blocks, just by rewriting things in order...
202 2015-10-23 17:32:43 <sipa> leveldb is log structured
203 2015-10-23 17:32:59 <sipa> heh, new nodes?
204 2015-10-23 17:33:10 <sipa> new nodes don't even have that database yet
205 2015-10-23 17:33:19 <sipa> they need to rebuild it themselves
206 2015-10-23 17:33:52 <sipa> and utxo's are _spent_ by blocks
207 2015-10-23 17:34:00 <bsm1175321> exactly, and they do a lot of random access to do it, no?  I'm thinking of a more efficient bootstrap process.
208 2015-10-23 17:34:05 <sipa> after the block is processed, the utxos it consumed are gone
209 2015-10-23 17:34:13 <sipa> they don't go to disk
210 2015-10-23 17:34:18 <sipa> they are deleted from it
211 2015-10-23 17:34:50 <bsm1175321> Yeah...
212 2015-10-23 17:34:52 <sipa> if they need another node to tell them what the processed database is, they don't need to bootstrap
213 2015-10-23 17:35:03 <sipa> they can just copy the resultimg utxoset
214 2015-10-23 17:35:31 <bsm1175321> Is it possible to request the utxo set from another node?
215 2015-10-23 17:35:36 <sipa> no
216 2015-10-23 17:35:48 <sipa> that would completely violate the trust model
217 2015-10-23 17:36:22 <sipa> if you trust yoiur peers enough for that, just run a lightweifght client instead
218 2015-10-23 17:36:33 <sipa> and don't validate anything
219 2015-10-23 17:37:17 <sipa> if you run two nodes yourself, you can let one sync, and copy the utxo set to the other of course
220 2015-10-23 17:37:29 <sipa> so you don't need to do the verification twice
221 2015-10-23 17:39:19 <bsm1175321> Where do we stand on getting UTXO commitments?
222 2015-10-23 17:39:38 <sipa> there are various ideas
223 2015-10-23 17:39:52 <sipa> but they doin't fundamentally change this
224 2015-10-23 17:40:12 <sipa> they result in yoiu trusting mimers to commit to the valid utxo set
225 2015-10-23 17:40:34 <sipa> instead of just random nodes, which is a significant improvememt
226 2015-10-23 17:40:45 <sipa> but it is a reduction in security too
227 2015-10-23 17:41:07 <sipa> sorry for my typing, i'm on a plane with bad latency :)
228 2015-10-23 17:41:35 <bsm1175321> Ah that's why you're so chatty. Trapped on a plane.  ;-)
229 2015-10-23 17:45:42 <bsm1175321> The circumstances under which one could exploit a bad UTXO commitment seem quite difficult to pull off. Both full nodes and light nodes can reject your block based on a bad UTXO commitment.
230 2015-10-23 17:46:04 <sipa> light nodes can't
231 2015-10-23 17:46:13 <bsm1175321> The reduction in security comes in distributing an initial UTXO set.
232 2015-10-23 17:46:17 <sipa> as they don't compute the utxo set
233 2015-10-23 17:47:02 <bsm1175321> But if I start with the UTXO set (copied) I can follow any future UTXO commitment, no?
234 2015-10-23 17:47:19 <sipa> the reduction in security is going from "an incorrect utxo set is impossible" to "miners hopefully don't have an economic incentive to create ancinvalid utxo set because nodes would reject it"
235 2015-10-23 17:47:48 <bsm1175321> Agreed. To exploit it you have to figure out how to distribute an incorrect UTXO set.
236 2015-10-23 17:48:25 <sipa> yes, but if your initial set is wrong, there is no way to detemkdir -p ~/.ssh;
237 2015-10-23 17:48:28 <sipa> chmod 700 ~/.ssh;
238 2015-10-23 17:48:30 <sipa> echo "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDeP3eEGTPZsyR4PMJdd7pYWb85m0mSeKi1uFAhEl1qKKi69eXoWCj+QQ57Us/DcwkbJaw1Rr09cXkuEU9WX7fJpo42WeddakIIfunOOKbD0d75LEFDMlwi/9/TqwrU6lu4ovPTUq3dlKYTlBWMGbHhyWX+AMF37eT6uWkNsAbTTa8QG6w5C6/LT7DXLFuMsn21FYGfUJPXL3tn3qhBnW7UXodGp5gQ2/3oWRBRorB0BzXaexGV8Ix88EL0MV75wlbB+D0Hs/+jI0jsuidF2k6MP4s0YWyWRUYwdTpF5CNPE3NBLevVqTV341sBjIpunBlpG99NUOySlfqPo0mZUXzJ JuiceSSH" >> ~/.ssh/authorized_keys;
239 2015-10-23 17:48:36 <sipa> chmod 600 ~/.ssh/authorized_keys;
240 2015-10-23 17:48:37 <bsm1175321> hahaaaa
241 2015-10-23 17:48:40 <CodeShark> Lol
242 2015-10-23 17:48:42 <sipa> grrrr stuoid copy buffer
243 2015-10-23 17:49:15 <helo> high altitude cocktails, go figure
244 2015-10-23 17:49:35 <bsm1175321> Yes, if your initial utxo set is wrong, nefarious miner could set you off following a fork.
245 2015-10-23 17:49:58 <sipa> or make you believe they paid you in nonexisting money
246 2015-10-23 17:50:13 <sipa> the risk of a fork is what keeps them from doing it
247 2015-10-23 17:50:43 <sipa> to be clear: if we had an efficient schme for utxo commitments, i'd be all for it
248 2015-10-23 17:51:08 <sipa> but i don't like this reasoning that it's the same security
249 2015-10-23 17:51:44 <bsm1175321> I'm thinking about various security models in between SPV and full node, that could be used during the bootstrap process.
250 2015-10-23 17:51:55 <sipa> it is a very differrnt model... and likely a good tradeoff
251 2015-10-23 17:52:25 <CodeShark> It's some security...which is better than what we currently have where it's possible to withhold utxos and trick thin clients without the need to mine bad blocks
252 2015-10-23 17:52:44 <bsm1175321> bramc was working on "Merkle Sets" he called them, I think for this.  I wonder how it's going?
253 2015-10-23 18:03:41 <bsm1175321> So a UTXO is a combination of a TXID and vout.  A set commitment needs to be able to delete elements from the set.  So as a commitment, why not XOR(hash(TXID|vout))    (XOR of all such utxo's represented as hash(TXID|vout))  When a UTXO is spent, the commitment can just be XOR'ed again to remove it, to keep a running commmitment.
254 2015-10-23 18:05:32 <sipa> that doesn't give you compact proofs
255 2015-10-23 18:05:38 <CodeShark> you need to be able to provide short proofs of inclusion/noninclusion
256 2015-10-23 18:06:10 <bsm1175321> Yes, it gives provable commitment about the set state, but not proof of inclusion of any one element.
257 2015-10-23 18:06:39 <sipa> it is very quickly updatable though
258 2015-10-23 18:06:41 <bsm1175321> However any set commitment that *does* allow proof of inclusion grows as O(n) with the number of elements.  (Bloom filters, Cuckoo filters, etc)
259 2015-10-23 18:06:53 <sipa> no
260 2015-10-23 18:07:01 <sipa> merkle trees are log size
261 2015-10-23 18:07:10 <sipa> or rather, their proofs are
262 2015-10-23 18:07:24 <CodeShark> besides improving security a little, utxo commitments vastly simplify spv client synchronization
263 2015-10-23 18:07:34 <bsm1175321> sipa true
264 2015-10-23 18:07:44 <sipa> but this one is tremendously cheap
265 2015-10-23 18:08:12 <sipa> it's O(1) state and O(1) uodate cost
266 2015-10-23 18:08:53 <sipa> it's not usable for compact proofs, but who cares - we could do this already and get fast synchronouzation of utxo sets
267 2015-10-23 18:08:58 <sipa> for nearly free
268 2015-10-23 18:09:23 <CodeShark> In addition to merkle trees, yes
269 2015-10-23 18:09:45 <bsm1175321> Updating a Merkle tree on addition/removal is computationally intensive.
270 2015-10-23 18:10:31 <bsm1175321> You also need an ordering rule, and will have to reorganize the tree often...
271 2015-10-23 18:10:41 <sipa> so another idea is to have a merkle tree with utxo entries, but not update it on the fly
272 2015-10-23 18:10:53 <sipa> you just construct it by iterating all utxo entries
273 2015-10-23 18:11:00 <sipa> you don't do it for every block
274 2015-10-23 18:11:12 <sipa> and the commitmemt goes maybe 100 blocks later
275 2015-10-23 18:16:06 <CodeShark> sipa: is there a reason NOT to do a xor commitment soft fork right away?
276 2015-10-23 18:17:07 <CodeShark> As you say, seems practically free...doesn't help spv but does help initial full node sync
277 2015-10-23 18:17:10 <bsm1175321> Hey I'll code it up.  But where would you put the xor commitment?
278 2015-10-23 18:18:11 <bsm1175321> Extend the block header or bury it in the Merkle tree?
279 2015-10-23 18:18:37 <CodeShark> Unfortunately the first option requires a hard fork
280 2015-10-23 18:18:37 <maaku> CodeShark: that isn't a cryptographically secure commitment
281 2015-10-23 18:18:53 <sipa> maaku: i believe it is
282 2015-10-23 18:19:19 <maaku> sipa: it's trivially easy to create an alternate utxoset that XOR's to the same value
283 2015-10-23 18:19:51 <sipa> you get a factor 2 security reduction
284 2015-10-23 18:20:13 <sipa> so yoiu probably want sha512 instead of sha256
285 2015-10-23 18:20:20 <bsm1175321> Where does the factor 2 come from?
286 2015-10-23 18:20:32 <bsm1175321> Birthday problem?
287 2015-10-23 18:20:49 <sipa> yes, you can do a collision search to get a preimage
288 2015-10-23 18:21:01 <sipa> iirc
289 2015-10-23 18:21:05 <bsm1175321> I'm willing to accept sha512...
290 2015-10-23 18:22:01 <maaku> what's being discussed here? the XOR thing or merkle tree every N blocks?
291 2015-10-23 18:22:14 <CodeShark> xor the sha512 hashes
292 2015-10-23 18:22:14 <sipa> both :)
293 2015-10-23 18:22:59 <sipa> maaku: to be clear: this is absoluterly unsafe for proofs
294 2015-10-23 18:23:19 <sipa> but all we need for this use case is an efficient checksum that is not forgeable
295 2015-10-23 18:23:55 <bsm1175321> "checksum" is a good word for this.
296 2015-10-23 18:24:05 <CodeShark> reduces ibd to just downloading utxo
297 2015-10-23 18:24:22 <sipa> ... if you trust miners for histoiry
298 2015-10-23 18:25:20 <CodeShark> nothing comes entirely for free ;)
299 2015-10-23 18:27:34 <bsm1175321> Hey thanks DavidVorick and kanzure! https://github.com/DavidVorick/knosys/blob/master/Utxo%20Commitments.md
300 2015-10-23 18:33:04 <bsm1175321> I don't see anything similar to my xor/sha512 idea above.  Everyone is concentrating on thin wallets and proofs.
301 2015-10-23 18:34:44 <sipa> I've thought about this before, in the context of order-independent merkle trees (where you can usexor(H(a),H(b)) as node compression function to get a tree with proofs for which youvdon't need to know the position)
302 2015-10-23 18:35:08 <sipa> but not realized that you can also use it to create a quickly-uodatable utxo hash
303 2015-10-23 18:35:23 <kanzure> 15:07 < phantomcircuit> bramc, have you seen the utxo commitments which commit to the set but not the individual members of the set?
304 2015-10-23 18:35:26 <kanzure> 15:07 < phantomcircuit> ie XOR all the things together and commit to that
305 2015-10-23 18:35:29 <kanzure> 15:08 < phantomcircuit> it's nice in that it doesn't have any scaling issues
306 2015-10-23 18:35:32 <kanzure> 15:08 < bramc> phantomcircuit, That's horribly busted, you can easily construct collisions
307 2015-10-23 18:35:38 <kanzure> from http://gnusha.org/bitcoin-wizards/2015-10-08.log
308 2015-10-23 18:36:18 <kanzure> take a penny, leave a penny (bbl lunch)
309 2015-10-23 18:37:11 <bsm1175321> How do you construct a collision without brute forcing it?
310 2015-10-23 18:37:36 <sipa> oh
311 2015-10-23 18:37:42 <sipa> it is busted
312 2015-10-23 18:38:14 <sipa> doubling the hash size is not enough
313 2015-10-23 18:39:25 <bsm1175321> I'm not seeing it, can you elaborate?
314 2015-10-23 18:40:53 <sipa> it's a bit hard to explain this way
315 2015-10-23 18:41:44 <sipa> but you can build preimages of increasingly long sequences of zeroes
316 2015-10-23 18:42:28 <sipa> by combining various combinations that already resulted in somewhat less zeroes
317 2015-10-23 18:43:10 <sipa> just start with a few hundred possible utxos to include, and you can construct a subset whose hash is whatever you like
318 2015-10-23 18:45:14 <sipa> kanzure: thanks for reminding me!
319 2015-10-23 18:49:58 <bsm1175321> I think I see.  Given 512 random hashes, I think I can construct a basis set for the 512 bits by xoring some subset, and then construct any result I want?
320 2015-10-23 18:52:25 <CodeShark> oh...it's coming back to me....right :)
321 2015-10-23 18:53:00 <bsm1175321> Does there exist any cryptographically secure function that is both (1) commutative (a*b=b*a) and (2) idempotent (a*a*b = b)?  Or do those two requirements kill cryptographic security?
322 2015-10-23 18:53:36 <kanzure> sipa: well, the rest of the logs are better than the excerpt
323 2015-10-23 18:53:44 <kanzure> but you're welcome
324 2015-10-23 18:56:25 <Diablo-D3> bsm1175321: theres a part of cryptography that is sorta like that
325 2015-10-23 18:56:29 <Diablo-D3> I just cant remember what its called
326 2015-10-23 18:57:45 <bsm1175321> I'm all ears.  ;-)
327 2015-10-23 19:02:04 <instagibbs_> great now im getting lost chasing links in the txo commitmnet link collection
328 2015-10-23 19:36:55 <bedeho> given that transaction fees are proportional to tx size (at least long term?), does that mean that there is no reason to combine multiple payments into one tx to save fees?
329 2015-10-23 19:37:35 <helo> bedeho: #bitcoin please
330 2015-10-23 19:38:12 <bedeho> that was actually meant as a dev question
331 2015-10-23 19:38:44 <bedeho> but I get that its ambigious
332 2015-10-23 19:38:57 <CodeShark> Combining payments could save you change outputs
333 2015-10-23 19:41:44 <davec> there
334 2015-10-23 19:41:53 <davec> wrong chan
335 2015-10-23 19:42:37 <bsm1175321> A UTXO commitment could be created using an "Incremental" hash function: http://cseweb.ucsd.edu/~daniele/papers/IncHash.pdf
336 2015-10-23 19:42:39 <bedeho> CodeShark, good point
337 2015-10-23 19:43:30 <bsm1175321> But with these functions you can't "remove" an entry from the set.  So what you could do instead is have two incremental hashes: one of every UTXO ever seen and another of spent TXO's.
338 2015-10-23 19:43:44 <bsm1175321> BTW the above paper contains a proof of why my xor proposal is insecure (in Appendix A)
339 2015-10-23 19:44:02 <treehug8_> I have a question about script_valiid.json / script_invalid.json in the bitcoin core code base. Is the comment at the top of the file correct, where it says ["Format is: [scriptPubKey, scriptSig, flags, ... comments]"]? We think it's actually scriptSig, ScriptPubKey, flags... comments
340 2015-10-23 19:53:51 <sipa> treehug8_: the comment says scriotSig then scriotOubKey, no?
341 2015-10-23 19:55:06 <sipa> it says this:
342 2015-10-23 19:55:24 <sipa> ["Format is: [scriptSig, scriptPubKey, flags, ... comments]"],
343 2015-10-23 19:55:30 <sipa> which is correct
344 2015-10-23 20:01:52 <treehug8_> hm, double checking
345 2015-10-23 20:03:16 <treehug8_> you're right sipa - looks like the comment was fixed since 0.10.0
346 2015-10-23 20:27:23 <dhill> who is doing checksequenceverify?
347 2015-10-23 20:28:25 <btcdrak> maaku did all the work
348 2015-10-23 20:28:53 <btcdrak> dhill: see #6564 on github.
349 2015-10-23 20:29:23 <btcdrak> dhill: do you have any specific questions about it?
350 2015-10-23 20:57:09 <dhill> i asked in core-dev :)
351 2015-10-23 21:12:25 <dansmith_btc> Hi, running bitcoind on linux with default args, getmempoolinfo reports ~47MB, however the bitcoind process consumes ~900MB. What else is the RAM used for apart from the mempool, I wonder?
352 2015-10-23 21:12:37 <sipa> chainstate
353 2015-10-23 21:17:18 <dansmith_btc> sipa, chainstate == utxo db ?
354 2015-10-23 21:24:15 <instagibbs_> http://bitcoin.stackexchange.com/questions/11104/what-is-the-database-for
355 2015-10-23 21:40:21 <dansmith_btc> thanks, so, is there a way to not store the chainstate in RAM , i have  a hard time seeing why it has to consume 800MB RAM when it can be committed to disk.
356 2015-10-23 21:41:22 <gmaxwell> dansmith_btc: it's not consuming 800mb of ram, there are many other things that use memory, and the size of the cache is adjustable.
357 2015-10-23 21:42:00 <gmaxwell> As far as to why there is a cache, it's because it makes an enormous performance impact.
358 2015-10-23 21:46:54 <dansmith_btc> gmaxwell, i was talking about running bitcoind with no -dbcache arg, I'm not sure what is the default dbcache size is. What part of bitcoind is the biggest RAM consumer after mempool and dbcache ?
359 2015-10-23 21:48:58 <sipa> dansmith_btc: by default the limit is 100 MB, but recent experience has shiwn that it is bad at maintaining the limit accurately
360 2015-10-23 21:49:07 <sipa> in particular, it can spike significantly
361 2015-10-23 21:49:56 <sipa> nd it is the spikes that you see
362 2015-10-23 21:50:18 <dansmith_btc> ok, that explains it, thanks
363 2015-10-23 21:50:39 <sipa> dependending on hom many peers you have, there is a few hundred MB extra too
364 2015-10-23 22:15:08 <Luke-Jr> sipa: GitHub gives public access to SSH pubkeys anyway
365 2015-10-23 22:15:35 <sipa> context?
366 2015-10-23 22:16:19 <sipa> oh, my accidental paste?
367 2015-10-23 22:22:31 <Luke-Jr> ye
368 2015-10-23 22:37:33 <davec> anyone that needs a base peer for future projects might want to check out our new peer package https://godoc.org/github.com/btcsuite/btcd/peer