1 2014-10-29 00:01:16 <BlueMatt> gmaxwell: I'm weary of using upnp to detect external ips...
  2 2014-10-29 00:01:49 <BlueMatt> gmaxwell: I personally have ISP-level nat that my isp enabled a few months back because they ran out of ipv4
  3 2014-10-29 00:02:32 <gmaxwell> BlueMatt: so? nothing changed there. Also, it gets ignored if the result is non-routable.
  4 2014-10-29 00:02:57 <BlueMatt> gmaxwell: I had only read the commitmsg as of yet, just figured Id bring it up :p
  5 2014-10-29 00:03:01 <gmaxwell> Go actually read what my PR does; it has nothing to do with UPNP.
  6 2014-10-29 00:03:12 <BlueMatt> ok, ok...
  7 2014-10-29 00:03:20 <lechuga_> cryptoki std needs support for bip32 key derivation :/
  8 2014-10-29 00:03:22 <BlueMatt> I was about to start reading, just hadnt yet
  9 2014-10-29 00:05:05 <lechuga_> wonder if coinkite has such an extension
 10 2014-10-29 00:39:54 <justanotheruser> Wouldn't sidechains have full node security given that you could have fraud proofs that prove A) A block is the incorrect size using a merkle sum tree, B) A transaction script doesn't evaluate to true, C) A transaction has a previous tx that wasn't in the utxo at the time (proof that it was already spent)
 11 2014-10-29 00:41:36 <gmaxwell> justanotheruser: Probably better in #bitcoin-wizards
 12 2014-10-29 00:41:42 <gmaxwell> I'll respond there.
 13 2014-10-29 00:42:18 <justanotheruser> ok
 14 2014-10-29 02:12:38 <imminent_overrun> Good day
 15 2014-10-29 02:12:45 <imminent_overrun> Is there anyone?
 16 2014-10-29 02:13:14 <imminent_overrun> Who are doing promotion of Bitcoin into world? Into different social services?
 17 2014-10-29 02:14:21 <imminent_overrun> I've discovered such perfect for Bitcoin Social Network: https://www.tsu.co/imminent_overrun
 18 2014-10-29 02:14:33 <imminent_overrun> Invite only for the first time
 19 2014-10-29 02:15:08 <imminent_overrun> there is $ for ads, which for example in Facebook taking company, divided proportionally between users
 20 2014-10-29 02:15:35 <imminent_overrun> This must be not a $, but Bitcoin.
 21 2014-10-29 02:16:09 <imminent_overrun> Who is main-promoter of Bitcoin currently?
 22 2014-10-29 02:16:43 <poutine> imminent_overrun, Do you really think there's an answer to that question?
 23 2014-10-29 02:17:25 <poutine> These are not dev questions
 24 2014-10-29 02:17:51 <Dizzle> And a better question for #bitcoin. Though if by promoter you mean people who can push commits to master of bitcoin-core on the dev side, that'd be gavinandresen and a few of the others in here.
 25 2014-10-29 02:22:53 <imminent_overrun> I'm actually think, that you guys are the core of #bitcoin, are not?
 26 2014-10-29 02:47:40 <lechuga_> bitcoin is like a bright flame to every would-be wacky entreprenuer moth on the planet
 27 2014-10-29 03:35:28 <aphorise> I keep getting 'error: {"code":-32601,"message":"Method not found (disabled)"}' for any wallet related API call - even though I built with wallet enabled.
 28 2014-10-29 03:35:37 <aphorise> What am I doing wrong?
 29 2014-10-29 03:36:22 <Luke-Jr> sounds like your build doesn't in fact have the wallet enabled - or maybe it's disabled at runtime?
 30 2014-10-29 03:38:33 <aphorise> sorry - damn trivial - it was a .conf issue - thanks.
 31 2014-10-29 04:16:42 <FRCJoe> Hello, I've been recently thinking that there may be a need for an update to the key alert system as has been highlighted by a few different posts.
 32 2014-10-29 04:16:43 <FRCJoe> I was wondering if others would think a multi-level alert key system would be of use. or at least giving each developer their own key
 33 2014-10-29 04:16:48 <FRCJoe> I know I am new to most of you, so instead of writing a straight BIP (I am writing a draft now), i wanted to hear thoughts if i should propose this officially.
 34 2014-10-29 04:16:51 <FRCJoe> I also was interested in hearing what requirements there might be for such a system to work efficiently
 35 2014-10-29 04:17:36 <FRCJoe> currently we only have one key and a few developers with the key, if one of those persons is compromised the entire key alert is compromised
 36 2014-10-29 04:17:37 <BlueMatt> wumpus: is currently working (IIRC) on alert key rotation support
 37 2014-10-29 04:17:41 <BlueMatt> not sure how much else we need
 38 2014-10-29 04:17:49 <FRCJoe> ah did not know that
 39 2014-10-29 04:17:56 <BlueMatt> yes, and a compromised alert key is not the end of the world
 40 2014-10-29 04:17:56 <FRCJoe> is there a bip i can look at?
 41 2014-10-29 04:18:11 <justanotheruser> Is a compromised alert key a DoS vuln?
 42 2014-10-29 04:18:26 <BlueMatt> (or shouldnt be, we've had bugs where it could disable functionality that it shouldnt (I think there is an override flag, or should be))
 43 2014-10-29 04:18:38 <BlueMatt> justanotheruser: moderately
 44 2014-10-29 04:18:49 <FRCJoe> i thought haulting tx was removed in a later version?
 45 2014-10-29 04:19:12 <FRCJoe> BlueMatt: can i pm you for one second
 46 2014-10-29 04:19:18 <BlueMatt> why?
 47 2014-10-29 04:19:26 <BlueMatt> umm...sure, but it can probably stay here?
 48 2014-10-29 04:20:06 <FRCJoe> okay. i can say it here. i met you in miami with Luke-Jr and such over proof of pizza recording
 49 2014-10-29 04:20:19 <FRCJoe> just in case you didn't know my handle ;)
 50 2014-10-29 04:20:59 <Luke-Jr> (speaking of Miami, deadline for early bird registration for 2015 is any day now)
 51 2014-10-29 04:21:03 <FRCJoe> but anyway, if i can be of help with that. please do let me know. i'll just pause on writing that proposal out since someone is working on it
 52 2014-10-29 04:21:10 <FRCJoe> thanks for the info Luke-Jr
 53 2014-10-29 04:21:39 <BlueMatt> FRCJoe: look at the stuff wumpus has been writing, ask him if you can help him when he wakes up, and review his work, Id say
 54 2014-10-29 04:21:54 <FRCJoe> okay. i'll try to find it. thanks BlueMatt
 55 2014-10-29 06:26:45 <wumpus> I didn't write a BIP, I just wrote some code to be able to generate new alert keys with a simple command https://github.com/bitcoin/bitcoin/pull/4883
 56 2014-10-29 06:28:07 <wumpus> we did have a discussion about how to rotate the key: it would mean accepting the old and new key for a long while, to be able to relay alerts to old versions and to make sure new versions aren't isolated between new versions that won't relay their alerts
 57 2014-10-29 06:28:41 <wumpus> would make sense to write a BIP on that, I may if I have some time...
 58 2014-10-29 06:28:49 <wumpus> I'm not convinced that anything beyond a rotation is needed, though
 59 2014-10-29 06:29:26 <gmaxwell> I would not like to see complexity added there, they're almost useless as is, and risking vulnerabilities to make them more complex would not be a good tradeoff.
 60 2014-10-29 06:30:01 <gmaxwell> if a signing group is compromised that only option is updating the software in any case; since thats the only way to remove them.
 61 2014-10-29 06:30:21 <wumpus> a compromised alert key would be annoying, but it'd indeed not be the end of the world, it'd only give people a way to spam messages
 62 2014-10-29 06:31:43 <gmaxwell> well not even that, since a maximum value alert triggers a fixed message.
 63 2014-10-29 06:31:47 <wumpus> seeing how rarely the alert system is actually used, and how little its code is maintained, I'd also find it a risk to make it overly complex
 64 2014-10-29 06:31:57 <wumpus> right, it'd at most be temporary
 65 2014-10-29 06:32:45 <poutine> the maximum value alert does not have an integer size, if bitcoin can ever be compiled 64 bit, one could trigger an alert that would only be permanent for 32 bit architectures
 66 2014-10-29 06:33:48 <poutine> https://github.com/bitcoin/bitcoin/blob/master/src/alert.cpp#L182
 67 2014-10-29 06:34:16 <wumpus> there are some smaller issues/improvements for the alert system open though:  https://github.com/bitcoin/bitcoin/issues/1005
 68 2014-10-29 06:34:47 <wumpus> "if bitcoin can ever be compiled 64 bit" ehh... like everyone does?
 69 2014-10-29 06:35:32 <poutine> err
 70 2014-10-29 06:35:53 <gmaxwell> poutine: your message is confusing, you're talking about ILP64 which has pretty much never been used. (though indeed, thats done in a dumb way there)
 71 2014-10-29 06:36:16 <gmaxwell> poutine: integers are not 64 bits on existant 64 bit systems.
 72 2014-10-29 06:36:37 <poutine> ok maybe I'm mistaken
 73 2014-10-29 06:36:51 <wumpus> oh, right, would have been better to use an explicit uint32_t there
 74 2014-10-29 06:37:22 <wumpus> although even with that: no guarantee *anything* will work on a hypothetical architecture that makes base ints 64 bit
 75 2014-10-29 06:37:48 <poutine> I think properly architectured code will not have issues
 76 2014-10-29 06:38:19 <poutine> but I was mistaken about several points here
 77 2014-10-29 06:38:36 <wumpus> you're thinking wrongly, in practice
 78 2014-10-29 06:38:59 <wumpus> just ask people with experience porting code to weird architectures
 79 2014-10-29 06:39:33 <gmaxwell> yea, the code is daft, but in a completely harmless way... (and a great amount will not work on ILP64)
 80 2014-10-29 06:39:58 <wumpus> (another fun one is if chars are 16 bits)
 81 2014-10-29 06:40:34 <gmaxwell> yea, thats actually existing in things I've targeted in the last couple years (TI DSPs...)
 82 2014-10-29 06:40:38 <poutine> Well I am admittedly rusty nowadays, but the only standard data size is sizeof(char) == 1, and you should not rely on the sizeof() for any other data type to be a fixed value unless the implementation dictates, and for the most part, if you use limits.h, or similar mechanisms, you should be using those hard defined values rather than setting arbitrary limits in-application, also care should be taken when doing bitwise operations
 83 2014-10-29 06:41:27 <poutine> but I've veered far from the original thing, thanks for the explanations
 84 2014-10-29 06:42:33 <Luke-Jr> 
ACTION wonders how difficult it would be to find an unexploitable way to relay alerts for other clients

 85 2014-10-29 06:42:40 <gmaxwell> poutine: The software is only supported on a subset of implementation defined behavior. (e.g. imagine: char == short == int == 8 bits is perfectly valid C)
 86 2014-10-29 06:42:59 <Luke-Jr> gmaxwell: no, short and int are guaranteed to be >=15-bit
 87 2014-10-29 06:43:32 <sipa> char can be more than 8 bits :p
 88 2014-10-29 06:43:50 <gmaxwell> sipa: yes just pointed that out above.
 89 2014-10-29 06:43:54 <Luke-Jr> ( >= 16-bit, unless you find a way to represent sign without a bit)
 90 2014-10-29 06:44:01 <wumpus> C's combination of low-levelness and high-levelness makes it *nearly* impossible to write code that will in practice work on all kind of weird architectures that people can think if. I'm sure with extremely rigorous programming and checking practices it may be possible given enough resources, but where's the incentive...
 91 2014-10-29 06:45:35 <poutine> Ok well C99 dictates that the result of sizeof(char) == 1
 92 2014-10-29 06:45:39 <gmaxwell> it's not that hard to write C that works on 16bit char platforms, but does some effort. I've never tried ILP64 (or SILP64)... (tried to find a cray..)
 93 2014-10-29 06:45:51 <Luke-Jr> poutine: that's because sizeof's "return value" is defined by multiples of char
 94 2014-10-29 06:45:57 <gmaxwell> poutine: yes but that doesn't mean what you think it does.
 95 2014-10-29 06:46:06 <gmaxwell> (as luke says)
 96 2014-10-29 06:46:09 <Luke-Jr> poutine: if char is 16-bit, then sizeof(int) can be 1
 97 2014-10-29 06:46:19 <poutine> Luke-Jr, true, I misspoke when I said its return value, I know it's done at compile time
 98 2014-10-29 06:46:22 <wumpus> no, it's not necessarily hard, that was not my point
 99 2014-10-29 06:46:42 <Luke-Jr> poutine: note C has the CHAR_BIT macro (usually 8) for this reason
100 2014-10-29 06:46:59 <Luke-Jr> poutine: my point was that char can be >8-bit validly :p
101 2014-10-29 06:47:33 <poutine> I'm seeing that, I'm glad I deal with higher level languages now
102 2014-10-29 06:47:40 <wumpus> in my experience, writing code for a specific architecture is easiest (no matter what the architecture is), but supporting all possible combos gets hairier and hairier
103 2014-10-29 06:48:07 <Luke-Jr> I target the generic "C language" architecture when it is easy to do so :P
104 2014-10-29 06:48:23 <Luke-Jr> which mainly means I use the right types for the job
105 2014-10-29 06:48:27 <wumpus> poutine: but do those even work on the weird platforms?
106 2014-10-29 06:48:47 <wumpus> if not, you've done the same as writing C code for a subset of platforms
107 2014-10-29 06:49:29 <wumpus> (I don't think many high level languages target TI DSPs, for example)
108 2014-10-29 06:50:15 <wumpus> (or DOS-style where ints are 16 bit)
109 2014-10-29 06:51:01 <Luke-Jr> heck, lots of stuff still breaks badly on x32
110 2014-10-29 06:51:14 <Luke-Jr> though I think we're clean on that front nowadays
111 2014-10-29 06:51:22 <poutine> You won't have nearly the same level performance as native code, but I remember using TCL compiled on some weird architectures. They make breakout boards with this all done for you like tessel.io if you want to do something simple
112 2014-10-29 06:54:17 <wumpus> "Tessel is a microcontroller that runs JavaScript. " hah
113 2014-10-29 06:55:29 <wumpus> but indeed; there have been microcontrollers with BASIC for a long time, so indeed, vendors tend to pick a high level language for developers that want to get started easily
114 2014-10-29 06:57:37 <wumpus> ... though usually proprietary dialects with weird data sizes that match their CPU's preferences :-)
115 2014-10-29 06:59:45 <BlueMatt> lol, we were treating isp-level nat addresses as publicly routable :(
116 2014-10-29 07:01:28 <gmaxwell> hm? really?
117 2014-10-29 07:02:06 <BlueMatt> my ip is 100.64.122.*
118 2014-10-29 07:02:08 <sipa> heh, i had never heard about "isp-level nat"
119 2014-10-29 07:02:13 <gmaxwell> I think there is some clown out there who is addr messaging for everyone who connects to him. :(
120 2014-10-29 07:02:21 <phantomcircuit> <Luke-Jr> poutine: my point was that char can be >8-bit validly :p
121 2014-10-29 07:02:25 <phantomcircuit> also 7 bits
122 2014-10-29 07:02:31 <phantomcircuit> because why not
123 2014-10-29 07:02:32 <lewellyn> sipa: there's talk of country nat now.
124 2014-10-29 07:02:34 <BlueMatt> sipa: my isp actually did itself run out of ipv4, so they're doing nat :(
125 2014-10-29 07:02:39 <BlueMatt> as of several months ago
126 2014-10-29 07:02:42 <sipa> lewellyn: y u no ipv6?
127 2014-10-29 07:02:50 <BlueMatt> sipa: afair its /incredibly/ common in india
128 2014-10-29 07:02:55 <BlueMatt> also, many cell phones in the us do it
129 2014-10-29 07:02:58 <phantomcircuit> BlueMatt, srsly? that's hilarious they only serve like 1k people
130 2014-10-29 07:03:00 <lewellyn> sipa: i've been pushing ipv6 for 15 years.
131 2014-10-29 07:03:07 <gmaxwell> there already is at least one whole contry behind a single nat...
132 2014-10-29 07:03:17 <wumpus> north korea?
133 2014-10-29 07:03:18 <sipa> BlueMatt: those people should be punished by death by pwnage for advertizing it as *internet*
134 2014-10-29 07:03:32 <gmaxwell> wumpus: quatar.
135 2014-10-29 07:03:33 <Luke-Jr> BlueMatt: T-Mobile just uses IPv6 :p
136 2014-10-29 07:03:42 <BlueMatt> phantomcircuit: webpass? no idea, sounds about right
137 2014-10-29 07:03:42 <gmaxwell> (I've cited this before as a reason not to ban concurrent connections from a single IP...
138 2014-10-29 07:03:45 <lewellyn> gmaxwell: there's talk of, instead of moving to ipv6, assigning a single ipv4 address per hosting provider. could you imagine "visit http://facebook.com:41432"
139 2014-10-29 07:03:46 <wumpus> some countries have crazily few IPv4 addresses in the first place compared to their number of people
140 2014-10-29 07:03:58 <BlueMatt> sipa: my vpn routes most ports back to my apartment...
141 2014-10-29 07:04:05 <Luke-Jr> phantomcircuit: well, char is defined such that it must be 8-bit at least
142 2014-10-29 07:04:08 <wumpus> (sure, IPv6 would be the solution, but it's still slow in being picked up)
143 2014-10-29 07:04:19 <BlueMatt> Luke-Jr: only if you pick the right apn, iirc
144 2014-10-29 07:04:24 <sipa> BlueMatt: so you're relying on vpn-over-noninternet to get you actual internet? :)
145 2014-10-29 07:04:25 <phantomcircuit> Luke-Jr, except it's practically only 7 bits on ancient systems
146 2014-10-29 07:04:34 <Luke-Jr> phantomcircuit: then they're not C compliant :P
147 2014-10-29 07:04:43 <lewellyn> wumpus: there's no good reason for slow adoption of ipv6 tbh. windows *xp* supported it fine. and it's finally dead and buried after a couple of stints at undead.
148 2014-10-29 07:04:57 <BlueMatt> sipa: something like that
149 2014-10-29 07:05:01 <Luke-Jr> char must be equivalent to signed char or unsigned char, both of which have limits needing 8 bits
150 2014-10-29 07:05:07 <BlueMatt> sipa: and yet its all still faster than anything you've got :p
151 2014-10-29 07:05:24 <Luke-Jr> doesn't Windows set up IPv6 tunnels by default these days?
152 2014-10-29 07:05:27 <gmaxwell> Luke-Jr: at least 8, to be clear. :P
153 2014-10-29 07:05:36 <Luke-Jr> gmaxwell: yes
154 2014-10-29 07:05:41 <wumpus> lewellyn: you're preaching to the choir :(
155 2014-10-29 07:05:56 <sipa> gmaxwell: wrt scalar.h: yeah, i realize that you could avoid the modulus in the non-over-order branch.... but we want to get rid of branches, right? :)
156 2014-10-29 07:06:02 <BlueMatt> sipa: my isp does, however, give me a globally routed /64 :)
157 2014-10-29 07:06:13 <BlueMatt> unlike many us isps which think it is ok to only give you one ipv6
158 2014-10-29 07:06:13 <Luke-Jr> BlueMatt: complain
159 2014-10-29 07:06:18 <sipa> BlueMatt: i've had a globally routed /64 for 10 years
160 2014-10-29 07:06:21 <Luke-Jr> BlueMatt: ISPs are supposed to give end users a /48
161 2014-10-29 07:06:22 <lewellyn> wumpus: more like ranting on friendly ears.
162 2014-10-29 07:06:27 <BlueMatt> Luke-Jr: wha?
163 2014-10-29 07:06:33 <Luke-Jr> BlueMatt: IANA guidelines
164 2014-10-29 07:06:43 <gmaxwell> I've thought it would be fun to define some weird arch where every decision is psycho, just to see how unportable things are. stack grows the parisc way, char is 12 bits, int is 24, long is 48.
165 2014-10-29 07:06:47 <BlueMatt> sipa: yes, I used to have a /48 in my basement via HE
166 2014-10-29 07:06:47 <lewellyn> BlueMatt: yes. a /64 can't be routed properly and have RA work.
167 2014-10-29 07:06:53 <wumpus> lewellyn: in the 90's I expected ipv6 adoption to be slow... but if you had asked me if it would have been adopted globally mainstream in 2014 I wouldn't have had a doubht
168 2014-10-29 07:06:57 <Luke-Jr> gmaxwell: just use trits :P
169 2014-10-29 07:07:04 <sipa> gmaxwell: would 47 bits be legal?
170 2014-10-29 07:07:09 <lewellyn> Luke-Jr: and a /56 is considered acceptable btw
171 2014-10-29 07:07:26 <lewellyn> wumpus: no shit. and i blame cisco above all :P
172 2014-10-29 07:07:29 <Luke-Jr> lewellyn: not for end users in general
173 2014-10-29 07:07:40 <lewellyn> Luke-Jr: it's not desired, but acceptable.
174 2014-10-29 07:07:42 <gmaxwell> sipa: there is one dsp arch where addresses are bit level.
175 2014-10-29 07:07:50 <BlueMatt> lewellyn: since when? last I heard /64 was the acceptable minimum
176 2014-10-29 07:07:52 <lewellyn> it's the smallest allocation which is deemed allowed at this time.
177 2014-10-29 07:07:56 <Luke-Jr> lewellyn: well, we can't *force* ISPs do to anything. but they should be doing /48s :p
178 2014-10-29 07:08:17 <Luke-Jr> lewellyn: it's allowed to do /127s for point-to-point
179 2014-10-29 07:08:18 <lewellyn> BlueMatt: i forget where it was allowed. but somewhere on either iana or arin's site, there is a document.
180 2014-10-29 07:08:37 <Luke-Jr> which sadly T-Mobile considers their cellular connection IIRC
181 2014-10-29 07:09:17 <lewellyn> also, there are isps assigning randomly as if this was the day of ipv4 and limited address space :P
182 2014-10-29 07:09:32 <lewellyn> just so they can get away with charging more for a static allocation :(
183 2014-10-29 07:09:37 <Luke-Jr> >_<
184 2014-10-29 07:09:54 <BlueMatt> how the fuck is -loadblock working but I'm not getting any UpdateTip printfs???
185 2014-10-29 07:10:04 <Luke-Jr> I wish 6LoWPAN didn't require IPv6 NAT
186 2014-10-29 07:10:30 <Luke-Jr> (effectively require)
187 2014-10-29 07:10:46 <lewellyn> nat sucks. it really irritates me, though, that iana hasn't set up a registration body for ULAs :P
188 2014-10-29 07:11:31 <lewellyn> ULA solves the case many sites would want nat (not the ones who think it means "security"), while still allowing as much point-to-point visibility as possible.
189 2014-10-29 07:11:57 <BlueMatt> sipa: did headers-first break -loadblock for bootstrap.dat???
190 2014-10-29 07:12:20 <Luke-Jr> it'd be nice if assignments didn't route via ISPs at all
191 2014-10-29 07:12:44 <lewellyn> Luke-Jr: they didn't used to. i blame arin on that one.
192 2014-10-29 07:14:07 <wumpus> BlueMatt: no reason it would
193 2014-10-29 07:15:45 <phantomcircuit> BlueMatt, iirc update tip can be delayed with headers first
194 2014-10-29 07:16:11 <BlueMatt> phantomcircuit: by 100k blocks with 0 connections?
195 2014-10-29 07:16:15 <BlueMatt> oh, wait, 0 connections?
196 2014-10-29 07:16:50 <phantomcircuit> that doesn't sound right
197 2014-10-29 07:16:54 <BlueMatt> lol, yep
198 2014-10-29 07:17:02 <BlueMatt> dont try to -loadblock with 0 connections
199 2014-10-29 07:23:07 <gmaxwell> I noticed a fun minor misbehavior, addnode some .onion peers (or ipv6 peers) and then run without onion/ipv6 support and watch it flood your debug log with complaints that it can't use the addresses configured.
200 2014-10-29 07:23:53 <gmaxwell> I wonder what broken node(s) are rumoring for things that connect into them against their consent. :(
201 2014-10-29 07:24:48 <gmaxwell> github is acting weird.
202 2014-10-29 07:24:59 <gmaxwell> seem to be getting MITMed?!
203 2014-10-29 07:25:18 <wumpus> no problem here
204 2014-10-29 07:25:24 <BlueMatt> nothing here
205 2014-10-29 07:25:29 <phantomcircuit> fun
206 2014-10-29 07:25:43 <phantomcircuit> https cert fingerprint 58:87:52:44:D8:60:12:B0:FB:D5:F6:C0:6E:F1:6E:FC:A2:0E:15:8D:58:E9:6E:6F:76:CE:DA:66:60:B5:9B:C2
207 2014-10-29 07:25:45 <BlueMatt> 
ACTION takes this opportunity to remind everyone to pgp-sign their acks

208 2014-10-29 07:25:45 <wumpus> cert SHA-256 fingerprint: 58:87:52:44:D8:60:12:B0:FB:D5:F6:C0:6E:F...
209 2014-10-29 07:26:04 <BlueMatt> 58 87 52 44 D8 60 12 B0 FB D5 F6 C0 6E F1 6E FC
210 2014-10-29 07:26:04 <BlueMatt> A2 0E 15 8D 58 E9 6E 6F 76 CE DA 66 60 B5 9B C2
211 2014-10-29 07:26:20 <phantomcircuit> gmaxwell, ssh?
212 2014-10-29 07:26:33 <BlueMatt> 
ACTION is coming at it from ntt

213 2014-10-29 07:26:37 <phantomcircuit> fingerprint matched for ssh here
214 2014-10-29 07:26:41 <gmaxwell> no, gah. nevermind. piece of shit DSL router here randomly MITM's connections if the DSL link burps.
215 2014-10-29 07:26:53 <phantomcircuit> ha
216 2014-10-29 07:26:56 <BlueMatt> get fiber!
217 2014-10-29 07:27:29 <wumpus> 'randomly MITMs connections'? that's an interesting router
218 2014-10-29 07:28:02 <BlueMatt> wumpus: you've never had one of those?
219 2014-10-29 07:28:03 <gmaxwell> wumpus: so it can display a totally useless notice... which isn't so helpful for https.
220 2014-10-29 07:28:04 <BlueMatt> they're fun
221 2014-10-29 07:28:23 <wumpus> maybe you've just not installed the necessary root cerficitate in your browser? :p
222 2014-10-29 07:28:34 <gmaxwell> wumpus: yea, sounds like a great plan there!
223 2014-10-29 07:28:53 <BlueMatt> wumpus: which, the one installed in the factory with 2 bits of entropy?
224 2014-10-29 07:29:29 <wumpus> hehe
225 2014-10-29 07:29:55 <gmaxwell> whats even more fun is that when this happens the router usually becomes unreachable in general because the ssl load from my network is too much for it. in any case, it's super awful. I have had switching it to pure bridge mode on my todo for some time, but its a lower priority than most other things.
226 2014-10-29 07:29:56 <wumpus> but yes, showing a notice sounds like a somewhat legitimate reason, although indeed useless for https
227 2014-10-29 07:29:57 <sipa> BlueMatt: hmm, loadblock doesn't work?
228 2014-10-29 07:30:48 <wumpus> gmaxwell: I've also switched my ISP's router to full bridge mode, it just crashed every few days needing a  manual reboot
229 2014-10-29 07:31:05 <BlueMatt> sipa: I dont see why, I'm debugging
230 2014-10-29 07:31:09 <wumpus> BlueMatt: with 0 connections, apparantly
231 2014-10-29 07:31:14 <wumpus> @sipa that was
232 2014-10-29 07:31:32 <gmaxwell> yea, when I had FiOS I had that issue too. actually in that case I was able to get them to switch to a straight up ethernet connection rather than going though the crappy router.
233 2014-10-29 07:31:51 <BlueMatt> gmaxwell: you switched away from FIOS? this seems like a bad idea?
234 2014-10-29 07:32:13 <BlueMatt> gmaxwell: what can you get on dsl out there?
235 2014-10-29 07:33:19 <gmaxwell> 25mbit/1.5mbit or 18mbit/3mbit in annex a mode. which is plenty fast...
236 2014-10-29 07:35:57 <BlueMatt> 
ACTION goes back to seeding bootstrap.dat at 3mbps

237 2014-10-29 07:37:49 <BlueMatt> gmaxwell: listening on torhs but no -discover? why not?
238 2014-10-29 07:38:02 <BlueMatt> I mean you dont want to discover your external ipv4, but it shouldnt if you're tor-only anyway?
239 2014-10-29 07:38:33 <gmaxwell> BlueMatt: because then you end up potentially leaking your v4 ip (e.g. learning it from upnp and giving it to a HS inbound peer)
240 2014-10-29 07:39:26 <wumpus> ouch
241 2014-10-29 07:39:32 <BlueMatt> hmm...indeed, forgot about that...maybe we should switch to never trying to find our ip and only use the ones peers tell us
242 2014-10-29 07:39:35 <BlueMatt> would be much safer
243 2014-10-29 07:41:21 <wumpus> although you should really disable upnp in case of a hiddenservice-only node
244 2014-10-29 07:41:42 <BlueMatt> indeed, but we also look at bind addresses and the addresses on network interfaces :(
245 2014-10-29 07:41:51 <wumpus> yes
246 2014-10-29 07:42:28 <wumpus> but in the normal, non-paranoid case that's not a problem, requiring people to give their external IPs will likely result in less connectable nodes advertising
247 2014-10-29 07:44:36 <BlueMatt> hmm?
248 2014-10-29 07:44:50 <BlueMatt> if you have a server with a public ip and are tor-ing, you dont want it to lookup your local address
249 2014-10-29 07:45:38 <wumpus> (also, for nodes running with dynamic external IPs,  if they're forced to provide it manually they'd have to change their configuration every time)
250 2014-10-29 07:45:51 <gmaxwell> BlueMatt: doing both ipv4 and tor is a totally reasonable thing to do (though not while you're trying to be hidden yourself, of course)
251 2014-10-29 07:46:23 <wumpus> gmaxwell: indeed; my nodes generally listen on tor as well as a service to tor peers, not to stay hidden
252 2014-10-29 07:47:08 <wumpus> (but due to the complex config I do provide my external IPs explicitly and have discover disabled)
253 2014-10-29 07:47:11 <BlueMatt> gmaxwell: absolutely, hence my suggestion of just rumoring your address that nodes give you when you want it heard
254 2014-10-29 07:47:19 <gmaxwell> BlueMatt: only using peer sourced IPs would be safer, but not always useful... since some network setups result in the outbound connections coming from another address. :(
255 2014-10-29 07:47:24 <BlueMatt> + probably updating it when your ip changes
256 2014-10-29 07:47:41 <gmaxwell> well one nice thing about using peer IPs is that it always updates when your IP changes.
257 2014-10-29 07:47:42 <BlueMatt> gmaxwell: how many network setups have that case?
258 2014-10-29 07:48:08 <BlueMatt> essentially only strange vpn setups, really?
259 2014-10-29 07:48:20 <gmaxwell> BlueMatt: several different people have shown up in #p2pool complaining they can never get inbound connections as a result (and in p2pool the protocol makes fixing it not possible without an incompatible change)
260 2014-10-29 07:48:51 <BlueMatt> gmaxwell: hmm? what kind of setup are they running?
261 2014-10-29 07:49:01 <wumpus> for bitcoin there is probably a larger-than-normal share that has weird vpn setups, so they have to be supported with some option
262 2014-10-29 07:49:11 <BlueMatt> -externalip
263 2014-10-29 07:49:12 <gmaxwell> router with multiple internet connections in one case, vpn in another.
264 2014-10-29 07:49:23 <wumpus> BlueMatt: yes
265 2014-10-29 07:49:34 <BlueMatt> hmm, yea, I'm not convinced that is common enough to care beyond -externalip
266 2014-10-29 07:49:49 <wumpus> as long as -externalip keeps working I'm fine with it
267 2014-10-29 07:49:54 <BlueMatt> ie never try to find our address, just re-send what each peer sends us, and then provide -externalip to force override that
268 2014-10-29 07:49:55 <gmaxwell> BlueMatt: in any case, all you're really asking for there is -discover off by default... which it will do if proxy is set.
269 2014-10-29 07:50:07 <BlueMatt> and remove all the code, I find it overkill
270 2014-10-29 07:50:13 <BlueMatt> but maybe just off by default is ok
271 2014-10-29 07:50:31 <gmaxwell> BlueMatt: also doesn't really gracefully handle cross protocol usage (peer only gives you the address they see you as)
272 2014-10-29 07:51:26 <BlueMatt> gmaxwell: indeed, though I'm too tired to try to model if thats a problem?
273 2014-10-29 07:51:41 <gmaxwell> e.g. you'll never be able to announce your IPv6 address unless you can first make it out to a v6 peer that gives you the right one. I think thats bad.
274 2014-10-29 07:52:26 <BlueMatt> yes, I see that, but I'm not sure how bad it is
275 2014-10-29 07:53:06 <BlueMatt> its not ideal, but...meh
276 2014-10-29 07:57:07 <gmaxwell> BlueMatt: yea, no great things there. In any case, we don't have to settle this all at once. Just getting rid of the horrifying external IP services will be a big improvement.
277 2014-10-29 07:57:29 <wumpus> gmaxwell: yes!
278 2014-10-29 07:57:40 <wumpus> and having a default that works in the most common setups
279 2014-10-29 07:58:04 <BlueMatt> gmaxwell: this is true, I just did not find it easy to audit as I hadnt looked at this code in a while and all of the interactions and ways we get ips are...complicated
280 2014-10-29 07:58:18 <BlueMatt> how we expect users to get this (eg if they're trying to be anonymous over tor!) I have no idea
281 2014-10-29 07:58:34 <BlueMatt> (and the help for -discover is useless)
282 2014-10-29 07:58:41 <gmaxwell> BlueMatt: yea... well, my stupid patch simplfied it some, e.g. reducing it from four places we advertise to two.
283 2014-10-29 07:58:41 <wumpus> this doesn't all have to be done in one pull!
284 2014-10-29 07:58:56 <wumpus> simplification is a worthy goal of course
285 2014-10-29 07:59:42 <wumpus> we have a document about running behind TOR, please keep it up to date
286 2014-10-29 08:01:02 <gmaxwell> It's important that the tor behavior be safe even if you don't read the docs too though. :)
287 2014-10-29 08:01:39 <wumpus> it's just that there are so many possible setups
288 2014-10-29 08:01:54 <wumpus> a document that explains one that *works* and is safe is very useful
289 2014-10-29 08:05:34 <wumpus> if you don't read the docs you don't really know how to configure bitcoind for tor, or should it auto-detect tor running as well?
290 2014-10-29 08:06:51 <gmaxwell> wumpus: what people do is they read the help output and then configure it.
291 2014-10-29 08:08:04 <wumpus> hm, in that case: the only place where we mention tor in the help output is "-onion=<ip:port>       Use separate SOCKS5 proxy to reach peers via Tor hidden services (default: -proxy)" ... and that's probably the wrong option to use
292 2014-10-29 08:08:41 <gmaxwell> yea, hm. well we used to call that -tor and it very much was getting misused, I renamed it to onion.
293 2014-10-29 08:08:45 <wumpus> (-onion is for advanced use-cases where you want to be able to connect to ipv4 and ipv6 directly but handle .onions through tor)
294 2014-10-29 08:08:50 <wumpus> right
295 2014-10-29 08:08:54 <gmaxwell> No _reports_ of misuse since the rename... but...
296 2014-10-29 08:09:24 <gmaxwell> perhaps best to move the word tor to the proxy line.
297 2014-10-29 08:10:23 <wumpus> maybe "Connect through SOCKS5 proxy (for example Tor)" ?
298 2014-10-29 08:10:55 <wumpus> we should have just had -tor=<ip:port> mean the right thing, but it's too late for that
299 2014-10-29 08:12:34 <BlueMatt> ugh, now I cant reproduce :(
300 2014-10-29 08:13:04 <BlueMatt> I had bitcoind in some strange state where it /had/ the blocks (and thus would not process them in -loadblock), but hadnt actually connected them, and so was going out and re-downloading them from the network
301 2014-10-29 08:14:14 <gmaxwell> wumpus: yea, oops. Well I think we can eventually repurpose -tor.  Perhaps 0.10 we should change it to reject -tor and tell you to use -proxy. Then in a later version we can make it just do the same as -proxy.
302 2014-10-29 08:14:35 <wumpus> gmaxwell: ack
303 2014-10-29 08:15:27 <gmaxwell> BlueMatt: wonder if those "random IPs" are uninitilized memory...
304 2014-10-29 08:15:54 <BlueMatt> heh, probably
305 2014-10-29 08:16:17 <wumpus> maybe try them as private keys ;)
306 2014-10-29 08:16:53 <iwilcox> wumpus: Does it hurt to use -onion where you could use -proxy?
307 2014-10-29 08:17:14 <wumpus> iwilcox: it does something completely different
308 2014-10-29 08:17:46 <iwilcox> Does that document explain in detail what each option really does?
309 2014-10-29 08:17:50 <wumpus> iwilcox: -proxy sends *all* outgoing communications over the proxy
310 2014-10-29 08:18:03 <gmaxwell> Yes, but _no one_ reads it.
311 2014-10-29 08:18:22 <wumpus> iwilcox: -onion (as the description above says) sets a proxy to use for .onion addresses
312 2014-10-29 08:18:24 <iwilcox> I hope -help points to it.
313 2014-10-29 08:18:47 <wumpus> that's why it says *separate* SOCKS5 proxy
314 2014-10-29 08:18:51 <gmaxwell> it doesn't. It could.
315 2014-10-29 08:19:11 <iwilcox> But what if you said -onion=... -onlynet=tor?  Perverse but achieving the same?
316 2014-10-29 08:19:24 <gmaxwell> so far I don't think I've seen anyone report misuse of -onion yet... but people don't notice.
317 2014-10-29 08:19:32 <gmaxwell> iwilcox: yes, that should be safe.
318 2014-10-29 08:19:36 <gmaxwell> I think. :)
319 2014-10-29 08:19:43 <gmaxwell> (well, it won't disable dnsseed)
320 2014-10-29 08:19:44 <wumpus> iwilcox: there are some parameter interactions for -proxy that you'll miss (such as -upnp=0, -listen=0)
321 2014-10-29 08:19:51 <BlueMatt> ok, I'm going to bed...if someone wants to track down a bug (should be obvious now, I'd think): if you start downloading from the network in headers-first, then kill it mid-download, then go back and try a -loadblock, it will load all the blocks but wont connect any of them!
322 2014-10-29 08:19:51 <wumpus> indeed, and dnsseed
323 2014-10-29 08:19:59 <BlueMatt> if no one gets to it by morning I'll do that tomorrow
324 2014-10-29 08:20:04 <BlueMatt> sipa: ^
325 2014-10-29 08:20:36 <sipa> BlueMatt: ack
326 2014-10-29 08:20:39 <iwilcox> wumpus: OK, well, I use -proxy anyway but I'm sure I've suggested onlynet+onion to people in the past; I'll just refrain from helping until I'm comfortable I understand the implications.
327 2014-10-29 08:20:53 <wumpus> iwilcox: oh.. no, it's not the same; -onlynet=tor connects *only* to hidden services
328 2014-10-29 08:20:57 <sipa> not sure if i'll get tot it today; i'm doing libsecp256k1 stuff currently
329 2014-10-29 08:21:23 <wumpus> iwilcox: it's safe, but more restrictive than normal proxy, which also connects to ipv4 nodes through exit nodes
330 2014-10-29 08:21:43 <wumpus> iwilcox: in any case: advice people to use proxy *unless* they know in detail the configuration
331 2014-10-29 08:21:59 <gmaxwell> darnit. a "See tor.md" in the -onion description ends up making it mention tor. :P
332 2014-10-29 08:22:20 <sipa> rename it
333 2014-10-29 08:22:44 <wumpus> tor.md is a good name
334 2014-10-29 08:22:59 <iwilcox> gmaxwell: Why is advertising it undesirable?
335 2014-10-29 08:23:40 <wumpus> iwilcox: even if you want that behavior (connect only to hidden services), I'd advice to use -proxy= and -onion= and -onlynet=tor just to be sure
336 2014-10-29 08:24:02 <wumpus> (oh, you can leave out the -onion= in that case)
337 2014-10-29 08:25:08 <gmaxwell> wumpus: thats the name of the file. :) doc/tor.md
338 2014-10-29 08:25:26 <wumpus> gmaxwell: I know :) but sipa said to rename it
339 2014-10-29 08:25:30 <gmaxwell> iwilcox: missing the context.
340 2014-10-29 08:26:00 <gmaxwell> wumpus: symlink it to onion.md for reference in that line. :P
341 2014-10-29 08:27:45 <wumpus> hah
342 2014-10-29 08:27:55 <wumpus> poor OSes without symlinks
343 2014-10-29 08:34:14 <SomeoneWeird> We don't care about those
344 2014-10-29 08:35:27 <phantomcircuit> ^
345 2014-10-29 08:37:13 <gmaxwell> heh, we do when the goal is to help more hapless users. :) it was a silly suggestion in any case.
346 2014-10-29 11:57:37 <aschildbach> Q about building bitcoin-qt on Ubuntu 14.10: For some reason, Boost is not detected any more. E.g.
347 2014-10-29 11:57:50 <aschildbach> checking whether the Boost::System library is available... no
348 2014-10-29 11:58:17 <aschildbach> But libboost-system is installed:
349 2014-10-29 11:58:19 <aschildbach> ii  libboost-system1.55.0:amd64                                     1.55.0+dfsg-1ubuntu3                 amd64
350 2014-10-29 11:59:07 <wumpus> what are you building, tag 0.9.3 or master?
351 2014-10-29 11:59:33 <wumpus> can you pastebin the config.log output?
352 2014-10-29 12:00:07 <wumpus> btw; libboost-system is not enough, you need the -dev packages to be able to compile code against it
353 2014-10-29 12:00:13 <wumpus> libboost-all-dev IIRC
354 2014-10-29 12:06:08 <aschildbach> wumpus: master, just pulled
355 2014-10-29 12:06:23 <aschildbach> ah ok let me check for the -dev packages
356 2014-10-29 12:11:31 <aschildbach> checking for exit in -lboost_filesystem... no
357 2014-10-29 12:11:31 <aschildbach> configure: error: Could not link against boost_filesystem !
358 2014-10-29 12:11:31 <aschildbach> wumpus: Ok installed the two boost -dev packages it complained about, but I now get this output:
359 2014-10-29 12:11:52 <wumpus> libboost-dev-all should get them all
360 2014-10-29 12:13:02 <aschildbach> Ah the script seems to be missing a check for Boost Filesystem
361 2014-10-29 12:13:11 <aschildbach> It just assumes its present and then fails
362 2014-10-29 12:13:32 <wumpus> you can try to do it one by one, but you'll spend a lot more time figuring out what exactly you need
363 2014-10-29 12:18:17 <aschildbach> wumpus: I don't care, as long as I can find out the proper dependencies
364 2014-10-29 12:19:10 <wumpus> ok
365 2014-10-29 12:21:26 <aschildbach> wumpus: The check for boost-filesystem is the only that's missing from the script
366 2014-10-29 12:22:08 <aschildbach> I'll try and submit a pull request for that
367 2014-10-29 12:23:35 <michagogo> Some people don't like installin libboost-all-dev because it's huge
368 2014-10-29 12:23:45 <michagogo> installing*
369 2014-10-29 12:25:05 <michagogo> at one point I was asking for that to be installed on a shared box where I had a shell, they made me tell them exactly which modules I needed
370 2014-10-29 12:25:05 <michagogo> hmm, actually
371 2014-10-29 12:25:19 <michagogo> That may be a way I can save some space on my almost-full VPS
372 2014-10-29 12:25:20 <wumpus> well, feel free to update doc/build-unix.md with exactly what modules you need
373 2014-10-29 12:25:34 <michagogo> (though it means figuring out exactly which those are again...)
374 2014-10-29 12:25:50 <wumpus> I guess the current assumption is that if you're going to run a full node, disk space shouldn't be much of an issue
375 2014-10-29 12:25:51 <michagogo> Well, I have something like 50G or 60G there
376 2014-10-29 12:25:53 <michagogo> Would be fine, if I weren't also seeding bootstrap :P
377 2014-10-29 12:26:13 <aschildbach> It's always important to know at least what the dependencies are. After all, you need to audit them for backdoors etc.
378 2014-10-29 12:26:21 <aschildbach> So its not only about cheap resources.
379 2014-10-29 12:27:05 <wumpus> plus it is less robust, the names of the individual packages change between ubuntu/debian releases
380 2014-10-29 12:29:44 <michagogo> Do they?
381 2014-10-29 12:29:44 <michagogo> It's not always `libboost-<name>-dev`?
382 2014-10-29 12:29:44 <wumpus> chrono*
383 2014-10-29 12:29:44 <wumpus> well some are more vague, like chrone
384 2014-10-29 12:29:44 <wumpus> yes I noticed that at some point when I tried to list them
385 2014-10-29 12:29:45 <wumpus> *depends on the boost package version, I suppose, not the distro version
386 2014-10-29 12:30:17 <wumpus> aschildbach: btw if you want the exact dependencies for that purpose it's better to look in depends/packages/boost.mk
387 2014-10-29 12:30:30 <wumpus> $(package)_config_libraries=chrono,filesystem,program_options,system,thread,test
388 2014-10-29 12:32:12 <aschildbach> wumpus: yeah that's what I just found out too. The only issue I have is that the non-presence of boost-filesystem isn't properly handled by the script.
389 2014-10-29 12:32:12 <wumpus> it's better to build from source anyhow if you want to audit the source
390 2014-10-29 12:32:43 <aschildbach> wumpus: Sure, but still you need to audit
391 2014-10-29 12:32:43 <wumpus> the depends system makes it very easy to build all depends from source
392 2014-10-29 12:33:46 <michagogo> Does it build all of boost, or just those?
393 2014-10-29 12:33:46 <michagogo> Oh, nvm
394 2014-10-29 12:33:46 <wumpus> aschildbach: that doesn't have anything to do with debian packages tho
395 2014-10-29 12:34:11 <aschildbach> wumpus: No, I'm talking about the build script that comes with bitcoin-qt
396 2014-10-29 12:34:11 <michagogo> (I assume that that line from the boost.mk means that that's what gets built?)
397 2014-10-29 12:34:35 <wumpus> aschildbach: I thought you got "configure: error: Could not link against boost_filesystem !", that looks clear enough to me
398 2014-10-29 12:34:58 <aschildbach> Try it -- remove the boost-filesystem from your system and try building
399 2014-10-29 12:34:58 <michagogo> aschildbach: less clear than configure: error: Could not find a version of the boost_program_options library!
400 2014-10-29 12:34:58 <michagogo> er
401 2014-10-29 12:34:58 <wumpus> michagogo: yes
402 2014-10-29 12:35:11 <michagogo> s/aschildbach/wumpus/
403 2014-10-29 12:35:39 <aschildbach> wumpus: No its not clear because the other libs get a clear message like "libboost-system not found" or something similar
404 2014-10-29 12:35:40 <wumpus> no time for that, but feel free to contribute improvements
405 2014-10-29 12:35:55 <aschildbach> And that message reads like Library is there, but could not link against it
406 2014-10-29 12:35:55 <michagogo> aschildbach: to be precise, configure: error: Could not find a version of the boost_program_options library!
407 2014-10-29 12:36:05 <michagogo> (I removed libboost-all-dev, then autoremoved, and then reinstalled the ones that I knew from memory are involved, and then reconfigured)
408 2014-10-29 12:36:16 <michagogo> (and then found I had missed some :) )
409 2014-10-29 12:36:23 <michagogo> also, ,,(lucky xkcd paranthetical emoticons)
410 2014-10-29 12:36:24 <gribble> http://xkcd.com/541/
411 2014-10-29 12:37:46 <aschildbach> Here is how the other libs are checked:
412 2014-10-29 12:37:47 <aschildbach> checking whether the Boost::System library is available... yes/no
413 2014-10-29 12:37:57 <aschildbach> That's a clear message
414 2014-10-29 15:33:24 <kanzure> i wonder if there's something other than a version number that can be used to indicate consensus library er, version.. like a feature vector or something.
415 2014-10-29 20:01:51 <coryfields_> michagogo: yes, those are the only libs necessary
416 2014-10-29 20:01:56 <coryfields_> re: boost deps
417 2014-10-29 20:47:49 <BlueMatt> ouchhh...headers-first broke restart after kill -9 during sync
418 2014-10-29 20:47:53 <BlueMatt> like...always
419 2014-10-29 20:55:29 <gwillen> BlueMatt: :-(
420 2014-10-29 21:01:37 <sipa> BlueMatt: meh, should be easy to fix
421 2014-10-29 21:01:44 <BlueMatt> sipa: should be, I'm looking
422 2014-10-29 21:01:52 <sipa> (but should be fixed!)
423 2014-10-29 21:02:26 <BlueMatt> naaaaaa
424 2014-10-29 22:44:54 <BlueMatt> ummm...have we /ever/ had a consistent db guarantee on kill -9? I dont think headers-first broke this, just made it obvious since it drastically increased times between writing block indexes and coins
425 2014-10-29 22:45:22 <BlueMatt> I think coins can get out-of-sync from headers, and then you get an assert on startup, but I think you could have always seen that.......
426 2014-10-29 22:45:32 <BlueMatt> s/headers/index/
427 2014-10-29 22:47:58 <gmaxwell> BlueMatt: during initial sync no.
428 2014-10-29 22:48:14 <gmaxwell> because we refrain from syncing block files before advancing the databases during initial sync.
429 2014-10-29 22:48:23 <gmaxwell> you could change that for testing, however.
430 2014-10-29 22:48:41 <BlueMatt> gmaxwell: no, I think it is always the case that they could get out of sync and blow up
431 2014-10-29 22:48:55 <BlueMatt> just less likely during !ibd
432 2014-10-29 22:48:59 <gmaxwell> hm. I think thats not the intention. It might be the case but it would be unfortunate if so.
433 2014-10-29 22:49:28 <BlueMatt> well, we have two databases, and they're written separately, and if they're out of sync, we assert on startup
434 2014-10-29 22:49:34 <gmaxwell> We should be appending the block, fsyncing, then updating the block database.. then updating the coins. The result should be that if things blow up it recovers.
435 2014-10-29 22:50:13 <gmaxwell> E.g. out of sync in one direction is fine, just means some redundant work. Out of sync in the other is not fine.
436 2014-10-29 22:50:18 <BlueMatt> yes, I think thats ~what we're doing, but if it gets killed before updating the coins, then we assert during startup and blow up
437 2014-10-29 22:50:22 <BlueMatt> it should be able to recover there
438 2014-10-29 22:53:50 <Luke-Jr> [22:44:54] <BlueMatt> ummm…have we /ever/ had a consistent db guarantee on kill -9? I dont think headers-first broke this, just made it obvious since it drastically increased times between writing block indexes and coins <-- yes, it's always worked consistently..
439 2014-10-29 22:54:08 <Luke-Jr> 9 times out of 10, I have to resort to kill -9 because my old version doesn't shutdown completely
440 2014-10-29 22:54:13 <BlueMatt> Luke-Jr: I didnt mean in practice, I meant if killed at /exactly/ the right place
441 2014-10-29 22:54:21 <Luke-Jr> ah
442 2014-10-29 22:54:59 <gmaxwell> hm. may just be too agressive checking.... I've wanted a test harness before that took disk write log of a running node and could restart it at random points in the write sequence to see that it recovers at every point. I was unable to find any good pre-existing tools for that; ... I don't know how you could create a durable database without such a test shim, so perhaps its the case that no OSS database software is actually durable.
443 2014-10-29 22:55:38 <BlueMatt> gmaxwell: they're durable up to their programmer's ability to reason about issues and production crash-testing
444 2014-10-29 22:55:38 <Luke-Jr> gmaxwell: worse, you'd have to randomise the order of all writes between fsyncs..
445 2014-10-29 22:55:46 <Luke-Jr> afaik that's not guaranteed
446 2014-10-29 22:56:17 <kruug> Hello, I'm looking to talk to someone with knowledge on scrypt/PoS coins.  I know this is off-limits here, so I'm looking to private message with them.  ##altcoin-dev is pretty dead, so I turn here.
447 2014-10-29 22:56:21 <gmaxwell> BlueMatt: yea, well "all untested software is broken"
448 2014-10-29 22:56:34 <BlueMatt> gmaxwell: including bitcoin! :)
449 2014-10-29 22:56:40 <Luke-Jr> (at least, I'd *hope* Linux is smart enough to plan writes to the same area of magnetic disks together..)
450 2014-10-29 22:56:55 <BlueMatt> Luke-Jr: last I heard it was very good at thait
451 2014-10-29 22:56:57 <Luke-Jr> kruug: if people were interested in talking about it, they'd be in ##altcoin-dev
452 2014-10-29 22:57:30 <kruug> Luke-Jr: thank you, I'll take that under consideration.
453 2014-10-29 22:58:28 <gmaxwell> Luke-Jr: sure, but a first step would be just every write. you could even realistically try every possible in-order recovery point... once you throw in reordering you can only do randomized tests.
454 2014-10-29 22:59:54 <phantomcircuit> gmaxwell, all databases software is only durable within certain constraints
455 2014-10-29 23:00:02 <phantomcircuit> most of which do not exist in the real world
456 2014-10-29 23:00:40 <phantomcircuit> fortunately we can work around that fairly easily for nearly everything
457 2014-10-29 23:00:47 <phantomcircuit> (well relatively easily)
458 2014-10-29 23:01:02 <gmaxwell> phantomcircuit: just simply ones like fail at any point in sequence. Sure once you start corrupting data on the way out, all assumptions go out the window.
459 2014-10-29 23:01:28 <phantomcircuit> in practice hdd's do weird random things at a fairly high rate
460 2014-10-29 23:02:26 <gmaxwell> ::nods:: but software often fails even without weird random things.
461 2014-10-29 23:03:14 <phantomcircuit> true, but the odds of hitting those weird random edge cases isn't necessarily higher than a hdd failure
462 2014-10-29 23:37:18 <coryfields_> mm, working on getting the pull-tester to run the rpc tests...
463 2014-10-29 23:37:53 <coryfields_> win32 chokes because bitcoin-cli outputs with CRLF at the end, throwing the scripts for a loop
464 2014-10-29 23:38:31 <coryfields_> getting rid of the \n in bitcoin-cli.cpp fixes the problem. Anyone have a suggestion for a more realistic fix/hack/workaround?
465 2014-10-29 23:38:45 <gmaxwell> just make the script more tolerant?
466 2014-10-29 23:39:25 <coryfields_> gmaxwell: that means a lot of sed's. I started that way, but apparently missed some
467 2014-10-29 23:39:46 <coryfields_> maybe i'm missing some dead-easy way of telling bash to ignore em?