1 2014-04-28 00:00:06 <jgarzik> sipa, yeah, I think that's pointless
  2 2014-04-28 00:00:20 <sipa> ndak: it means you built from a non-release source
  3 2014-04-28 00:00:36 <jgarzik> sipa, anybody who's using getwork is using ancient software unlikely to survive such a change
  4 2014-04-28 00:00:38 <sipa> ndak: so it warns you that the code may be insufficiently tested
  5 2014-04-28 00:00:44 <jgarzik> sipa, so it is functionally equivalent to deleting it
  6 2014-04-28 00:00:52 <sipa> jgarzik: i doubt that, actually
  7 2014-04-28 00:01:38 <jgarzik> sipa, a lot of miners based off of cpuminer required those fields, and failed without them (== bfgminer, cgminer).  It did not _use_ those field, but nonetheless triggered a failure if they were missing.
  8 2014-04-28 00:02:11 <sipa> oh
  9 2014-04-28 00:02:24 <jgarzik> Hopefully that was fixed long ago in modern miner software, but the point remains:  ancient software unlikely to change is the most likely to die
 10 2014-04-28 00:13:46 <gmaxwell> if jeff supports removing getwork then I think we're ready to og
 11 2014-04-28 01:04:53 <warren> anyone know if bitcoind will gracefully handle debug.log rotating with logrotate?
 12 2014-04-28 01:05:08 <sipa> yes
 13 2014-04-28 01:05:26 <warren> michagogo|cloud: I ended up buying two useless USB sticks to mine testnet when needed
 14 2014-04-28 01:43:45 <HaltingState> jcorgan, his code is fine; i try to keep it up to date
 15 2014-04-28 01:45:03 <HaltingState> jcorgan, i tested sipa 's library and fuzzed it and the changes that will be made in future wont change binary; mostly just bug fixes, updates, but its robust for inputs and matches openssl for all known inputs; i tried 10^6 random inputs for each function and is fine
 16 2014-04-28 01:45:43 <HaltingState> sipa, i think your library is at the right level
 17 2014-04-28 01:46:34 <HaltingState> the gmp dependency has caused some issues on OSX however
 18 2014-04-28 02:16:45 <HaltingState> sipa, if you make library more complex or change interface, can cause problems in terms of success/failure conditions varying between libraries ; its already very clean, but some things return error code that should not be allowed to fail
 19 2014-04-28 02:18:42 <HaltingState> you input 32 bytes and you get private key and its just base raised to that power; but if you add check and failure condition, suddenly some inputs fail and you have to handle condition, it affects deterministic wallet implementations that are generating sequences of keys through iteration
 20 2014-04-28 03:12:33 <jcorgan> HaltingState: As a learning exercise for Go, I'm replicating my Python BIP32 library and generator.  It will be nice to base that off a fast EC library rather than the pure Python stuff it uses now.
 21 2014-04-28 03:46:26 <HaltingState> jcorgan, i used python alot; like go now
 22 2014-04-28 03:49:36 <jcorgan> i'm finding the concurrency really attractive; similar to erlang without the functional religion.  haven't quite grokked the choice of no-shared libraries, but i'm sure i'll understand eventually.
 23 2014-04-28 04:04:16 <uiop> jcorgan: i'd bet it's less a choice than a shared-libraries-and-language-abis-are-hard-lets-go-shopping thing
 24 2014-04-28 04:08:12 <jcorgan> sure, one step at a time
 25 2014-04-28 05:00:27 <jcorgan> question--incoming Tor hidden nodes always connect as 127.0.0.1, yet bitcoind specifically exempts this address from ban disconnects since it thinks it's local.  Is this intended?
 26 2014-04-28 05:00:41 <wumpus> yes
 27 2014-04-28 05:01:10 <wumpus> otherwise it would ban all nodes connecting from tor
 28 2014-04-28 05:02:11 <jcorgan> ah, i thought once the threshold was exceeded on a connection, it would just disconnect, i didn't realize it also prevented future connections
 29 2014-04-28 05:02:25 <gmaxwell> We have no way to distinguish tor peers from each other, to make it costly to attack that way. A way to do that is desirable, and I've given it some thought but there isn't a nice compact way to do it.
 30 2014-04-28 05:02:30 <gmaxwell> jcorgan: yes for 24 hours.
 31 2014-04-28 05:02:49 <gmaxwell> disconnecting in this case might still be good— won't stop an attack but might clear off some brokenness.
 32 2014-04-28 05:03:12 <wumpus> right, I was not aware that disconnection was also prevented, just banning
 33 2014-04-28 05:03:46 <gmaxwell> yea, disconnection is prevented— which is good when the peer is— say— a local p2pool node, you don't want it getting disconnected since it might not get a connection slot on reconnect.
 34 2014-04-28 05:04:19 <gmaxwell> jcorgan: here is an example where I was working on grinding out some ideas that would help for making it expensive to attack a HS listening node: https://bitcointalk.org/index.php?topic=310323.0
 35 2014-04-28 05:06:18 <wumpus> as there is no (portable) way to distinguish localhost connections, the way to avoid that would be to add a local-bound 'whitelist port' to which local p2pool nodes and such would connect, but not TOR
 36 2014-04-28 05:06:29 <michagogo> cloud|Well, the node knows whether a connection is tor or actual-local, doesn't it?
 37 2014-04-28 05:06:38 <wumpus> no
 38 2014-04-28 05:06:48 <michagogo> cloud|Huh?
 39 2014-04-28 05:06:54 <gmaxwell> wumpus: yup thats precisely what I think we should do (you could also allow access to that port to other 'trusted peers')
 40 2014-04-28 05:07:02 <wumpus> tor just connects to localhost like any other local connection
 41 2014-04-28 05:07:05 <gmaxwell> michagogo|cloud: it's _is_ a local connection at the tcp level.
 42 2014-04-28 05:07:05 <michagogo> cloud|What's addrlocal?
 43 2014-04-28 05:07:16 <gmaxwell> 127.0.0.1 of course. it's a socket out of the tor process.
 44 2014-04-28 05:07:36 <michagogo> cloud|gmaxwell: ...I see my onion address in getpeerinfo for each tor peer
 45 2014-04-28 05:08:04 <michagogo> cloud|Why can't it use that?
 46 2014-04-28 05:08:23 <jcorgan> was just digging through my logs, i see a 'not banning local node' every 2-3 hours
 47 2014-04-28 05:08:30 <gmaxwell> michagogo|cloud: because most tor peers are not sending anything there (they aren't themselves HS listeners) and they can send anything they want there.
 48 2014-04-28 05:08:56 <gmaxwell> which is precisely what an attacker wouldn't do, so it's pointless to ban on it.
 49 2014-04-28 05:09:00 <gmaxwell> er would do
 50 2014-04-28 05:09:20 <michagogo> cloud|gmaxwell: yes, but as someone just said it's useful for removing brokenness
 51 2014-04-28 05:09:27 <michagogo> cloud|Even if it won't stop an attack
 52 2014-04-28 05:09:47 <michagogo> cloud|(It was you, actually)
 53 2014-04-28 05:09:47 <wumpus> tor works like a reverse proxy, it accepts connections on the tor network and redirects them to a ip/port pair, distinguishing them would require specific integration with the tor software (and it's not like they export a library....)
 54 2014-04-28 05:09:59 <gmaxwell> michagogo|cloud: yes, but hanging up is probably good enough for that.
 55 2014-04-28 05:10:12 <gmaxwell> wumpus: tor has no idea where an inbound connection came from.
 56 2014-04-28 05:10:17 <michagogo> cloud|gmaxwell: uh?
 57 2014-04-28 05:10:18 <gmaxwell> so there is nothing to integrate with there.
 58 2014-04-28 05:10:25 <michagogo> cloud|That's what I'm talking about
 59 2014-04-28 05:10:28 <wumpus> gmaxwell: but we would know that it came from tor
 60 2014-04-28 05:10:36 <gmaxwell> oh I see what you're saying.
 61 2014-04-28 05:10:38 <jcorgan> i'd be happy with a hangup only
 62 2014-04-28 05:10:47 <gmaxwell> well I think that the seperate port thing is probably the best general tool.
 63 2014-04-28 05:10:53 <wumpus> agreed
 64 2014-04-28 05:10:54 <gmaxwell> and doesn't require adding functionality to tor.
 65 2014-04-28 05:11:35 <michagogo> cloud|Can't we check what the process is on the other side of the socket?
 66 2014-04-28 05:11:38 <wumpus> no
 67 2014-04-28 05:11:44 <michagogo> cloud|Or does that require root?
 68 2014-04-28 05:11:52 <wumpus> it's OS specific
 69 2014-04-28 05:12:23 <wumpus> on linux you can see what process is on the other side of a localhost socket, if it belongs to the same user AFAIK
 70 2014-04-28 05:12:34 <wumpus> but that would get very messy and really isn't an option
 71 2014-04-28 05:13:23 <michagogo> cloud|Fair enough.
 72 2014-04-28 05:13:41 <gmaxwell> besides, there are more cases where you don't want hangupgs happening.
 73 2014-04-28 05:13:58 <wumpus> (there are also quite a few race conditions and security considerations in doing it, as I remember)
 74 2014-04-28 05:14:03 <gmaxwell> listening on a second port is easy, and you can tell which one the connections came in on.
 75 2014-04-28 05:14:11 <gmaxwell> oh the days of ident.
 76 2014-04-28 05:14:15 <wumpus> right, it's the best solution
 77 2014-04-28 05:14:36 <jcorgan> gmaxwell: and finger
 78 2014-04-28 05:14:48 <michagogo> cloud|Maybe make that localhost-only, or like rpcallowip?
 79 2014-04-28 05:15:20 <wumpus> localhost only would be obvious
 80 2014-04-28 05:15:31 <michagogo> cloud|gmaxwell: the days of ident? That's still around...
 81 2014-04-28 05:15:46 <gmaxwell> michagogo|cloud: yes but it used to matter a lot more when multiuser shells were common.
 82 2014-04-28 05:15:48 <wumpus> if you really need to expose it to the network, you could set up some redirect
 83 2014-04-28 05:16:05 <gmaxwell> wumpus: I think the bind should be configurable and should just default to localhost.
 84 2014-04-28 05:16:17 <gmaxwell> (we already have configurable binds for other stuff)
 85 2014-04-28 05:16:17 <michagogo> cloud|wumpus: why not just add a -wlallowip=
 86 2014-04-28 05:16:22 <wumpus> gmaxwell: but if you allow a bind, you'd need something like rpcallowip, and that rabbit hole goes deep
 87 2014-04-28 05:16:38 <wumpus> before you know it you're implementing a firewall
 88 2014-04-28 05:16:53 <wumpus> I really believe such functionality belongs outside of bitcoind
 89 2014-04-28 05:17:22 <michagogo> cloud|So are you against having rpcallowip?
 90 2014-04-28 05:17:29 <wumpus> yes
 91 2014-04-28 05:17:36 <wumpus> RPC should have been localhost-only as well
 92 2014-04-28 05:18:22 <wumpus> (also wouldn't have needed rpcssl in that cast)
 93 2014-04-28 05:18:26 <wumpus> case*
 94 2014-04-28 05:19:29 <michagogo> cloud|IMHO, if we add a whitelist port, we should either drop rpcallowip or have a wlallowip
 95 2014-04-28 05:19:54 <gmaxwell> wumpus: we still don't need rpcssl.  Running an extra process to just proxy connections is ugly and failure prone, your bind might be connecting you to vpns and filtered by firewalls, etc. I can't think of any other daemon which is localhost only off the top of my head. Even ntp and dns administration, also databases, all work fine across the network if you enable them.
 96 2014-04-28 05:20:07 <gmaxwell> michagogo|cloud: p2p whitelist is distinct from rpc in any case.
 97 2014-04-28 05:20:34 <wumpus> gmaxwell: bitcoin is  a*lot* more senstive though
 98 2014-04-28 05:20:44 <wumpus> gmaxwell: at least, if built with wallet
 99 2014-04-28 05:21:29 <gmaxwell> There are systems handling more dollars in value than the whole bitcoin network running in software like oracle which does allow network administration, if enabled. :)
100 2014-04-28 05:21:45 <michagogo> cloud|gmaxwell: of course it's distinct, but I do think we should have a wlallowip
101 2014-04-28 05:22:34 <gmaxwell> michagogo|cloud: our allowip functionality is really cruddy. I mean ... wildcard and not cidr? 0_o.   Though I don't disagree with you. I just thought you were saying they should be the same setting and I strongly disagree.
102 2014-04-28 05:22:57 <michagogo> cloud|Either we do fooallowip or we don't, IMO -- unless you want to remove rpcallowip...
103 2014-04-28 05:23:07 <michagogo> cloud|gmaxwell: ah, yeah, of course not
104 2014-04-28 05:23:23 <michagogo> cloud|Heh, we don't do cidr? :-/
105 2014-04-28 05:23:39 <michagogo> cloud|
ACTION wishes he could C++

106 2014-04-28 05:24:50 <gmaxwell> michagogo|cloud: nope, no cidr, see WildcardMatch in util.cpp. It's boggling.
107 2014-04-28 05:25:04 <wumpus> gmaxwell: that's what I meant; you'll start implementing 'firewall' functionality
108 2014-04-28 05:25:49 <wumpus> of course a wildcard match is not good enough
109 2014-04-28 05:25:53 <gmaxwell> wumpus: sure, and ntp, bind, postgres, etc. do. At least basic functionality. Though many things only implement a controllable bind.
110 2014-04-28 05:26:12 <gmaxwell> The reason you often want it in the application is when it's more than just port access you want to control.
111 2014-04-28 05:26:21 <gmaxwell> e.g. only these features, only these users.
112 2014-04-28 05:26:21 <wumpus> gmaxwell: that they do it doesn't mean that we have to; it grates me that there is so much duplication of functionality
113 2014-04-28 05:26:57 <wumpus> many different implementations of the same thing, usually an afterthought, some are probably buggy in one way or another
114 2014-04-28 05:27:25 <gmaxwell> There is usually a right level of duplication that makes things loosely coupled, so that you're not having to learn how everything in the universe works just to setup a simple thing.
115 2014-04-28 05:28:51 <wumpus> hehe
116 2014-04-28 05:30:25 <wumpus> it starts harmless but people want more and more duplication, until you have to integrate the entire universe
117 2014-04-28 05:31:04 <wumpus> and then you are at the same point, it's no longer a simple thing but an entire universe :p
118 2014-04-28 05:31:47 <wumpus> in any case I'd be ok with configurable bind functionality
119 2014-04-28 05:32:01 <wumpus> but adding all kinds of ACLs seems like a step too far
120 2014-04-28 05:32:08 <gmaxwell> Sure and navigating the tensions and tradeoffs of conflicting requirements is at the core of engineering. :) I think it's okay, we can just work these things out.
121 2014-04-28 05:32:25 <gmaxwell> Sounds okay to me.
122 2014-04-28 05:32:46 <gmaxwell> bind is enough that it makes it easy to use external acl functionality (e.g. firewalls) without excessive complexity.
123 2014-04-28 05:33:00 <wumpus> right
124 2014-04-28 05:34:22 <gmaxwell> the other place where internal 'acl' functionality comes into play is with things like banning, where the application itself is making these decisions, — we wouldn't want to be making callouts to firewalls. :)
125 2014-04-28 05:34:53 <gmaxwell> but at least today we don't do too much there.
126 2014-04-28 05:35:36 <wumpus> hey, callouts to firewalls would make sense, if you really want to blackhole an IP instead of just banning it *ducks*
127 2014-04-28 05:35:56 <wumpus> sure if the decision is with the application it's really something unique to that and it makes sense
128 2014-04-28 05:36:10 <michagogo> cloud|wumpus: so should we drop rpcallowip?
129 2014-04-28 05:36:21 <wumpus> michagogo|cloud: that wouldn't be nice would it
130 2014-04-28 05:36:48 <wumpus> michagogo|cloud: I'd love to drop it, but once you give something people hate it if you take it back :p
131 2014-04-28 05:37:24 <michagogo> cloud|If anything, I'd think wlallowip would be more useful/good to have than rpcallowip
132 2014-04-28 05:37:54 <wumpus> I think we need a port to send allowip requests and then add a allowipallowip
133 2014-04-28 05:38:18 <michagogo> cloud|Since if you need to make rpc calls from a remote machine, you should know how to set up a firewall and access rules
134 2014-04-28 05:38:26 <wumpus> or a turing complete allowip language
135 2014-04-28 05:38:40 <michagogo> cloud|But a wlallowip can be useful even to people who don't necessarily know how to do that
136 2014-04-28 05:38:55 <wumpus> or a neural network to train which IP patterns to allow
137 2014-04-28 05:39:37 <michagogo> cloud|e.g. If someone wants to use p2pool, getting into access rules, firewall settings, etc may be too complicated for them
138 2014-04-28 05:39:47 <wumpus> I'm not denying that it can be useful michagogo|cloud...
139 2014-04-28 05:40:24 <wumpus> if it listens by default on localhost, they already can use p2pool without configuring anything
140 2014-04-28 05:41:15 <michagogo> cloud|I suspect that, if they both existed, wlallowip would get more use than rpcallowip
141 2014-04-28 05:41:33 <gmaxwell> wumpus: there is a lot of software that just suports a lost of cidr masks, thats hardly skynet level complexity. :)
142 2014-04-28 05:41:49 <gmaxwell> s/lost/list/
143 2014-04-28 05:42:19 <wumpus> gmaxwell: I agree
144 2014-04-28 05:43:25 <michagogo> cloud|Is this easy for a user without much technical knowledge to set up on all supported OSes?
145 2014-04-28 05:44:02 <michagogo> cloud|Without having to go out and find a tool that does it?
146 2014-04-28 05:44:49 <uiop> just train a neural net until either it learns how to implem the whole software itself or the sun burns out, whichever first
147 2014-04-28 05:45:04 <uiop> *grr, was scrolled up)
148 2014-04-28 05:45:08 <wumpus> uiop: ah yes I forgot self-modifying
149 2014-04-28 05:46:05 <uiop> best to implem it in selfmod pdp11 asm for good measure
150 2014-04-28 05:47:03 <wumpus> so there are two pulls that implement whitelisting, but neither adds a seperate port
151 2014-04-28 05:47:05 <uiop> (with callout to system("sudo iptables ..."))
152 2014-04-28 05:47:42 <wumpus> they do have on-the-fly configurability, for if your static ACLs aren't good enough :-)
153 2014-04-28 05:48:28 <wumpus> uiop: sudo? no way, we'd just make it part of the kernel
154 2014-04-28 05:51:43 <uiop> theuniverse.ko
155 2014-04-28 05:53:42 <wumpus> gmaxwell: why do you want to remove getblock?
156 2014-04-28 05:54:57 <wumpus> gmaxwell: oh, you probably mean getwork :)
157 2014-04-28 05:54:58 <jcorgan> gee, all i asked about was hanging up on misbehaving onion nodes; but the conversation since has been very interesting :)
158 2014-04-28 05:55:08 <uiop> on a serious note though, i can see it being a *very* fine line between "just enough" (implem of selected parts of existing non-portable subsytems in bitcoin*) and having a "half-working reimplem of solved systems that might have egregious bugs, i'm not sure"
159 2014-04-28 05:56:24 <uiop> especially/for-example with the acls exploitability probably increases much faster than acl-language complexity
160 2014-04-28 05:56:46 <wumpus> uiop: a very fine line indeed, and it's easy to overstep it just trying to be 'user friendly'
161 2014-04-28 05:56:53 <gmaxwell> erp
162 2014-04-28 05:57:37 <gmaxwell> I think uniformity should be the tie breaker, other infrastructure daemons implement bind and basic cidr style acls, bugs in them seem fairly infrequently.
163 2014-04-28 05:57:56 <gmaxwell> and if we don't have a wallet we're functionally similar to bind or ntp.
164 2014-04-28 05:58:15 <wumpus> without a wallet things would be so much easier
165 2014-04-28 05:58:20 <gmaxwell> a somewhat security sensitive, reliability sentitive, public network service.
166 2014-04-28 05:59:19 <wumpus> it could even be mostly unauthenticated in that case, except for control RPCs
167 2014-04-28 05:59:48 <wumpus> well ok there's ddos risk
168 2014-04-28 06:00:15 <wumpus> but all in all, things would be much prettier
169 2014-04-28 06:02:11 <wumpus> CIDR seems simpler than the crazy wildcard matching that we use now
170 2014-04-28 06:07:21 <wumpus> and it can be matched directly without converting the address to a string, making it less expensive than the (recursive) string matching, yes I think it would be a good thing
171 2014-04-28 06:08:51 <wumpus> I wonder if there really are users using strange wildcards like *.1
172 2014-04-28 06:13:59 <uiop> wumpus: BUG REPORT: "before i did *.1, not you're telling me i've got to enumerate all 16777216 ip addr explicitly!?!"
173 2014-04-28 06:14:17 <uiop> no no, it would be:
174 2014-04-28 06:15:05 <uiop> wumpus: BUG REPORT: "bitcoind is exhausting all my memory and swap trying to read my config file, it's 314G because i had to enumerate ..."
175 2014-04-28 06:16:11 <wumpus> uiop: yes that would be typical, first the bug report would just contain 'bitcoind is exhausting all my memory !', then after much prodding they reveal their config file (what, am I not supposed to put 16 million IPs in there? YOU DON'T SCALE)
176 2014-04-28 06:16:48 <uiop> hehe
177 2014-04-28 06:20:12 <wumpus> just like Qt becomes slow if you paste >32000 characters into a line edit. Not much of a use case, until bitcoin came along, and people started pasting raw hex transactions > 16k
178 2014-04-28 06:25:00 <wumpus> hm even the *.1 case could be handled by allowing the use of netmasks
179 2014-04-28 06:34:12 <_andares> hey, what is the most appropriate library for inspecting blockchain history in java?
180 2014-04-28 06:34:17 <_andares> bitcoinj seems to run in SPV mode only
181 2014-04-28 06:39:44 <wumpus> connecting to a bitcoind instance using RPC and using getblock is simplest, although it can be slow if you need lots of block data
182 2014-04-28 06:40:58 <_andares> are there RPC libraries for bitcoind written in Java?
183 2014-04-28 06:41:36 <_andares> oh, regular JSON-RPC
184 2014-04-28 06:41:36 <wumpus> in that case I suppose bitcoinj has a way to request whole blocks from your node through the P2P port, not using bloom filtering
185 2014-04-28 06:42:41 <_andares> it gets worse than that.
186 2014-04-28 06:42:52 <_andares> I want to look up every transaction a given address has been a party to.
187 2014-04-28 06:43:22 <wumpus> right, then you will have to scan over all blocks, or make an index
188 2014-04-28 06:44:36 <jcorgan> patch your bitcoind with pull request #3652, set addrindex=1 in your bitcoin.conf, re-run with -reindex, then you'll have what you need
189 2014-04-28 06:44:55 <_andares> wow, awesome. thanks jcorgan
190 2014-04-28 06:45:32 <jcorgan> sipa hates me about now :)
191 2014-04-28 06:47:14 <_andares> I should use the "listtransactions" API on bitcoind, I guess
192 2014-04-28 06:47:42 <jcorgan> well, after than reindex you can do:
193 2014-04-28 06:47:45 <jcorgan> ./bitcoin-cli searchrawtransactions 1KwDYMJMS4xq3ZEWYfdBRwYG2fHwhZsipa
194 2014-04-28 06:47:56 <jcorgan> and get what you want out as json
195 2014-04-28 06:48:20 <jcorgan> downside: the blockchain+txindex+addrindex is now 26G
196 2014-04-28 06:50:18 <_andares> that is... pretty heavy
197 2014-04-28 06:50:52 <jcorgan> price you pay for having those indexes at your beck and call
198 2014-04-28 06:51:24 <_andares> and I guess there's no way that I could verify that a web service gave me the full tx history for an address
199 2014-04-28 06:54:12 <uiop> oh nice
200 2014-04-28 06:54:18 <uiop> 
ACTION gets bitcoin-cli

201 2014-04-28 06:55:58 <arowser> arowser tiange
202 2014-04-28 06:58:20 <_andares> I'm in a situation where I want a bunch of essentially thin clients, but I need this search functionality. if my clients could verify that the history returned for an address was complete, they could rely on a central server like blockchain to do the lookup without trusting it.
203 2014-04-28 06:59:46 <Belxjander> _andares: do you mean all the ThinClients are on a LAN setup with a central "HUB" for that LAN also acting as the Internet Gateway + BlockChain Service providing machine?
204 2014-04-28 06:59:51 <jcorgan> you could have a single, local bitcoind that manages that data and acts as an RPC server for all the thin clients
205 2014-04-28 07:00:08 <jcorgan> assuming the thin clients were on a fast local network
206 2014-04-28 07:00:17 <wumpus> verifying completeness is hard; not even SPV clients have this guarantee (although they can improve reliablity by asking multiple peers)
207 2014-04-28 07:01:35 <_andares> Belxjander: no, it's a decentralized P2P network. but one of the features of the nodes requires looking up the entire history of an address
208 2014-04-28 07:02:13 <jcorgan> unless each node maintains its own database, it will require trusting another node that does
209 2014-04-28 07:03:17 <_andares> without the indexes you mentioned, how efficient is listtransactions?
210 2014-04-28 07:03:56 <jcorgan> i think that still requires one of those indexes (txindex)
211 2014-04-28 07:04:20 <jcorgan> (not sure on that last)
212 2014-04-28 07:07:21 <_andares> how much space without the indexes does the blockchain take up?
213 2014-04-28 07:09:15 <wumpus> about 17GB
214 2014-04-28 07:09:54 <_andares> and about how large is each block?
215 2014-04-28 07:10:09 <wumpus> up to 1MB
216 2014-04-28 07:11:02 <anton000> hey wumps
217 2014-04-28 07:14:41 <wumpus> hello
218 2014-04-28 07:17:36 <sipa> _andares: listtransactions has nothing to do with those indexes
219 2014-04-28 07:17:53 <sipa> _andares: listtransactions queries the wallet
220 2014-04-28 07:18:11 <sipa> _andares: searchrawtransactions queries the blockchain
221 2014-04-28 07:20:32 <_andares> ah, thanks for the clarification. I was wondering why it returned a null set.
222 2014-04-28 07:21:20 <sipa> it also does not actually list transactions, ilists their effect on the wallet
223 2014-04-28 07:22:12 <gmaxwell> 00:01 < _andares> Belxjander: no, it's a decentralized P2P network. but one of the features of the nodes requires looking up the entire history of an address
224 2014-04-28 07:22:28 <gmaxwell> nodes have no such facility in general (a wallet knows only about its own transactions)
225 2014-04-28 07:22:51 <gmaxwell> oh you're saying you want it. sorry.
226 2014-04-28 07:24:51 <_andares> ah yes. I am wondering if I can make do in SPV mode somehow
227 2014-04-28 07:25:16 <sipa> no
228 2014-04-28 07:25:45 <sipa> there is no way to query for all transactions related to an address
229 2014-04-28 07:26:01 <sipa> there is not even a way to query a specific transaction by txid
230 2014-04-28 07:26:11 <_andares> right, you can only request blocks
231 2014-04-28 07:26:23 <sipa> because nodes should not be burdened by needing to index all that
232 2014-04-28 07:26:37 <_andares> agreed.
233 2014-04-28 07:26:37 <sipa> as ot is no requirement for a functioning network
234 2014-04-28 07:27:05 <sipa> can anyone do a 32-bit build of #4100?
235 2014-04-28 07:27:19 <sipa> i can't reproduce pulltester's failure
236 2014-04-28 07:27:59 <_andares> but an SPV client can ask a peer for a transaction from its memory pool, can't it?
237 2014-04-28 07:28:02 <_andares> I seet hat in the bitcoinj source code
238 2014-04-28 07:31:47 <sipa> _andares: yes
239 2014-04-28 07:43:58 <coder123> if i build bitcoind in ubuntu and use testnet will it still try to download the main chain in the background?
240 2014-04-28 07:44:22 <coder123> i'm trying to start coding but don't want to wait 2-3 days for the block chain to sync ;(
241 2014-04-28 07:44:35 <coder123> just want to start testing my code with bitcoind
242 2014-04-28 07:44:42 <buZz> a) you can use bootstrap.dat, b) this is not a topic for #bitcoin-dev
243 2014-04-28 07:44:56 <coder123> buZz: which channel would be better to dicsuss this?
244 2014-04-28 07:45:01 <buZz> #bitcoin ;)
245 2014-04-28 07:45:33 <coder123> thanks i'll post there... does the test net chain take days to sync as well?
246 2014-04-28 07:45:42 <sipa> if you run bitcoind in testnet mode, it will obviously only sync the testnet network
247 2014-04-28 07:48:56 <coder123> thanks sips, any idea how many hrs it takes to sync testnet?
248 2014-04-28 07:48:58 <_andares> are transactions being pruned off the Merkle tree today?
249 2014-04-28 07:49:00 <coder123> sipa*
250 2014-04-28 07:51:48 <sipa> coder123: maybe one
251 2014-04-28 07:51:56 <coder123> great, thanks!
252 2014-04-28 07:52:47 <sipa> _andares: spv nodes downloaded filtered blocks, with just the one relevant to them in it
253 2014-04-28 07:53:33 <sipa> _andares: and receive a partial merkle tree with it to prove that those transactions actually belong to the downloaded blkck
254 2014-04-28 07:55:02 <sipa> but that is only on the network
255 2014-04-28 07:55:10 <sipa> they are always stored in full
256 2014-04-28 07:58:35 <_andares> sipa: so if an SPV node's 8 peers (somehow) lied to it and returned zero block headers, the node would conclude it had 0 BTC in its wallet?
257 2014-04-28 07:58:53 <_andares> assuming it had no cached blockchain data
258 2014-04-28 08:02:26 <sipa> _andares: correct
259 2014-04-28 08:02:45 <_andares> how do the spv nodes ask for all transactions their wallet has been in?
260 2014-04-28 08:02:54 <sipa> _andares: read bip 37
261 2014-04-28 08:14:50 <_andares> so my client could set a bloom filter for the accounts it's interested in, and then start requesting block headers from the peers
262 2014-04-28 08:15:12 <_andares> I guess I have to query 210k blocks though
263 2014-04-28 08:17:57 <maaku> _andares: wallet birthdays
264 2014-04-28 08:18:55 <sipa> not block headers; blocks
265 2014-04-28 08:34:41 <coder123> interestingly, SPV wallets leak privacy, -- by only asking for headers to blocks of relevant transactions -- attackers can connect to all nodes and then see who's asking for what, and try to deduce which addresses are controlled by the wallets of each node
266 2014-04-28 08:42:01 <_andares> sipa: you're right, blocks. but only the blocks participating in a transaction with that account will be returned to me
267 2014-04-28 08:42:04 <_andares> which may not be too many
268 2014-04-28 08:43:09 <ndak> 
ACTION [A MESSAGE] BRB

269 2014-04-28 08:45:27 <kinlo> coder123: bloom filters should obscure that
270 2014-04-28 08:49:08 <coder123> kinlo:  does multi bit use bloom filters?
271 2014-04-28 08:49:32 <kinlo> coder123: either it does, or it downloads entire blocks and filters locally
272 2014-04-28 08:49:46 <kinlo> the bitcoin protocol doesn't allow to query for 1 address
273 2014-04-28 08:49:55 <kinlo> so in both cases your privacy is guaranteed
274 2014-04-28 08:50:25 <coder123> seems like some data would still be leaked -- but i'll do some further reading
275 2014-04-28 09:34:24 <michagogo> cloud|Why is undo data in blocks/ and not in chainstate/?
276 2014-04-28 09:34:39 <michagogo> cloud|(Or do I misunderstand what that data is?)
277 2014-04-28 09:34:52 <wumpus> michagogo|cloud: I always wonder about that too
278 2014-04-28 09:35:20 <wumpus> michagogo|cloud: might because of the access pattern (rarely accessed, big files)
279 2014-04-28 09:44:28 <sipa> because it is per-block data
280 2014-04-28 09:45:06 <sipa> and it is referenced by blocks/index
281 2014-04-28 09:45:22 <sipa> the blocks directory and chainstate directory are independent from eachother
282 2014-04-28 09:45:34 <sipa> you can copy one without copying the other
283 2014-04-28 09:49:22 <sipa> or another way of seeing it: blocks are forward patches to the chainstate
284 2014-04-28 09:49:32 <sipa> undo data are reverse patches
285 2014-04-28 09:49:47 <sipa> michagogo|cloud, wumpus: does that make sense?
286 2014-04-28 09:50:32 <michagogo> cloud|sipa: maybe...
287 2014-04-28 09:50:56 <sipa> the reason is chainstate and blocks being independent
288 2014-04-28 09:51:16 <sipa> so you can copy the chainstate without blocks, or the other way around
289 2014-04-28 09:51:35 <sipa> everything in blocks/ is about the block tree, not the active chain
290 2014-04-28 09:51:52 <sipa> chainstate/ is about the active chain only and not the blocks
291 2014-04-28 09:54:19 <sipa> kinlo, coder123: in practice bloom filters gain you very little privacy
292 2014-04-28 09:55:40 <sipa> as the false positive rate goes up very rapidly if you don't reset the filter from time to time
293 2014-04-28 10:00:14 <hearn> sipa: that’s a quirk of the current implementation though, right? we could fix it by tracking hits on outpoints outside the filter and making them non probabilistic
294 2014-04-28 10:00:27 <hearn> or at least make the problem less severe
295 2014-04-28 10:11:07 <sipa> hearn: yeah, it can certainly be improved i think
296 2014-04-28 10:36:22 <michagogo> cloud|sipa: ah, I see what you're saying
297 2014-04-28 10:36:55 <michagogo> cloud|The information in chainstate/ is only what's current and being used
298 2014-04-28 10:37:25 <michagogo> cloud|Erm, maybe I don't
299 2014-04-28 10:37:49 <sipa> michagogo|cloud: chainstate/ contains data per-txout, blocks/ contains data per-block
300 2014-04-28 10:37:52 <ndak> hi. i want to install a Openvpnserver + PEM for my VPS. I am new and i need this asap, I want use my own VPN because its cheaper. :) I am using Debian 7 and I will pay you for the work /msg me
301 2014-04-28 10:37:55 <michagogo> cloud|Are the rev*.dat files not chainstate-specific?
302 2014-04-28 10:38:07 <sipa> michagogo|cloud: no
303 2014-04-28 10:38:27 <sipa> michagogo|cloud: not more than blk*.dat are chainstate-specific
304 2014-04-28 10:38:45 <michagogo> cloud|What is in rev?
305 2014-04-28 10:38:59 <sipa> how to unapply the effects done by a block to the chainstate
306 2014-04-28 10:39:20 <sipa> but they're not specific to *your* chainstate directory
307 2014-04-28 10:39:37 <sipa> everyone has byte-for-byte the same undo data for each block
308 2014-04-28 10:40:03 <michagogo> cloud|Why are they needed?
309 2014-04-28 10:40:16 <michagogo> cloud|Can't you use the block data?
310 2014-04-28 10:40:16 <michagogo> cloud|Oh, wait
311 2014-04-28 10:40:24 <michagogo> cloud|You'd need txindex
312 2014-04-28 10:40:42 <michagogo> cloud|Because a transaction deletes outputs from the chainstate
313 2014-04-28 10:40:55 <sipa> you'd need to find the earlier block that added them
314 2014-04-28 10:41:20 <michagogo> cloud|So is it: when a transaction in a block is processed, it's inputs are deleted from chainstate and added to rev*.dat?
315 2014-04-28 10:41:40 <michagogo> cloud|The STXO set, as it were?
316 2014-04-28 10:42:12 <sipa> indeed, it's a list of the TXOs spent by a particular block
317 2014-04-28 10:42:29 <sipa> and it's small, because it has no signatures
318 2014-04-28 10:42:36 <michagogo> cloud|Why is- Ahhhh
319 2014-04-28 10:42:55 <michagogo> cloud|You anticipated my question
320 2014-04-28 10:43:40 <michagogo> cloud|And the scriptSig is a big part of transactions
321 2014-04-28 10:43:47 <michagogo> cloud|Since it's a signature and a pubkey
322 2014-04-28 10:44:04 <michagogo> cloud|Okay, I think I understand now :)
323 2014-04-28 10:44:24 <michagogo> cloud|So the rev*.dats are append-only like blk*.dat are?
324 2014-04-28 10:44:27 <sipa> indeed
325 2014-04-28 10:44:39 <sipa> and they contain corresponding data
326 2014-04-28 10:44:49 <michagogo> cloud|So it's just more data about each block
327 2014-04-28 10:44:54 <sipa> correct
328 2014-04-28 10:45:02 <sipa> everything in blocks/ is block-specific data
329 2014-04-28 10:45:07 <michagogo> cloud|Aha.
330 2014-04-28 10:45:20 <michagogo> cloud|Why isn't it just in blk*.dat?
331 2014-04-28 10:45:43 <sipa> backwards compatibility, i guess
332 2014-04-28 10:45:51 <sipa> with tools that expected block files in a particular format
333 2014-04-28 10:46:23 <wumpus> sipa: yes, makes sense
334 2014-04-28 10:46:25 <michagogo> cloud|Oh, was it not always there?
335 2014-04-28 10:46:48 <wumpus> I think it's nice that the blkXXX files contain just the block data received from the network, nothing more
336 2014-04-28 10:48:05 <michagogo> cloud|Hm. Is rev*.dat trusted data?
337 2014-04-28 10:48:11 <sipa> yes
338 2014-04-28 10:48:29 <sipa> michagogo|cloud: until 0.7, we only had blk*.dat and blkindex.dat
339 2014-04-28 10:48:43 <michagogo> cloud|No utxo db?
340 2014-04-28 10:48:46 <sipa> michagogo|cloud: and blkindex.dat contained for each transaction output where it was spent (if spent at all)
341 2014-04-28 10:48:50 <michagogo> cloud|Ahhh
342 2014-04-28 10:48:50 <sipa> no
343 2014-04-28 10:49:03 <michagogo> cloud|Oh, interesting
344 2014-04-28 10:49:18 <sipa> i wonder how long 0.7 would take to sync today :)
345 2014-04-28 10:49:34 <michagogo> cloud|Wait, so blkindex.dat was a txindex?
346 2014-04-28 10:49:38 <sipa> yes
347 2014-04-28 10:49:45 <sipa> and more than that
348 2014-04-28 10:49:47 <michagogo> cloud|Interesting
349 2014-04-28 10:49:57 <sipa> it took 12 bytes per txout ever created
350 2014-04-28 10:52:15 <sipa> it would be a database of several GB today, i think
351 2014-04-28 10:52:22 <michagogo> cloud|So with the introduction of the utxo set, the txindex became optional and off-by-default, I assume?
352 2014-04-28 10:53:23 <sipa> indeed
353 2014-04-28 10:53:44 <sipa> blkindex.dat functioned as UTXO set, txindex, and block undo data
354 2014-04-28 10:53:53 <sipa> in 0.8, those 3 functions were split up
355 2014-04-28 10:54:11 <sipa> so that the UTXO set (which is vastly more frequently accessed than the others) could be optimized
356 2014-04-28 10:56:47 <michagogo> cloud|So right now, by default, is there a record of which block a given utxo is in?
357 2014-04-28 10:56:57 <sipa> indirectly
358 2014-04-28 10:57:04 <sipa> the UTXO encodes the height at which it was created
359 2014-04-28 10:57:13 <sipa> which uniquely identifies a block in the chai
360 2014-04-28 10:57:38 <michagogo> cloud|Okay. So, how come undo data includes the whole txin?
361 2014-04-28 10:57:50 <michagogo> cloud|Why not just hash, vout, and height?
362 2014-04-28 10:58:11 <sipa> because it's about utxo entries that were deleted
363 2014-04-28 10:58:22 <sipa> the undo data is necessary to recreate them
364 2014-04-28 10:58:46 <sipa> ah, wait
365 2014-04-28 10:59:01 <ndak> hi. i want to install a Openvpnserver + PEM for my VPS. I am new and i need this asap, I want use my own VPN because its cheaper. :) I am using Debian 7 and I will pay you for the work /msg me
366 2014-04-28 10:59:07 <sipa> ndak: not here
367 2014-04-28 10:59:54 <sipa> michagogo|cloud: that'd require seeking through perhaps hundreds of block files to undo one block
368 2014-04-28 11:00:24 <michagogo> cloud|sipa: ah, right.
369 2014-04-28 11:01:16 <michagogo> cloud|When that thought came to me, I missed the fact that the utxos would need to be recovered in large batches
370 2014-04-28 11:01:57 <michagogo> cloud|Hm, how about storing the full txo for the past n blocks and beyond that just storing the block pointer?
371 2014-04-28 11:02:56 <wumpus> that'd imply a lot of copying
372 2014-04-28 11:03:29 <michagogo> cloud|Oh, right. Copying or fragmentation
373 2014-04-28 11:03:45 <michagogo> cloud|(Or empty blocks, which defeats the purpose)
374 2014-04-28 11:03:55 <wumpus> 'reverse patches' only contain what is changed, so that's the more efficient way given the access pattern (ie, almost write-only)
375 2014-04-28 11:03:56 <michagogo> cloud|Er, wait, why?
376 2014-04-28 11:04:23 <michagogo> cloud|Why not jus-oh, wait
377 2014-04-28 11:04:23 <wumpus> (more efficient than writing the whole UTXO set to disk for each block)
378 2014-04-28 11:04:27 <sipa> to disconnect a block, all you need to read is the block and its undo data
379 2014-04-28 11:04:32 <sipa> and apply that to the UTXO set
380 2014-04-28 11:04:37 <sipa> no seeking whatsoever
381 2014-04-28 11:04:39 <wumpus> right, I think it's quite ingenious
382 2014-04-28 11:04:48 <sipa> :)
383 2014-04-28 11:04:59 <michagogo> cloud|I was kinda thinking in terms of disk space
384 2014-04-28 11:05:11 <sipa> if we care about that, we should implement pruning
385 2014-04-28 11:05:20 <michagogo> cloud|Right
386 2014-04-28 11:06:12 <michagogo> cloud|By the time it gets to the point that changing full txouts for block pointers will make a big difference, we should already have pruning
387 2014-04-28 11:06:18 <michagogo> cloud|(Or put it in)
388 2014-04-28 11:06:33 <wumpus> I suppose you could remove the undo files for more than 1000 blocks ago, and do a full reindex in case a reorganization that big happens, but it would be disastrous if *everyone* had to do that
389 2014-04-28 11:06:37 <michagogo> cloud|And in the meantime this is better for performance
390 2014-04-28 11:07:39 <sipa> so, when designing the undo data stuff, there were two choices: either make it standalone (so you don't need the actual block to undo a block), or make it complementary
391 2014-04-28 11:07:56 <sipa> i tried both in disk usage, and the standalone was around 1/3 of the size of actual blocks
392 2014-04-28 11:08:16 <sipa> the other (which is what is used now) is around 1/10
393 2014-04-28 11:08:22 <michagogo> cloud|Why do you need the actual block again?
394 2014-04-28 11:08:41 <sipa> to know which txouts were created by the block, so you can remove them
395 2014-04-28 11:08:46 <michagogo> cloud|Oh, of course.
396 2014-04-28 11:08:52 <churchill2> i downloaded the client but the sha256sum isn't as in the signature
397 2014-04-28 11:08:54 <sipa> really just the txids of the block
398 2014-04-28 11:09:21 <wumpus> churchill2: what file did you download?
399 2014-04-28 11:09:34 <churchill2> wumpus, bitcoin-0.9.1-linux.tar.gz
400 2014-04-28 11:09:39 <sipa> from where?
401 2014-04-28 11:10:08 <churchill2> https://bitcoin.org/en/download
402 2014-04-28 11:10:21 <michagogo> cloud|What hash did you get?
403 2014-04-28 11:10:45 <wumpus> whoa
404 2014-04-28 11:11:00 <churchill2> michagogo|cloud, 4e0d202d6b14a....
405 2014-04-28 11:11:17 <uiop> dwarf's CFI (call frame information) system for unwinding the callstack (which uses a little stack machine language too) is nicely done imo, and is kind of related at idea level. reading about it if full of thoroughly debugged and honed ideas
406 2014-04-28 11:11:17 <wumpus> mine matches the signature
407 2014-04-28 11:11:33 <wumpus> just downloaded and got the sha256sum 3fabc1c629007b465a278525883663d41a2ba62699f2773536a8bf59ca210425  bitcoin-0.9.1-linux.tar.gz.1
408 2014-04-28 11:11:51 <michagogo> cloud|I match as well
409 2014-04-28 11:11:56 <michagogo> cloud|Try redownload ong
410 2014-04-28 11:12:03 <churchill2> my file size is 35,667,653 bytes
411 2014-04-28 11:12:03 <wumpus> can you upload your 'corrupted' file somewhere?
412 2014-04-28 11:12:07 <wumpus> I'd like to compare it
413 2014-04-28 11:12:08 <michagogo> cloud|Redownloading*
414 2014-04-28 11:12:15 <michagogo> cloud|Yep, too small
415 2014-04-28 11:12:18 <uiop> (re: undo-trails, and (second-order-relatedness) a special-purpose stack vm bytecode lang)
416 2014-04-28 11:12:21 <wumpus> -rw-rw-r-- 1 orion orion 45665893 Apr  8 20:47 bitcoin-0.9.1-linux.tar.gz.1
417 2014-04-28 11:12:22 <michagogo> cloud|I have 45665893
418 2014-04-28 11:12:33 <wumpus> ok, probably just got cut off
419 2014-04-28 11:12:41 <churchill2> michagogo|cloud, strange, it didn't happen before. thanks
420 2014-04-28 11:12:51 <michagogo> cloud|wumpus: try chopping off the end?
421 2014-04-28 11:13:03 <michagogo> cloud|(Not on a machine with Linux or Cygwin)
422 2014-04-28 11:14:06 <wumpus> 4e0d202d6b14ad8ec1a2f90be623ffdb4a787472e6972e0f3cb04c85a67dce46  bitcoin-0.9.1-linux.tar.gz.2
423 2014-04-28 11:14:10 <wumpus> yep... chopped off it matches
424 2014-04-28 11:14:11 <wumpus> phew
425 2014-04-28 11:14:41 <wumpus> we can breathe again :)
426 2014-04-28 11:14:44 <uiop> heh, those short moments of dread when you think the hash doesn't match
427 2014-04-28 11:14:52 <sipa> i'm happy to see that people actually check that hash
428 2014-04-28 11:15:02 <michagogo> cloud|Indeed!
429 2014-04-28 11:15:07 <stonecoldpat> +1
430 2014-04-28 11:15:20 <stonecoldpat> +1
431 2014-04-28 11:15:58 <uiop> the dread is so huge when it takes 30 minutes an attempt to compute the hash (i just backed up my hdd and upgrades laptop yesterday, the horror)
432 2014-04-28 11:21:32 <uiop> here's the enum for DW_OP dwarf cfi opcodes http://gcc.gnu.org/git/?p=gcc.git;a=blob;f=include/dwarf2.def;h=71a37b30c9f87b429bab08a4a420caa9a6385503;hb=HEAD#l411
433 2014-04-28 11:23:59 <uiop> also, i'm a big fan of the {.def,.h} system thing going on here too (for things like this) http://gcc.gnu.org/git/?p=gcc.git;a=blob;f=include/dwarf2.h;h=120e2c16b4820f823c75ab8b2aa7c00bab6291fa;hb=HEAD#l77
434 2014-04-28 11:27:20 <uiop> (DW_OP_form_tls_address,DW_OP_call_frame_cfa,DW_OP_GNU_push_tls_address) are some of the analogous insns to the crypto/sig/verification macro-opcodes in bitcoin script
435 2014-04-28 11:31:44 <wumpus> uiop: yes, dwarf is pretty crazy, it's a hell of a lot of work to interpret (and there have been exploits in some versions of the interpreters), but it's like that for a good reason
436 2014-04-28 11:32:39 <uiop>  how it works is: the stack vm opcode script computes the values that each register needs to be restored with, along with the address of the next callframe. run script on curr call frame, now your simulated (registers,retaddr) machine state is unwound one frame, locate cfi program associated with callframe you just computed, loop.
437 2014-04-28 11:32:41 <wumpus> it's a hard problem so it really needs a complex solution like this
438 2014-04-28 11:34:54 <wumpus> the same language is used to reconstruct values of parameters and variables at arbitrary locations of the source code, even sometimes if optimized out
439 2014-04-28 11:35:07 <uiop> wumpus: i read this one paper that talked about the possibility to write malicious cfi programs, but in practice all the dwarf interpreters in the actual std toolchain(s) won't, for instance, execute loops of subroutine calls
440 2014-04-28 11:35:08 <wumpus> (though, even with all this C++ debugging manages to suck a lot of the time)
441 2014-04-28 11:35:24 <uiop> *exec loops _or_ subroutine calls
442 2014-04-28 11:36:08 <churchill2> hi
443 2014-04-28 11:36:29 <uiop> wumpus: right, it also can reconstruct the (conceptual, never fully computed at once) line number table...
444 2014-04-28 11:36:39 <uiop> dwarf does so much stuff it's crazy
445 2014-04-28 11:37:05 <churchill2> i'm trying to run the client but i get: bitcoin-qt: Syntax error: "(" unexpected
446 2014-04-28 11:38:05 <sipa> run? you're not compiling it, are you?
447 2014-04-28 11:38:23 <churchill2> no
448 2014-04-28 11:39:39 <sipa> you're not accidentally typing ". bitcoin-qt" rather than "./bitcoin-qt" ?
449 2014-04-28 11:40:17 <churchill2> sipa, all's good now
450 2014-04-28 11:40:25 <uiop> wumpus: totally, a complicated multiobjective problem that doesn't have a simple solution
451 2014-04-28 11:41:00 <churchill2> sipa, what's the difference between ./ and sh ?
452 2014-04-28 11:41:25 <sipa> this is not #unix
453 2014-04-28 11:45:49 <uiop> here's a marathon run or 5 http://gcc.gnu.org/git/?p=gcc.git;a=blob;f=gcc/dwarf2out.c;h=12723265e879ce34852e12e284bbbd5aae1490d1;hb=HEAD
454 2014-04-28 11:46:24 <uiop> 24397 lines gcc /dwarf2out.c
455 2014-04-28 11:46:30 <uiop> ouch
456 2014-04-28 11:47:11 <sipa> fun
457 2014-04-28 11:49:23 <uiop> iirc there's one DW_OP interpreter in each of gdb, glibc (i think), libffi, it's been a while since i looked
458 2014-04-28 11:49:35 <uiop> sipa: it hurts so good
459 2014-04-28 11:49:41 <sipa> in glibc? :o
460 2014-04-28 11:49:42 <sipa> why
461 2014-04-28 11:49:55 <uiop> for backtrace*()
462 2014-04-28 11:52:47 <wumpus> I think it's the one in glibc that was used for an exploit at some point
463 2014-04-28 11:56:39 <uiop> here's execute_stack_op(..) http://repo.or.cz/w/glibc.git/blob/HEAD:/sysdeps/generic/unwind-dw2.c#l326
464 2014-04-28 11:57:00 <uiop> interp switch(){}
465 2014-04-28 11:57:54 <uiop> (the _Unwind_*(..) functions are standardized)
466 2014-04-28 11:58:55 <uiop> oh, and exec_cfa_program(..) here http://repo.or.cz/w/glibc.git/blob/HEAD:/sysdeps/generic/unwind-dw2.c#l731
467 2014-04-28 11:59:14 <uiop> wumpus: nice, that'd be interesting to see
468 2014-04-28 12:03:23 <uiop> fs->regs.reg[reg].loc.offset          /*C is awesome*/
469 2014-04-28 12:10:08 <uiop> sipa: (re: why in glibc) backtrace*(..) and the c++ abi runtime support functions make use of a standardized (c interface) lib that glibc provides
470 2014-04-28 12:10:23 <uiop> sipa: reading that interpreter code made me remember
471 2014-04-28 12:10:56 <uiop> (the _Unwind* stuff)
472 2014-04-28 12:11:51 <uiop> that interpreter gets run every time a C++ exception is thrown!
473 2014-04-28 12:12:31 <uiop> (every?)
474 2014-04-28 12:53:51 <melvster> does anyone know if there's an open source / free software faucet out there?
475 2014-04-28 14:20:19 <lclc> Where and how do you (the Bitcoin Core developers) decide e.g. what technologie you want to use for xy?   How to implement something when there are several ways? Who's going to make the decision? do you vote (e.g. each core dev) in a merocratic group like Debian?
476 2014-04-28 14:24:52 <Emcy_> consensus
477 2014-04-28 14:24:57 <Emcy_> common sense
478 2014-04-28 14:25:18 <Emcy_> vote as a last resort
479 2014-04-28 14:25:58 <sipa> lclc: if it's just local implementation (without impact on the network, or protocol changes), it's usually just the one who implements it that decides (of course, with input from others, but the one coding it is often also the one who understands the problem best)
480 2014-04-28 14:26:25 <sipa> for protocol changes (which are quite rare...), it always after discussion on the bitcoin-development mailinglist
481 2014-04-28 14:27:26 <lclc> Ok, but for example you decide to use boost for networking and multithreading instead of ZMQ and c++11. Who decides that?  (I know, that was more historic, but just as an example)
482 2014-04-28 14:28:48 <gavinandresen> Satoshi decided that, then inertia (and years of security/peer review that we don't want to throw out) keeps us from changing to the latest&greatest new thing-a-ma-bob
483 2014-04-28 14:29:14 <sipa> i don't think you should see it as "decisions"; every step is an increment change
484 2014-04-28 14:29:28 <sipa> and sometimes someone suggests making a change, because it has certain benefits
485 2014-04-28 14:29:42 <sipa> and that's either rejected or accepted by consensus
486 2014-04-28 14:32:29 <Emcy_> assuming every aspect of bitcoin core was desinged and blueprinted from scratch is wrong
487 2014-04-28 14:32:52 <sipa> for example, nobody ever decided "against" ZMQ as far as I know... there was an implementation at some point, but it wasn't maintained
488 2014-04-28 14:33:10 <Emcy_> it has evolved much more organically than that, according to what people wanted/needed to work on at a particular time. Its been interestng to watch
489 2014-04-28 14:33:14 <lclc> sure, that wasn't the question. It's more about comming decisions about technologies for specific tasks (which might affect later tasks)
490 2014-04-28 14:33:51 <Emcy_> the project lead?
491 2014-04-28 14:34:34 <Emcy_> i think this project operates on what is known in oss as the BDFL model......
492 2014-04-28 14:35:00 <lclc> that wouldn't be very FOSS-like IMHO. I expected the core devs to vote on such things
493 2014-04-28 14:35:12 <sipa> i don't think we've ever needed a vote
494 2014-04-28 14:35:28 <Emcy_> some things are. coinbase voting has been done
495 2014-04-28 14:35:37 <sipa> Emcy_: that was about a protocol change
496 2014-04-28 14:35:47 <sipa> that doesn't seem to be what lclc is asking about
497 2014-04-28 14:36:25 <Emcy_> I think if devs cant come to consensus due to fundamental difference of opinion or trolling or w/e, everyone just gets sick of it and puts the issue ont he back burner for a while
498 2014-04-28 14:38:03 <Emcy_> lclc there is a dev mailing list you know. Also a lot of things are hammered out right here
499 2014-04-28 14:38:24 <sipa> very little bitcoin-core-internal implementation stuff though
500 2014-04-28 14:38:25 <Emcy_> pressing circumstances tend to be handled here
501 2014-04-28 14:38:28 <sipa> that doesn't belong there
502 2014-04-28 14:39:00 <maraoz> gmaxwell: regarding p2sh multisig bip32 structure discussion on mailing list, should I submit a BIP draft to the list, or request a BIP number and issue a pull request to bitcoin/bips?
503 2014-04-28 14:39:23 <Emcy_> i would say probably the implementation nitty gritty is done on github pull threads right?
504 2014-04-28 14:39:26 <lclc> ok, so, so far technology decisions in bitcoin have never been a discussion since they were always clear. But they would if needed
505 2014-04-28 14:39:36 <Luke-Jr> maraoz: read BIP 1
506 2014-04-28 14:39:46 <sipa> lclc: hell yes there have been discussions
507 2014-04-28 14:39:53 <sipa> lclc: but if there is no consensus, a change doesn't happen
508 2014-04-28 14:40:15 <lclc> ok thanks.
509 2014-04-28 14:40:17 <Luke-Jr> sipa: that's not true..
510 2014-04-28 14:40:27 <lclc> just wanted to know how BItcoin does it for another FOSS project
511 2014-04-28 14:40:41 <sipa> (talking about core implementation stuff here, and consensus among core devs)
512 2014-04-28 14:40:45 <Emcy_> im not sure i would say bitcoin core has something that could be considered a roadmap.........
513 2014-04-28 14:40:51 <Emcy_> unles there is something i havent seen
514 2014-04-28 14:40:55 <sipa> not about network level changes, where far more people are affected and involved
515 2014-04-28 14:42:17 <Emcy_> prptocol changes have been put out to vote on the blockchain once or twice, which is a very interesting way to do things for foss
516 2014-04-28 14:42:51 <Emcy_> and necessary, no other project is quite like this once
517 2014-04-28 14:46:06 <Emcy_> apart from perculiarities like that, i dont hink this project operates much differently from any other lively and successful foss project
518 2014-04-28 14:46:24 <maraoz> Luke-Jr: thanks, found it
519 2014-04-28 14:46:33 <Emcy_> that is to say getting shit done with a minimum of politics if anyone can help it
520 2014-04-28 15:00:36 <tyrick> where are cs_Shutdown and lockShutdown defined in the Shutdown() function within init.cpp
521 2014-04-28 15:02:01 <sipa> TRY_LOCK(a, b) tries to lock a, and puts the result in a new variable b
522 2014-04-28 15:02:14 <sipa> so b can be tested to see whether locking succeeded
523 2014-04-28 15:02:22 <sipa> it's defined in sync.h
524 2014-04-28 15:02:47 <tyrick> Well, I see that TRY_LOCK is defined there
525 2014-04-28 15:03:01 <tyrick> but where is cs_Shutdown definied
526 2014-04-28 15:03:10 <sipa> the line above
527 2014-04-28 15:03:11 <tyrick> the 'a' that is being locked
528 2014-04-28 15:03:42 <sipa> static CCriticalSection cs_Shutdown;
529 2014-04-28 15:06:11 <tyrick> Okay, I just explored it a bit more
530 2014-04-28 15:06:38 <tyrick> So there is a recusive mutex lock on cs_Shutdown
531 2014-04-28 15:07:14 <sipa> yes
532 2014-04-28 15:07:41 <tyrick> Then what? I can not find any other uses of the cs_Shutdown resource in the project
533 2014-04-28 15:08:02 <sipa> there can't be, the variable is only defined within Shutdown()
534 2014-04-28 15:08:22 <sipa> it prevents multiple invocations of that function from operating at the same time
535 2014-04-28 15:08:34 <tyrick> Ah!
536 2014-04-28 15:08:51 <tyrick> lol... I was totally making a big deal out of nothing
537 2014-04-28 15:12:18 <tyrick> There are better ways of ensuring that a function is only called once
538 2014-04-28 15:12:52 <tyrick> Is there a reason for this particular way?
539 2014-04-28 15:21:35 <sipa> tyrick: for example?
540 2014-04-28 15:22:59 <tyrick> A static variable within the function that manages its usage
541 2014-04-28 15:23:59 <tyrick> I guess the benefit of using a mutex would be that it is an atomic operation
542 2014-04-28 15:24:29 <Arnavion> Go read up what a critical section is
543 2014-04-28 15:24:34 <tyrick> I suppose an interrupt could happen while the program was in the process of checking that functions static variable
544 2014-04-28 15:24:48 <sipa> c++11 has atomic variables, which would allow that
545 2014-04-28 15:25:06 <sipa> but without, just a static variable isn't thread safe for this usage
546 2014-04-28 15:49:06 <tyrick> Another question, Shutdown() is called in AppInit(), and unless lockShutdown is false, the program will close.  if TRY_LOCK(a, b) places the result in b, does this mean that a was not able to be locked on startup?
547 2014-04-28 15:56:11 <wumpus> the idea is that only one thread can get the shutdown lock (and thus perform the shutdown)
548 2014-04-28 15:57:05 <wumpus> a try-lock is used so that other threads will fall through and exit so that they're not in the way, instead of waiting for the lock
549 2014-04-28 15:57:26 <yoyoceramic> Are there any gists or guides on how to use a proof of assets scheme on a series of multisignature addresses?
550 2014-04-28 16:01:42 <tyrick> Thanks for the answers btw.  But the shutdown() that is called in AppInit() is on a single thread right?
551 2014-04-28 16:02:06 <tyrick> Or rather, AppInit() is on a single thread
552 2014-04-28 16:03:11 <tyrick> and is the first function to call Shutdown(), and I would assume the main thread would get the lock
553 2014-04-28 16:14:30 <dexx> hey sipa, i'm using a modified version of your address indexed branch and noticed in the case of orphaned blocks some transactions have a reference to the orphan instead of one to the active chain in which they were included nevertheless. i was wondering, if i introduced this somehow or if it was known before
554 2014-04-28 16:56:23 <petertodd> dexx: when I tested it ages ago there was some weirdness going on with incorrect indexing that I never figured out; sounds vaguely plausible
555 2014-04-28 17:06:25 <dexX7> petertodd: what kind of weirdness?
556 2014-04-28 17:08:02 <petertodd> dexX7: IIRC all things missing from the index; my comments should be on github somewhere
557 2014-04-28 17:09:27 <dexX7> the only problem i faced until now besides the orphaned block references are crashes due to too many open files, but this is probably unrelated to the addrindex branch itself
558 2014-04-28 17:10:12 <petertodd> interesting, maybe the problems I saw got fixed?
559 2014-04-28 17:10:45 <petertodd> like I say, I was testing this right when it came out
560 2014-04-28 17:14:03 <dexX7> well, only because i'm not aware of problems doesn't mean there are none
561 2014-04-28 17:15:51 <dexX7> https://github.com/dexX7/bitcoin/tree/addrindex based on 0.9-ish
562 2014-04-28 17:32:13 <maaku> hearn: For a non-bitcoin purpose I need to modify the payment protocol to add an additional field. I'd rather maintain future compatability with upstream, if possible. How should I select the protocol buffer field number for this new field?
563 2014-04-28 17:32:36 <hearn> https://github.com/bitcoin/bips/blob/master/bip-0070.mediawiki#Extensibility
564 2014-04-28 17:33:03 <hearn> if your payment requests will go outside your own controlled world, pick a number or range of numbers, and reserve them on that page
565 2014-04-28 17:33:39 <hearn> if you want to make an extension that could become a standard, well, i guess just pick the next available numbers after the ones BIP 70 use. as long as nobody else is going to use those numbers, it’s not a big deal
566 2014-04-28 17:33:55 <hearn> just announce that you’re doing it on the -development list i guess
567 2014-04-28 17:34:54 <maaku> well it's a field (reference height) which only has to do with demurrage / interest rates on Freicoin, so it's exceedingly unlikely to ever make it back into bitcoin
568 2014-04-28 17:36:04 <hearn> then just grab an extension field by submitting a pull req to that file
569 2014-04-28 17:36:04 <maaku> but if you're willing to reserve an extension value for that then it's no problem, as that's the best solution
570 2014-04-28 17:36:11 <hearn> sure, of course. they’re only numbers
571 2014-04-28 17:36:17 <maaku> thank you
572 2014-04-28 17:38:57 <petertodd> maaku: https://github.com/bitcoin/bips/blob/master/bip-0070/extensions.mediawiki <- that's specifically the extensions page FWIW
573 2014-04-28 17:39:09 <maaku> petertodd: thanks
574 2014-04-28 17:41:39 <hearn> oops, thought i pasted that. thanks.