1 2014-09-04 00:00:01 <uiop> Luke-Jr: if you're not passing it overlapping memory, https://sourceware.org/bugzilla/describecomponents.cgi?product=glibc :)
  2 2014-09-04 00:00:53 <Luke-Jr> uiop: No, it segfaults 100% of the time
  3 2014-09-04 00:01:42 <Luke-Jr> uiop: I'd report it, but I don't really have time to investigate *why*
  4 2014-09-04 00:02:07 <Luke-Jr> I presume the fact that nobody else has reported it means it only affects x86_32
  5 2014-09-04 00:02:29 <uiop> Luke-Jr: i'd be very surprised if this is a glibc bug
  6 2014-09-04 00:02:35 <Luke-Jr> (although IIRC I did see a Qt bugreport on a similar nature)
  7 2014-09-04 00:02:51 <Luke-Jr> uiop: it's trying to use SSSE3 when it segfaults
  8 2014-09-04 00:02:55 <uiop> Luke-Jr: oh, are you sure it wasn't an "illegal instruction" bug
  9 2014-09-04 00:03:00 <Luke-Jr> yes
 10 2014-09-04 00:03:11 <Luke-Jr> either Haswell's SSSE3 is broken, or glibc is using it in an unsupported way
 11 2014-09-04 00:03:16 <Luke-Jr> (maybe unaligned?)
 12 2014-09-04 00:04:29 <Luke-Jr> … or there's some obscure POSIX rule requiring memcpy to be aligned, but I find that unlikely O.o\
 13 2014-09-04 00:05:04 <uiop> ah, possibly (movdqa/etc assume aligned, but can't recall eventual signal linux translates that into). but not sure how that would happen differently on i386 than x86_64
 14 2014-09-04 00:05:46 <uiop> probably some combination-of-code-on-a-multiarch-system-resulting-in-ridiculousness-segfault or something
 15 2014-09-04 00:05:59 <Luke-Jr> uiop: I suspect the stack is "more" aligned on x86_64 ABI
 16 2014-09-04 00:06:07 <Luke-Jr> my system isn't multiarch
 17 2014-09-04 00:06:20 <Luke-Jr> well, except the kernel - that's 64-bit
 18 2014-09-04 00:06:39 <uiop> that sounds like a not un-plausible theory
 19 2014-09-04 00:06:59 <sipa> kdomanski_: https://github.com/sipa/bitcoin/tree/fastmap
 20 2014-09-04 00:07:15 <sipa> kdomanski_: replaces mapBlockIndex with an unordered map, and uses memcmp for equality/inequality
 21 2014-09-04 00:08:07 <gmaxwell> sipa: sorry, I should have just given you my patch for that, would have saved you some typing in changing all those typedefs.
 22 2014-09-04 00:09:09 <sipa> gmaxwell: it's 2am; what else would i be doing now? :p
 23 2014-09-04 00:10:29 <jrick> I thought std::unordered_map was a thing? why use boost?
 24 2014-09-04 00:10:43 <sipa> jrick: c++11
 25 2014-09-04 00:10:44 <gmaxwell> jrick: it's a C++11 (or 14?) thing.
 26 2014-09-04 00:10:48 <jrick> oh
 27 2014-09-04 00:11:04 <gmaxwell> the boost thing is functionally equal, so it can be changed directly when the codebase someday becomes C++11.
 28 2014-09-04 00:11:23 <sipa> the boost implementation was even changed to the std one, once it was standardized
 29 2014-09-04 00:11:55 <jrick> libstdc++?
 30 2014-09-04 00:12:22 <sipa> libstdc++ is GCC's implementation of the C++ standard library
 31 2014-09-04 00:12:31 <jrick> right
 32 2014-09-04 00:12:34 <sipa> or is it apple's?
 33 2014-09-04 00:12:42 <jrick> that's libc++ iirc
 34 2014-09-04 00:12:49 <sipa> right
 35 2014-09-04 00:14:07 <phantomcircuit> gmaxwell, didn't you say the unordered map didn't really help?
 36 2014-09-04 00:14:45 <sipa> not for memory usage; and likely not much for performance either
 37 2014-09-04 00:14:52 <sipa> but hey; the code becomes cleaner too
 38 2014-09-04 00:15:05 <gmaxwell> You can .reserve now too.
 39 2014-09-04 00:15:51 <sipa> won't do that much, i think
 40 2014-09-04 00:16:00 <gmaxwell> didn't.
 41 2014-09-04 00:16:24 <sipa> as that only allocates the actual hashmap, not the actual entries (which are independently allocated)
 42 2014-09-04 00:16:37 <sipa> and certainly not the CBlockIndex objects
 43 2014-09-04 00:16:55 <gmaxwell> sipa: I thought about making mapBlockIndex an unordered map of CBlockIndex instead of CBlockIndex* (thus avoiding a bunch of randomly timed allocations, perhaps) but it required too many changes to justify for just an expirement.
 44 2014-09-04 00:17:18 <gmaxwell> ah, scratch the "avoiding" part too then I guess!
 45 2014-09-04 00:17:20 <sipa> gmaxwell: i've been wanting to do that too; if anything, it would save us a few MB of malloc overhead
 46 2014-09-04 00:17:25 <kdomanski_> sipa: I'd change <string.h> to <cstring> and use memcpy on operator{<,>,>=} as well
 47 2014-09-04 00:17:32 <sipa> kdomanski_: can't
 48 2014-09-04 00:17:38 <kdomanski_> huh?
 49 2014-09-04 00:17:41 <gmaxwell> sipa: yep. Also fragmentation.
 50 2014-09-04 00:17:56 <kdomanski_> sipa: which part?
 51 2014-09-04 00:18:06 <sipa> kdomanski_: memcmp does "big endian" comparison
 52 2014-09-04 00:18:20 <gmaxwell> kdomanski_: memcmp is byte oritended. The numbers are LE bigints.
 53 2014-09-04 00:18:27 <sipa> kdomanski_: uint256's operator< _must_ obey to mathematical integer ordering for PoW
 54 2014-09-04 00:18:29 <gmaxwell> for POW the order matters greatly.
 55 2014-09-04 00:18:49 <kdomanski_> sipa: oh, I thought they were BE
 56 2014-09-04 00:18:57 <sipa> kdomanski_: x86 is LE
 57 2014-09-04 00:19:21 <sipa> and the uint32_t inside uint256 are also LE ordered
 58 2014-09-04 00:19:33 <gmaxwell> I suppose we could swizzle the values in memory so memcmp worked.
 59 2014-09-04 00:20:01 <sipa> that would break the ability to do fast arithmetic
 60 2014-09-04 00:20:04 <phantomcircuit> gmaxwell, plz2not
 61 2014-09-04 00:20:14 <phantomcircuit> :P
 62 2014-09-04 00:20:56 <gmaxwell> sipa: but we don't do any arithmetic on them outside of POW checking, IIRC.
 63 2014-09-04 00:21:24 <sipa> gmaxwell: i believe that
 64 2014-09-04 00:21:37 <sipa> actually no; we also use it for comparing chainwork
 65 2014-09-04 00:21:44 <kdomanski_> gmaxwell: I don't think the benefit is worth messing with it
 66 2014-09-04 00:21:51 <gmaxwell> chainwork compares to itself, not block hashes.
 67 2014-09-04 00:22:26 <sipa> gmaxwell: true, but chainwork also needs arithmetic compatible with that comparison
 68 2014-09-04 00:23:02 <gmaxwell> true enough.
 69 2014-09-04 00:26:10 <sipa> kdomanski_: github's input sanitization is funny
 70 2014-09-04 00:26:23 <sipa> i get an email from you with "How about instead?"
 71 2014-09-04 00:29:02 <kdomanski_> that's because i wrote "How about <cstring> instead?" instead of "How about \<cstring> instead?"
 72 2014-09-04 00:29:23 <sipa> ah, you actually can escape it?
 73 2014-09-04 00:29:27 <kdomanski_> then I edited
 74 2014-09-04 00:29:29 <teward> sipa, yeah you can
 75 2014-09-04 00:31:18 <uiop> if you guys are talking about comparing arrays of uint256's, that would be an excellent candidate for custom sse. <sketch>could use pcmpgtX/similar, movmaskY, then something-appropriate</sketch>
 76 2014-09-04 00:32:00 <uiop> could completely eliminate all branches (other than the loop backedge)
 77 2014-09-04 00:32:34 <gmaxwell> I don't think we ever actually do that.
 78 2014-09-04 00:32:37 <uiop> *that is, eliminate the "if(..) break;" equivalent part of loop
 79 2014-09-04 00:32:48 <uiop> gmaxwell: ah, too bad
 80 2014-09-04 00:34:03 <uiop> **err, you could elim *some* branch, though i can't quite pin down a generic all-applicable way to describe which
 81 2014-09-04 00:36:55 <uiop> ***oh, right. scratch that. the gain wouldn't be elim of a branch (you can already only have one in this case), what you would gain would be doing N=4 (or 2 or 8, depending) comparisons in parallel, thus needing current_num_iterations/N to do comparison (which gmaxwell points out, you don't actually do anyways, so this is just me being anal :)
 82 2014-09-04 00:37:48 <uiop> (so roughly an Nx speedup)
 83 2014-09-04 00:40:01 <uiop> this sort of ninja sse comparison code is currently beyond gcc's abilities (which is not that surprising actually, since to achieve it gcc would need to transform NON-sse-like-branchless-comparison-code into it, which seems pretty nontrivial, even at second or third glance)
 84 2014-09-04 00:44:53 <phantomcircuit> gmaxwell, github's caching is overly aggressive
 85 2014-09-04 00:45:08 <phantomcircuit> afaict they blindly set cache timeout values on assets which should be using etags
 86 2014-09-04 01:34:36 <lewellyn> screw noriega. mussolini was more effective.
 87 2014-09-04 01:36:14 <lewellyn> Luke-Jr: problem with what?
 88 2014-09-04 01:36:31 <Luke-Jr> lewellyn: anything around the licensing
 89 2014-09-04 02:18:04 <lewellyn> Luke-Jr: what's the reference for that highlight? i lack context?
 90 2014-09-04 05:30:11 <earlz> If you have a signed transaction.. Will the hash of the transaction be the same in block X, if it later is orphaned and appears in block Y?
 91 2014-09-04 05:32:00 <Luke-Jr> not necessarily
 92 2014-09-04 05:32:11 <Luke-Jr> it might not be the same in block X
 93 2014-09-04 05:32:28 <earlz> Well, I mean is it /possible/ for the hash to be the same
 94 2014-09-04 05:33:01 <Luke-Jr> sure
 95 2014-09-04 05:33:05 <Luke-Jr> often is
 96 2014-09-04 05:33:36 <earlz> ok.. I still don't completely understand transaction malleability heh
 97 2014-09-04 05:34:13 <Luke-Jr> earlz: basically any transaction can be modified without making it invalid, so long as the recipients remain the same
 98 2014-09-04 05:34:32 <Luke-Jr> earlz: mining it just locks it to a specific hash in that blockchain
 99 2014-09-04 05:35:34 <earlz> so what exactly is signed then
100 2014-09-04 05:37:34 <gmaxwell> earlz: on a normal transaction the whole transaction is signed, but obviously the signature itself cannot be signed.
101 2014-09-04 05:41:27 <earlz> but how would the signature be modified? Like what can be changed in a transaction while keeping the singature valid?
102 2014-09-04 05:42:58 <Luke-Jr> earlz: think numbers. add leading zeros.
103 2014-09-04 05:43:07 <Luke-Jr> that's one obvious way
104 2014-09-04 05:47:35 <earlz> mmm..
105 2014-09-04 05:47:49 <earlz> Everything has to keep the same semantics then at least
106 2014-09-04 05:47:57 <gmaxwell> yes, does exactly the same thing.
107 2014-09-04 05:51:10 <Luke-Jr> bitcoin would be so much easier if we just used the hash of the scriptPubKey as the input :P
108 2014-09-04 05:51:15 <Luke-Jr> (and txid)
109 2014-09-04 05:53:23 <gmaxwell> Luke-Jr: can't prevent duplicate transaction ids if you do that.
110 2014-09-04 05:53:39 <Luke-Jr> gmaxwell: prevent why? ;)
111 2014-09-04 05:54:09 <gmaxwell> man, and some people think mutability screws up accounting, ... good luck with duplicate txids!
112 2014-09-04 05:54:09 <Luke-Jr> I guess there's some DoSes that might be possible
113 2014-09-04 05:54:27 <Luke-Jr> gmaxwell: no duplicate txids if addresses are used once ;)
114 2014-09-04 09:24:14 <aphorise> */*-
115 2014-09-04 09:35:31 <fcw> Hello. Somebody can tell me where I can put a bugreport for Bitcoin Core 0.9.2.1?
116 2014-09-04 09:36:04 <fcw> It is not a severe bug, only a annoying bug.
117 2014-09-04 09:36:19 <gmaxwell> fcw: https://github.com/bitcoin/bitcoin/issues/
118 2014-09-04 09:36:31 <fcw> Thank
119 2014-09-04 13:59:38 <jgarzik> http://rocksdb.org/
120 2014-09-04 13:59:59 <jgarzik> Supposedly this is faster than leveldb, for datasets that do not fit 100% in memory
121 2014-09-04 14:00:34 <jgarzik> I imagine most of our database fits in page cache...
122 2014-09-04 14:05:16 <kinlo> created by facebook, I assume they know what they are talking about....
123 2014-09-04 14:56:15 <wumpus> jgarzik: bitcoin's database usage is pretty atypical in that everything that is read is usually written too, and it has large batches... but if it's not too hard to plug it in, it'd be worth a benchmark of course
124 2014-09-04 14:57:47 <jgarzik> wumpus, "everything that is read is usually writte too" -- can you elaborate?
125 2014-09-04 14:58:24 <wumpus> well in the utxo set, an entry is read and usually marked as spent and written again
126 2014-09-04 14:59:06 <jgarzik> wumpus, sure, but the database is hit (read) a few times to determine spendability, before that write
127 2014-09-04 15:00:23 <wumpus> but entries that are read before eventually being written again are rare
128 2014-09-04 15:00:54 <wumpus> of course it is read first, then it is cached, and eventually flushed back in a write
129 2014-09-04 15:01:09 <sipa> there are 3 layers: disk, leveldb's cache, application cache
130 2014-09-04 15:01:21 <wumpus> the block database has different behavior, but is much less interesting here
131 2014-09-04 15:01:56 <sipa> the block database is write-only
132 2014-09-04 15:02:06 <sipa> except when you use txindex
133 2014-09-04 15:02:11 <wumpus> right
134 2014-09-04 15:04:20 <wumpus> but entries that are read before eventually being written again are rare <- ehh, I mean, entries that are read, but *not* written back eventually
135 2014-09-04 15:04:51 <sipa> inputs for mempool transactions that don't confirm
136 2014-09-04 15:04:54 <wumpus> this is also the mistake I made when implementing a LRU read cache in CCoinsView.... a read cache is useless
137 2014-09-04 15:05:04 <sipa> wumpus: no, it's not useless
138 2014-09-04 15:05:05 <wumpus> (well, mostly)
139 2014-09-04 15:05:13 <sipa> you just don't see its effect in a reindex test
140 2014-09-04 15:05:22 <sipa> but it's certainly relevant for a block relay speed test
141 2014-09-04 15:05:45 <sipa> which is arguably more important for the network, actually - just harder to benchmark
142 2014-09-04 15:05:57 <wumpus> well I mean at the application level; the disk cache at the OS level and the cache at the DB level takes care there too
143 2014-09-04 15:06:43 <sipa> if it has to hit (spinning) disk, you lose 10ms at least
144 2014-09-04 15:06:45 <wumpus> even for block relay I'm not so sure; we'll have seen all those transactions previously, so they'll probably be still in the write cache
145 2014-09-04 15:07:14 <wumpus> (as it's implemented right now everything ends up in the write cache)
146 2014-09-04 15:07:28 <sipa> well yes, there only exists one cache
147 2014-09-04 15:07:38 <sipa> but as you implemented it, those would be in the read cache
148 2014-09-04 15:07:47 <wumpus> anyhow, it may be that it would have helped in some other benchmark
149 2014-09-04 15:09:42 <wumpus> what bothered me about the current implementation is that if you read an utxo entry just to look at it, it is still written back in the next flush, even if it didn't change... but it seems pretty ok  for the behavior we really have
150 2014-09-04 15:10:14 <wumpus> (something like a getutxo P2P command may change that, though... thinking about it)
151 2014-09-04 15:10:43 <sipa> wumpus: #4834 fixes that too (in a simpler way), but i don't understand why it fails pulltester
152 2014-09-04 15:16:21 <wumpus> not sure it's much simpler than what I did :)
153 2014-09-04 15:16:57 <sipa> oh, i don't mean the code; it does more too (removing spent unwritten entries from the cache)
154 2014-09-04 15:17:03 <wumpus> can't see anything obviously wrong with it though
155 2014-09-04 15:17:14 <sipa> but conceptually, just keeping a dirty flag rather than having two independent caches
156 2014-09-04 15:18:50 <wumpus> makes sense as long as the expiration policy for either is the same
157 2014-09-04 15:19:52 <sipa> heh; it's still "dump everything when grown too large" - i'm just trying to optimize memory usage here, not performance
158 2014-09-04 15:20:04 <sipa> s/dump/drop/
159 2014-09-04 15:20:41 <wumpus> unknown location(0): fatal error in "CreateNewBlock_validity": memory access violation at address: 0xffffffe1: no mapping at fault address    .. it's not even a "normal" null derefence
160 2014-09-04 15:21:25 <sipa> that looks like a 32-bit address
161 2014-09-04 15:22:06 <hearn> subtraction from null to find metadata before the block, perhaps
162 2014-09-04 15:22:42 <wumpus> or a dangling pointer, or use-after-free, ... C++ sucks :p
163 2014-09-04 15:23:19 <hearn> have you ever considered switching memory management to use the boehm gc?
164 2014-09-04 15:23:28 <wumpus> no
165 2014-09-04 15:23:31 <hearn> it's something i was thinking about lately as a way to reduce risk of changes.
166 2014-09-04 15:23:42 <hearn> i.e. eliminate memory leaks and double frees, use after frees
167 2014-09-04 15:23:50 <sipa> i think we've had surprisingly few memory related problems in bitcoind, actually
168 2014-09-04 15:23:57 <sipa> compared to my experience with C projects before
169 2014-09-04 15:24:00 <wumpus> sipa: agreed
170 2014-09-04 15:24:02 <hearn> yes because the way satoshi wrote it, there's hardly any dynamic memory allocation
171 2014-09-04 15:24:06 <hearn> it's all RAII
172 2014-09-04 15:24:18 <hearn> unfortunately as the code becomes more optimised and multi-threaded that becomes harder to maintain.
173 2014-09-04 15:24:26 <wumpus> bitcoind has less strange memory issues than any C/C++ project I've worked on before, so at least that's on the right track
174 2014-09-04 15:24:28 <hearn> introducing threads, caches, etc, all introduce the risk of problems.
175 2014-09-04 15:25:18 <hearn> the good thing is, because it's written in that "modern" c++ style , load on a GCd heap would be very low
176 2014-09-04 15:25:46 <wumpus> I really wouldn't want to add a dependency on a GC to bitcoind
177 2014-09-04 15:25:48 <sipa> i don't like GCs because they introduce runtime indeterminism
178 2014-09-04 15:26:01 <hearn> indeterminism? you mean in terms of time?
179 2014-09-04 15:26:14 <timothy> garbage collectors are useless if you know how to code :)
180 2014-09-04 15:26:24 <sipa> latency when the GC runs, memory spikes, ...
181 2014-09-04 15:26:31 <wumpus> if we go that far, I'd rather just switch to a garbage collected language like Java or Go :-)
182 2014-09-04 15:26:32 <hearn> malloc introduces random latency as well
183 2014-09-04 15:26:40 <sipa> fair enough
184 2014-09-04 15:26:52 <hearn> if you want hard-real time performance, i don't think we can get that in an app like bitcoind
185 2014-09-04 15:26:52 <wumpus> timothy: I hate statements like that, really
186 2014-09-04 15:27:04 <wumpus> 'C is useless if you know how to write assembly!' :p
187 2014-09-04 15:27:08 <sipa> C++ is also useless when you know assebly
188 2014-09-04 15:27:11 <sipa> lol
189 2014-09-04 15:27:11 <wumpus> HAH
190 2014-09-04 15:27:14 <hearn> timothy: and airbags are useless if you know how to drive, hard hats are useless if you remember to look upwards, etc ...
191 2014-09-04 15:27:27 <timothy> lol
192 2014-09-04 15:27:51 <wumpus> only people with too much time on their hands AND overestimate their own preciseness of working think like that
193 2014-09-04 15:27:55 <hearn> smart programmers find every way to eliminate entire classes of errors when possible. that's basically how we make progress ... find ways to eliminate groups of bugs at once, this lets us build more complicated software, and up we go
194 2014-09-04 15:28:05 <daybyter> wumpus: you use java for bitcoin stuff?
195 2014-09-04 15:28:19 <hearn> the nice thing about the boehm GC is that it's drop in for C++ and you can keep the delete statements, if you like.
196 2014-09-04 15:28:23 <wumpus> daybyter: not me, but many people do
197 2014-09-04 15:28:24 <hearn> or remove them
198 2014-09-04 15:28:30 <daybyter> I do...
199 2014-09-04 15:28:35 <hearn> however it does not require much in the way of code changes.
200 2014-09-04 15:28:55 <hearn> quite a few large, well known programs use GC'd C++ so it's not as bizarre as it sounds
201 2014-09-04 15:29:09 <hearn> for example the Unreal Engine uses a GCd core heap, even though it's a game with hard latency requirements (gotta hit that frame rate)
202 2014-09-04 15:29:12 <hearn> Inkscape as well
203 2014-09-04 15:29:50 <hearn> there's another flip side - GC can make some kinds of concurrent data structures a lot easier to write.
204 2014-09-04 15:30:18 <timothy> btw I don't like the usage of garbage collectors in critical/security code
205 2014-09-04 15:30:21 <wumpus> hearn, are you trying to sell us something? :P
206 2014-09-04 15:30:51 <hearn> i'm interested in ways to drive down the risk of code changes, as that's kind of paralysing the project at the moment. memory management vulnerabilities is obviously one problem that's plagued C++ apps since forever.
207 2014-09-04 15:31:10 <wumpus> timothy:  I don't see what security has to do with them?
208 2014-09-04 15:31:15 <hearn> bitcoin has done pretty well, but it's inevitable that at some point a double free or use after free somewhere
209 2014-09-04 15:31:32 <hearn> timothy: it's the opposite, GC makes code more secure. just go look at how many exploits are based on memory management bugs.
210 2014-09-04 15:31:46 <wumpus> timothy: for hard real-time code I'd agree, but bitcoin is far from that
211 2014-09-04 15:31:52 <hearn> GC + automatic bounds checking are the best things that ever happened for computer security, by far.
212 2014-09-04 15:32:06 <wumpus> timothy: I wouldn't want my pacemaker to go to sleep for a few minutes to sweep memory :P
213 2014-09-04 15:32:08 <timothy> and what about boehm GC bugs? :P
214 2014-09-04 15:32:11 <sipa> wumpus: not real-time code, but we do have unusual resource constraints
215 2014-09-04 15:32:33 <timothy> you have to trust a third-party library
216 2014-09-04 15:32:36 <hearn> it's a rather old and mature codebase. i haven't heard of boehm GC bugs ever introducing exploits.
217 2014-09-04 15:32:40 <sipa> timothy: we have to trust g++ too
218 2014-09-04 15:32:49 <sipa> and the linux kernel
219 2014-09-04 15:32:49 <timothy> and openssl, infact ...
220 2014-09-04 15:32:54 <timothy> :P
221 2014-09-04 15:32:56 <sipa> don't mention openssl
222 2014-09-04 15:33:15 <timothy> more code means more bug, imho
223 2014-09-04 15:33:30 <hearn> the one possibility for problems is that gc in c++ is conservative. if you can create an integer that looks like a pointer, it can pin that object in memory
224 2014-09-04 15:33:34 <sipa> timothy: that's a very simplistic view
225 2014-09-04 15:33:43 <hearn> on 64 bit systems with randomized heaps that's probably very difficult. on 32 bit without aslr not sure ...
226 2014-09-04 15:34:52 <wumpus> hearn: that's what has always scared me about GC in languages without pointer tagging
227 2014-09-04 15:35:17 <sipa> boehm even avoids memory locations to which an integer-that-looks-like-a-pointer points, afaik
228 2014-09-04 15:35:22 <hearn> in practice it doesn't seem to cause problems. i mean, i've never heard of this actually causing problems in real apps because the list of conditions needed to get a false positive is pretty high
229 2014-09-04 15:35:43 <hearn> it has to be within the heap, it has to be aligned, it has to point to an actual allocated piece of data, etc
230 2014-09-04 15:36:20 <hearn> integers that look the same as heap addresses are rare, in most apps. for bitcoin, the only risk i can see is that someone manages to create a protocol message that contains a pointer-like item ... but again, modern systems should all be randomizing the address space
231 2014-09-04 15:36:21 <wumpus> not that pinning an object is usually a very big problem, but it just feels wrong to me that similar-looking data can influence things
232 2014-09-04 15:36:38 <hearn> so it'd be really tough to do that reliably. and on 64 bit forget it
233 2014-09-04 15:36:39 <wumpus> especially if you have data that is controlled by external parties (ie read from disk or network)
234 2014-09-04 15:36:43 <sipa> compile-time garbage collection ftw!
235 2014-09-04 15:37:09 <hearn> i think it'd be worth it, at least to experiment.
236 2014-09-04 15:37:34 <wumpus> it makes it too much like voodoo... now we have this bit pattern, and even though we never interpret it as pointer, magically it influences the program!
237 2014-09-04 15:37:56 <hearn> you can also tell the GC not to scan certain areas.
238 2014-09-04 15:38:08 <wumpus> so IMO, if you want garbage collection, use a proper garbage collected language
239 2014-09-04 15:38:16 <hearn> for example it is quite simple to mark any data deserialized from the network as non-scanned.
240 2014-09-04 15:40:00 <wumpus> (microsoft actually tried C++ with pointer tagging, as 'managed C++', no idea where that went though, not that we can use it anyway...)
241 2014-09-04 15:41:15 <hearn> it was quite successful
242 2014-09-04 15:41:25 <hearn> (on windows)
243 2014-09-04 15:42:04 <wumpus> hearn: sure, it's interesting and if you want to experiment with it, you're welcome, but unless it really solves a problem that we have I don't see it being merged any time soon
244 2014-09-04 15:43:14 <wumpus> and if it allows removing say half of the code it becomes interesting :-)
245 2014-09-04 15:43:24 <jrick> adding a gc to c++ doesn't instantly solve all your dangling pointers
246 2014-09-04 15:43:25 <wumpus> but we already have a lot of automated memory management using RAII
247 2014-09-04 15:43:58 <hearn> i see it more as preventing problems we may have but don't know about yet, or may introduce in future.
248 2014-09-04 15:44:11 <hearn> after all this conversation started because you are debugging some kind of memory management problem
249 2014-09-04 15:44:13 <sipa> i think it will more cause us to not problems :)
250 2014-09-04 15:44:19 <sipa> *see
251 2014-09-04 15:45:26 <wumpus> hearn: yes, but it's the first time in ages that this happens, and it's in unmerged, hardly tested and mostly unreviewed code
252 2014-09-04 15:46:05 <wumpus> it's not as we spent hours per day chasing obscure memory issues, at least that we don't have in common with the kernel
253 2014-09-04 15:46:30 <wumpus> jrick: a gc will do nothing against dangling pointers
254 2014-09-04 15:47:05 <jrick> right, you need escape analysis
255 2014-09-04 15:47:39 <sipa> we hardly do manual new/free anyway
256 2014-09-04 15:47:44 <wumpus> capability-based addresing :)
257 2014-09-04 15:49:07 <hearn> GC is supposed to prevent dangling pointers
258 2014-09-04 15:49:14 <hearn> unless you're using the term differently to how i usually see it used
259 2014-09-04 15:49:30 <jrick> hearn: only if the language performs escape analysis and punts the pointed to object to the heap
260 2014-09-04 15:49:38 <hearn> because by definition you cannot have a dangling pointer - the GC will not free the memory it points to until it becomes unreachable.
261 2014-09-04 15:49:49 <hearn> what are you talking about? escape analysis has nothing to do with GC
262 2014-09-04 15:50:10 <hearn> you're talking about pointers onto the stack, here?
263 2014-09-04 15:50:15 <jrick> yes
264 2014-09-04 15:50:18 <hearn> where someone returns a pointer to their local stack frame and then returns?
265 2014-09-04 15:50:22 <jrick> sure
266 2014-09-04 15:50:38 <hearn> yes, but modern compilers warn about that or treat it as an error. that's not a mistake that's easy to make at all.
267 2014-09-04 15:51:11 <hearn> (well not in c++ anyway)
268 2014-09-04 15:51:13 <wumpus> hearn: compilers aren't very smart in that, for example try returning a pointer to part of an object on the stack
269 2014-09-04 15:52:06 <jrick> rust fixes this without needing a gc by adding lifetimes to the type system
270 2014-09-04 15:52:09 <wumpus> or assigning a pointer to an object on the stack to something passed in, instead of returning it, c++ has lots of ways to shoot yourself in the foot
271 2014-09-04 15:53:08 <hearn> yes, pointers are risky.
272 2014-09-04 15:53:31 <daybyter> 
ACTION likes java....no pointers...

273 2014-09-04 15:53:59 <hearn> java is full of pointers. you just aren't allowed to do arithmetic with them
274 2014-09-04 15:54:07 <hearn> along with other misc restrictions that make them safe
275 2014-09-04 15:54:22 <wumpus> right, you're not allowed to use them as arrays, that helps a lot :-)
276 2014-09-04 15:54:34 <hearn> they also always point into the heap
277 2014-09-04 15:54:43 <hearn> or rather, if they don't, that's hidden from the programmer
278 2014-09-04 15:54:50 <hearn> (modern jvms do stack allocation)
279 2014-09-04 15:54:55 <daybyter> java is good for old senile coders...
280 2014-09-04 15:55:30 <sipa> please
281 2014-09-04 15:55:41 <jrick> http://is.gd/oBvs4q <- attempting a dangling pointer in rust
282 2014-09-04 15:56:25 <wumpus> hehe <anon>:7:14: 7:15 error: `i` does not live long enough
283 2014-09-04 15:56:41 <wumpus> is that watertight? or can you still cheat it into causing segmentation faults?
284 2014-09-04 15:57:03 <jrick> you can cheat it with an unsafe {} block
285 2014-09-04 15:57:44 <wumpus> at least you have to use the word unsafe
286 2014-09-04 15:59:23 <wumpus> so you know where to concentrate code review
287 2014-09-04 15:59:29 <hearn> interestingly you *can* do unsafe memory management in java. there's a class called Unsafe that lets you do raw memcpys, atomic compare/sets, manual allocation of memory blocks etc. but i don't know many apps that use it.
288 2014-09-04 16:00:35 <wumpus> and when you've fixed memory issues.. there's still integer overflows as a source of unexpected behavior
289 2014-09-04 16:01:25 <hearn> yes. i have been trying to convince the developers of the kotlin language to make all arithmetic checked by default. but that's not something you can easily fix in C++ given how much software relies on it
290 2014-09-04 16:03:33 <kdomanski_> sipa: your patch seems to have cut CompareTo calls in half
291 2014-09-04 16:03:51 <daybyter> my biggest problem is the BigDecimal syntax in java...looks ugly...
292 2014-09-04 16:12:29 <wumpus> sipa: hey, it passed now
293 2014-09-04 16:12:38 <sipa> wumpus: i removed the last commit, sadly
294 2014-09-04 16:12:46 <sipa> wanted to find out which commit broke things
295 2014-09-04 16:13:34 <wumpus> sipa: oh!
296 2014-09-04 16:14:00 <wumpus> sipa: do you still have the last commit available? I was going to test it on a 32-bit system
297 2014-09-04 16:16:19 <sipa> wumpus: just pushed it again
298 2014-09-04 16:17:13 <wumpus> looks like github immediately removes unreferenced commits, I still knew the commit id, but couldn't pull it before you pushed it again to a branch
299 2014-09-04 16:17:55 <sipa> i'm actually trying to run the tests myself on the jenkins machine now
300 2014-09-04 16:20:14 <wumpus> no errors detected...
301 2014-09-04 16:27:47 <sipa> ok, reproduced
302 2014-09-04 16:32:50 <sipa> seems like a corrupted unordered_map; it's inside ::find that it segfaults...
303 2014-09-04 16:33:04 <wumpus> I've tested on ubuntu 12.04 32 bit x86 and Debian GNU/Linux jessie/sid 32 bit arm, no issues in tests
304 2014-09-04 16:33:15 <sipa> seems pulltester is 10.04 64 bit
305 2014-09-04 16:33:25 <sipa> older boost, presumably
306 2014-09-04 16:33:34 <sipa> i hope we didn't hit another boost::unordered_map bug
307 2014-09-04 16:33:55 <wumpus> well on my 64-bit system I couldn't reproduce either
308 2014-09-04 16:34:06 <wumpus> I don't hope so either... :/
309 2014-09-04 16:59:29 <andytoshi> are there any wallets who have implemented RFC6979 deterministic sigs? i am curious what decisions they made for message hash and HMAC hash
310 2014-09-04 17:07:16 <andytoshi> ah, electrum did, they just used regular (not doubled) SHA256
311 2014-09-04 17:07:36 <andytoshi> and used the sha256d(msg) as the input to the deterministic nonce gen
312 2014-09-04 17:09:15 <gmaxwell> weird choices.
313 2014-09-04 17:09:49 <gmaxwell> oh I misread that, well not that weird.
314 2014-09-04 17:09:51 <andytoshi> well, most crypto libs don't have sha256d, and they want you to pass a hash fn to the deterministic sign function (it doesn't choose for you)
315 2014-09-04 17:10:09 <andytoshi> so either you implement their interface for your sha256d, or you just pass it the existing sha256
316 2014-09-04 17:13:39 <gmaxwell> while talking to sipa I think I'd expressed a preference for using HMAC-SHA512, in that any code implementing BIP32 has it anyways... though perhaps there is a complaint to be raised about truncating its output.
317 2014-09-04 17:15:37 <gmaxwell> the hmac-sha256^2 that the RFC arguably prefers you use is kinda excessively weird.
318 2014-09-04 17:16:08 <gmaxwell> And it's not a MUST.  It's actually important to be consistent in this, since we'd want to verify implementations are working right.
319 2014-09-04 17:16:19 <andytoshi> rust-crypto has a generic HMAC, i can use any hash i want :P. my feeling was HMAC-SHA512 too because of bip32
320 2014-09-04 17:16:28 <andytoshi> but if other wallets have developed a convention then i'll do that
321 2014-09-04 17:17:08 <andytoshi> truncation is in the RFC, it's assumed to happen because apparently a lot of EC groups have weird orders
322 2014-09-04 17:19:07 <andytoshi> i was going to make my nonce-generator generic (it is going in the rust secp256k1 bindings) but i can't genericize over the hash size and allocating is a nogo
323 2014-09-04 17:19:48 <gmaxwell> really this should move into libsecp256k1, the current interface is unsafe for mortals.
324 2014-09-04 17:20:05 <gmaxwell> It's not there yet due to the lack of internal hash functions.
325 2014-09-04 17:20:13 <andytoshi> the rust interface is much nicer :)
326 2014-09-04 17:25:17 <andytoshi> i think armory uses random nonces, the c++ code is tangley but it creates a prng and gives it to the signer object..
327 2014-09-04 17:27:00 <andytoshi> so my feeling is there is no convention, i should use HMAC-SHA512
328 2014-09-04 17:27:03 <andytoshi> sipa, gmaxwell: ^
329 2014-09-04 17:28:28 <sipa> sgtm
330 2014-09-04 18:11:09 <Eliel> gmaxwell: truncating hash function output is bad?
331 2014-09-04 18:27:15 <ma11011s> When people like O’Brien say things like: “Before we know it bitcoin is going to be critical to our lives the way the Internet or mobile communication is critical”. I hope this reaches all of you and realize you are doing a fantastic job. I am learning that bitcoin has huge potential and lights up this dark world. Thank you all.
332 2014-09-04 18:30:12 <cfields> michagogo: around?
333 2014-09-04 18:30:54 <sipa> wumpus: fixed!
334 2014-09-04 18:31:22 <sipa> wumpus: map1.find(...) != map2.end() ... seems to work on newer boost, but not on older boost
335 2014-09-04 18:33:50 <cfields> sipa: yikes!
336 2014-09-04 18:49:41 <michagogo> cfields: ish
337 2014-09-04 18:49:43 <michagogo> Why?
338 2014-09-04 18:50:01 <michagogo> (Going through a tunnel atm, messages may be delayed)
339 2014-09-04 18:51:25 <andytoshi> any ideas for how i can build a test corpus for RFC6979 nonces with secp256k1 and HMAC-SHA512?
340 2014-09-04 18:51:34 <andytoshi> i guess, openssl at the prompt..
341 2014-09-04 18:51:55 <cfields> michagogo: it'd be great if you could give #4727 a try now, I'm satisfied with it
342 2014-09-04 18:52:11 <andytoshi> grr, why doesn't the openssl prompt support variables
343 2014-09-04 18:52:25 <michagogo> cfields: link?
344 2014-09-04 18:52:42 <cfields> michagogo: https://github.com/bitcoin/bitcoin/pull/4727
345 2014-09-04 18:54:02 <michagogo> Thanks. Don't think I'll get around to it tonight (it's 10 PM and I'm not home yet), but I'll open it in a tab for when I can
346 2014-09-04 18:54:14 <cfields> np. thanks
347 2014-09-04 19:06:48 <cfields> sipa: are there any operand size constraints i can assume when implementing secp256k1_num_mul ?
348 2014-09-04 19:07:16 <sipa> cfields: inputs at most 256 bits; output at most 512 bits
349 2014-09-04 19:08:52 <cfields> sipa: thanks. I guess the more accurate question: any operations where a single input can be >256bit ?
350 2014-09-04 19:10:51 <sipa> cfields: yes, division
351 2014-09-04 19:10:55 <sipa> cfields: and shifts
352 2014-09-04 19:11:18 <sipa> though #21 removes division entirely...
353 2014-09-04 19:11:21 <sipa> 
ACTION is tempted to merge it

354 2014-09-04 19:12:07 <gmaxwell> should just figure out why it isn't faster. It should be faster.
355 2014-09-04 19:12:22 <sipa> yup
356 2014-09-04 19:12:42 <cfields> sipa: great, thanks.
357 2014-09-04 19:13:36 <kdomanski_> sipa: is there a specific compiler targeted on OSX?
358 2014-09-04 19:14:07 <sipa> kdomanski_: ask cfields
359 2014-09-04 19:14:17 <kdomanski_> cfields: is there a specific compiler targeted on OSX?
360 2014-09-04 19:15:30 <cfields> kdomanski_: any should work.
361 2014-09-04 19:16:02 <cfields> kdomanski_: releases use clang 3.2 and 10.7sdk, but they're actually cross-built from Linux
362 2014-09-04 19:20:27 <sipa> gmaxwell: it's strange that even the microbenchmark (his test which runs the split 10x instead of once) is only 1.8x times faster
363 2014-09-04 19:20:51 <sipa> i'd expected a few multiplications and shifts to be at least an order of magnitude less work than a division
364 2014-09-04 19:22:43 <melvster> anyone know, is redeemScript optional in a normal P2SH signrawtransaction command?
365 2014-09-04 19:23:29 <gmaxwell> melvster: IIRC it's only optional if the p2sh was addmultisiged.
366 2014-09-04 19:23:41 <melvster> gmaxwell: ah ok thanks
367 2014-09-04 19:25:30 <sipa> melvster: apparently, it's required if you specify your own private keys
368 2014-09-04 19:25:49 <sipa> or if your own wallet doesn't know about it, of course
369 2014-09-04 19:26:01 <melvster> sipa: ah right, yes I'm adding a privatekey array
370 2014-09-04 19:26:27 <melvster> and it normally looks like 76a914  XXXX 88ac?
371 2014-09-04 19:27:21 <gmaxwell> sipa: hm. this isn't the implementation I expected this to use, this is just an approximation. Is it increasing the weight of the split?
372 2014-09-04 19:28:00 <sipa> gmaxwell: i haven't checked the math; peter claimed that it is very close or equal to the previous implementation
373 2014-09-04 19:29:12 <sipa> melvster: that looks like a normal pay-to-pubkey-hash script
374 2014-09-04 19:29:43 <gmaxwell> exact division should be possible with less work given a modular multiply large enough to contain the whole range of numbers.
375 2014-09-04 19:31:26 <gmaxwell> Here is the citation, http://dl.acm.org/citation.cfm?id=1078059
376 2014-09-04 19:34:46 <kanzure> here's a copy http://libgen.org/scimag/get.php?doi=10.1109%2FARITH.2005.31
377 2014-09-04 19:34:54 <kanzure> (maybe)