1 2014-02-06 00:01:06 <jiffe98> hmm, genesis block seems to use some opcodes in a higher range than you defined there
2 2014-02-06 00:03:11 <andytoshi> jiffe98: yeah, sorry, i never got around to learning what the deal there is..
3 2014-02-06 00:04:07 <andytoshi> i think it may just be merged-mining gibberish
4 2014-02-06 00:26:38 <jiffe98> that's cool
5 2014-02-06 01:04:12 <skinnkavaj> sipa: Is bitmessage secure? Do you trust it?
6 2014-02-06 01:07:24 <sipa> i've never used it
7 2014-02-06 02:57:54 <jiffe98> is there anything special about the genesis block that it needs to be hard coded?
8 2014-02-06 02:59:27 <benkay> it'd be confusing to the network if you jumped on and said "the last block i've received is NULL"
9 2014-02-06 02:59:39 <benkay> but feel free to send a PR ;)
10 2014-02-06 03:06:04 <jiffe98> that's just a function of the peer communication
11 2014-02-06 03:06:43 <sipa> peers need to agree on where the block tree starts, in order to be able to find a path through it they agree upon
12 2014-02-06 03:07:09 <sipa> it's just as hardcoded as the rule that we don't want to spend coins without valid signature
13 2014-02-06 03:08:23 <jiffe98> well what if you have a checkpoint set for the genesis block but don't hard code the parameters of the block
14 2014-02-06 03:09:11 <sipa> jiffe98: ok, but what difference is that in practice
15 2014-02-06 03:09:27 <sipa> you're right - we could checkpoint the genesis block and fetch it from the network
16 2014-02-06 03:09:41 <sipa> but at least the genesis block had to be hardcoded in the first bitcoin node that ever ran
17 2014-02-06 03:09:48 <sipa> no reason to remove it
18 2014-02-06 03:09:58 <sipa> (while we are planning to remove checkpoints at some point...)
19 2014-02-06 03:10:24 <jiffe98> checkpoints not necessary anymore?
20 2014-02-06 03:10:31 <sipa> they are not necessary
21 2014-02-06 03:10:38 <sipa> they're a hack to avoid a dos attack
22 2014-02-06 03:11:01 <sipa> while synchronizing, if we disable signature checking, a node could be fed an invalid chain
23 2014-02-06 03:11:36 <sipa> the correct solution is first synchronizing the headers, so we know the best chain in advance, and only then start verifying it
24 2014-02-06 03:11:58 <sipa> and for example disable signature checking if they're one month worth of proof of work known to be on top of it
25 2014-02-06 03:12:09 <sipa> instead of relyong an centrally-set checkpoints in the source code
26 2014-02-06 03:13:28 <jiffe98> gotcha, yeah I always thought checkpointing kind of centralized the whole thing but thought it to be a necessary evil to keep things on track
27 2014-02-06 03:13:53 <sipa> if anything, you must understand that checkpoints are not a means to prevent reorganizing the chain
28 2014-02-06 03:14:09 <sipa> if they end up doing that, bitcoin's consensus system has already failed
29 2014-02-06 03:16:04 <jiffe98> makes sense
30 2014-02-06 03:16:20 <jiffe98> looks like a checkpoint was set not too long ago
31 2014-02-06 03:16:49 <sipa> we don't set checkpoints less than 2016 blocks ago
32 2014-02-06 06:24:05 <Lightsword> PSA https://bitcoinaverage.com/markets.htm#USD seems to have been compromised, do not use their API
33 2014-02-06 06:25:00 <jcorgan> Lightsword: details?
34 2014-02-06 06:25:10 <Lightsword> look at the bid price
35 2014-02-06 06:25:30 <Lightsword> was causing our calculations to be significantly off
36 2014-02-06 06:25:38 <jcorgan> what's the API URL
37 2014-02-06 06:25:54 <jcorgan> oh geez, i see it now
38 2014-02-06 06:26:15 <Lightsword> https://api.bitcoinaverage.com/ticker/USD/
39 2014-02-06 06:26:17 <Lightsword> yeah
40 2014-02-06 06:26:19 <jcorgan> you think it is a compromise and not just a screwup?
41 2014-02-06 06:26:39 <Lightsword> not sure, but it caused all of our prices to be way lower than they should have been
42 2014-02-06 06:26:50 <Lightsword> just happened this evening a few hours ago
43 2014-02-06 06:26:58 <Lightsword> we switched to a different API
44 2014-02-06 06:27:54 <Lightsword> we caught it before any damage was done
45 2014-02-06 06:29:29 <Lightsword> anyone have any idea how to contact their operator? I sent an email but have not heard back
46 2014-02-06 06:30:45 <Imbue> oh wow, that's interesting
47 2014-02-06 06:32:52 <jcorgan> i sure don't. i use the api personally for some stuff, but the last data has been good, so i didn't notice
48 2014-02-06 06:34:30 <Lightsword> jcorgan I'm not sure what happened but it seems not many have noticed, we just happened to use the bid price for calculations
49 2014-02-06 06:35:03 <jcorgan> good catch though, thanks for the PSA
50 2014-02-06 06:43:48 <cortexA9> hello
51 2014-02-06 06:45:38 <cortexA9> how work the bitcoin network?
52 2014-02-06 06:50:47 <cortexA9> *works
53 2014-02-06 06:55:04 <Apocalyptic> cortexA9, #bitcoin
54 2014-02-06 06:59:39 <Lightsword> what would be the best way to replace bitcoind's leveldb with postgres for a production server?
55 2014-02-06 07:00:01 <Lightsword> would I just need to replace the wrapper? https://github.com/bitcoin/bitcoin/blob/master/src/leveldbwrapper.cpp
56 2014-02-06 07:01:10 <Luke-Jr> Lightsword: two different things
57 2014-02-06 07:01:15 <Luke-Jr> LevelDB is a key/value store
58 2014-02-06 07:01:20 <Luke-Jr> PostgreSQL is a relational database
59 2014-02-06 07:03:16 <Lightsword> I would like to be able to perform rapid searches of the blockchain and since LevelDB doesn't have indexes certain things are very slow.
60 2014-02-06 07:04:19 <Luke-Jr> what kind of index is it missing?
61 2014-02-06 07:05:46 <Lightsword> it doesn't have any index support at all as far as I can tell
62 2014-02-06 07:06:07 <uiop> Luke-Jr: from http://code.google.com/p/leveldb/ : "Limitations: ... , and it has no support for indexes. ..."
63 2014-02-06 07:06:54 <Luke-Jr> it's a key/value store..
64 2014-02-06 07:07:08 <Lightsword> For a production server LevelDB is far too slow for searches, thats why I want to use postgres
65 2014-02-06 07:07:09 <Luke-Jr> it has whatever index you make
66 2014-02-06 07:07:44 <Luke-Jr> Lightsword: if something else could perform better, we'd be using it
67 2014-02-06 07:08:03 <sipa> depends on your use case
68 2014-02-06 07:08:14 <Lightsword> I thought LevelDB was just used since it is lightweight
69 2014-02-06 07:08:31 <Lightsword> and good enough for a single wallet
70 2014-02-06 07:08:38 <Lightsword> but it doesn't seem to scale
71 2014-02-06 07:08:38 <Luke-Jr> sipa: well, I suspect his "use case" itself may lack a use case ;p
72 2014-02-06 07:08:38 <petertodd> Lightsword: leveldb isn't for the wallet
73 2014-02-06 07:08:53 <sipa> we don't use leveldb for the wallet at all...
74 2014-02-06 07:08:54 <Luke-Jr> Lightsword: if your problem is wallet performance, that's entirely unrelated
75 2014-02-06 07:09:12 <Lightsword> isn't leveldb used to store the blockchain?
76 2014-02-06 07:09:19 <petertodd> Lightsword: yes, blockchain != wallet
77 2014-02-06 07:09:22 <uiop> Luke-Jr: SELECT * FROM blockchain WHERE (txin_count > 7 AND scriptype = 'p2sh') OR txin_0_addr IN (txout_0_addr,txout_0_addr);
78 2014-02-06 07:09:38 <uiop> e.g.
79 2014-02-06 07:09:47 <Luke-Jr> uiop: explain why that is useful.
80 2014-02-06 07:09:48 <Lightsword> by wallet I meant including the blockchain
81 2014-02-06 07:09:55 <Lightsword> bitcoin-qt
82 2014-02-06 07:10:23 <sipa> bitcoin-qt's wallet performance has nothing to do with its storage engine
83 2014-02-06 07:10:32 <sipa> it is admittedly slow
84 2014-02-06 07:10:54 <Lightsword> right, but I want I need to be able to search the blockchain
85 2014-02-06 07:10:54 <sipa> but that's because it recomoutes many things over andnover again
86 2014-02-06 07:10:58 <Lightsword> which is stored in leveldb
87 2014-02-06 07:11:00 <uiop> Luke-Jr: for ad-hoc queries of the blockchain, for e.g. visualization-with-graphviz software, blockchain statistic generation, datamining for not-necessarily "nefarious" purposes
88 2014-02-06 07:11:21 <petertodd> Lightsword: ok, then add more indexes
89 2014-02-06 07:11:45 <Lightsword> I want to just put it in a database that actually supports indexes natively
90 2014-02-06 07:11:53 <Lightsword> ex postgres
91 2014-02-06 07:12:10 <uiop> Luke-Jr: (i don't know what Lightsword's use case(s) are, just speaking to my own personal off-the-cuff use cases)
92 2014-02-06 07:12:11 <petertodd> Lightsword: really, you'll probably find that just making your own blockchain database serves your purposes better - just get the blocks from another node
93 2014-02-06 07:12:42 <sipa> yeah, bitcoind's storage is designed to be very fast for exactly what it does
94 2014-02-06 07:12:50 <lianj> Lightsword: fyi, blockchain in postgres is about 200GB with indexes
95 2014-02-06 07:13:00 <Lightsword> lianj thats not a problem
96 2014-02-06 07:13:09 <petertodd> lianj: what type of indexes do you have?
97 2014-02-06 07:13:41 <lianj> petertodd: lots. full tx/inputs/outputs/addresses
98 2014-02-06 07:13:53 <Lightsword> petertodd I want to make my own blockchain database using postgres
99 2014-02-06 07:14:10 <uiop> that's on my to-eventually-get-around-to list also
100 2014-02-06 07:14:21 <petertodd> lianj: huh, 200GB is a lot bigger than 15GB... didn't realize the indexes blew it up so much
101 2014-02-06 07:14:50 <Lightsword> for a production server 200GB is not a problem
102 2014-02-06 07:14:51 <uiop> good thing 200GB is small wrt modern disks
103 2014-02-06 07:15:04 <sipa> bitcoind + TTximndex + addrindex is only a few GB extra
104 2014-02-06 07:15:12 <sipa> txindex
105 2014-02-06 07:15:32 <alfer> so it appears the linux distro of the bitcoin client has statically linked binaries included - how can I build a statically linked version of bitcoind from the sources?
106 2014-02-06 07:15:39 <Lightsword> the closest I found to what I want to do is https://github.com/bitcoin-abe/bitcoin-abe but it still has to read leveldb, I want to outright replace leveldb
107 2014-02-06 07:16:36 <petertodd> Lightsword: light I said, just read blocks into your database - you can get them off the RPC interface - and do whatever you want with them in your own program. mucking about int he bitcoin core code won't do what you want
108 2014-02-06 07:16:37 <sipa> still has to read leveldb? what?
109 2014-02-06 07:16:41 <lianj> petertodd: yea, lets say 100-200GB :D depending on what additional indexes you might put in for exotic quieres that require to be fast
110 2014-02-06 07:17:32 <uiop> Lightsword: my plan was (is) to just dump the blockchain (w/ txindex) out of bitcoin-qt, parse it, xform to format suitable for input into postgres (COPY much faster than INSERT*10^alot)
111 2014-02-06 07:18:01 <uiop> it doesn't seem to be highly difficult
112 2014-02-06 07:18:09 <Luke-Jr> Lightsword: you're going to have to start from scratch I think (unless bitcoin-ruby has the code lianj uses..)
113 2014-02-06 07:18:21 <petertodd> uiop: you can do even better reading the blocks direct from the on-disk blockchain*.dat files - doing that is trivial
114 2014-02-06 07:18:35 <petertodd> uiop: I wrote some python code to do that in ~10min with pyton-bitcoinlib
115 2014-02-06 07:18:38 <sipa> or use the P2P protocol to read them
116 2014-02-06 07:18:57 <uiop> petertodd: oh nice, noted
117 2014-02-06 07:18:58 <Lightsword> it seems simpler to just replace the database controller in bitcoin-qt with one for postgres, how tied to leveldb is bitcoin-qt?
118 2014-02-06 07:19:00 <lianj> Luke-Jr: i don't use the storage part of bitcoin-ruby, but it has postgres too
119 2014-02-06 07:19:00 <petertodd> uiop: though keep in mind orphans are stored in blockchain*.dat too if your node saw them
120 2014-02-06 07:19:24 <Luke-Jr> Lightsword: you'd end up rewriting Bitcoin-Qt
121 2014-02-06 07:19:32 <petertodd> uiop: IIRC even python was doing something like 1MB/sec full serialization/deserialization roundtrip
122 2014-02-06 07:19:43 <lianj> like others said, just get the blocks and push the in the database how you want/need to
123 2014-02-06 07:20:26 <Luke-Jr> Lightsword: you can't just "unplug" a key/value db and "plug in" a relational db in its place..
124 2014-02-06 07:21:10 <Luke-Jr> well, you *could*, but you'd not gain anything from it, and lose a lot of performance..
125 2014-02-06 07:21:16 <Lightsword> any particular reason you can't? postgres supports key/value, it just also has additional functionality
126 2014-02-06 07:21:23 <petertodd> uiop: oh, and you know, you could do better with parallelism: a full index type thing can always have a "txout spent" index that's separate from what txouts actually exist :)
127 2014-02-06 07:21:47 <lianj> Lightsword: bitcoind doesn't store its blocks in leveldb
128 2014-02-06 07:21:47 <petertodd> Lightsword: touching consensus critical code when you don't have to is a sign of madness
129 2014-02-06 07:21:47 <sipa> Lightsword: you could use postgres to store what we're currently store in leveldb just fine
130 2014-02-06 07:22:14 <sipa> Lightsword: bjt it wouldn't give you any benefit, as we don't store things in a way you want them
131 2014-02-06 07:22:26 <sipa> worse, we don't store that data at all
132 2014-02-06 07:22:32 <uiop> petertodd: on a related note, "make -j24" never ceases to amaze me :)
133 2014-02-06 07:22:34 <petertodd> lianj: and bitcoind only appends to the blockchain files, so you can just store indexes and use those same files
134 2014-02-06 07:22:53 <lianj> yea, but he didn't want that
135 2014-02-06 07:22:53 <petertodd> uiop: lol, me too, and I've only got 8 cores :)
136 2014-02-06 07:23:14 <petertodd> uiop: raid1 intel ssds though, heh
137 2014-02-06 07:23:37 <Lightsword> basically the issue I want to avoid is having to keep to completely different databases in sync for production servers, thats why I want to have bitcoind directly use postgres
138 2014-02-06 07:23:40 <uiop> petertodd: ssds really did change the game
139 2014-02-06 07:23:48 <sipa> Lightsword: in leveldb we currently store block headers, disk positions of blocks, and for every txid its unspent outputs
140 2014-02-06 07:23:54 <Luke-Jr> make -j 200 is always fun..
141 2014-02-06 07:24:04 <sipa> Lightsword: all in custom-designed compact formats
142 2014-02-06 07:24:37 <petertodd> Lightsword: to get reorgs right on your postgres implementation within bitcoind would take more work than doing a implementation separate from bitcoind
143 2014-02-06 07:24:38 <sipa> Lightsword: and optionally, for every transaction where it is on disk
144 2014-02-06 07:25:01 <uiop> ACTION just realizes Lightsword wants to replace leveldb w/ postgres *IN* bitcoind ..
145 2014-02-06 07:25:37 <sipa> Lightsword: notice in particular that nothing in there has even remotely anything to do with addresses
146 2014-02-06 07:26:08 <sipa> so no postgres index will help you an address-related query
147 2014-02-06 07:26:37 <sipa> uiop: yeah
148 2014-02-06 07:27:17 <Lightsword> well, I would be creating additional tables, I just can't figure out a proper way to deploy leveldb
149 2014-02-06 07:27:54 <Luke-Jr> Lightsword: addresses don't exist in the blockchain.
150 2014-02-06 07:28:20 <Luke-Jr> they're merely an abstraction
151 2014-02-06 07:28:30 <sipa> and as bitcoind's database is essential for its validation, if you'd make any mistake in how to deal with reorganizationa wrt your database, you risk rejecting valid blocks or the other way around
152 2014-02-06 07:28:46 <Lightsword> well yeah, I would have a table for that which relates to the blockchain I think
153 2014-02-06 07:29:02 <Luke-Jr> Lightsword: so just make that table independent from bitcoind
154 2014-02-06 07:29:32 <Lightsword> but then I have to deploy 2 databases which is difficult since there aren't really sysops tools for leveldb from what I can tell
155 2014-02-06 07:29:39 <Luke-Jr> â¦
156 2014-02-06 07:30:02 <sipa> the main reason is that leveldb is not a database server
157 2014-02-06 07:30:11 <Luke-Jr> compared to what you're talking about, deploying 2 databases is TOTALLY TRIVIAL
158 2014-02-06 07:30:13 <sipa> it's a library to maintain a key value store
159 2014-02-06 07:30:30 <sipa> it doesn't give you a database yoiu can access from different processes
160 2014-02-06 07:31:39 <sipa> bitcoind's database is designed very specifically for what bitcoind needs, and is not intended to be used by anything else
161 2014-02-06 07:31:47 <petertodd> sipa: oh, leveldb can't be accessed with multiple threads?
162 2014-02-06 07:31:48 <Lightsword> hmm, well what would be the best way to replicate and sync the 2 databases in a deployment?
163 2014-02-06 07:32:01 <sipa> petertodd: threads yes, processes no
164 2014-02-06 07:32:01 <uiop> Lightsword: my off-the-cuff idea based on an incomplete idea of your use case is to mirror the db in postgres, and sync it at the desired rate
165 2014-02-06 07:32:17 <petertodd> Lightsword: you don't have to "sync" them, just keep track of what block headers are current in your thing, and handle reorgs yourself
166 2014-02-06 07:32:37 <petertodd> Lightsword: I've done this myself - it's not hard
167 2014-02-06 07:33:04 <Lightsword> does leveldb just have to be copied manually then?
168 2014-02-06 07:33:14 <Lightsword> or just as a file
169 2014-02-06 07:33:35 <petertodd> Lightsword: you don't have to be copying anything; replay the blockchain on your second server in the same way as the first
170 2014-02-06 07:34:03 <Lightsword> as in wait for it to sync from scratch?
171 2014-02-06 07:34:21 <petertodd> you can copy the blockchain*.dat files from one machine to another
172 2014-02-06 07:34:43 <petertodd> should be able to copy the whole .bitcoin database
173 2014-02-06 07:35:04 <Lightsword> petertodd, I suppose, was trying to avoid that though
174 2014-02-06 07:35:06 <uiop> Lightsword: in the same manner you'd create the postgres db to begin with
175 2014-02-06 07:35:18 <Lightsword> postgres has replication though
176 2014-02-06 07:35:29 <petertodd> Lightsword: huh? that's the same as what you're proposing, size wise
177 2014-02-06 07:36:06 <Lightsword> but postgres replication is more advanced than just copying a file
178 2014-02-06 07:36:19 <uiop> Lightsword: oh sure, but you need to update at least one of your replicated postgres dbs from the *.dat files on a recurring basis
179 2014-02-06 07:36:32 <petertodd> sheesh, you don't have to replicate anything, just run bitcoind on the second machine
180 2014-02-06 07:37:49 <petertodd> honestly, you've been arguing about this for sufficiently long enough you could have whipped up a independent postgres index creator thing in the same time with python-bitcoinlib
181 2014-02-06 07:38:47 <Luke-Jr> heck, without bitcoinlib..
182 2014-02-06 07:39:07 <Lightsword> just seems like there must be a better option than using a straight file transfer
183 2014-02-06 07:39:20 <petertodd> Lightsword: why do you think file transfer comes into this?
184 2014-02-06 07:39:32 <Lightsword> copying leveldb .dat files
185 2014-02-06 07:39:56 <Lightsword> resyncing with bitcoin network from scratch takes far too long
186 2014-02-06 07:39:58 <Pucilowski> How would I go about achieving some sort of atomicity with bitcoin withdrawals over rpc?
187 2014-02-06 07:40:00 <petertodd> Lightsword: hell, creat postgres on one machine that runs bitcoind and a python script, and use postgres replication on the other machine from the postgres database
188 2014-02-06 07:40:12 <Pucilowski> A way to issue a withdrawal and receive a guarantee it happened
189 2014-02-06 07:40:55 <Lightsword> petertodd: well that would replicate the postgres part, but I would want bitcoind running on both as well
190 2014-02-06 07:41:00 <petertodd> Pucilowski: can't do that right now - what you probably want is a way to have the wallet spit out transactions, but not actually add them to the wallet until something else broadcasts them on the network
191 2014-02-06 07:41:16 <Pucilowski> i think that'd achieve the same goal, yes
192 2014-02-06 07:41:17 <petertodd> Lightsword: if bitcoind is running on both, run your python postgres updater on both...
193 2014-02-06 07:41:34 <petertodd> Pucilowski: pull-reqs accepted :) (it's a feature I've been kinda wanting too)
194 2014-02-06 07:41:40 <Luke-Jr> Lightsword: if you want a wallet, without an independently maintained copy of the blockchain, use Armory
195 2014-02-06 07:41:43 <Pucilowski> also, suppose that I wish to run a service where I'll hold bitcoins on behalf of users, deposits/withdrawals, the works
196 2014-02-06 07:41:55 <Luke-Jr> Pucilowski: good luck with the licensing :/
197 2014-02-06 07:41:58 <Pucilowski> What are the cons/pros of relying on bitcoind to maintain account balances?
198 2014-02-06 07:42:05 <Luke-Jr> Pucilowski: it doesn't scale.
199 2014-02-06 07:42:19 <Pucilowski> What's the most common bottleneck?
200 2014-02-06 07:42:25 <petertodd> Pucilowski: key management is easy to screw up and screwing up means you lose your funds
201 2014-02-06 07:42:28 <Luke-Jr> Pucilowski: the wallet
202 2014-02-06 07:42:29 <Lightsword> petertodd: I would be starting with one completely empty server so I would not initially have bitcoind on both, I would want to sync both bitcoind and postgres
203 2014-02-06 07:42:34 <Pucilowski> which specific function though
204 2014-02-06 07:42:43 <uiop> Lightsword: you can simply just run bitcoind on one, sync that pg db from the *.dat's, then replicate with your desired method(s)
205 2014-02-06 07:42:51 <petertodd> Lightsword: then don't start it on an empty server, copy the files over...
206 2014-02-06 07:44:00 <uiop> Lightsword: or store the bitcoind files on nfs, or ...
207 2014-02-06 07:44:26 <Lightsword> petertodd, I suppose that would work, just don't like deploying with file copy, prefer native database syncing
208 2014-02-06 07:44:28 <Luke-Jr> LOL @ NFS
209 2014-02-06 07:44:42 <uiop> Luke-Jr: heh, /me has become a recent nfs lover
210 2014-02-06 07:44:48 <Luke-Jr> I can't imagine that working well.
211 2014-02-06 07:44:58 <petertodd> Lightsword: look, we've told you the right way to do it, you don't get it; I've got better things to do. Bye
212 2014-02-06 07:45:01 <Luke-Jr> certainly not with multiple clients trying to use it
213 2014-02-06 07:45:43 <uiop> Luke-Jr: oh yeah, didn't mean to suggest that (although in retrospect i should have made that clear)
214 2014-02-06 07:47:49 <uiop> i'd imagine running bitcoind w/ data file over nfs would possibly be non-ideal (e.g. mmap) or possibly problematic (dependent upon what leveldb code is up to)
215 2014-02-06 07:49:08 <petertodd> uiop: I think what's most interesting about nfs and the even more complex network filesystems after it, is how much better rcs's like git work in comparison
216 2014-02-06 07:49:37 <petertodd> uiop: fundementally merging changes is really hard, and you wind up either doing it manually or failing spectacularly every so often
217 2014-02-06 07:50:03 <Luke-Jr> +1
218 2014-02-06 07:50:14 <Luke-Jr> I use git for my email
219 2014-02-06 07:50:23 <petertodd> same here! git annex specifically
220 2014-02-06 07:50:26 <uiop> git is good for everything
221 2014-02-06 07:50:35 <petertodd> uiop: even large files with git annex
222 2014-02-06 07:51:00 <petertodd> I've got ~1TB of media files with it
223 2014-02-06 07:51:09 <uiop> heh
224 2014-02-06 07:51:30 <Luke-Jr> I'm too disorganised for that :<
225 2014-02-06 07:51:43 <petertodd> uiop: awesome feeling to be able to type git annex fsck, wait a few hours, and know that every last byte is safe
226 2014-02-06 07:52:20 <Luke-Jr> petertodd: I thoguht git-annex specifically disclaimed being a backup system? :P
227 2014-02-06 07:53:06 <uiop> ACTION looks into git annex
228 2014-02-06 07:53:07 <petertodd> Luke-Jr: that's why I have amazon glacier as a backend, and then backup my (much smaller) git annex directories to multiple places
229 2014-02-06 07:53:23 <wumpus> it isn't a backup system, it just keeps track of files that you have to backup somewhere yourself
230 2014-02-06 07:54:09 <petertodd> wumpus: very good description
231 2014-02-06 07:54:19 <Luke-Jr> wumpus: those two things sound equivalent to me..
232 2014-02-06 07:55:32 <petertodd> Luke-Jr: point is, using git annex doesn't magically make anything redundent, even if you store them on multiple backends - you need the master git repo to get them back and if you lose all copies of that you're in trouble
233 2014-02-06 07:55:58 <Luke-Jr> petertodd: I can't git annex my git annex repoâ
234 2014-02-06 07:56:08 <wumpus> ...lol
235 2014-02-06 07:56:18 <Luke-Jr> I'd think without the repo, you could still dig through the objects manually..
236 2014-02-06 07:56:19 <petertodd> Luke-Jr: heh, well, you sync that repo to multiple machines
237 2014-02-06 07:56:36 <petertodd> Luke-Jr: git annex supports encrypted backends, so...
238 2014-02-06 07:57:11 <petertodd> Luke-Jr: like, my setup for my mail, is I sync the repo to three machines, and the files get stored on one machine as well as amazon glacier
239 2014-02-06 07:57:18 <wumpus> putting git annex in git annex would be pointless it just moves the problem a level up :p
240 2014-02-06 07:57:36 <Luke-Jr> wumpus: yes, I was joking :p
241 2014-02-06 07:57:38 <petertodd> (I also have completely different backup mechanisms for redundency - don't want to just trust git annex!)
242 2014-02-06 07:57:42 <wumpus> it's git annexes all the way!
243 2014-02-06 07:58:08 <petertodd> wumpus: nah, at some point it just dumps everything in the dogecoin blockchain
244 2014-02-06 07:58:16 <Luke-Jr> btw, are you guys going to Texas? we're plotting to invent proof-of-steak there.
245 2014-02-06 07:58:34 <petertodd> Luke-Jr: whens texas?
246 2014-02-06 07:58:37 <Luke-Jr> petertodd: March
247 2014-02-06 07:58:42 <petertodd> Luke-Jr: what dates?
248 2014-02-06 07:58:56 <Luke-Jr> March 5-6
249 2014-02-06 07:59:17 <petertodd> gonna be at the financial crypto conf
250 2014-02-06 08:00:08 <Luke-Jr> meh
251 2014-02-06 08:00:17 <Luke-Jr> are we at the point where we have overlapping conferences now? :/
252 2014-02-06 08:00:38 <petertodd> well, financial crypto conf is not just bitcoin
253 2014-02-06 08:00:46 <petertodd> ...yet
254 2014-02-06 08:01:31 <Luke-Jr> I guess on the bright side, at this rate I'll be able to just attend ones I can drive to <.<
255 2014-02-06 08:01:52 <petertodd> ha
256 2014-02-06 08:02:19 <petertodd> in april I'll have one I can walk to
257 2014-02-06 09:21:44 <michagogo> cloud|Wimpish: something just occurred to me
258 2014-02-06 09:21:50 <michagogo> cloud|wumpus*
259 2014-02-06 09:22:13 <michagogo> cloud|What is your lxc environment like?
260 2014-02-06 09:22:13 <wumpus> ACTION slaps michagogo|cloud's autocorrect
261 2014-02-06 09:22:38 <wumpus> the default created by gitian
262 2014-02-06 09:22:47 <michagogo> cloud|No, I mean
263 2014-02-06 09:22:51 <michagogo> cloud|What environment are
264 2014-02-06 09:23:00 <michagogo> cloud|You running gitian in?
265 2014-02-06 09:23:00 <wumpus> ubuntu 12.04
266 2014-02-06 09:23:19 <michagogo> cloud|My DO droplet is also running 12.04
267 2014-02-06 09:23:39 <michagogo> cloud|...but the VM I was using is running 13.04
268 2014-02-06 09:24:05 <wumpus> and that's the one with the assembly errors?
269 2014-02-06 09:24:13 <michagogo> cloud|No
270 2014-02-06 09:24:20 <michagogo> cloud|That's the one that didn't match
271 2014-02-06 09:24:41 <michagogo> cloud|The one with assembly errors was KVM running in a VM
272 2014-02-06 09:44:51 <michagogo> cloud|wumpus: I'm spinning up a raring droplet to test if that's the cause
273 2014-02-06 09:45:22 <wumpus> what didn't match? yesterday everything matched, right?
274 2014-02-06 09:45:39 <wumpus> except for that stupid .tar.gz but that was explainable
275 2014-02-06 09:49:23 <The_Fly> hi, anyone here up for helping me test this bitcoin-related site im about to launch?
276 2014-02-06 09:49:44 <The_Fly> i want to do a gentle soft-launch first to catch any show-stopping bugs
277 2014-02-06 09:49:55 <The_Fly> before i tweet/redit it out to a wider audience
278 2014-02-06 09:50:15 <michagogo> cloud|wumpus: No, not when I built in my VM
279 2014-02-06 09:50:30 <The_Fly> give me a shout, any time, if you are able to help i would be most grateful
280 2014-02-06 09:50:32 <wumpus> michagogo|cloud: I don't remember that
281 2014-02-06 09:51:01 <wumpus> michagogo|cloud: so you build 0.9.0rc1 using the descriptors in #3622 and got non-matching outputs?
282 2014-02-06 09:51:12 <michagogo> cloud|wumpus: look back to 22:00-sh UTC
283 2014-02-06 09:51:18 <wumpus> can you send me the files ?
284 2014-02-06 09:51:42 <michagogo> cloud|Let me check what's there
285 2014-02-06 09:51:44 <wumpus> I'm about to give up this gitian stuff completely, it drives me mad
286 2014-02-06 09:51:56 <wumpus> days on end I've only be looking at builds
287 2014-02-06 09:52:02 <michagogo> cloud|I'm setting up a DO 13.04 gitian environment
288 2014-02-06 09:52:17 <michagogo> cloud|(holy crap, it's fast on an SSD)
289 2014-02-06 09:53:34 <michagogo> cloud|wumpus: Any preferred method of getting you the file?
290 2014-02-06 09:53:45 <wumpus> just put it on some server
291 2014-02-06 09:53:48 <michagogo> cloud|k
292 2014-02-06 09:55:00 <michagogo> cloud|wumpus: Looking back to the logs from last night...
293 2014-02-06 09:55:19 <michagogo> cloud|looks like I built deps and boost, and each of those didn't match
294 2014-02-06 09:55:20 <Imbue> hello. I am playing around with ECC, creating functions that add points, multiply points by scalars, and so on
295 2014-02-06 09:55:29 <wumpus> did you also build the final result?
296 2014-02-06 09:55:41 <michagogo> cloud|No, doesn't look like it
297 2014-02-06 09:55:59 <Imbue> I am pretty much there, but I am unsure how to represent the infinity point - it seems that it's not possible - should I just create a special case?
298 2014-02-06 09:56:25 <Imbue> what I want to do is basically make it so that my point_add function returns A if it is asked the result of A + O
299 2014-02-06 09:56:32 <Imbue> but how to manage that escapes me
300 2014-02-06 09:58:12 <wumpus> michagogo|cloud: it's very well possible it's another sorting or timestamp issue, in which case the final result will match... but do send me the intermediate files
301 2014-02-06 09:58:55 <wumpus> I mean we can't even trust the sort command to behave the same with different locales, what else could there be sneaking in....
302 2014-02-06 09:59:59 <wumpus> computers are just not meant to be deterministic :p
303 2014-02-06 10:00:35 <Imbue> lol
304 2014-02-06 10:02:44 <wumpus> maybe gitian should use a language such as python, or even C++, to drive the build commands instead of bash script, something that relies less on the GNU utilities with all their wonderful customizability through the environment
305 2014-02-06 10:03:46 <wumpus> or maybe LXC is just not 'isolated' enough
306 2014-02-06 10:04:03 <Imbue> hm, it seems more like the code I am looking at is just daft, it shouldn't need to reference the point at infinity
307 2014-02-06 10:04:32 <wumpus> Imbue: yes, I don't remember special cases for that in other ecdsa code, it's just an error if it arises
308 2014-02-06 10:05:11 <michagogo> cloud|wumpus: see PM
309 2014-02-06 10:05:31 <Imbue> http://pastebin.com/tTHWGTFc
310 2014-02-06 10:05:37 <Imbue> this is the pseudocode I have
311 2014-02-06 10:06:03 <Imbue> the general principle makes sense to me, I just need to find a way of having some sensible 'start point'
312 2014-02-06 10:07:54 <Imbue> ah wait, it's kind of obvious. if R exists, do point add, if it doesn't yet, initialize it as N
313 2014-02-06 10:11:18 <sipa> Imbue: typical implementations have a boolean variable along with the coordinates to denote infinity
314 2014-02-06 10:13:07 <michagogo> cloud|wumpus: 1e534dc2e337bc0ac35ebd6544e15c4641b245c621706cc2c03436de545c600a build/out/boost-linux32-1.55.0-gitian-r1.zip
315 2014-02-06 10:13:14 <michagogo> cloud|That's on raring DO
316 2014-02-06 10:13:23 <sipa> DO?
317 2014-02-06 10:14:18 <michagogo> cloud|DigitalOcean
318 2014-02-06 10:14:18 <wumpus> can you check yourself wether it matches, I'm busy comparing the other boost...
319 2014-02-06 10:14:22 <michagogo> cloud|It doesn't
320 2014-02-06 10:14:35 <wumpus> uhm actually it does
321 2014-02-06 10:14:41 <michagogo> cloud|Erm
322 2014-02-06 10:14:46 <wumpus> see the op in #3622
323 2014-02-06 10:14:47 <michagogo> cloud|It doesn't match my VM, that ia
324 2014-02-06 10:14:50 <michagogo> cloud|that is*
325 2014-02-06 10:15:18 <michagogo> cloud|Okay, so the variable isn't which release of Ubuntu is used as the host
326 2014-02-06 10:15:50 <wumpus> oh man... you know what is different?
327 2014-02-06 10:15:53 <michagogo> cloud|I don't
328 2014-02-06 10:15:56 <wumpus> the file permissions
329 2014-02-06 10:16:01 <michagogo> cloud|ffff
330 2014-02-06 10:16:05 <wumpus> of the .o file in the .a in the zip!
331 2014-02-06 10:16:11 <wumpus> wowohwohfsdjfadskljfasdkljfjklddfjklsdfjld *crash*
332 2014-02-06 10:16:17 <sipa> *TILT*
333 2014-02-06 10:16:21 <wumpus> idd
334 2014-02-06 10:16:23 <michagogo> cloud|I think cfields was seeing something like that with LXC when he was working on the OS X
335 2014-02-06 10:16:42 <sipa> i didn't even know that .a files stored permissions...
336 2014-02-06 10:16:49 <michagogo> cloud|devrandom made some change, or something, I think
337 2014-02-06 10:16:53 <cfields> it's the umask
338 2014-02-06 10:17:04 <michagogo> cloud|But that didn't fix it
339 2014-02-06 10:17:06 <cfields> it changed in ubuntu
340 2014-02-06 10:17:15 <wumpus> all that metadata... let's kill all the metadata....
341 2014-02-06 10:17:23 <wumpus> just the meat not the crap
342 2014-02-06 10:17:41 <sipa> find -type f | sort | xargs cat >bitcoin.cat
343 2014-02-06 10:18:09 <wumpus> but yes adding an umask at the top of the descriptors is probably the way to go
344 2014-02-06 10:18:16 <cfields> wumpus: already done
345 2014-02-06 10:18:18 <wumpus> lol sipa
346 2014-02-06 10:18:22 <michagogo> cloud|wumpus: I think that didn't help
347 2014-02-06 10:18:33 <michagogo> cloud|sipa: Do you mean >>?
348 2014-02-06 10:18:36 <cfields> wumpus: it doesn't affect things early enough in the build
349 2014-02-06 10:18:47 <sipa> michagogo|cloud: no
350 2014-02-06 10:18:56 <michagogo> cloud|Oh
351 2014-02-06 10:18:56 <wumpus> hmm? how do you mean early enough?
352 2014-02-06 10:18:59 <cfields> the main output dir still gets created 002
353 2014-02-06 10:19:00 <michagogo> cloud|Right, nvm
354 2014-02-06 10:19:31 <michagogo> cloud|wumpus: 0473f96aa382d4448994fde4192155f3a96454a7436efdc04c5fc693b185d864 boost-linux64-1.55.0-gitian-r1.zip
355 2014-02-06 10:19:33 <wumpus> ok but this is about the build directory, I think I create that myself
356 2014-02-06 10:19:35 <cfields> or 022, i forget which is backwards now
357 2014-02-06 10:19:41 <michagogo> cloud|(match)
358 2014-02-06 10:20:15 <cfields> be careful, busted lxc will match another busted lxc
359 2014-02-06 10:20:15 <michagogo> cloud|Hmm
360 2014-02-06 10:20:34 <cfields> (busted is relative)
361 2014-02-06 10:20:42 <wumpus> at least this doesn't affect the executables
362 2014-02-06 10:20:44 <michagogo> cloud|So I wonder, what's the difference between lxc on a DO raring droplet and lxc on my raring Vbox VM?
363 2014-02-06 10:21:10 <wumpus> vbox doesn't leak the umask
364 2014-02-06 10:21:19 <michagogo> cloud|Hm?
365 2014-02-06 10:21:48 <cfields> michagogo|cloud: your user's umask may be different between vbox and droplet
366 2014-02-06 10:21:51 <wumpus> if you build in a vbox you get a bog-default Ubuntu 12.04 image with a well-defined default umask, if you use LXC then somehow your parent system's umask gets leaked in
367 2014-02-06 10:21:58 <cfields> i don't remember exactly where it pulls from
368 2014-02-06 10:22:04 <michagogo> cloud|wumpus: No, I don't mean a vbox VM
369 2014-02-06 10:22:09 <michagogo> cloud|(as the build system)
370 2014-02-06 10:22:23 <michagogo> cloud|I mean, build in LXC on a droplet vs build in LXC on a Vbox VM
371 2014-02-06 10:22:40 <cfields> either way, i'd recommending verifying with kvm, then tracking down the real bug
372 2014-02-06 10:22:45 <michagogo> cloud|cfields: How do you check umask?
373 2014-02-06 10:22:54 <cfields> as it kind of defeats the purpose of gitian otherwise
374 2014-02-06 10:22:59 <michagogo> cloud|And we've verified with kvm
375 2014-02-06 10:23:00 <wumpus> cfields: all the kvm builds match
376 2014-02-06 10:23:01 <cfields> michagogo|cloud: touch file && ls -al file
377 2014-02-06 10:23:13 <michagogo> cloud|KVM matches the droplet
378 2014-02-06 10:23:36 <michagogo> cloud|-rw-rw-r-- 1 micha micha 0 פ×ר 6 12:23 file
379 2014-02-06 10:23:57 <michagogo> cloud|-rw-rw-r-- 1 micha micha 0 Feb 6 10:23 file
380 2014-02-06 10:24:08 <michagogo> cloud|It's a match
381 2014-02-06 10:24:11 <wumpus> there is no problem when using KVM build system, it's only with lxc on raring
382 2014-02-06 10:24:21 <michagogo> cloud|wumpus: LXC on raring in my VM
383 2014-02-06 10:24:32 <michagogo> cloud|LXC on raring in the DO droplet is fine
384 2014-02-06 10:25:08 <michagogo> cloud|I wonder
385 2014-02-06 10:25:18 <michagogo> cloud|What if I create a 12.04 VM on my machine
386 2014-02-06 10:25:23 <wumpus> I'm going to try adding 'umask 022' at the top of the descriptors, I think it will work in this case as the .o files are created by the build process which should use the umask
387 2014-02-06 10:25:45 <cfields> wumpus: see current gitian, it injects that before the descriptors
388 2014-02-06 10:26:01 <wumpus> cfields: okay
389 2014-02-06 10:26:17 <cfields> wumpus: you're free to try it, ofc :)
390 2014-02-06 10:26:36 <cfields> but i've already poked at this one a bit a while back
391 2014-02-06 10:26:40 <wumpus> I'm giving up on determinism in the intermediate outputs in that case... this is just too much
392 2014-02-06 10:26:49 <michagogo> cloud|38658a215429992a8d066f827b6f46ca63abc14173983647d16ac0e21a9ddfb7 build/out/bitcoin-deps-linux32-gitian-r3.zip
393 2014-02-06 10:26:57 <michagogo> cloud|That's the raring droplet
394 2014-02-06 10:27:05 <cfields> what distro is used for build these days?
395 2014-02-06 10:27:11 <cfields> er, version
396 2014-02-06 10:27:13 <wumpus> ubuntu 12.04
397 2014-02-06 10:27:14 <michagogo> cloud|cfields: precise VMs
398 2014-02-06 10:27:44 <michagogo> cloud|My local VM, though, has c609049c60d96e193a172d0c5c0f44914032f562ee1691d18218bb3c0275ec32 build/out/bitcoin-deps-linux32-gitian-r3.zip
399 2014-02-06 10:27:50 <cfields> mm
400 2014-02-06 10:28:21 <cfields> wumpus: i wouldn't worry about the intermediate ones, instead, bump up versions after release
401 2014-02-06 10:28:32 <cfields> trusty should have all of my binutils patches included
402 2014-02-06 10:28:41 <cfields> so everything is deterministic
403 2014-02-06 10:29:09 <wumpus> michagogo|cloud: have you built the executables yet with that VM?
404 2014-02-06 10:29:25 <michagogo> cloud|wumpus: linux64 deps building now
405 2014-02-06 10:29:44 <wumpus> I'm 99% sure those will match, but let's see
406 2014-02-06 10:29:58 <michagogo> cloud|aea5d842642b299c7ca89daa1e1df477f43e04d413243df94ed5b5a0c526c5a0 bitcoin-deps-linux64-gitian-r3.zip
407 2014-02-06 10:29:58 <michagogo> cloud|erm, built*
408 2014-02-06 10:30:10 <wumpus> forget the intermediate outputs for now
409 2014-02-06 10:30:30 <michagogo> cloud|micha@Raring-Virtualbox-amd64:~/gitian-lxc/gitian-builder$ ./bin/gbuild ../bitcoin/contrib/gitian-descriptors/gitian-linux.yml --commit bitcoin=v0.9.0rc1
410 2014-02-06 10:30:33 <michagogo> cloud|Oh, god dammit
411 2014-02-06 10:30:49 <michagogo> cloud|I forgot to move the zips to the inputs folder
412 2014-02-06 10:31:04 <michagogo> cloud|And apparently gitian nukes the build folder right at the start
413 2014-02-06 10:31:07 <wumpus> I suggest including that in your script :p
414 2014-02-06 10:31:10 <wumpus> yes it does
415 2014-02-06 10:31:34 <michagogo> cloud|Hit ctrl-c as soon as I saw sha256sum: bitcoin-deps-linux32-gitian-r3.zip: No such file or directory
416 2014-02-06 10:31:34 <michagogo> cloud|sha256sum: bitcoin-deps-linux64-gitian-r3.zip: No such file or directory
417 2014-02-06 10:31:39 <michagogo> cloud|But it was already gone
418 2014-02-06 10:31:43 <michagogo> cloud|Script? WHat script?
419 2014-02-06 10:31:52 <michagogo> cloud|Right now there's no script
420 2014-02-06 10:31:58 <michagogo> cloud|(other than gbuild)
421 2014-02-06 10:32:01 <wumpus> the script that does your gitian builds, or are you typing the commands every time?
422 2014-02-06 10:32:24 <michagogo> cloud|Typing, for now
423 2014-02-06 10:32:29 <michagogo> cloud|It's not that much to type
424 2014-02-06 10:32:37 <wumpus> no but it's very easy to make mistakes
425 2014-02-06 10:32:51 <wumpus> and mistakes cost a lot of time
426 2014-02-06 10:33:07 <michagogo> cloud|bin/gb<tab> ../bi<tab>contr<tab>git<tab>e<tab>de<tab>l<tab>
427 2014-02-06 10:33:17 <wumpus> lol
428 2014-02-06 10:33:40 <wumpus> also I build them in a loop by default to see if the runs match
429 2014-02-06 10:35:20 <wumpus> but you're right, it includes umask 002 by default
430 2014-02-06 10:35:38 <wumpus> which should result in permissions of 664
431 2014-02-06 10:36:15 <wumpus> the KVM-built dependencies have this correctly, michagogo|cloud's LXC-built (on raring) have 644
432 2014-02-06 10:36:24 <cfields> wumpus: can you run an 'ar --help' on a target?
433 2014-02-06 10:36:38 <cfields> to see if the deterministic option exists there?
434 2014-02-06 10:36:42 <michagogo> cloud|The odd thing is
435 2014-02-06 10:36:57 <wumpus> so somehow it's forcing the umask to 022 independent of what gitian builder is doing
436 2014-02-06 10:37:00 <wumpus> cfields: sure
437 2014-02-06 10:37:05 <michagogo> cloud|touchfile && ls -la file, on the raring vm, returned 664
438 2014-02-06 10:38:55 <wumpus> cfields: http://www.hastebin.com/cekimegiha.vhdl
439 2014-02-06 10:39:54 <cfields> wumpus: hmm, it's there
440 2014-02-06 10:40:02 <wumpus> what's there?
441 2014-02-06 10:40:03 <cfields> ar -D should do it
442 2014-02-06 10:40:44 <wumpus> [D] - use zero for timestamps and uids/gids
443 2014-02-06 10:40:53 <wumpus> that also normalizes the permissions?
444 2014-02-06 10:41:04 <cfields> at least, in recent versions, -D means deterministic
445 2014-02-06 10:41:26 <cfields> hmm, that might've changed though. you don't have -U which reverses that
446 2014-02-06 10:41:38 <wumpus> all files. When this option is used, if ar is used with identical options and identical input files, multiple runs will create identical output
447 2014-02-06 10:41:38 <wumpus> D Operate in deterministic mode. When adding files and the archive index use zero for UIDs, GIDs, timestamps, and use consistent file modes for
448 2014-02-06 10:41:38 <wumpus> files regardless of the input files' owners, groups, file modes, or modification times.
449 2014-02-06 10:41:44 <wumpus> that's the manpage on Ubuntu 12.04
450 2014-02-06 10:41:51 <wumpus> so should be OK
451 2014-02-06 10:42:07 <cfields> right
452 2014-02-06 10:42:29 <cfields> so if you can sneak that in for whatever's making the .a's, it should avoid the problem
453 2014-02-06 10:43:34 <wumpus> I suppose the easiest way to sneak it in would be to make our own 'ar' script that passes through arguments and adds -D
454 2014-02-06 10:43:57 <wumpus> otherwise we'll have to check all dependencies independently how to do it
455 2014-02-06 10:44:06 <wumpus> or maybe there is some environment variable for it
456 2014-02-06 10:45:43 <wumpus> I vaguely remember that for qt there was no other way to override ar arguments, but that may have changed in 5.x...
457 2014-02-06 10:46:07 <cfields> it didn't change
458 2014-02-06 10:46:19 <cfields> i had a patch ready to submit
459 2014-02-06 10:46:27 <cfields> but there was a CLA involved, so i stopped there
460 2014-02-06 10:46:41 <sipa> CLA?
461 2014-02-06 10:47:03 <wumpus> that's also where I'd stop
462 2014-02-06 10:47:07 <cfields> https://en.wikipedia.org/wiki/Contributor_License_Agreement
463 2014-02-06 10:47:22 <sipa> ic
464 2014-02-06 10:47:30 <TD> sipa: google also has one, btw
465 2014-02-06 10:47:37 <sipa> TD: i know
466 2014-02-06 10:53:05 <michagogo> cloud|wumpus: Okay, rebuilt deps -- gitian-linux.yml from your repo building now in the VM
467 2014-02-06 10:54:45 <Imbue> sipa: thanks for the help, managed to fix the problem now
468 2014-02-06 10:54:59 <Imbue> my tool takes 1 second to get a pubkey from a privkey... super efficient! :p
469 2014-02-06 10:55:33 <michagogo> cloud|Hm, I want to create a 12.04 VM to test with
470 2014-02-06 10:55:52 <michagogo> cloud|But at some point today, Ubuntu 12.04.4 is going to be relased
471 2014-02-06 10:55:59 <sipa> Imbue: i believe my library does that in around 0.1ms
472 2014-02-06 10:56:24 <Imbue> sipa: I'm doing it in pretty much the least efficient manner that would actually complete
473 2014-02-06 10:56:41 <michagogo> cloud|Imbue: Wow, the least efficient manner is only 10,000 times slower?
474 2014-02-06 10:56:42 <michagogo> cloud|:-P
475 2014-02-06 10:56:44 <sipa> yeah, first make it working :)
476 2014-02-06 10:57:37 <Imbue> well, i'm sure I could run it on a NES, or deliberately gimp my algorithms michagogo|cloud :P
477 2014-02-06 10:59:23 <Imbue> what's confusing me is that my mult algorithm seems to sometimes be operating on 512bit integers, which makes me think I've forgotten a modulo somewhere
478 2014-02-06 11:00:11 <Imbue> in fact that's silly, I certainly have :p
479 2014-02-06 11:00:52 <sipa> what language?
480 2014-02-06 11:01:08 <Imbue> python, i'm surprised by how easy it is to use infinite size integers
481 2014-02-06 11:01:15 <Imbue> though that's likely one of the reasons it's so slow
482 2014-02-06 11:01:57 <cfields> heh
483 2014-02-06 11:02:34 <sipa> here's my lowest-level part for the field multiplication: https://github.com/sipa/secp256k1/blob/master/src/impl/field_5x52_int128.h
484 2014-02-06 11:02:50 <sipa> it uses 5 52-bit integers to represent the number
485 2014-02-06 11:02:55 <Imbue> think I might have nailed it, looks like my inverse algorithm does not mod
486 2014-02-06 11:03:04 <Imbue> hopefully this fixes the 512-bit thing
487 2014-02-06 11:04:17 <wumpus> cfields: hmm boost has <archiveflags>, let's hope it does what it's supposed to
488 2014-02-06 11:04:37 <cfields> wumpus: heh, good luck with that :)
489 2014-02-06 11:04:47 <cfields> wumpus: iirc, there's a mis-spelling in there somewhere
490 2014-02-06 11:04:54 <cfields> there was for osx, anyway
491 2014-02-06 11:05:01 <wumpus> thanks, I'll need it...
492 2014-02-06 11:05:09 <cfields> it was archivflags or something
493 2014-02-06 11:05:14 <wumpus> hahaha
494 2014-02-06 11:05:31 <wumpus> boost is really built to drive people crazy, who uses jam to build a c++ library
495 2014-02-06 11:05:59 <cfields> it's insane that the most useful lib has the most worthless buildsystem
496 2014-02-06 11:06:50 <cfields> it's really a shame. takes some serious meditation to decide if you can commit to that dependency or not
497 2014-02-06 11:07:19 <wumpus> also the lack of either .la or .pc to sanely find the libs/dependencies is.. disturbed
498 2014-02-06 11:07:32 <wumpus> yes
499 2014-02-06 11:07:38 <cfields> heh, no shit
500 2014-02-06 11:08:55 <Imbue> yep, nailed it, 0.4 sec now :p
501 2014-02-06 11:08:57 <wumpus> but I agree it's one of C++ most useful libraries, it makes the language from a roll-everything-yourself masochism outlet to actually a convenient and useful lanaguage
502 2014-02-06 11:09:23 <cfields> wumpus: https://lists.gnu.org/archive/html/autoconf-archive-maintainers/2013-12/msg00010.html
503 2014-02-06 11:09:42 <cfields> i tried :)
504 2014-02-06 11:09:46 <wumpus> cfields: heh, nuff said
505 2014-02-06 11:11:12 <michagogo> cloud|Erm, my internet connection just dropped... I wonder what that did to the build?
506 2014-02-06 11:11:26 <wumpus> should do nothing to the build
507 2014-02-06 11:11:38 <michagogo> cloud|Will it continue to run in the background?
508 2014-02-06 11:11:43 <michagogo> cloud|(the one on DO)
509 2014-02-06 11:11:45 <wumpus> it wouldn't be very deterministic if it relied on a continuous internet connection
510 2014-02-06 11:12:01 <michagogo> cloud|No, but I lost the terminal the build was running in
511 2014-02-06 11:12:01 <wumpus> only if you started it in screen/tmux I guess
512 2014-02-06 11:12:08 <michagogo> cloud|Nope :-/
513 2014-02-06 11:12:26 <wumpus> most scripts/applications don't really like killing the controlling terminal
514 2014-02-06 11:12:34 <wumpus> not sure if gitian is different
515 2014-02-06 11:13:07 <wumpus> usually they die the next time they try to print something
516 2014-02-06 11:13:19 <michagogo> cloud|Hmm, looks like it's still running
517 2014-02-06 11:13:42 <cfields> with gitian, i usually have to take a hammer to it to make it stop running
518 2014-02-06 11:13:55 <cfields> so it'll probably finish the build at least :)
519 2014-02-06 11:14:16 <wumpus> well I can kill it using ctrl-c when building using KVM, LXC is very hard to convince to stop
520 2014-02-06 11:14:36 <michagogo> cloud|tailf var/build.log shows the build still running
521 2014-02-06 11:14:50 <cfields> wumpus: yea, that's probably the case here too
522 2014-02-06 11:20:41 <wumpus> either ar or boost is doing a good job ignoring the -D flag
523 2014-02-06 11:21:07 <wumpus> boost, it seems
524 2014-02-06 11:22:45 <michagogo> cloud|wumpus: http://paste.ubuntu.com/6884655/
525 2014-02-06 11:22:52 <michagogo> cloud|That's on the DO raring droplet
526 2014-02-06 11:23:04 <michagogo> cloud|Looks like gitian died as soon as the build finished
527 2014-02-06 11:23:29 <michagogo> cloud|But I was able to use libexec/copy-from-target with a little help from a reading of the gbuild script
528 2014-02-06 11:23:29 <wumpus> it did still copy the outputs
529 2014-02-06 11:23:32 <wumpus> oh
530 2014-02-06 11:24:17 <wumpus> seems they match!
531 2014-02-06 11:24:27 <michagogo> cloud|Again, though, this is the droplet
532 2014-02-06 11:25:13 <michagogo> cloud|Where the deps and boost matched as well
533 2014-02-06 11:25:45 <michagogo> cloud|Here are the 32-bit build results from my VM, which is currently running the 64-bit build: http://paste.ubuntu.com/6884665/
534 2014-02-06 11:27:49 <wumpus> cfields: did you ever get archiveflags to work?
535 2014-02-06 11:27:55 <wumpus> or, archivflags
536 2014-02-06 11:28:26 <michagogo> cloud|wumpus: Looks like a match, except for the tarball
537 2014-02-06 11:28:42 <wumpus> yes, the tarball still doesn't work
538 2014-02-06 11:28:56 <michagogo> cloud|Ah, okay
539 2014-02-06 11:29:14 <michagogo> cloud|So despite boost and deps not being deterministic, looks like the build is
540 2014-02-06 11:29:18 <michagogo> cloud|bitcoin build*
541 2014-02-06 11:29:29 <wumpus> yes the actual compiledcode is determnistic, that's great
542 2014-02-06 11:29:32 <michagogo> cloud|bitcoin linux build*
543 2014-02-06 11:32:24 <torokun> I haven't looked at it for a while, but does boost really give you that much you couldn't do pretty straightforwardly with stl?
544 2014-02-06 11:32:48 <wumpus> yes, it does
545 2014-02-06 11:33:25 <wumpus> http://www.boost.org/doc/libs/1_55_0/
546 2014-02-06 11:37:58 <torokun> I guess the networking, image, and inter process stuff seems useful
547 2014-02-06 11:38:46 <wumpus> also parsing, threading, boost::signals, filesystem
548 2014-02-06 11:42:18 <michagogo> cloud|wumpus: Should I take a look at the Windows stuff now?
549 2014-02-06 11:42:45 <wumpus> michagogo|cloud: sure, but I'm only interested in the final output for now, the intermediate files will probably not match
550 2014-02-06 11:44:59 <michagogo> cloud|wumpus: even with http://paste.ubuntu.com/6884665/?
551 2014-02-06 11:45:00 <michagogo> cloud|erm
552 2014-02-06 11:45:12 <michagogo> cloud|wumpus: even with laanwj:2014_02_gitian_windows_deps_fix?
553 2014-02-06 11:46:09 <wumpus> well as we just discovered it's still not complete
554 2014-02-06 11:46:16 <michagogo> cloud|We did?
555 2014-02-06 11:46:29 <wumpus> yes...
556 2014-02-06 11:46:53 <wumpus> ar and the .a files
557 2014-02-06 11:47:11 <michagogo> cloud|Oh, right
558 2014-02-06 11:47:28 <wumpus> let's not forget it that quickly
559 2014-02-06 11:53:41 <michagogo> cloud|Okay, here's my local VM: http://paste.ubuntu.com/6884762/
560 2014-02-06 11:54:31 <sipa> wumpus: why is that a problem? can't you do something like touch all .o files with a particular timestamp before ar'ing?
561 2014-02-06 11:54:55 <wumpus> sipa: passing -D on ar would do, really
562 2014-02-06 11:55:06 <sipa> so, why don't we?
563 2014-02-06 11:55:09 <wumpus> sipa: except that doing that for every dependency is a lot of research
564 2014-02-06 11:55:11 <wumpus> yes, why don't we...
565 2014-02-06 11:55:18 <sipa> oh right, not in our code
566 2014-02-06 11:55:18 <wumpus> I'm trying it for boost now
567 2014-02-06 11:55:21 <wumpus> it's ignoring everything I try
568 2014-02-06 11:55:31 <wumpus> boost's build system is a hell
569 2014-02-06 11:56:19 <wumpus> it's just another small thing of many, after I fixed this, who knows what will be the next thing that is slightly different based on the environment that will mess with it
570 2014-02-06 11:56:22 <wumpus> it's just a never-ending rabbit hole
571 2014-02-06 11:57:15 <sipa> we need a fully deterministic system emulator
572 2014-02-06 11:57:24 <wumpus> KVM does it fine :)
573 2014-02-06 11:57:26 <sipa> with a clock that ticks once per executed instruction
574 2014-02-06 11:57:40 <sipa> and deterministic process scheduler in the kernel
575 2014-02-06 11:57:42 <wumpus> indeed that'd make sense
576 2014-02-06 11:57:45 <michagogo> cloud|sipa: and it should have no hardware requirements
577 2014-02-06 11:57:56 <sipa> yeah implement it as turing machine
578 2014-02-06 11:58:02 <sipa> we don't care about runtime, right? :p
579 2014-02-06 11:58:07 <michagogo> cloud|(i.e. no need for hardware virt)
580 2014-02-06 11:58:26 <wumpus> at least all the toolchains should stop inserting all kind of nonsense metadata in their output
581 2014-02-06 11:58:47 <sipa> yeah, timestamps, who came up with that??
582 2014-02-06 11:59:09 <wumpus> yep... and file permissions and users/groups in a shared library, who came up with that?
583 2014-02-06 11:59:31 <wumpus> I can understand it for archives, if you want to use it for backups...
584 2014-02-06 11:59:41 <wumpus> eh, static library*
585 2014-02-06 11:59:56 <wumpus> though I wouldn't be surprised if shared libraries contained the same kind of cruft
586 2014-02-06 12:01:40 <torokun> not having used boost in like 10 years, can't you bypass this stuff and just include what you use?
587 2014-02-06 12:02:14 <wumpus> yes, I can just go do something else, the end result is deterministic, but it frustrates me
588 2014-02-06 12:03:57 <wumpus> hah I've even got boost build to segfault, how's that
589 2014-02-06 12:04:13 <sipa> boost _build_ ??
590 2014-02-06 12:04:56 <warren> wumpus: no overheating? that happens randomly while I gitian
591 2014-02-06 12:05:22 <wumpus> yes... I'm trying to convince it to pass flags to the archiver, I'm trying to debug what happens, I've added debug statements in the gcc.jam now it segfaultss
592 2014-02-06 12:05:44 <wumpus> warren: no overheating, the crashes are... deterministic!
593 2014-02-06 12:12:37 <michagogo> cloud|00736b701c01d71680b52a0a2682e38e4bfb05f80563511d358b9cc4e77ee874 bitcoin-deps-win32-gitian-r10.zip
594 2014-02-06 12:12:37 <michagogo> cloud|a1e1f8729bd69add02c8c6ccadadba9e95de65d295889e6c12399abd7c91ee9e bitcoin-deps-win64-gitian-r10.zip
595 2014-02-06 12:22:01 <michagogo> cloud|those were from the DO droplet
596 2014-02-06 12:22:37 <michagogo> cloud|15cbe05b6c0399030dc50be5b2bb4e673e9a84732ebcb3cd5bc17ec514bb4eac bitcoin-deps-win64-gitian-r10.zip
597 2014-02-06 12:22:37 <michagogo> cloud|These are from the VM: e86cac569e5fb7bcfbd80c0a9bd83be45eb1f896e1a94128d5ebb1a2c16a0ac1 bitcoin-deps-win32-gitian-r10.zip
598 2014-02-06 12:31:12 <c--O-O> hi
599 2014-02-06 12:31:43 <c--O-O> I was wondering, did the interface of bitcoins undergone changes?
600 2014-02-06 12:32:10 <c--O-O> is that why mtgox is having problem processing bitcoins? because the padding changed?
601 2014-02-06 12:33:57 <michagogo> cloud|Droplet: d02d2f32d0621315c683edf151f820d3fc362c99e8d9b3d9acf3b8080f114282 boost-win32-1.55.0-gitian-r6.zip
602 2014-02-06 12:33:57 <michagogo> cloud|ed7f66cedb05ca4787aae1a2716d09e2f371ccc412163aa41fa68d8185da6063 boost-win64-1.55.0-gitian-r6.zip
603 2014-02-06 12:34:19 <andytoshi> c--O-O: nothing has changed in the bitcoin protocol. mtgox has no excuse
604 2014-02-06 12:34:35 <c--O-O> I see.
605 2014-02-06 12:34:42 <sturles> c--O-O: No, the padding hasn't changed, but bitcoind has become more picky about what it accepts after 0.8.0: https://bitcointalk.org/index.php?topic=8392.0;all
606 2014-02-06 12:34:44 <c--O-O> I thought so too
607 2014-02-06 12:34:59 <sipa> "padding" ?