1 2013-01-26 01:52:07 <Shinra> hey guys
  2 2013-01-26 01:53:20 <Shinra> anyone on?
  3 2013-01-26 02:10:51 <HM> this Facebook hacker cup is fun
  4 2013-01-26 02:14:19 <HM> https://www.facebook.com/hackercup/problems.php?round=185564241586420
  5 2013-01-26 02:14:27 <HM> might interest some here, even though it's offtopic
  6 2013-01-26 08:47:30 <BTCOxygen> !seen jgarzik
  7 2013-01-26 08:47:30 <gribble> jgarzik was last seen in #bitcoin-dev 13 hours, 36 minutes, and 57 seconds ago: * jgarzik glares at grau :)
  8 2013-01-26 08:48:03 <BTCOxygen> jgarzik: hi
  9 2013-01-26 08:48:19 <BTCOxygen> jgarzik: Did you get your Avalon ASIC ?
 10 2013-01-26 08:48:47 <jgarzik> jgarzik ASIC arrival FAQ of the day, https://bitcointalk.org/index.php?topic=137534.msg1478626#msg1478626
 11 2013-01-26 08:50:22 <BTCOxygen> jgarzik: What do you plan to do with your ASIC once they arrive ?
 12 2013-01-26 08:50:37 <BTCOxygen> SOLO mine or POOL mine ?
 13 2013-01-26 08:51:49 <jgarzik> I plan to use my ASIC to rule this land.  And I will call it... This Land.
 14 2013-01-26 08:53:37 <BTCOxygen> jgarzik: lol
 15 2013-01-26 08:53:52 <BTCOxygen> jgarzik: How many GH/s in total have you ordered ?
 16 2013-01-26 09:10:21 <BTCOxygen> jgarzik: How many GH/s in total have you ordered ?
 17 2013-01-26 10:58:50 <JyZyXEL> does blockchain have to be deleted when upgrading from 0.6.2 to 0.7.2 to get the new database format?
 18 2013-01-26 11:05:45 <JyZyXEL> the synchronization is so insanely slow im thinking something has to be wrong here
 19 2013-01-26 11:12:21 <sipa> JyZyXEL: no changes between 0.6.x and 0.7.x
 20 2013-01-26 11:12:31 <sipa> to the block chain format
 21 2013-01-26 11:12:49 <JyZyXEL> well something was supposed to be faster
 22 2013-01-26 11:13:00 <sipa> yes, a bit
 23 2013-01-26 11:13:03 <JyZyXEL> im not sure if it is but its still really slow
 24 2013-01-26 11:13:13 <sipa> how slow?
 25 2013-01-26 11:13:28 <JyZyXEL> what command can i use to get performance information?
 26 2013-01-26 11:14:10 <sipa> no such command really... but you can see the number of blocks per second, or seconds per block or so
 27 2013-01-26 11:14:48 <JyZyXEL> GUI only shows number of blocks remaining, no other peformance information
 28 2013-01-26 11:15:15 <sipa> well how fast does that change?
 29 2013-01-26 11:15:19 <JyZyXEL> im thinking it must then be accesible trough the console
 30 2013-01-26 11:16:11 <sipa> not really, no
 31 2013-01-26 11:16:36 <sipa> i just want to get an idea of what you call slow
 32 2013-01-26 11:16:58 <JyZyXEL> its about 5 blocks in one minute
 33 2013-01-26 11:18:14 <CodeShark> JyZyXEL: you can check the debug.log file to follow what's happening
 34 2013-01-26 11:18:15 <JyZyXEL> its silly that no one though to add any sort of metrics for the syncronization
 35 2013-01-26 11:18:34 <CodeShark> tail -f <datadir>/debug.log
 36 2013-01-26 11:18:37 <sipa> JyZyXEL: 0.8 has such metrics
 37 2013-01-26 11:18:47 <JyZyXEL> nice
 38 2013-01-26 11:18:51 <sipa> and it should be a lot faster too
 39 2013-01-26 11:19:12 <sipa> how many blocks remaining?
 40 2013-01-26 11:19:18 <JyZyXEL> 1000
 41 2013-01-26 11:19:33 <JyZyXEL> it was 300 at one point but jumped to 2000
 42 2013-01-26 11:19:44 <JyZyXEL> i don't know how many times it has jumped like that
 43 2013-01-26 11:21:30 <JyZyXEL> with this rate, if i started from an empty block chain, it would take me 30 days to get the full blockchain
 44 2013-01-26 11:22:30 <JyZyXEL> i think i started at around 90% and i've been at it for days
 45 2013-01-26 11:23:15 <JyZyXEL> not constantly though because the load on the hard drives makes so much audible noise and makes the system a bit sluggish
 46 2013-01-26 11:23:33 <sipa> are you interested in trying a 0.8 prerelease?
 47 2013-01-26 11:23:51 <JyZyXEL> would i need to start from the beginning?
 48 2013-01-26 11:23:55 <CodeShark> no
 49 2013-01-26 11:24:20 <sipa> actually, yes
 50 2013-01-26 11:24:36 <sipa> but it imports the blocks you already have from disk
 51 2013-01-26 11:24:37 <CodeShark> oh, the bdb -> leveldb was between 0.7 and the current HEAD?
 52 2013-01-26 11:24:52 <sipa> yes, and ultraprune as well
 53 2013-01-26 11:25:46 <CodeShark> yeah, ditch 0.7.2, JyZyXEL :)
 54 2013-01-26 11:26:18 <sipa> 0.8 isn't released yet, and i don't recommend it for anything but testing
 55 2013-01-26 11:26:45 <JyZyXEL> most people are using 0.7.2 though?
 56 2013-01-26 11:26:53 <sipa> but depending on the hardware, it shouldn't take more than a few hours to sync from scratch
 57 2013-01-26 11:27:00 <JyZyXEL> one would think such slowness would be a big issue?
 58 2013-01-26 11:27:20 <sipa> yes, but it requires a very fundamental change to improve that
 59 2013-01-26 11:27:42 <sipa> and the stakes are high when there's a bug
 60 2013-01-26 11:27:45 <JyZyXEL> sipa: well the experience for me so far has been slower by multiple orders of magnitude
 61 2013-01-26 11:28:11 <sipa> i know, i'm talking about 0.8
 62 2013-01-26 11:29:22 <sipa> what hardware are you on?
 63 2013-01-26 11:29:31 <JyZyXEL> i5-750
 64 2013-01-26 11:29:54 <sipa> ram/disk?
 65 2013-01-26 11:30:19 <JyZyXEL> 4GB and 2x1TB Raid-1
 66 2013-01-26 11:30:55 <sipa> i wonder if the raid impacts disk write latency
 67 2013-01-26 11:31:09 <JyZyXEL> not by much
 68 2013-01-26 11:31:17 <JyZyXEL> the full disk encryption has a bigger hit
 69 2013-01-26 11:31:29 <sipa> oh, with disk encryption
 70 2013-01-26 11:32:20 <sipa> i'm not sure why, but iirc it kills disk i/o latency
 71 2013-01-26 11:32:49 <sipa> and since 0.7 does (excessively) much shnchronous i/o, it slows tgings down a lot
 72 2013-01-26 11:33:12 <JyZyXEL> whats the normal sync speed for 0.7?
 73 2013-01-26 11:33:36 <sipa> depends a lot on hardware
 74 2013-01-26 11:33:47 <sipa> in a ram drive, pretty fast
 75 2013-01-26 11:34:04 <sipa> other than that, it's limited by disk speed really
 76 2013-01-26 11:34:06 <JyZyXEL> yeah but for a recent dekstop machine similar to mine
 77 2013-01-26 11:35:43 <sipa> i can't say
 78 2013-01-26 11:36:09 <sipa> i've heen working on and using 0.8 code for almost half a year i think
 79 2013-01-26 11:36:33 <sipa> and even there the reported speeds vary between minutes and hours
 80 2013-01-26 11:38:51 <finway> What's the data struture in /coins ?
 81 2013-01-26 11:39:02 <finway> ('c\\x01(i\\xda\\xb5\\xaa\\xb8\\xa1\\x0275O\\xa1r\\xb0n\\xfcrb\\x14\\xbc\\x16\\xdb\\x1d\\xbb\\x80sA\\xb6\\xe8\\x1d\\xf9', '\\x01\\x032\\x05\\xa6\\xd4\\x92\\x8d\\x89\\xd5\\x14\\x1d\\xf9?\\xdf\\x05\\x8e7\\xcf<x\\x9e<"\\xeb\\xad\\xba7,\\x0ca\\xe8#\\x03\\x14\\xae\\x80\\xbem')
 82 2013-01-26 11:39:29 <finway> What does this mean?
 83 2013-01-26 11:41:01 <sipa> finway: the key is 'c' + 32-byte transaction id
 84 2013-01-26 11:41:04 <JyZyXEL> i wish i had some v0.7 synchronization statistics to compare with
 85 2013-01-26 11:41:36 <sipa> finway: for the value, see class CCoins in main.h
 86 2013-01-26 11:42:01 <JyZyXEL> though its quite obvious 5 blocks / minute is way slower than intended
 87 2013-01-26 11:42:46 <JyZyXEL> there would be hardly any new users if they have to wait a month for their copy of the blockchain
 88 2013-01-26 11:43:29 <gmaxwell> 04:32 < sipa> i'm not sure why, but iirc it kills disk i/o latency
 89 2013-01-26 11:43:33 <finway> thankssipa
 90 2013-01-26 11:43:36 <gmaxwell> Truecrypt does??? but linux dmcrypt doesn't.
 91 2013-01-26 11:44:34 <CodeShark> gmaxwell: I was able to reproduce that issue of a change address matching a sending address when two instances of the same wallet are used on the master branch: http://blockchain.info/tx/2ff65a89b87e2ed43d5156c6875e62ef0c3ff30996320d79c4e9cd907c4a255d
 92 2013-01-26 11:44:39 <gmaxwell> truecrypt does because it does multiple sync writes for every write to avoid blocks getting torn when writing because it doesn't know for sure how its blocks line up with disk sectors.
 93 2013-01-26 11:45:00 <gmaxwell> CodeShark: sweet, how did you do that? (also! test with testnet!)
 94 2013-01-26 11:45:49 <sipa> JyZyXEL: on my own laptop, i see several blocks per second at least on recent code
 95 2013-01-26 11:46:14 <gmaxwell> CodeShark: it _should_ retire the address from the pool when it sees that transaction.
 96 2013-01-26 11:46:18 <sipa> with parallel signature checking even closer to 10 blocks/s iirc
 97 2013-01-26 11:46:22 <CodeShark> yes, agreed, gmaxwell
 98 2013-01-26 11:46:48 <CodeShark> the correct behavior would be for it to look through the pool each time it sees a transaction and remove the address from the pool
 99 2013-01-26 11:46:49 <sipa> gmaxwell: i know of no code that does that, but for bip32 wallets it certainly should get that
100 2013-01-26 11:47:30 <sipa> no response from hal, btw :(
101 2013-01-26 11:48:40 <gmaxwell> CodeShark: it should probably also retire all intermediate addresses..
102 2013-01-26 11:48:43 <JyZyXEL> im not sure if the performance hit could be so massive
103 2013-01-26 11:48:52 <CodeShark> gmaxwell: yes
104 2013-01-26 11:49:17 <gmaxwell> JyZyXEL: from truecrypt? That much is well known (and not unique to bitcoin??? look up random write numbers for it)
105 2013-01-26 11:49:19 <sipa> JyZyXEL: as said, 0.7 uses bdb for its database, which requires tons of synchronous disk writes
106 2013-01-26 11:49:47 <sipa> so much thay it is easily the limiting factor
107 2013-01-26 11:50:05 <sipa> 0.8 uses leveldb which does less i/o and mostly sequential
108 2013-01-26 11:50:10 <JyZyXEL> i never realized TC can have such a giantic effect
109 2013-01-26 11:50:26 <sipa> and uses a different database layout that needs less storage
110 2013-01-26 11:50:39 <gmaxwell> The TC effect is mostly on random writes.. reads are not really impacted.
111 2013-01-26 11:50:49 <sipa> and it caches changes in memory and writes them in bulk
112 2013-01-26 11:51:01 <gmaxwell> TC suffers because its a block device on a file system, I guess.
113 2013-01-26 11:51:26 <JyZyXEL> hmm so i guess thats why it doesn't show up really bad in normal use
114 2013-01-26 11:51:46 <JyZyXEL> i guess its time to go SSD
115 2013-01-26 11:52:00 <sipa> bitcoin is quite a non-usual load for consumer hardware
116 2013-01-26 11:52:24 <CodeShark> I like the idea of having two separate key pools: one for change addresses, the other for receiving addresses
117 2013-01-26 11:52:40 <sipa> i doubt many people maintain >GiB databases on their home computer
118 2013-01-26 11:52:48 <gmaxwell> CodeShark: yea, bip32 gets you that.
119 2013-01-26 11:52:51 <CodeShark> right
120 2013-01-26 11:53:52 <CodeShark> as it is right now, it's a little problematic - if another instance of the wallet sends change, there's really no way for the first instance to know whether it is change or whether the user deliberately sent to that address
121 2013-01-26 11:54:47 <CodeShark> if the change addresses could never show up when you call getnewaddress, it wouldn't be an issue
122 2013-01-26 11:55:08 <CodeShark> (unless the user decided to look in wallet.dat directly)
123 2013-01-26 11:56:04 <CodeShark> wallet synchronization issues are far more serious than just the issue of a change address being reused
124 2013-01-26 11:56:16 <CodeShark> in fact, the change reuse issue is arguably one of the lesser issues :)
125 2013-01-26 11:56:39 <gmaxwell> yea, but most of them are not solvable. (e.g. account mapping)
126 2013-01-26 11:57:37 <CodeShark> account mapping could be solved by separating the account mapping stuff from the keystore entirely and letting the user sync the accounts by storing them somewhere in the cloud
127 2013-01-26 11:58:44 <CodeShark> but without deterministic wallets, there's still a key pool divergence issue
128 2013-01-26 12:00:31 <CodeShark> and merging pseudorandom keystores doesn't sound like a very elegant solution
129 2013-01-26 12:02:06 <CodeShark> it would be best to reduce the amount of exchanging of private keys as much as possible
130 2013-01-26 12:02:50 <CodeShark> compromised account mappings aren't as serious as compromised private keys
131 2013-01-26 12:03:00 <CodeShark> in most instances :)
132 2013-01-26 12:04:11 <CodeShark> but I suppose even with random keys, we could support a merge operation...but we're also giving users plenty of opportunities to shoot themselves in the foot
133 2013-01-26 12:06:23 <CodeShark> sipa: so you don't have any other channels through which to try to contact hal or find out what his situation is like?
134 2013-01-26 12:08:56 <gmaxwell> CodeShark: We know in general what his situation is like, and it implies long communication times.
135 2013-01-26 12:09:13 <sipa> CodeShark: first time i mailed him, i got an answer within a day or two
136 2013-01-26 12:09:20 <sipa> after that, nothing
137 2013-01-26 12:09:30 <CodeShark> sipa: when was the first time you mailed him?
138 2013-01-26 12:11:23 <sipa> december 8th
139 2013-01-26 12:11:51 <gmaxwell> I would imagine that there is some level of nagging that is optimal for his wellbeing. Keep him aware that there are people out here who need him, without flooding him with I/O he has a hard time keeping up with.
140 2013-01-26 12:14:33 <CodeShark> I hope he's ok. I don't think we should be solely relying on him to move forward with this, though.
141 2013-01-26 12:14:49 <gmaxwell> Well, we won't.
142 2013-01-26 12:45:06 <CodeShark> gmaxwell: another simpler example of the change address reuse issue I just did: http://blockchain.info/tx/270915c405dab1a97668f190e7509a2049c12ff3060f275a3b110379af363a57
143 2013-01-26 12:47:28 <CodeShark> change.
144 2013-01-26 12:47:28 <CodeShark> steps to produce it: 1) start bitcoind with no wallet.dat file and let it create one with at least several addresses in its pool (I used 100). 2) stop bitcoind. 3) make a copy of wallet.dat. 4) restart bitcoind, getnewaddress, send to it from another wallet, wait for it to confirm. 5) stop bitcoind. 6) swap the two wallet files. 7) restart bitcoind with -rescan. 8) send to another wallet an amount that will require
145 2013-01-26 15:32:56 <Eliel> there's a question on reddit about a transaction that's been waiting for confirmation for over 12 hours already. It even has the standard fee in it. http://blockchain.info/tx/504b21f2eee449a867dcc4e18b307e8da68ead68164603068aca01da9ae1cdfc
146 2013-01-26 15:33:23 <Eliel> I can't figure out why this transaction would end up not being confirmed for this long.
147 2013-01-26 15:37:32 <Luke-Jr> Eliel: that one depends on an unconfirmed txn
148 2013-01-26 15:37:35 <CodeShark> either it conflicts with another transaction or it didn't propagate very widely
149 2013-01-26 15:37:47 <CodeShark> or it depends on a transaction that conflicts
150 2013-01-26 15:39:00 <Luke-Jr> Eliel: how did they send that?
151 2013-01-26 15:39:44 <Eliel> it's apparently from coinbase.
152 2013-01-26 15:39:47 <CodeShark> looks like the result of a long chain of unconfirmed transactions
153 2013-01-26 15:39:53 <Luke-Jr> sigh
154 2013-01-26 15:39:55 <Eliel> http://www.reddit.com/r/Bitcoin/comments/17bhkn/coinbase_transaction_still_has_not_confirmed/
155 2013-01-26 15:39:58 <CodeShark> heh
156 2013-01-26 15:40:05 <ThomasV> how does https://en.bitcoin.it/w/images/en/7/70/Bitcoin_OpCheckSig_InDetail.png generalize to op_checkmultisig?
157 2013-01-26 15:40:23 <ThomasV> etotheipi_: ^^
158 2013-01-26 15:41:38 <ThomasV> more precisely, how do I build the subscript for signing?
159 2013-01-26 15:41:56 <sipa> ThomasV: exactly the same way
160 2013-01-26 15:42:23 <sipa> as the input being signed is erased entirely before signing, the actual script there doesn't matter
161 2013-01-26 15:42:29 <Luke-Jr> Eliel: \\http://blockchain.info/tx/0743aa0cf0241c4138d9ff9bdec6262d0a80c5b2dd91696731351ce895e0dac8
162 2013-01-26 15:42:58 <ThomasV> sipa: what is erased entirely?
163 2013-01-26 15:43:05 <sipa> ThomasV: the scriptSig
164 2013-01-26 15:44:03 <sipa> ThomasV: step 7
165 2013-01-26 15:44:12 <Luke-Jr> Eliel: want me to CC you to an email to Brian?
166 2013-01-26 15:44:28 <Eliel> Luke-Jr: sure, I'm interested in seeing the conversation.
167 2013-01-26 15:44:38 <Luke-Jr> Eliel: PM me your email ;)
168 2013-01-26 15:44:43 <Luke-Jr> ^ anyone else who wants CC
169 2013-01-26 15:45:07 <Luke-Jr> Eliel: got a link to the reddit btw?
170 2013-01-26 15:45:07 <sipa> brian?
171 2013-01-26 15:45:11 <Luke-Jr> sipa: Coinbase guy
172 2013-01-26 15:45:14 <sipa> ah
173 2013-01-26 15:45:17 <Eliel> I pasted it above already
174 2013-01-26 15:45:29 <Luke-Jr> ah, was analyzing it while you posted that :P
175 2013-01-26 15:45:48 <ThomasV> sipa: my question is more concerning step 8.. is the subscript something like [op_hash106,  push 20_bytes,  op_equal] ?
176 2013-01-26 15:46:41 <sipa> ThomasV: step 8 is just copying the output script of the txout being spent
177 2013-01-26 15:46:48 <sipa> ThomasV: i dislike the term 'subscript'
178 2013-01-26 15:47:16 <ThomasV> I dislike it too, but I've been trying to do that for hours, no success :P
179 2013-01-26 15:47:58 <ThomasV> I am trying to verify the signature manually, for this tx: http://blockchain.info/tx-index/30888901
180 2013-01-26 15:48:36 <ThomasV> so try to build the string to sign, and to check the signatures against the sigs in that tx
181 2013-01-26 15:49:37 <ThomasV> sipa: this is the string I assume should be verified for that tx: 0100000001aca7f3b45654c230e0886a57fb988c3044ef5e8f7f39726d305c61d5e818903c0000000017a914f815b036d9bbbce5e9f2a00abd1bf3dc91e9551087ffffffff0140420f00000000001976a914ae56b4db13554d321c402db3961187aed1bbed5b88ac0000000001000000
182 2013-01-26 15:50:57 <sipa> ThomasV: you didn't forget to append the hashtype?
183 2013-01-26 15:51:11 <ThomasV> no, maybe the chat cut it
184 2013-01-26 15:51:22 <ThomasV> it ends with 01000000
185 2013-01-26 15:51:38 <sipa> ah, my screen session is messed up
186 2013-01-26 15:51:44 <sipa> i'll have a look later, gtg now
187 2013-01-26 15:51:54 <ThomasV> ok thanks
188 2013-01-26 15:54:07 <Luke-Jr> http://www.reddit.com/r/Bitcoin/comments/17bhkn/coinbase_transaction_still_has_not_confirmed/c83za7g fwiw
189 2013-01-26 15:54:52 <CodeShark> yeah, tx 0743aa0cf0241c4138d9ff9bdec6262d0a80c5b2dd91696731351ce895e0dac8 is the problem
190 2013-01-26 15:55:56 <Luke-Jr> the way I see it, considering it will be confirmed by Eligius eventually, the fee system is working as intended here
191 2013-01-26 15:56:01 <CodeShark> my nodes haven't even seen it
192 2013-01-26 15:56:22 <Luke-Jr> CodeShark: odd, the 0.0005 BTC fee *should* have been enough to get it relayed
193 2013-01-26 15:56:45 <CodeShark> oh wait...nvm
194 2013-01-26 15:56:52 <CodeShark> http://blockhawk.net/bitcoin/tx/0743aa0cf0241c4138d9ff9bdec6262d0a80c5b2dd91696731351ce895e0dac8
195 2013-01-26 15:57:35 <Luke-Jr> CodeShark: you know B??? is standard Unicode, right? :P
196 2013-01-26 15:58:18 <CodeShark> beh, my text editors don't like it very much so it's easier to just use &#3647;
197 2013-01-26 15:59:10 <CodeShark> so why did this transaction not confirm yet?
198 2013-01-26 16:08:17 <jeremias> http://blockchain.info/tx/604192720889218c6767a96c74315c863b80a0a8d883d6bf999d472c85b45960
199 2013-01-26 16:08:30 <jeremias> could there be a reason why this transaction is not included to block chain?
200 2013-01-26 16:15:06 <Luke-Jr> jeremias: is it from Coinbase?
201 2013-01-26 16:41:18 <jeremias> Luke-Jr: yeah it was from the coinbase
202 2013-01-26 16:41:48 <Luke-Jr> jeremias: seems Coinbase has a bug; I've contacted them about it
203 2013-01-26 16:41:58 <jeremias> ok, good to know, thanks
204 2013-01-26 16:42:19 <lianj> his tx is confirmed now though
205 2013-01-26 16:42:42 <Luke-Jr> yeah, looks like Coinbase transactions are reliant on Eligius for now :P
206 2013-01-26 16:42:51 <sipa> how so?
207 2013-01-26 16:43:05 <Luke-Jr> sipa: they're not paying fees meeting other pools/miners' rules
208 2013-01-26 16:43:11 <sipa> ic
209 2013-01-26 16:44:08 <Luke-Jr> and only Eligius is running the child-pays-for-parent code
210 2013-01-26 17:03:21 <CodeShark> really? other miners are not doing child-pays-for-parent?
211 2013-01-26 17:04:05 <CodeShark> I would think that's super important in how you prioritize transactions
212 2013-01-26 17:04:09 <Luke-Jr> CodeShark: I'm not aware of many, at least
213 2013-01-26 17:04:25 <Luke-Jr> CodeShark: it's not merged in mainline yet
214 2013-01-26 17:04:45 <CodeShark> so you're saying that if there's a transaction that pays no fee, then there's another transaction depending on it that pays an enormous fee, most miners will simply ignore it?
215 2013-01-26 17:05:00 <Luke-Jr> yes
216 2013-01-26 17:05:19 <CodeShark> lol
217 2013-01-26 17:05:51 <CodeShark> I would think that the priority calculation is about maximizing the fee to size ratio
218 2013-01-26 17:06:03 <Luke-Jr> it is
219 2013-01-26 17:06:13 <Luke-Jr> but it doesn't consider dependencies w/o the patch
220 2013-01-26 17:07:19 <CodeShark> you sort transactions by fee, regardless of size - then you recurse back on the inputs
221 2013-01-26 17:07:23 <sipa> Luke-Jr: regarding that, how up-to-date is #1647?
222 2013-01-26 17:07:35 <sipa> Luke-Jr: i must admit i never managed to go through the code changes involved
223 2013-01-26 17:07:46 <Luke-Jr> sipa: should be completely up to date
224 2013-01-26 17:08:56 <Luke-Jr> sipa: although admittedly, it is NOT the exact code running on Eligius
225 2013-01-26 17:09:17 <Luke-Jr> sipa: Eligius is still using 0.6.0.x-based code, without Gavin's sort-by-fee change stuff
226 2013-01-26 17:10:02 <Luke-Jr> so one of these days, I'll have to try yet again to make master work???
227 2013-01-26 17:11:42 <sipa> Luke-Jr: in case a parent and a child are detected
228 2013-01-26 17:12:01 <sipa> does the code try to just mine the parent in case parent+child fail priority checks?
229 2013-01-26 17:13:08 <Luke-Jr> sipa: basically, it just includes any unconfirmed parents in the child's calculations
230 2013-01-26 17:13:22 <sipa> that's not an answer to my question :)
231 2013-01-26 17:13:39 <Luke-Jr> I didn't understand the question :P
232 2013-01-26 17:13:52 <sipa> imagine you have a parent and a child in your mempool
233 2013-01-26 17:14:18 <sipa> so at some point you try to include the combination
234 2013-01-26 17:14:41 <sipa> but this fails the priority checks
235 2013-01-26 17:14:47 <sipa> so it isn't included
236 2013-01-26 17:14:55 <sipa> does it try to just mine the parent itself?
237 2013-01-26 17:15:05 <Luke-Jr> no, they're included only after it passes the priority checks
238 2013-01-26 17:15:20 <sipa> still not an answer
239 2013-01-26 17:15:25 <Luke-Jr> together or not at all (unless the parent gets in independently)
240 2013-01-26 17:15:33 <sipa> ok
241 2013-01-26 17:16:08 <Luke-Jr> if there are multiple children, another one might include the parent, which can cause the first child to be reevaluated independently, as well
242 2013-01-26 17:17:51 <sipa> why does pmapInfoById need to be included in CTxInfo?
243 2013-01-26 17:17:57 <sipa> there is just one such map, no?
244 2013-01-26 17:18:13 <Luke-Jr> sipa: one map per CreateBlock call
245 2013-01-26 17:19:04 <sipa> ok, sure, but from the scope of the CreateBlock call, every CTxInfo that it reaches will always point to its own mapInfoById?
246 2013-01-26 17:19:24 <Luke-Jr> yes
247 2013-01-26 17:19:47 <Luke-Jr> probably the whole code should be refactored so it maintains a list independent of CreateBlock calls, tbh
248 2013-01-26 17:20:01 <Luke-Jr> but that can be better done on its own, not trying to introduce a new functionality at the same time
249 2013-01-26 17:20:31 <sipa> sure, that's something else
250 2013-01-26 17:20:55 <sipa> but independent from that, having CTxInfo entries that need to know which map they are part of seems very ugly
251 2013-01-26 17:32:01 <CodeShark> where is this CreateBlock and CTxInfo code?
252 2013-01-26 17:32:24 <sipa> #1647
253 2013-01-26 17:32:44 <CodeShark> oh
254 2013-01-26 18:59:55 <gavinandresen> I just made the patch for the 0-confirmation vulnerability public: https://github.com/bitcoin/bitcoin/pull/2223
255 2013-01-26 19:05:07 <petertodd> testing it now
256 2013-01-26 19:05:52 <petertodd> FWIW: "far in the future" meant 7000 years when I attacked blockchain's mixer...
257 2013-01-26 19:14:29 <jgarzik> petertodd: meta: when addressing others in a github comment, use "@" in front of their nick
258 2013-01-26 19:14:45 <jgarzik> petertodd: e.g. "@jgarzik: your last comment was completely insane, sod off"
259 2013-01-26 19:15:10 <jgarzik> petertodd: github does a bit of highlighting and linking and notifications with '@'
260 2013-01-26 19:17:15 <petertodd> jgazik: thanks, I didn't know that. Where did I have a comment addressing someone? I'll try it out.
261 2013-01-26 19:19:19 <jgarzik> petertodd: https://github.com/bitcoin/bitcoin/pull/2160
262 2013-01-26 19:19:38 <jgarzik> petertodd: s/gavin:/@gavinandresen/ etc.
263 2013-01-26 19:20:40 <petertodd> ah right
264 2013-01-26 19:20:58 <petertodd> (I was hoping it was a comment to you, so I could edit it to say you're insane)
265 2013-01-26 19:21:24 <sipa> you still can :p
266 2013-01-26 19:21:52 <petertodd> lol
267 2013-01-26 19:23:40 <petertodd> done
268 2013-01-26 19:27:45 <petertodd> any suggestions on limiting memory usage for bitcoind? just noticed one I have on an ec2 micro instance died (512MiB)
269 2013-01-26 19:28:28 <petertodd> (0.7.2)
270 2013-01-26 19:29:09 <sipa> -maxconnections
271 2013-01-26 19:29:20 <sipa> network memory buffers tend to grow large
272 2013-01-26 19:30:21 <petertodd> thanks, I'll try that
273 2013-01-26 19:30:33 <petertodd> std::bad_alloc is almost certainly mem exhaustion right?
274 2013-01-26 19:30:47 <sipa> yes
275 2013-01-26 21:06:55 <HM> my bitcoind instance runs on a VPS with 512MiB
276 2013-01-26 21:07:06 <HM> seems to gobble 300 MiB after a day or so
277 2013-01-26 21:09:44 <gmaxwell> HM: you're probably not listening to inbound connections.
278 2013-01-26 21:15:08 <HM> gmaxwell: i have 98 connections atm
279 2013-01-26 21:15:21 <HM> inbound is definitely working
280 2013-01-26 21:15:31 <sipa> that's very impressive with only 300 MiB
281 2013-01-26 21:15:42 <HM> yeah well... there might be 100 GiB of swap
282 2013-01-26 21:15:44 <HM> i haven't checked
283 2013-01-26 21:17:33 <sipa> gavinandresen: any reason why excessive coinbase output value doesn't trigger DoS?
284 2013-01-26 21:17:42 <sipa> imho, it should
285 2013-01-26 21:17:55 <sipa> you can't do that accidentally
286 2013-01-26 21:18:41 <gavinandresen> sipa: that seems right, I can't think of any reason NOT to DoS for a bad coinbase
287 2013-01-26 21:18:55 <petertodd> HM: is that 0.7 or 0.8?
288 2013-01-26 21:19:25 <sipa> gavinandresen: i'm working on a patch to improve dealing with errors during validation - i'll include a commit to change that
289 2013-01-26 21:20:03 <gavinandresen> sipa:  although??? you might be able to use it as an oracle to determine whether a peer is a SPV node or full node.  There are probably much cheaper ways of finding that out, though
290 2013-01-26 21:20:19 <HM> petertodd: 0.7.2
291 2013-01-26 21:20:57 <sipa> gavinandresen: like looking at their announced service bits?
292 2013-01-26 21:21:09 <gavinandresen> yeah???.
293 2013-01-26 21:21:12 <CodeShark> isn't that what they're for? :)
294 2013-01-26 21:21:18 <sipa> it's not like we're trying to hide that information
295 2013-01-26 21:21:30 <HM> there is quite a bit of swap actually
296 2013-01-26 21:21:34 <HM> eesh
297 2013-01-26 21:21:59 <HM> oh well, drive space is cheap
298 2013-01-26 21:22:41 <sipa> using disk swap because an application using a too large network buffer seems silly though :p
299 2013-01-26 21:24:29 <petertodd> HM: ahh, yeah, I don't have swap on the machine I was having problems with (swap on ec2 can be expensive becuase there is a small fee per io req)
300 2013-01-26 21:25:01 <HM> yeah
301 2013-01-26 21:25:46 <HM> but the guys here will soon have it running on a Z80 :P
302 2013-01-26 21:26:24 <sipa> if you're willing to wait a century per block, fine
303 2013-01-26 21:29:03 <CodeShark> I'm working on an abacus implementation :p
304 2013-01-26 21:29:46 <CodeShark> ever do ECDSA validation on an abacus?
305 2013-01-26 21:29:46 <petertodd> I'm genetically engineering a breed of ants that implement a turing machine, so if any of you want to do a port...
306 2013-01-26 21:29:57 <CodeShark> lol
307 2013-01-26 21:30:09 <D34TH> CodeShark: just need fast fingers is all
308 2013-01-26 21:30:11 <CodeShark> I think such ants already exist - if we only knew how to program them
309 2013-01-26 21:30:18 <sipa> seems easy to port that to DragonFly BSD
310 2013-01-26 21:30:20 <CodeShark> something to do with pheromones
311 2013-01-26 21:31:00 <HM> http://alpha.zimage.com/~ant/antfarm/ants/ChipAnt2.jpg
312 2013-01-26 21:31:03 <CodeShark> and for our next lesson: implementing a NAND gate using a colony of ants and synthetic pheromones :p
313 2013-01-26 21:31:55 <petertodd> odd... 0.7.1 checks that txin's are unspent in the mempool on sendrawtransaction right? because I'm finding I can send a tx multiple times on my node without it getting rejected the second time, which is what usually happens
314 2013-01-26 21:32:26 <petertodd> for that matter, anyone else see afd97dd4fe30b35f1cc98a394562f7402fc7c4c4be0a01b5422af33cb9f8fcdf in their mempool?
315 2013-01-26 21:32:45 <gavinandresen> exactly the same transaction?  It'll notice it is in the mempool, say "OK", then rebroadcast it.
316 2013-01-26 21:33:07 <petertodd> exactly, ah, ok, I probably didn't notice that before
317 2013-01-26 21:33:10 <gavinandresen> that's a feature??? otherwise there's no way to rebroadcast a transaction every once in a while
318 2013-01-26 21:33:18 <CodeShark> if you restart bitcoind, though, it will forget, right? so you can try to resend a conflicting transaction
319 2013-01-26 21:33:38 <petertodd> yeah, this is a non-conflicting one
320 2013-01-26 21:33:51 <gavinandresen> yes, or just call signrawtransaction twice and you'll get two different txids for the same inputs, and the second will be rejected
321 2013-01-26 21:34:39 <petertodd> gavin: the sign twice bit is probably why I didn't realize the rebroadcast behavior existed
322 2013-01-26 21:35:24 <gavinandresen> yeah, signing creates a random nonce, so different signature, so different txid every time
323 2013-01-26 21:36:41 <gavinandresen> CodeShark:  it will forget memory pool transactions that aren't to or from keys in your wallet.
324 2013-01-26 21:36:44 <petertodd> related: any plans on deriving the nonce from the tx+private key or something to avoid duplicating nonces?
325 2013-01-26 21:36:59 <CodeShark> gavinandresen: right - the wallet keeps its own transaction history that is separate from the mempool
326 2013-01-26 21:37:03 <sipa> petertodd: i wouldn't object, if some cryptographer can sign off on that
327 2013-01-26 21:37:09 <sipa> petertodd: i know reason why not, though
328 2013-01-26 21:37:22 <sipa> *no
329 2013-01-26 21:38:05 <petertodd> sipa: could do the first implementation as tx+private key derivation mixed in with the existing random bytes mechanism
330 2013-01-26 21:38:06 <CodeShark> there needs to be a random number used to generate a signature
331 2013-01-26 21:38:22 <Luke-Jr> sipa: it's quite simple to accidentally make a block that's got the wrong coinbase output, sadly :/
332 2013-01-26 21:38:24 <petertodd> sipa: then either could fail
333 2013-01-26 21:38:53 <sipa> Luke-Jr: if you have a buggy implementation, i don't mind disconnecting you
334 2013-01-26 21:39:10 <sipa> Luke-Jr: what i want to avoid if you being able to broadcast data that will cause others to be disconnected
335 2013-01-26 21:39:37 <CodeShark> I suppose if you have a good hash function on the tx (without signature) you could derive the nonce from that
336 2013-01-26 21:39:43 <sipa> CodeShark: i don't think so - ECDSA is defined to use a random input, but there are very similar schemes which don't
337 2013-01-26 21:40:03 <sipa> CodeShark: and the nonce must definitely be based on the secret key
338 2013-01-26 21:40:17 <sipa> if not, any validator can know the nonce, and sign anything he wants
339 2013-01-26 21:40:25 <CodeShark> the problem with ECDSA and nonrandom inputs is if you have two signatures generated with the same random input it converts the difficult discrete log problem into a simple algebraic one
340 2013-01-26 21:40:29 <sipa> wait, it's not quite that bad
341 2013-01-26 21:40:57 <gavinandresen> The benefits of switching to a non-random nonce would have to be pretty darn big to convince me the risk is worth it
342 2013-01-26 21:41:56 <sipa> gavinandresen: not depending on the presence of good cryptographic randomness is a good reason, imho
343 2013-01-26 21:42:20 <Luke-Jr> sipa: could always XOR random with deterministic-pseudo-random..
344 2013-01-26 21:42:46 <petertodd> Granted, if the mechanisms proposed are just message and private key derivation, you might as well just add incoming tx's and your private keys to your entropy pool.
345 2013-01-26 21:43:05 <petertodd> Same effect, but much less prone to buggy implementations.
346 2013-01-26 21:43:17 <gavinandresen> sipa: oh, I can imagine switching to a new signature algorithm that doesn't need a good source of pseudo-randomness.  But, as you say, ECDSA is defined as "has a random nonce"
347 2013-01-26 21:44:11 <petertodd> gavin: We're lucky that it's perfectly ok for us to sign a given message exactly the same way every time.
348 2013-01-26 21:44:27 <petertodd> gavin: Better really, so the tx hashes are consistent.
349 2013-01-26 21:46:15 <CodeShark> sha(sha(private key + tx without sigscripts))?
350 2013-01-26 21:46:18 <HM> what's an example of an algo that doesn't require a good PRN?
351 2013-01-26 21:47:01 <gavinandresen> petertodd: but an attacker could still create two versions of a transactions, with two different nonces???  so I think having them be consistent would, at best, lull people into writing code that ASSUMES that there is only one signature possible
352 2013-01-26 21:47:36 <CodeShark> right, gavinandresen makes a good point - in order for there to be any benefit, the consistency would have to be enforced...and there's no way to enforce it
353 2013-01-26 21:48:23 <CodeShark> without using some other signature scheme that does not require a random nonce :)
354 2013-01-26 21:48:25 <sipa> HM: Ed25519 is very similar to ECDSA, but uses a well-defined scheme to generate its nonce deterministically from public key, private key, and message
355 2013-01-26 21:48:44 <petertodd> gavin: true, lulling people is a problem, although once consistency is enforced it'd be good to have deterministic sigs
356 2013-01-26 21:49:07 <CodeShark> sipa: but is it possible to verify that the nonce is in fact correct without knowing the private key?
357 2013-01-26 21:49:22 <sipa> CodeShark: no
358 2013-01-26 21:49:27 <CodeShark> so then it's pointless :)
359 2013-01-26 21:49:30 <sipa> it's not
360 2013-01-26 21:49:35 <CodeShark> no way to enforce it
361 2013-01-26 21:49:40 <sipa> no need to enforce it
362 2013-01-26 21:50:25 <CodeShark> the point is that nodes could choose to ignore the nonce derivation scheme
363 2013-01-26 21:50:36 <CodeShark> and there'd be no way for other nodes to detect it
364 2013-01-26 21:50:40 <sipa> so, then they're hurting themself?
365 2013-01-26 21:51:32 <CodeShark> if you're only really interested in your own transactions consistently hashing and don't rely on this feature for security I suppose it could be useful
366 2013-01-26 21:53:14 <CodeShark> not really sure how useful this is in practice, though
367 2013-01-26 21:53:16 <sipa> gavin is right that it may cause people to assume there is only one signature possible
368 2013-01-26 21:53:33 <sipa> but that's not an inherent lack of security in the system
369 2013-01-26 21:53:52 <sipa> just one more way for people to screw up in implementations
370 2013-01-26 21:53:58 <CodeShark> yes
371 2013-01-26 21:54:01 <sipa> but so is depending on good randomness right now
372 2013-01-26 21:54:09 <sipa> (i.e., the bitcoincard people...)
373 2013-01-26 21:54:26 <CodeShark> good randomness is required for any crypto system
374 2013-01-26 21:54:42 <sipa> for many systems, not while signing
375 2013-01-26 21:54:53 <CodeShark> but you still need it to generate master keys and such
376 2013-01-26 21:55:07 <petertodd> sipa: note that the bitcoincard people said on the forums they understood that problem, and it was just that their test setup delibrately seeds the prng with a fixed vector
377 2013-01-26 21:55:27 <sipa> petertodd: oh, good
378 2013-01-26 21:55:32 <petertodd> CodeShark: yes, but in many applications you don't generate masters keys very often
379 2013-01-26 21:55:33 <sipa> i wasn't aware
380 2013-01-26 21:55:57 <sipa> petertodd: though i could have stolen some coins
381 2013-01-26 21:55:59 <sipa> in mainnet
382 2013-01-26 21:56:04 <HM> can't always trust the platform generator either
383 2013-01-26 21:56:04 <petertodd> sipa: yeah, still a bit sloppy to not disable that for a demo, but it was their money to lose
384 2013-01-26 21:56:39 <CodeShark> how much total was there to steal?
385 2013-01-26 21:56:55 <sipa> very little
386 2013-01-26 21:56:59 <petertodd> sipa: I've sent SIGHASH_NONE tx's out on mainnet before, and the txout has never been changed on me, yet...
387 2013-01-26 21:57:15 <sipa> CodeShark: maybe 1 BTC in total
388 2013-01-26 21:57:34 <sipa> (there were more vulnerable outputs, but those were already spent)
389 2013-01-26 21:57:59 <HM> SIGHASH_NONE is for offline transaction storage i take it
390 2013-01-26 21:58:09 <HM> like a blank cheque?
391 2013-01-26 21:58:24 <petertodd> HM: exactly, although offline transaction storage isn't the use-case
392 2013-01-26 21:58:41 <sipa> you still need a trustable miner to mine it for you
393 2013-01-26 21:58:48 <HM> well, not for immediate broadcast then
394 2013-01-26 21:58:53 <sipa> and be sure it won't get reverted afterwards
395 2013-01-26 21:59:12 <petertodd> SIGHASH_NONE + nLockTime has some interesting properties though
396 2013-01-26 21:59:33 <petertodd> I'm using that for the fidelity bond "throw away money" mechanism
397 2013-01-26 22:00:16 <HM> I was asking about this the other day. lock time is checked and signed, so it's safe to use on mainnet?
398 2013-01-26 22:00:37 <petertodd> HM: yes, it is signed
399 2013-01-26 22:01:19 <HM> tasty
400 2013-01-26 22:03:51 <petertodd> HM: speaking of, here, have fun: 01000000019da2ad399cd0b6319f0af3f4458380dff3d2524a1aa9cd2eb963bb5bad8509eb000000006b48304502204a22085f63e6cbbe5e02b03ef973fa71ecb636e58762cee2e1cb4d86cdb82144022100dfea0352c40e61a730b118ecf03704b9ebc8d140850333d1c83cc43aecb215a8822102aede205575263dfeb45226509a1d65c231893ade090fa9cd7eb71e90a4293c7dfffffffe0061540300
401 2013-01-26 22:17:19 <HM> petertodd: i didn;'t think you felt that way
402 2013-01-26 22:17:56 <sipa> ?
403 2013-01-26 22:18:43 <HM> sipa: just exchanging steganographic love notes
404 2013-01-26 22:19:01 <sipa> great
405 2013-01-26 22:19:12 <petertodd> lol
406 2013-01-26 22:22:33 <sipa> ;;bc,blocks
407 2013-01-26 22:22:34 <gmaxwell> I think SIGHASH_NONE is for "we added a new kind of fancy signature type in the future, and its softforking"
408 2013-01-26 22:22:35 <gribble> 218201
409 2013-01-26 22:23:44 <petertodd> gmaxwell: maybe, but there are a lot of uses for it, just one of many
410 2013-01-26 22:24:14 <petertodd> for instance you can donate money by giving someone a encrypted, none-signed transaction
411 2013-01-26 22:24:23 <petertodd> (ideally none + anyonecanpay)
412 2013-01-26 22:26:03 <HM> why is so much bitcoind code in header files?
413 2013-01-26 22:26:24 <petertodd> HM: lets the compiler inline more stuff
414 2013-01-26 22:26:35 <HM> uh hu
415 2013-01-26 22:26:46 <sipa> HM: some part has good reasons, but the major reason is historic
416 2013-01-26 22:27:18 <gmaxwell> HM: thats a very common C++ pattern.
417 2013-01-26 22:28:23 <HM> for template heavy code sure
418 2013-01-26 22:28:41 <sipa> most of our header code has no good reasons
419 2013-01-26 22:30:15 <gmaxwell> HM: even without templates "zomg inlining!"  (I'm not saying its a good reason, only that its common)
420 2013-01-26 22:31:23 <petertodd> gmaxwell: speaking of, is "compiled inlining", IE inlining stuff not in the header, supported yet on gcc? I seem to remember Microsoft's c++ compiler being able to do that
421 2013-01-26 22:31:27 <HM> funswitch-loops w00t
422 2013-01-26 22:32:04 <sipa> petertodd: -flto
423 2013-01-26 22:32:31 <HM> GNU binutils has a link optimiser, and there's -fwhole-program
424 2013-01-26 22:32:33 <HM> or something
425 2013-01-26 22:32:47 <gmaxwell> petertodd: sure, LTO.
426 2013-01-26 22:33:25 <petertodd> ah, cool, so you don't need inlined header stuff at all now? (modulo templates)
427 2013-01-26 22:33:40 <gmaxwell> Correct.
428 2013-01-26 22:34:04 <gmaxwell> At least in so far as if your have enough ram to LTO your whole program.
429 2013-01-26 22:35:05 <petertodd> True... bitcoind can't compile on a ec2 micro instance (512MiB ram)
430 2013-01-26 22:35:48 <HM> it's a slow ass compile as well
431 2013-01-26 22:35:54 <HM> due, i think, to bloaty headers
432 2013-01-26 22:36:02 <HM> very few forward decls
433 2013-01-26 22:37:07 <HM> anyone tried with clang?
434 2013-01-26 22:38:30 <gmaxwell> I wouldn't be too surprised if you couldn't compile hello world on an EC2 instance while pulling in the libraries we pull in. (the crazy json template library and boost).
435 2013-01-26 22:38:57 <HM> i hate boost spirit
436 2013-01-26 22:39:03 <HM> i'm fond of xpressive though
437 2013-01-26 22:39:57 <HM> xpressive could probably handle limited amounts of json
438 2013-01-26 22:41:23 <petertodd> given that you can compile python with cython I find it hard to think of a good reason to use C++ that doesn't involve compatibility
439 2013-01-26 22:42:14 <sipa> and efficiency?
440 2013-01-26 22:43:46 <petertodd> sipa: for efficiency, use cython
441 2013-01-26 22:44:13 <gmaxwell> Only works when you have obvious innerloops.
442 2013-01-26 22:44:37 <petertodd> writing your whole program in cython is quite reasonable
443 2013-01-26 22:44:51 <petertodd> and what cython compiles to is pretty good
444 2013-01-26 22:45:05 <sipa> not following (i'm not very python-aware, though)
445 2013-01-26 22:45:18 <sipa> do you mean writing stuff in python and having that compiled to machine code
446 2013-01-26 22:45:28 <sipa> or linking with subroutines written in C?
447 2013-01-26 22:45:59 <HM> I disagree, i think C++ is the way to go. If you wanted you could bolt on Lua for scripting
448 2013-01-26 22:46:08 <gmaxwell> scripting?!
449 2013-01-26 22:46:20 <HM> yes, scripting your daemon
450 2013-01-26 22:46:20 <petertodd> well, basically cython is a cython compiler that accepts a language very similar to python, similar enough that most python code is valid cython as is, yet at the same time you can add declarations in cython for variable types and the like
451 2013-01-26 22:46:51 <petertodd> the cython compiler outputs C code that targets the python C api, and at that point you've got code that is very close to the efficiency of C
452 2013-01-26 22:48:18 <gmaxwell> petertodd: yes, if you don't use a fair amount of python's features, like any of the library except numpy, strings. list comprehensions. generators. lambdas... or I assume a lot of stuff I'm not aware of.
453 2013-01-26 22:48:27 <petertodd> the numpy/scipy project - numerical computation and some symbolic and other stuff - uses cython extensively
454 2013-01-26 22:49:03 <HM> what would the benefit of moving to python be
455 2013-01-26 22:49:45 <petertodd> gmaxwell: list comprehensions have been added to cython IIRC, generators they're working on, same with lambdas, more to the point implementing a lot of that stuff in C++ efficiently is hard too (although boost has right?)
456 2013-01-26 22:50:12 <HM> c++ has lambdas, and generators
457 2013-01-26 22:50:27 <gmaxwell> petertodd:  Yes, thats why I mentioned numpy.. maybe it's gotten a lot faster since I last looked, but it was only like 20% faster on pybench than cypthon too.
458 2013-01-26 22:50:51 <gmaxwell> petertodd: sure but forcing you into a subdialect of the language further restricts the developers on the project.
459 2013-01-26 22:51:25 <petertodd> gmaxwell: You sure that's due to cython though? I thought numpy's internals were compiled only.
460 2013-01-26 22:51:43 <petertodd> gmaxwell: Yes, but the alternative is C++ :P
461 2013-01-26 22:52:32 <petertodd> It's not a perfect solution by any means, but at least from what I've seen it's a very good one that should be considered a lot more. Heck, even just sticking with pure-Python (or some other hll) would be a good solution for a lot of projects.
462 2013-01-26 22:55:05 <gmaxwell> A couple years back I had decided "my time is more valueable than the computer's I'll stop using C for one time programs and use python" ... but after too many cases where naive python was hundreds or even as much as 2000x slower than my C code, and finding I was wasting a lot of time twiddling the python to make it even remotely fast enough ( 'of of course, use xrange instead of range.. how does anyone not know that' ) I gave up.  I'm sympat
463 2013-01-26 22:55:55 <gmaxwell> Of course, C++ is a mess too. C isn't though. :P
464 2013-01-26 22:56:21 <sipa> I'm sympat[...] ?
465 2013-01-26 22:56:41 <gmaxwell> I'm sympathetic, but skeptical.
466 2013-01-26 22:57:28 <petertodd> Yeah, I don't do very much stuff where even being thousands of times slower matters, and the few that do I find I can get away with Cython. But hey, my day job is analog electronics, so take what I say with a grain of salt!
467 2013-01-26 22:57:52 <petertodd> My coworkers are mostly moving to Python/numpy/scipy from matlab and mathematical FWIW.
468 2013-01-26 22:58:29 <sipa> i'd be impressed if you could replace analog electronics by cython!
469 2013-01-26 22:58:34 <petertodd> s/mathematical/mathematica/
470 2013-01-26 22:59:01 <HM> i code c++ for a living, i guess we all have our own biases
471 2013-01-26 22:59:11 <petertodd> sipa: lol
472 2013-01-26 23:00:24 <petertodd> I think much of my bias is C++ is complex enough that I find it hard to "just jump in" and get stuff done, which isn't true of Python, and even Cython compiles to C-code that's readable enough to relatively quickly understand what it's doing under the hood.
473 2013-01-26 23:01:22 <sipa> well, in several place the use of C++ is restricted to a subset of the language
474 2013-01-26 23:01:41 <petertodd> I spent a summer learning lisp once actually, and my main complaint about the lisp compiler I was using was that while it was utterly magical, reading the assembler output was difficult at best.
475 2013-01-26 23:01:41 <sipa> which makes sense i think, as it means you don't need to force everyone to understand every feature
476 2013-01-26 23:01:58 <petertodd> sipa: yup, which is fine, but it means now working with two different projects is difficult
477 2013-01-26 23:02:04 <sipa> agree
478 2013-01-26 23:02:09 <gmaxwell> Python/numpy/scipy  vs matlab/octave is probably a more fair comparison. In mathlab your own code is generally horribly slow unless it spends all its time inside the matlab operators (which are fast). So long as numpy/scipy are fast enough then anything using them will be compariable to matlab.
479 2013-01-26 23:02:38 <sipa> s/mathlab/matlab/
480 2013-01-26 23:02:52 <HM> that's like PHP, it powers half the worlds blogs, it's dog slow...but most of its functions are just slim wrappers in to C libraries
481 2013-01-26 23:03:09 <HM> (which is why it's so inconsistent)
482 2013-01-26 23:03:15 <petertodd> gmaxwell: For sure. Early numpy used C libraries for everything speed critical and the rest was Python, it's only more recently that Cython's been used.
483 2013-01-26 23:03:23 <HM> any scripting language will be fast if it just wraps optimised C libraries
484 2013-01-26 23:03:44 <petertodd> HM: usually, although memory management overhead can be a big problem
485 2013-01-26 23:03:52 <sipa> HM: and all tight loops of the logic are inside the C code
486 2013-01-26 23:04:24 <HM> yeah i think i read every single PHP value consumes something like 100 bytes
487 2013-01-26 23:04:33 <petertodd> Remember that every time you get your working set small enough to fit in another level of the memory hiarchy, your access speed goes up by at least an order of magnitude.
488 2013-01-26 23:05:51 <HM> like translating python to C....you're bypassing the interpreter (bytecode or otherwise), and getting direct dispatch through compile time linking
489 2013-01-26 23:06:29 <petertodd> HM: exactly, and in particular your instance attributes can become direct struct accesses in cython
490 2013-01-26 23:06:42 <HM> Oh...someone should port bitcoind to Chromes NaCl so we can run it in a browser :P
491 2013-01-26 23:06:56 <gmaxwell> HM: picocoin, not bitcoind
492 2013-01-26 23:06:57 <petertodd> (although not enough people know about Python's __slots__ feature as it is)
493 2013-01-26 23:08:20 <HM> I'm not really a python guy. i've started playing with it again recently for some web projects
494 2013-01-26 23:08:39 <HM> Django it's ORM is weak compared to what i'm used to in C++...
495 2013-01-26 23:08:49 <HM> *Django is nice but it's ORM...
496 2013-01-26 23:09:09 <sipa> you mean 'its' ?
497 2013-01-26 23:09:22 <CodeShark> no, I think he means it's
498 2013-01-26 23:09:34 <CodeShark> or at least in the second instance
499 2013-01-26 23:09:44 <CodeShark> or perhaps not p
500 2013-01-26 23:09:47 <HM> yeah its
501 2013-01-26 23:09:52 <sipa> seems to me the second instance is a replacement for the prefix of the first
502 2013-01-26 23:10:02 <CodeShark> yes
503 2013-01-26 23:10:10 <HM> I now need a third :P but i'll spare you
504 2013-01-26 23:10:28 <petertodd> What's weak about the ORM? Lack of object types and the like? (never used Django myself)
505 2013-01-26 23:12:07 <CodeShark> I find in the time it takes me to fully evaluate and master a new framework, I can write a couple of my own :p
506 2013-01-26 23:12:26 <sipa> writing code is easier than reading code
507 2013-01-26 23:12:30 <CodeShark> for sure
508 2013-01-26 23:12:55 <sipa> but the problem is that newly written code doesn't work yet, while code you're reading often already does :p
509 2013-01-26 23:14:17 <HM> petertodd: I just find it lacks the versatility
510 2013-01-26 23:14:28 <HM> it's also magic variable based
511 2013-01-26 23:14:42 <CodeShark> the problem with frameworks is that you end up investing a lot of time and effort into learning something that isn't really fundamental and doesn't really carry over to any project
512 2013-01-26 23:15:03 <CodeShark> whereas learning how to implement stuff from scratch does :)
513 2013-01-26 23:15:25 <Scrat> this might become big: http://appjs.org/
514 2013-01-26 23:15:29 <CodeShark> but there's a place for frameworks, of course
515 2013-01-26 23:15:49 <Scrat> hate js or not, it's a great idea
516 2013-01-26 23:16:02 <CodeShark> it's just that people often end up overrelying on them - and once they master the use of the hammer, everything starts to look like a nail
517 2013-01-26 23:16:23 <HM> Scrat, another server side javascript framework?
518 2013-01-26 23:16:25 <Scrat> using chromium as a multiplatform abstraction layer
519 2013-01-26 23:16:37 <Scrat> HM: desktop
520 2013-01-26 23:17:17 <sipa> desktop what?
521 2013-01-26 23:17:26 <HM> I avoid webdev wherever possible
522 2013-01-26 23:17:30 <CodeShark> the nicest thing about node.js is that it has HTTP, websockets, and JSON as built-in features of the language
523 2013-01-26 23:17:55 <Scrat> sipa: desktop application where the UI is html/css and code is JS
524 2013-01-26 23:18:05 <petertodd> HM: ah, yeah I've done almost no framework using code at all, analog electronics isn't exactly known for it :P although that said I have been dragged into some of the equivilent with fpga/microprocessor stuff, which makes hll frameworks look sane by comparison...
525 2013-01-26 23:18:17 <Scrat> making use of the great work that has been done in chromium/v8
526 2013-01-26 23:18:49 <HM> I think Google want to push 1) their extension APIs and 2) NaCl.
527 2013-01-26 23:19:01 <HM> Mozilla tried browser applications a decade ago with XUL
528 2013-01-26 23:19:14 <HM> nothing on the web gets saner with time
529 2013-01-26 23:20:12 <jgarzik> I dunno.  XML is slowly fading away, thankfully.
530 2013-01-26 23:20:18 <CodeShark> I'm still not sure about the HTML 5 approach to desktop app development - it's been talked about for a long time...but it still doesn't quite work like it was intended to
531 2013-01-26 23:20:26 <HM> yeah so now we have Json, which gives you structured data...
532 2013-01-26 23:20:34 <HM> but no standardised way to transform or query it
533 2013-01-26 23:21:07 <HM> so we're emulated binary protocols using json over websockets like a bunch of numpties
534 2013-01-26 23:21:11 <HM> *emulating
535 2013-01-26 23:22:33 <petertodd> isn't progress wonderful?
536 2013-01-26 23:22:36 <CodeShark> lol
537 2013-01-26 23:22:36 <Scrat> CodeShark: your UI will look exactly the same on all platforms, have hardware acceleration, have the huge node/libuv api which can do whatever you want AND the entire npm repo at your disposal
538 2013-01-26 23:22:40 <upb> < CodeShark> the nicest thing about node.js is that it has HTTP, websockets, and JSON as built-in features of the language
539 2013-01-26 23:22:41 <Scrat> I think that it will take off
540 2013-01-26 23:22:42 <jgarzik> websockets is awful; avoid at all costs.
541 2013-01-26 23:22:42 <upb> ^- how are they built int othe language?!?!
542 2013-01-26 23:22:53 <upb> theyre more supported by the standard lib
543 2013-01-26 23:23:01 <HM> jgarzik: you have no alternative for low latency comms in the browser
544 2013-01-26 23:23:10 <HM> but i agree
545 2013-01-26 23:23:14 <CodeShark> Scrat: I don't necessarily want my UI to look exactly the same on all platforms - different platforms have different look-and-feel and I sort of like my apps on the platform to have an inner consistency
546 2013-01-26 23:23:50 <Scrat> in that case dont style your input boxes or buttons and chromium will show OS widgets
547 2013-01-26 23:24:03 <Scrat> well for the most part :p
548 2013-01-26 23:24:30 <sipa> is OS understood to include desktop environment?
549 2013-01-26 23:24:44 <HM> world moves to web GUIs, 2 decades of GUI and widget dev wasted
550 2013-01-26 23:24:55 <CodeShark> it's more subtle than that - on some platforms, navigation involves a toolbar on the bottom - on others, it might be more customary to use pop-up menus
551 2013-01-26 23:25:09 <CodeShark> depending on the pointing device and such
552 2013-01-26 23:27:39 <CodeShark> sipa: O
553 2013-01-26 23:27:53 <CodeShark> sipa: I'd say "platform" could refer to the windowing environment
554 2013-01-26 23:28:06 <CodeShark> but I don't usually understand OS to mean that
555 2013-01-26 23:29:08 <Scrat> CodeShark: I see what you mean. but I don't think it's more work to incorporate these conventions compared to Qt for example
556 2013-01-26 23:32:06 <CodeShark> Scrat: I suppose at that level of detail you're right. However, I have to say that by default, Qt controls do tend to blend in pretty well with the windowing environment in which they run
557 2013-01-26 23:33:34 <HM> Gtkmm isn't bad to work with but tends to look like crud on Windows
558 2013-01-26 23:33:43 <HM> and feel very foreign
559 2013-01-26 23:33:56 <Scrat> browser is the new OS
560 2013-01-26 23:34:14 <CodeShark> Scrat: people have been saying that since the days of netscape vs. internet explorer
561 2013-01-26 23:34:14 <HM> Since Qt liberalised their license i've moved to it exclusively
562 2013-01-26 23:35:46 <sipa> Scrat: you're referring to Active Desktop? :p
563 2013-01-26 23:36:44 <HM> Desktop widgets didn't last long in Windows Vista/7
564 2013-01-26 23:36:55 <Scrat> sipa: damn that was aweful
565 2013-01-26 23:38:15 <sipa> hmm, never realized how similar the words awesome and aweful are
566 2013-01-26 23:38:51 <HM> I think you mean 'awful'
567 2013-01-26 23:39:19 <HM> take that you it's/its Nazi :P
568 2013-01-26 23:39:30 <Scrat> HM: you're right (lol at first search hit of "aweful")
569 2013-01-26 23:40:02 <sipa> HM: didn't intend to be a grammar nazi (i was honestly struggling to parse your sentence) - and you're right about awful :D
570 2013-01-26 23:40:50 <CodeShark> now we live in an age where practically every device comes with some form of webview built into its software stack - yet the trend has not been to abandon the platform's graphical shell in favor of a webview...but rather, supporting the webview embedded into its graphical shell, which often has unifying components that go beyond a URL bar, back, and forward buttons
571 2013-01-26 23:42:17 <CodeShark> I still don't see the webview replacing the shell altogether
572 2013-01-26 23:42:31 <HM> Best innovation i've seen recently on the web is Mozilla Persona
573 2013-01-26 23:42:45 <Scrat> but really imagine a game (not talking crysis here) where you don't have to write a single line of platform specific code and all your code is in high level interpreted languages. you might ask what's the difference between that and a "web app". the difference is that you control the experience, you have a ton of low level libs at your disposal and it will work 100% of the time
574 2013-01-26 23:43:24 <Scrat> web apps also can't spawn real tcp sockets (screw you websocket)
575 2013-01-26 23:43:52 <HM> I think NaCl is the bees knees as well because it means you don't have to port your language of choice to JVM (which is the only real portable runtime in the browser atm)
576 2013-01-26 23:44:41 <HM> Then you have people trying to make JS the 'assemby language of the web'
577 2013-01-26 23:44:47 <HM> which is kind of like admitting defeat
578 2013-01-26 23:45:33 <CodeShark> JS was originally just intended to provide hover-over effects and do web-form validation :p
579 2013-01-26 23:45:38 <Scrat> if it can run nacl in the future then it will broaden it's potential
580 2013-01-26 23:45:54 <Scrat> right now chrome is at 33% market share
581 2013-01-26 23:46:36 <HM> 
ACTION shrugs

582 2013-01-26 23:47:05 <Scrat> CodeShark: but but. google invested millions on V8 and the result is that it's only about 4 times slower than native C (on average)
583 2013-01-26 23:47:25 <Scrat> and I'll agree with you that it's horrible (wasn't it made in a week?)
584 2013-01-26 23:47:38 <HM> v8 is slower than luajit, which is a 1 man development effort
585 2013-01-26 23:47:41 <CodeShark> it's pretty remarkable how much faster JS runtime has gotten in the last few years
586 2013-01-26 23:48:05 <sipa> Scrat: what is made in a week?
587 2013-01-26 23:48:25 <Scrat> JS
588 2013-01-26 23:48:37 <sipa> :o
589 2013-01-26 23:48:43 <CodeShark> erm...
590 2013-01-26 23:49:00 <porquilho> fuck you talltim
591 2013-01-26 23:49:04 <porquilho> you ruined a good channel
592 2013-01-26 23:49:13 <sipa> talltim?
593 2013-01-26 23:49:15 <porquilho> there is always a talltim to ruin things
594 2013-01-26 23:49:19 <porquilho> i go sleep now
595 2013-01-26 23:49:20 <porquilho> bye
596 2013-01-26 23:49:29 <CodeShark> mmm...ok..
597 2013-01-26 23:49:36 <sipa> wth is he talking about
598 2013-01-26 23:49:40 <Scrat> HM now you're just trolling
599 2013-01-26 23:49:42 <petertodd> s/talltim/gmaxwell/ ?
600 2013-01-26 23:49:55 <porquilho> talltim
601 2013-01-26 23:49:58 <porquilho> not gmaxwell
602 2013-01-26 23:50:04 <sipa> wth is talltim
603 2013-01-26 23:50:22 <porquilho> he is here
604 2013-01-26 23:50:30 <porquilho> oh i guess he left
605 2013-01-26 23:50:33 <porquilho> he was here
606 2013-01-26 23:50:36 <porquilho> anyway i go sleep
607 2013-01-26 23:50:41 <CodeShark> yes, please do
608 2013-01-26 23:50:54 <porquilho> bye
609 2013-01-26 23:50:55 <porquilho> good night
610 2013-01-26 23:54:05 <CodeShark> I like the native client stuff - I tend to prefer the approach of using languages and libraries that compile to multiple native clients and having each client fetch a version optimized for it...than the approach of using clunky interpreted languages or JVM-like middleware that eats at your performance
611 2013-01-26 23:54:49 <CodeShark> and can make your programs behave in strange ways
612 2013-01-26 23:54:56 <HM> Some JITs are good enough for even performance sensitive applications
613 2013-01-26 23:55:28 <HM> Garbage collection is tricky though, i think it can make or break a language whether it works right or just 'ok'
614 2013-01-26 23:56:18 <CodeShark> JITs have gotten much better in the last decade or so - but most still are at least a little quirky
615 2013-01-26 23:56:33 <HM> Mozillas Rust looks damned interesting, the language itself has pointer types that help the garbage collector work more efficiently
616 2013-01-26 23:56:50 <CodeShark> reference counting?
617 2013-01-26 23:57:04 <CodeShark> does the developer have to give it clues?
618 2013-01-26 23:57:39 <sipa> i imagine it must mean something like smart pointers
619 2013-01-26 23:57:41 <HM> I don't know how it's implemented
620 2013-01-26 23:58:00 <HM> they call them borrowed pointers
621 2013-01-26 23:58:13 <CodeShark> probably something like boost::shared_ptr
622 2013-01-26 23:58:21 <HM> you mean std::shared_ptr :P
623 2013-01-26 23:58:33 <CodeShark> or std::shared_ptr :P
624 2013-01-26 23:58:51 <HM> it's similar semantically but moving between unique_ptr and shared_ptr is possible in Rust and moves the object between memory heaps
625 2013-01-26 23:59:09 <HM> this means the garbage collector can operate on each thread without stopping the world
626 2013-01-26 23:59:13 <HM> or something along those lines
627 2013-01-26 23:59:36 <HM> i think there are 4 pointer types...all with arcane syntax