1 2012-12-18 01:31:06 <aethero> Any devs alive?
2 2012-12-18 01:31:13 <aethero> Got something you might want to take a look at
3 2012-12-18 01:32:17 <gmaxwell> "don't ask to ask" :P
4 2012-12-18 01:32:42 <aethero> This is something that should be private, trust me :P
5 2012-12-18 01:32:44 <aethero> pming you
6 2012-12-18 03:06:50 <stealth222> are there any differences in message structures between testnet and the main network? I've built a client that connects fine to the main network but when I try to connect to testnet bitcoind gives me PROCESSMESSAGE MESSAGESTART NOT FOUND
7 2012-12-18 03:07:37 <stealth222> is the version message different?
8 2012-12-18 03:09:40 <Luke-Jr> the messagestart is different
9 2012-12-18 03:09:48 <stealth222> howso?
10 2012-12-18 03:10:04 <Luke-Jr> https://en.bitcoin.it/wiki/Protocol_specification#Message_structure
11 2012-12-18 03:10:35 <stealth222> I'm using the testnet magic bytes
12 2012-12-18 03:10:40 <stealth222> any other differences?
13 2012-12-18 03:11:38 <lianj> Luke-Jr: isn't testnet3 magic bytes different from testnet?
14 2012-12-18 03:11:50 <stealth222> I know the magic bytes are different
15 2012-12-18 03:12:15 <stealth222> any other differences?
16 2012-12-18 03:12:25 <Luke-Jr> lianj: unfortunately no
17 2012-12-18 03:12:35 <lianj> stealth222: address_version, p2sh_version, privkey_version
18 2012-12-18 03:13:02 <stealth222> ok, I'm also using address version 0x6f instead of 0x00
19 2012-12-18 03:13:34 <stealth222> if my client doesn't do any transaction signing, do I also need to worry about p2sh_version and privkey_version?
20 2012-12-18 03:13:35 <lianj> stealth222: sounds right then
21 2012-12-18 03:13:43 <stealth222> I guess not
22 2012-12-18 03:14:11 <stealth222> ok, then let me make absolute sure I'm not botching up the magic bytes
23 2012-12-18 03:14:27 <gmaxwell> 20:11 < lianj> Luke-Jr: isn't testnet3 magic bytes different from testnet?
24 2012-12-18 03:14:29 <gmaxwell> yes
25 2012-12-18 03:14:33 <gmaxwell> 20:12 < Luke-Jr> lianj: unfortunately no
26 2012-12-18 03:14:38 <Luke-Jr> gmaxwell: O.o since when?
27 2012-12-18 03:14:41 <stealth222> oh? it is?
28 2012-12-18 03:14:42 <stealth222> lol
29 2012-12-18 03:14:45 <stealth222> that would explain everything
30 2012-12-18 03:14:47 <gmaxwell> it got changed a bit before the first testnet3 release.
31 2012-12-18 03:15:00 <gmaxwell> because 'unfortunately', as you said.
32 2012-12-18 03:15:00 <stealth222> so what are the testnet3 magic bytes?
33 2012-12-18 03:15:10 <Luke-Jr> nice
34 2012-12-18 03:15:40 <stealth222> and did anything else change in testnet3? address_version?
35 2012-12-18 03:15:53 <Luke-Jr> stealth222: address_version was supposed to, but didn't <.<
36 2012-12-18 03:15:54 <gmaxwell> stealth222: no, address version did not change.
37 2012-12-18 03:15:55 <lianj> testnet3 is "\\x0b\\x11\\x09\\x07" right?
38 2012-12-18 03:16:09 <Luke-Jr> lianj: you taking care of wiki update?
39 2012-12-18 03:16:12 <gmaxwell> that looks correct. involves lots of '11' as gavin wanted.
40 2012-12-18 03:16:27 <stealth222> is that the network byte order?
41 2012-12-18 03:16:34 <lianj> Luke-Jr: got no account, but was about to suggest that, that was why i asked in the first place
42 2012-12-18 03:16:53 <gmaxwell> lianj: make one
43 2012-12-18 03:16:58 <gmaxwell> don't worry, we'll pay for t.
44 2012-12-18 03:17:00 <gmaxwell> er it
45 2012-12-18 03:17:44 <Luke-Jr> lol
46 2012-12-18 03:18:10 <Luke-Jr> are the magic bytes used big-endian anywhere? if not, I'd like to remove "magic value" from the wiki page <.<
47 2012-12-18 03:18:11 <gmaxwell> stealth222: this is bitcoin, you use rand() to pick the byte order.
48 2012-12-18 03:18:48 <stealth222> haha
49 2012-12-18 03:18:58 <lianj> oh, found my account. wth, the email reminder sent me the password in cleartext. great success
50 2012-12-18 03:19:16 <stealth222> time to whip out the packet sniffer, I guess...
51 2012-12-18 03:19:56 <lianj> stealth222: "\\x0b\\x11\\x09\\x07" is a string, so \\x0b is sent first
52 2012-12-18 03:20:14 <stealth222> ok
53 2012-12-18 03:20:57 <lianj> Luke-Jr: thanks for updating???
54 2012-12-18 03:21:11 <Luke-Jr> XD
55 2012-12-18 03:22:17 <stealth222> horray!
56 2012-12-18 03:22:18 <stealth222> thanks, guys
57 2012-12-18 03:22:32 <lianj> btw neverind what i said about the cleartext password, misread it, it ofcourse is a temporary password
58 2012-12-18 03:23:34 <lianj> stealth222: cool, nice to know its working for you now
59 2012-12-18 03:23:57 <Luke-Jr> holy crap, that's a long macro name: AVAILABLE_MAC_OS_X_VERSION_10_6_AND_LATER_BUT_DEPRECATED_IN_MAC_OS_X_VERSION_10_7
60 2012-12-18 03:53:45 <stealth222> LOL - block 0000000000bc915505318327aa0f18568ce024702a024d7c4a3ecfe80a893d6c in testnet3 has 6287 transactions
61 2012-12-18 03:54:32 <stealth222> does testnet also have the same block size limitations that the main network has?
62 2012-12-18 03:59:28 <jgarzik> stealth222: yes
63 2012-12-18 03:59:48 <stealth222> jgarzik: thanks
64 2012-12-18 03:59:50 <jgarzik> stealth222: testnet3 has many blocks and transactions that intentionally push the limits of the system. that's the point of testing, after all...
65 2012-12-18 04:00:23 <jgarzik> stealth222: test away, we only ask that you don't point ASICs or a huge GPU farm at it. that just bloats the chain for no reason.
66 2012-12-18 04:00:44 <jgarzik> (we'll reset it, so it's just a lot of annoyance for everyone, when a big miner fails to do their own local testnet)
67 2012-12-18 04:01:02 <jgarzik> but any other bitcoin software... please do use testnet
68 2012-12-18 04:03:22 <stealth222> since the magic bytes and genesis block hash are hardcoded into bitcoind, I guess resets require version upgrades?
69 2012-12-18 04:04:16 <stealth222> couldn't we set the magic bytes, genesis block hash, address version, etc... inside the config file?
70 2012-12-18 04:05:31 <stealth222> rather than testnet=1, we could have multiple named network configurations and have network=main, or network=testnet, or network=myprivatetestnet
71 2012-12-18 04:06:20 <stealth222> the main network would have to be treated specially anyhow in bitcoind with hardcoded checkpoints and all that
72 2012-12-18 04:06:28 <stealth222> but the other networks could be completely arbitrary
73 2012-12-18 04:07:55 <Luke-Jr> stealth222: would make it too easy for scamcoins
74 2012-12-18 04:08:15 <stealth222> scamcoins?
75 2012-12-18 04:09:15 <stealth222> can you elaborate?
76 2012-12-18 04:11:38 <stealth222> is "scamcoin" the codename for alternate block chains?
77 2012-12-18 04:11:41 <stealth222> lol
78 2012-12-18 04:14:45 <jgarzik> stealth222: Sure, that is possible. testnet is compiled in for simplicity's sake. So few will need anything else, it is not worth the bother.
79 2012-12-18 04:15:17 <stealth222> well, you gave an example earlier of ASIC miners wanting to build their own testnet
80 2012-12-18 04:15:47 <jgarzik> stealth222: Yes. Anybody testing mining would just use testnet3-in-a-box, and mine privately.
81 2012-12-18 04:16:12 <jgarzik> stealth222: No need to create a new genesis block or anything, with TNIAB.
82 2012-12-18 04:16:14 <stealth222> as things stand, it would require at the very least using a different set of magic bytes and probably the use of a different port if they weren't testing on a LAN
83 2012-12-18 04:16:26 <jgarzik> stealth222: no, just node isolation
84 2012-12-18 04:17:17 <stealth222> yeah, I suppose so - but there's greater risk that they'll leave out that connect=my.super.host line in the conf file
85 2012-12-18 04:17:35 <jgarzik> stealth222: TNIAB provides specialized bitcoin.conf setups
86 2012-12-18 04:18:04 <stealth222> yeah, I guess it isn't really a highly sought-after feature
87 2012-12-18 04:18:08 <jgarzik> stealth222: http://sourceforge.net/projects/bitcoin/files/Bitcoin/testnet-in-a-box/
88 2012-12-18 04:18:20 <jgarzik> automatically a closed network
89 2012-12-18 04:18:42 <jgarzik> stealth222: Correct. If you need something more specialized, you are likely skilled enough to do it yourself. :)
90 2012-12-18 04:18:52 <jgarzik> Changing is not difficult.
91 2012-12-18 04:19:39 <stealth222> does the current version support dynamically adding or removing nodes?
92 2012-12-18 04:19:44 <jgarzik> stealth222: Patch one file in libccoin, for example: https://github.com/jgarzik/picocoin/blob/master/lib/coredefs.c
93 2012-12-18 04:19:46 <stealth222> that's to say, can I tell it to add a node at runtime?
94 2012-12-18 04:20:14 <jgarzik> stealth222: Not in current bitcoin.git codebase. There is a pull request that adds dynamic-node-add support.
95 2012-12-18 04:20:59 <jgarzik> stealth222: Current code tries hard to kick any node that behaves strangely, so little call for dynamic-removal at this time. Would not be difficult to add.
96 2012-12-18 04:24:42 <stealth222> one feature I'd really like to add to bitcoind (I've actually written my own client out of the absolute need for this feature) is the ability to subscribe URLs to HTTP POST notifications when certain events occur
97 2012-12-18 04:25:04 <stealth222> I'd be willing to create a fork and submit a pull request and all that - never gone through the process, though
98 2012-12-18 04:26:11 <Luke-Jr> stealth222: which events? bitcoind already has a -blocknotify option, and probably will soon have a -walletnotify
99 2012-12-18 04:27:20 <stealth222> also notifications of particular scripts in inputs or outputs (arbitrary scripts or bitcoin addresses) and notifications of when a particular transaction has gotten a certain number of confirmations, for instance
100 2012-12-18 04:27:46 <stealth222> not just wallet addresses
101 2012-12-18 04:28:11 <stealth222> I've got an architecture that completely separates the wallet management from the alert system
102 2012-12-18 04:28:32 <stealth222> the alert system doesn't need to know any private keys for any wallets
103 2012-12-18 04:28:37 <stealth222> lower security requirements
104 2012-12-18 04:29:26 <stealth222> easier to deploy highly redundant alerter nodes to provide a notification service without the need to be passing around any private keys
105 2012-12-18 04:29:41 <jgarzik> stealth222: That's been the long term plan, to separate the "block chain engine" from wallet, with clear lines of communication and separation between them
106 2012-12-18 04:29:54 <stealth222> I'd be willing to help out in this pursuit
107 2012-12-18 04:30:16 <stealth222> it's been something I've been needing for many, many months which led me to develop my own protocol library and client
108 2012-12-18 04:30:30 <stealth222> as well as my own wallet
109 2012-12-18 04:31:07 <stealth222> which sucks...it would be better if all this could be part of the daemon itself
110 2012-12-18 04:31:08 <jgarzik> stealth222: BlueMatt already did a lot of work on that, but it was highly invasive -- as it must be -- yet unfortunately a lower development priority than replacing the keeling-over BDB
111 2012-12-18 04:31:19 <jgarzik> stealth222: grab his stuff, maybe he already did the work for you
112 2012-12-18 04:31:35 <jgarzik> stealth222: BlueMatt is also the one who wrote the dynamic-addnode pull req
113 2012-12-18 04:32:09 <stealth222> so what are you going to be replacing BDB with?
114 2012-12-18 04:32:15 <jgarzik> stealth222: my picocoin client does something similar, with fork()-enforced separation between network and wallet
115 2012-12-18 04:33:11 <jgarzik> stealth222: Replaced, past tense. "ultraprune" consists of two large updates: (1) LevelDB, (2) highly efficient unspent-transaction output (UTXO) data structure work
116 2012-12-18 04:33:41 <jgarzik> stealth222: sipa's ultraprune work is already upstream in bitcoin.git
117 2012-12-18 04:34:38 <stealth222> I've written a class that connects to an arbitrary bitcoin node and provides callbacks for onTx and onBlock
118 2012-12-18 04:34:57 <stealth222> and have used it for my alerter system
119 2012-12-18 04:35:52 <stealth222> yeah, the better database would be nice
120 2012-12-18 04:36:10 <stealth222> another feature I wanted was the ability to perform more sophisticated queries
121 2012-12-18 04:36:20 <jgarzik> stealth222: If it's just monitoring, a lot of people just use https://github.com/jgarzik/pynode/tree/mini-node to watch for tx's and blocks on the network
122 2012-12-18 04:36:30 <stealth222> that's ugly - lol
123 2012-12-18 04:36:45 <stealth222> what I have runs much faster
124 2012-12-18 04:36:50 <jgarzik> More sophisticated queries -- more indices, unused by most
125 2012-12-18 04:37:20 <jgarzik> There is thought about an optional all-tx index
126 2012-12-18 04:37:21 <stealth222> not to insult your code, j - I'm just not as much of a fan of py for daemons
127 2012-12-18 04:38:07 <jgarzik> ACTION shrugs
128 2012-12-18 04:38:35 <stealth222> the code is actually pretty clean
129 2012-12-18 04:39:56 <stealth222> I mean your py code
130 2012-12-18 04:42:34 <stealth222> is BlueMatt's stuff in a public repo/
131 2012-12-18 04:42:36 <stealth222> ?
132 2012-12-18 04:44:53 <stealth222> I mean, the block chain engine / wallet separation
133 2012-12-18 04:45:12 <stealth222> and will it be difficult to merge it back in after the database updates?
134 2012-12-18 04:46:27 <stealth222> if so, I guess yeah, we should definitely do the database stuff first
135 2012-12-18 04:55:38 <stealth222> ultraprune is a higher priority to mass adoption
136 2012-12-18 04:56:23 <stealth222> and has a very different set of requirements to the block chain engine stuff
137 2012-12-18 04:57:39 <stealth222> oh well, I'm not that interested in hacking away at a fork that will be impossible to merge back later
138 2012-12-18 04:58:18 <stealth222> I prefer just having a separate process that connects to bitcoind like I have it right now
139 2012-12-18 05:09:41 <dparrish> hey has someone got a spare testnet coin or 2 they can send me pls?
140 2012-12-18 05:10:13 <dparrish> n1JZA2Z86UWESpJRDB8zabWSgeaVxpfk6n
141 2012-12-18 05:13:38 <stealth222> I would but the ones I have haven't matured yet
142 2012-12-18 05:13:54 <dparrish> heh ok
143 2012-12-18 05:13:59 <stealth222> if you wait four more blocks I can send you a couple
144 2012-12-18 05:14:01 <stealth222> lol
145 2012-12-18 05:14:48 <dparrish> i really thought i would have been able to mine some in a few days of trying
146 2012-12-18 05:14:54 <jgarzik> stealth222: BlueMatt's stuff is definitely in a public repo. You'll have to bug him for the branch name. somewhere on github/bluematt...
147 2012-12-18 05:14:58 <dparrish> the difficulty is only 170 ffs
148 2012-12-18 05:15:14 <jgarzik> dparrish: one sec
149 2012-12-18 05:15:19 <stealth222> I've been mining for just a few hours on 6 CPU cores and have already mined three blocks
150 2012-12-18 05:15:28 <stealth222> using the bitcoind built-in miner
151 2012-12-18 05:15:38 <dparrish> stealth222: that's what i did! 8 cores, 3 days, 0 blocks :(
152 2012-12-18 05:15:45 <jgarzik> dparrish: a1fe56a7e261d49145541805206b679ceb458cce9aab6d8f566166300772cce5
153 2012-12-18 05:15:51 <dparrish> jgarzik: thanks!
154 2012-12-18 05:15:52 <stealth222> what's your hashrate, dparrish?
155 2012-12-18 05:16:37 <dparrish> stealth222: erm i actually don't know.. does the built-in miner let me see that? IIRC it's about 800kh/s which is useless for the main chain
156 2012-12-18 05:16:45 <stealth222> bitcoind getmininginfo
157 2012-12-18 05:17:01 <dparrish> 6105126
158 2012-12-18 05:17:08 <dparrish> that's more than i thought
159 2012-12-18 05:17:15 <stealth222> hmm...that's not much less than I'm getting: 6481582
160 2012-12-18 05:17:36 <stealth222> three days? you must just have been horribly unlucky - lol
161 2012-12-18 05:17:45 <dparrish> heh that's my life
162 2012-12-18 05:28:18 <stealth222> are the goals of ultraprune and a database that supports complex queries on the entire block chain and memory pool at odds?
163 2012-12-18 05:28:55 <stealth222> perhaps it would be better architecturally to separate the peer discovery / networking component first
164 2012-12-18 05:29:11 <stealth222> since both the wallet and the block chain engine will need to use it
165 2012-12-18 05:29:47 <stealth222> and to write the wallet and block chain engine apps as separate apps
166 2012-12-18 05:30:51 <stealth222> furthermore, for the wallet, there are two main use scenarios - one is for enduser lightweight clients - the other is for commerce applications
167 2012-12-18 05:32:05 <stealth222> ideally, the signing component should be implemented in hardware :)
168 2012-12-18 05:32:35 <stealth222> tracking unspent outputs doesn't require knowledge of private keys
169 2012-12-18 05:33:35 <stealth222> a small USB device that connects to the enduser's computer (or a MicroSD card on their mobile device) could store private keys and do signing
170 2012-12-18 05:34:28 <lianj> an then blindly trusting/signing what it is given? :P
171 2012-12-18 05:34:42 <stealth222> no, the user would have to be prompted
172 2012-12-18 05:35:18 <stealth222> and strict limits could be set in the protocol - like transaction volume
173 2012-12-18 05:35:57 <stealth222> these parameters could be stored on the hardware device and require the enduser to physically authorize any changes to them
174 2012-12-18 05:37:34 <stealth222> furthermore, transaction signing and sending only requires a database of unspent outputs - doesn't require storing any other structures at all
175 2012-12-18 05:38:13 <stealth222> verification of receipt of payment would require storing other stuff
176 2012-12-18 05:38:30 <stealth222> but that doesn't have to be done on all the user's devices
177 2012-12-18 05:39:35 <stealth222> in fact, most thin clients will probably end up trusting servers for these kinds of verification services
178 2012-12-18 05:40:30 <stealth222> all that's needed locally are private keys and unspent outpoints
179 2012-12-18 05:43:03 <stealth222> even without connecting to a centralized server, the client can always query a bunch of other nodes it's fairly confident are not in collusion with one another at random and see if they agree
180 2012-12-18 05:43:43 <stealth222> it wouldn't take too many such queries to establish, with extremely high probability, that a transaction did in fact confirm
181 2012-12-18 05:44:35 <stealth222> as for nodes that relay transactions, these would need to do some level of verification
182 2012-12-18 05:44:49 <stealth222> but signing nodes do not need to relay anyone else's transactions
183 2012-12-18 05:45:17 <stealth222> in fact, it almost makes sense that they don't - it's far easier to isolate them and make them more secure
184 2012-12-18 05:50:53 <stealth222> furthermore, verification that a transaction confirmed does not require any private keys
185 2012-12-18 05:51:29 <stealth222> nor does relaying verified transactions
186 2012-12-18 05:52:48 <stealth222> as a business, I would want to have a server or two that verify all transactions as rigorously as possible and that can give me high-level query capabilities. but on my mobile device, I just want a simple signing component
187 2012-12-18 05:55:48 <stealth222> I guess ultraprune would be OK for mobile devices, too - some level of verification
188 2012-12-18 05:58:57 <stealth222> but even the ultraprune piece could be separate from the signing piece
189 2012-12-18 05:59:07 <stealth222> and probably should be
190 2012-12-18 06:01:47 <stealth222> that way I can disconnect the signing component and still track and verify payments
191 2012-12-18 06:01:59 <stealth222> without having to worry about my node getting hacked
192 2012-12-18 06:17:46 <stealth222> blah, all this has probably been talked about to death before - I'm a relative newcomer into this whole thing, so I apologize if I'm just repeating stuff that everyone has already heard a million times
193 2012-12-18 06:19:21 <stealth222> setting aside all the philosophical stuff for a moment, I have a specific question: what's the multisig address version for testnet?
194 2012-12-18 06:19:39 <stealth222> I know for the main network it's 0x05
195 2012-12-18 06:21:59 <stealth222> is it 0xC4?
196 2012-12-18 07:21:14 <stealth222> why COINBASE_MATURITY+20?
197 2012-12-18 07:21:15 <stealth222> int CMerkleTx::GetBlocksToMaturity() const
198 2012-12-18 07:21:16 <stealth222> if (!IsCoinBase())
199 2012-12-18 07:21:18 <stealth222> return 0;
200 2012-12-18 07:21:20 <stealth222> return max(0, (COINBASE_MATURITY+20) - GetDepthInMainChain());
201 2012-12-18 07:21:22 <stealth222> }
202 2012-12-18 07:45:41 <jaromil> guys.... this is OT but also too funny to not post it here http://www.indiegogo.com/projects/272646
203 2012-12-18 07:46:13 <jaromil> the crew doing this "turbofilm" is a funny crowd, they live around NYC let me know if someone likes to get in touch
204 2012-12-18 07:46:38 <jaromil> I've played a part in the next teaser and made some nicknames of developers here :)
205 2012-12-18 07:47:14 <jaromil> they are fundraising it with perks one could also join them in the actual filming in porto rico
206 2012-12-18 08:36:35 <sipa> midnightmagic: i just commented out that line
207 2012-12-18 10:27:00 <gmaxwell> 00:21 < stealth222> why COINBASE_MATURITY+20?
208 2012-12-18 10:27:51 <stealth222> yes
209 2012-12-18 10:27:57 <gmaxwell> Because if you try to spend one right when your peers aren't yet up to date with the chain, it'll just get dropped. 20 is probably too high for the network today, 2 works alright.
210 2012-12-18 10:28:10 <gmaxwell> But its not like it matters to many people.
211 2012-12-18 10:28:23 <stealth222> it's inconsistent with the documentation, though
212 2012-12-18 10:28:30 <stealth222> might be a good idea to make a note of it in the wikis
213 2012-12-18 10:29:41 <gmaxwell> How is it inconsistent?
214 2012-12-18 10:30:14 <stealth222> https://en.bitcoin.it/wiki/Protocol_rules#.22tx.22_messages
215 2012-12-18 10:30:18 <gmaxwell> The protocol rule is COINBASE_MATURITY. The client adds some margin on that to avoid bumping into poor forwarding due to spending right at the edge.
216 2012-12-18 10:30:43 <gmaxwell> It's very important for you to keep in mind that not everything the reference client does is a protocol rule.
217 2012-12-18 10:31:09 <stealth222> then it might be a good idea to put together a document explaining how the client implementation differs
218 2012-12-18 10:31:18 <gmaxwell> It doesn't differ.
219 2012-12-18 10:31:20 <gmaxwell> ugh.
220 2012-12-18 10:31:47 <gmaxwell> stealth222: It will happily forward txn that spend at COINBASE_MATURITY. It will happily accept blocks that include txn spending at COINBASE_MATURITY.
221 2012-12-18 10:32:23 <stealth222> ok, I see. so it will relay but it won't initiate a transaction
222 2012-12-18 10:32:25 <gmaxwell> But while choosing inputs to build a new transaction it won't choose ones untl COINBASE_MATURITY+20.
223 2012-12-18 10:32:35 <stealth222> ok, got it.
224 2012-12-18 10:33:29 <stealth222> the relay rules and the create new transaction rules are different
225 2012-12-18 10:33:32 <gmaxwell> Similar to how it won't spend a payment from someone else until 6 confirms if it can help it, 1 otherwise. Though nodes will (at least potentially) relay at 0.
226 2012-12-18 10:33:53 <sipa> stealth222: pne is a
227 2012-12-18 10:34:16 <sipa> stealth222: one is a network rule, the other is a wallet lolicy
228 2012-12-18 10:34:21 <gmaxwell> (in that case, it's to avoid propagating fraud??? if someone rips you off with a payment that gets reversed you shouldn't rip off unrelated third parties)
229 2012-12-18 10:34:35 <gmaxwell> "lolicy" explains it. :P
230 2012-12-18 10:34:50 <stealth222> ok :)
231 2012-12-18 10:35:03 <sipa> ACTION needs typing lessons
232 2012-12-18 10:35:31 <sipa> in my defence: a 3.2" on-screen keyboard is tiny
233 2012-12-18 10:36:34 <gmaxwell> stealth222: and yea, having some better docs on 'good' client behavior as seperate from the protocol rules would be nice, but it's basically impossible to document everything it does, and far less important than having good documentation of the protocol rules.
234 2012-12-18 10:36:46 <gmaxwell> stealth222: it's also worth noting that the lists on the wiki are incomplete.
235 2012-12-18 10:38:51 <drizztbsd> hi, who sign SHA256SUM file?
236 2012-12-18 10:40:50 <sipa> drizztbsd: if you can't find that out by verifying the signature, your verification isn't very thorough :)
237 2012-12-18 10:41:33 <drizztbsd> yes, I'm asking if there can be more than one person
238 2012-12-18 10:41:54 <drizztbsd> 0.7.2 is signed by Gavin Andresen (CODE SIGNING KEY) <gavinandresen@gmail.com>
239 2012-12-18 10:45:43 <sipa> stealth222: btw, about your earlier monologue: it's indeed a long-term goal to separate the wallet code and block verification code more (for example as separate processes, or even as separate network components); a database for complexer queries... my preference is keeping the reference client code light (as in: not more complex than it needs to be for implementing full verification), but perhaps add optional indexes for people who need lookup by...
240 2012-12-18 10:45:49 <sipa> address or txid
241 2012-12-18 10:47:24 <sipa> drizztbsd: not sure if PGP supports multiple signatures simultaneously, though adding one on top of the other should work
242 2012-12-18 10:47:57 <sipa> drizztbsd: note that you can find signatures for the deterministic build process on https://github.com/bitcoin/gitian.sigs
243 2012-12-18 10:48:05 <sipa> typically with 2-3 signatures for every release
244 2012-12-18 11:21:15 <stam7> i don't recognize you, gribble
245 2012-12-18 11:21:18 <stam7> you are a problem, to me
246 2012-12-18 11:21:23 <stam7> and whoever is controlling you
247 2012-12-18 11:22:56 <stam7> stop ban-placing and i might forgive you
248 2012-12-18 11:23:05 <stam7> ok, dear gribble?
249 2012-12-18 11:24:43 <stam7> but that's no guarantee. you see i MAY or MAY NOT forgive you. it's all my choice
250 2012-12-18 11:25:14 <stam7> also, you need to apologize for your rude behavior
251 2012-12-18 11:47:50 <stam7> it's interesting how i'm never banned from -dev, ever since i said i'd rather not be here
252 2012-12-18 12:43:48 <stam7> it's as if you are only here to torture me
253 2012-12-18 12:43:59 <stealth222> please die
254 2012-12-18 15:33:29 <lianj> in bitcoind mainnet, is opcode op_eval used or is it just interepreted as op_nop1?
255 2012-12-18 15:34:00 <sipa> OP_EVAL doesn't exist
256 2012-12-18 15:35:45 <lianj> cool thanks, so handle op_nop1 as nop then?
257 2012-12-18 15:37:43 <sipa> indeed
258 2012-12-18 15:38:18 <lianj> thank you. just asking because of this mainnet tx http://blockexplorer.com/tx/f003f0c1193019db2497a675fd05d9f2edddf9b67c59e677c48d3dbd4ed5f00b
259 2012-12-18 15:38:42 <sipa> OP_NOP1 was redefined to OP_EVAL by BIP12, but it was replaced by BIP16
260 2012-12-18 15:39:18 <lianj> right
261 2012-12-18 15:39:34 <sipa> that looks like a BIP12 transaction, but following current rules, that is just a pubkeyhash check
262 2012-12-18 15:40:25 <lianj> ok, yea, that was my question, because as bip12 it seems broken anyway oO :D
263 2012-12-18 15:40:43 <lianj> "04e19cb94dab9efa1e4507c17c81d4fdb0fc9d03c01caac970995ca4788f6e3fd3b2eb0efba75a98b1e1a62f9bdcb71430ce066869facb4f1e20b9ee1d1669b356 OP_DUP OP_HASH160 07e761706c63b36e5a328fab1d94e9397f40704d OP_EQUALVERIFY OP_EVAL"
264 2012-12-18 15:42:35 <sipa> how do you mean BIP12 seems broken?
265 2012-12-18 15:43:12 <lianj> nah, only the above bip12 script
266 2012-12-18 15:43:33 <gmaxwell> BIP12 doesn't exist.
267 2012-12-18 15:43:47 <gmaxwell> ACTION looks for his men in black flashy thing
268 2012-12-18 15:43:57 <lianj> ^^
269 2012-12-18 15:44:35 <sipa> lianj: well you can't really judge brokenness of a rule that isn't implemented by looking at mainnet
270 2012-12-18 15:45:34 <lianj> ACTION deletes all traces of op_eval in his code
271 2012-12-18 15:49:02 <lianj> done, all gone. :) well, if someone does Script.from_string("foo bar OP_EVAL").to_string it at least doesnt faild but outputs "foo bar OP_NOP1" now
272 2012-12-18 15:49:41 <sipa> what code?
273 2012-12-18 15:50:02 <lianj> just my silly bitcoin lib
274 2012-12-18 16:36:35 <Enrico1> YOU ARE A PIECE OF SHIT, GREG MAXWELL, AND I AM GOING TO KEEP SAYING THIS UNTIL YOU DO WHAT YOU ARE SUPPOSED TO DO.
275 2012-12-18 16:37:25 <drizztbsd> italian lamer?
276 2012-12-18 16:38:01 <Enrico1> no, they just ban all "stams" now
277 2012-12-18 16:38:37 <Enrico1> can't join as "stam" anymore. it's forbidden
278 2012-12-18 16:39:41 <drizztbsd> not really, check the ban list
279 2012-12-18 16:40:00 <Enrico1> they do it per IP
280 2012-12-18 16:40:12 <Enrico1> they see "stam" and they ban the IP
281 2012-12-18 16:40:37 <D34TH> stam?
282 2012-12-18 16:40:46 <Enrico1> hello D34TH
283 2012-12-18 16:40:57 <D34TH> Enrico1, sup
284 2012-12-18 16:41:07 <Enrico1> i used a nickname generator to get the "Enrico"
285 2012-12-18 16:41:21 <Enrico1> but it's taken already, so it became Enrico1
286 2012-12-18 16:41:42 <MC1984> youre only feeding gregs ego, you know that right
287 2012-12-18 16:41:57 <MC1984> if youre accruing haters, youre doing great and important things :)
288 2012-12-18 16:42:48 <Enrico1> "accruing haters"? don't know how that works
289 2012-12-18 16:47:24 <gavinandresen> Enrico1: ad-hominem attacks are not welcome here. Please don't do that again.
290 2012-12-18 16:47:26 <helo> 1) scam a few people 2) wait
291 2012-12-18 16:48:02 <D34TH> so noone ever answered my question, what is stam?
292 2012-12-18 16:48:05 <Enrico1> it's not necessarily the truth, what i am saying, it's just my *opinion* of him!
293 2012-12-18 16:48:21 <gavinandresen> This IRC channel is not for opinions, it is for technical discussions.
294 2012-12-18 16:48:43 <Enrico1> i just want him to just remove the WoT rating that says i am a pervert
295 2012-12-18 16:49:39 <D34TH> why didnt he just pm him
296 2012-12-18 16:49:43 <D34TH> that is a private matter
297 2012-12-18 16:49:44 <gavinandresen> thanks jeff
298 2012-12-18 16:49:48 <D34TH> oh well
299 2012-12-18 16:49:50 <D34TH> back to bitcoin
300 2012-12-18 16:50:01 <jgarzik> back to non-bitcoin xmas family shite ;p
301 2012-12-18 16:50:27 <D34TH> pullreq #2109 please
302 2012-12-18 16:50:34 <gavinandresen> speaking of technical discussions... anybody have good ideas on what I should get my wife for xmas? :)
303 2012-12-18 16:51:03 <D34TH> technically you should get shiny stuff
304 2012-12-18 16:52:12 <MC1984> bowling ball
305 2012-12-18 16:52:25 <D34TH> suggestion: include compiled leveldb files in with bitcoin-deps-0.0.5.zip
306 2012-12-18 16:55:44 <gavinandresen> D34TH: are you Diapolo on github?
307 2012-12-18 16:55:46 <D34TH> no
308 2012-12-18 16:55:49 <D34TH> im grimd34th
309 2012-12-18 16:55:53 <D34TH> but those changes help me also
310 2012-12-18 16:56:01 <gavinandresen> you're developing on windows?
311 2012-12-18 16:56:05 <D34TH> yep
312 2012-12-18 16:56:07 <gavinandresen> mingw?
313 2012-12-18 16:56:19 <D34TH> mingw + qt
314 2012-12-18 16:56:44 <gavinandresen> cool. I've been having all sorts of trouble compiling sipa's latest leveldb in an old XP VM
315 2012-12-18 16:56:55 <gavinandresen> have you had luck?
316 2012-12-18 16:57:02 <D34TH> sec ill pull sipa's latest
317 2012-12-18 16:57:22 <gavinandresen> D34TH: his leveldb17 branch, if I recall
318 2012-12-18 16:58:43 <D34TH> building
319 2012-12-18 16:59:28 <D34TH> build.h forced?
320 2012-12-18 17:00:36 <D34TH> oh nevermind i accidentally put unix instead of mingw
321 2012-12-18 17:05:24 <D34TH> hmm
322 2012-12-18 17:05:29 <D34TH> threw unknown platform
323 2012-12-18 17:06:19 <gavinandresen> I got around that by hacking makefile.mingw to pass $(MAKE) TARGET_OS=NATIVE_WINDOWS libleveldb.a libmemenv.a
324 2012-12-18 17:07:36 <D34TH> i just pulled my uname -s
325 2012-12-18 17:07:53 <gavinandresen> what IS your uname -s ?
326 2012-12-18 17:08:01 <gavinandresen> I get: MINGW32_NT-5.1
327 2012-12-18 17:08:10 <D34TH> MINGW32_NT-6.1
328 2012-12-18 17:08:16 <D34TH> windows version
329 2012-12-18 17:08:18 <D34TH> im running 7
330 2012-12-18 17:09:01 <gavinandresen> my vm is XP SP3
331 2012-12-18 17:09:14 <D34TH> i think i need to upgrade boost
332 2012-12-18 17:10:07 <D34TH> yep
333 2012-12-18 17:10:09 <gavinandresen> D34TH: does the current git HEAD leveldb compile for you?
334 2012-12-18 17:10:15 <D34TH> yea
335 2012-12-18 17:10:24 <gavinandresen> ok, good to know, I haven't tried that
336 2012-12-18 17:10:33 <D34TH> want the compiled .a's?
337 2012-12-18 17:10:44 <sipa> ~my leveldb17 version doesn't touch the windows makefile at all, since i'd have no clue what to put there anyway
338 2012-12-18 17:10:47 <gavinandresen> D34TH: no, I'm trying to test/debug the NEW code!
339 2012-12-18 17:11:50 <gavinandresen> D34TH: error I'm running into is db/c.cc:137:42: error: 'strdup' was not declared in this scope
340 2012-12-18 17:12:03 <D34TH> im getting linking errors
341 2012-12-18 17:12:10 <D34TH> but im relating it to boost 1.5.0
342 2012-12-18 17:12:13 <D34TH> **1.50
343 2012-12-18 17:12:49 <sipa> you have to use 1.51 or 1.52
344 2012-12-18 17:13:05 <D34TH> i saw that after i started
345 2012-12-18 17:13:16 <sipa> 1.50 and before have a bug in the thread library for thread
346 2012-12-18 17:13:42 <gavinandresen> sipa: actually, the strdup error might be the root cause of the bug: if the cross-compile environment is picking up /usr/include .h files instead of the cross-compiler ones, then that would explain why vsprintf is crashing
347 2012-12-18 17:14:19 <sipa> gavinandresen: hmm, indeed
348 2012-12-18 17:16:44 <gavinandresen> lunch time, I'll dig in more when I'm not so hungry
349 2012-12-18 18:37:10 <D34TH> sipa, gavinandresen :time to sort through http://pastebin.com/K0u5yXPp
350 2012-12-18 18:39:05 <sipa> D34TH: what's the compile command line that caused that?
351 2012-12-18 18:39:19 <D34TH> make -f makefilemingw
352 2012-12-18 18:39:23 <D34TH> make -f makefile.mingw
353 2012-12-18 18:39:50 <sipa> the actual command, like the g++ invocation
354 2012-12-18 18:40:17 <D34TH> sec, lemme see if i can recreate it more
355 2012-12-18 18:40:23 <D34TH> under a seperate env
356 2012-12-18 18:42:35 <D34TH> sipa: http://pastebin.com/tX71LW5A
357 2012-12-18 18:45:59 <gavinandresen> sipa: making some progress with the leveldb17 branch; the strdup problem is because strdup is not __STRICT_ANSI__ . I modified build_detect_platform to pass -std=c++0x -U__STRICT_ANSI__
358 2012-12-18 18:46:21 <gavinandresen> ... now my problem is the #include <dbghelp.h> in util/env_win.cc
359 2012-12-18 18:46:30 <gavinandresen> Anybody know what <dbghelp.h> is ?
360 2012-12-18 18:46:56 <sipa> gavinandresen: dbghelp.h is the single reason i switched to mingw-w64, since mingw32 doesn't seem to have it
361 2012-12-18 18:47:24 <sipa> alos, i never saw any strdup problem...
362 2012-12-18 18:48:27 <sipa> D34TH: no clue
363 2012-12-18 18:48:41 <D34TH> i think it is libdb
364 2012-12-18 18:48:47 <D34TH> **leveldb
365 2012-12-18 18:48:48 <gavinandresen> hmm... I wonder if mingw64 will work in my XP vm... (somehow I think not)
366 2012-12-18 18:48:58 <D34TH> it only has a 32kb .a
367 2012-12-18 18:49:42 <D34TH> sipa: what defines -DLEVELDB_PLATFORM_WINDOWS
368 2012-12-18 18:50:18 <sipa> D34TH: the makefile calling it, i suppose
369 2012-12-18 18:50:25 <sipa> D34TH: unless it can autodetect
370 2012-12-18 18:50:50 <D34TH> i can't find it at all in any of the leveldb17 files
371 2012-12-18 18:50:57 <D34TH> just build_detect_platform
372 2012-12-18 18:51:26 <sipa> yeah, than you'll need to set it in the calling makefile
373 2012-12-18 18:51:29 <gavinandresen> build_detect_platform creates build_config.mk, which the makefile includes
374 2012-12-18 18:51:51 <sipa> oh, i assumed build_detect_platform wouldn't be executed at all on windows
375 2012-12-18 18:52:38 <sipa> gavinandresen: i think mingw-w64 should work on XP...
376 2012-12-18 18:52:47 <sipa> (the 32-bit version, presumably, but still)
377 2012-12-18 18:58:42 <gavinandresen> aargh, whenever I try to understand mingw I feel like I'm in a maze of twisty compilers, all alike
378 2012-12-18 18:59:40 <sipa> the '64' in the mingw-w64 name is very confusing... it seems to just be a new generation of mingw which *also* supports 64-bit
379 2012-12-18 18:59:47 <D34TH> gavinandresen, dont feel bad
380 2012-12-18 18:59:52 <D34TH> that is actually how it is
381 2012-12-18 19:00:40 <gavinandresen> sipa: mmm. So I've got a development environment all setup with mingw32.... I don't suppose there are instructions somewhere for how to upgrade to mingw-w64....
382 2012-12-18 19:01:17 <sipa> gavinandresen: uninstall mingw32, install mingw-w64
383 2012-12-18 19:01:26 <sipa> it's really a separate project, i think
384 2012-12-18 19:02:40 <gavinandresen> ok, that's a project for another day. I assume I'll have to start over and recompile dependencies, too.
385 2012-12-18 19:02:56 <sipa> likely, i think
386 2012-12-18 19:04:58 <phantomcircuit> hmm
387 2012-12-18 19:52:57 <killerstorm> hi. anybody got testnet coins? muLWgg2Wn4GTbqU44uLLJUNGunWyxai48G
388 2012-12-18 20:00:07 <stealth222> just sent you a couple
389 2012-12-18 20:00:18 <stealth222> 8a9ce79e7b92192b0558e8e9e8acad2b59c2a4e308275dcbb69b3f7a0e3a8ad5
390 2012-12-18 20:03:44 <killerstorm> stealth222: thanks!
391 2012-12-18 20:07:07 <stealth222> np
392 2012-12-18 20:25:43 <donnchac> Hi guys.
393 2012-12-18 20:26:15 <donnchac> I'm looking at developing a site where each user will have an individual account and address where they can send bitcoin's too.
394 2012-12-18 20:26:52 <donnchac> I'm just wondering is it best practice to use the "accounts" feature and is this scalable for a large number of users?
395 2012-12-18 20:27:17 <TD> effectively you'll have one giant wallet
396 2012-12-18 20:27:30 <TD> from what i know, bitcoins current code doesn't scale well to huge wallets
397 2012-12-18 20:27:48 <TD> but is that really the best way to do your service? there are usually better ways than running online wallets. what is your overall goal?
398 2012-12-18 20:30:13 <donnchac> They overall goal it to allow users to agree on freelance work. The money is paid in bitcoins which are stored in escrow by the server.
399 2012-12-18 20:30:51 <donnchac> The buer would pay into their account on the website and then when they agree a deal, those funds would be locked in.
400 2012-12-18 20:32:31 <donnchac> *the buyer
401 2012-12-18 20:33:03 <donnchac> I'd like to get the solution correct from the start to allow it to potentially scale to ~10k accounts.
402 2012-12-18 20:33:43 <gmaxwell> I'm not aware of any reason that having lots of accounts themselves has major scaling issues??? but generally if you're thinking of using accounts for anything serious you should reconsider that decision.
403 2012-12-18 20:34:01 <gmaxwell> There are major challenges with accounts for large scale operations, e.g. backing them up.
404 2012-12-18 20:34:45 <gmaxwell> For many reasons you need to basically implement the account logic in your own database/app anyways, and then using them in bitcoind just buys you additional bug exposure.
405 2012-12-18 20:34:56 <sipa> the largest problem is a large number of transactions
406 2012-12-18 20:35:17 <sipa> as many account-based RPCs are linear in the number of wallet transactions
407 2012-12-18 20:35:18 <helo> if you are building a system that can send bitcoin at a user's request, you are the perfect target for hackers
408 2012-12-18 20:35:29 <helo> good luck dealing with that...
409 2012-12-18 20:37:44 <donnchac> I understand that. I don't really see another way of dealing with the issue.
410 2012-12-18 20:39:29 <donnchac> gmaxwell: So you would recommend I should just implement bitcoin account management functionality nativily in my app and then just store bitcoins in one account.
411 2012-12-18 20:39:43 <gmaxwell> Correct.
412 2012-12-18 20:39:59 <gmaxwell> Accounts in bitcoind are just bookkeeping in any case.
413 2012-12-18 20:43:27 <donnchac> Okay, thank you gmaxwell. I'm still trying to figure out the best way of keeping my database synced with the current bitcoind balance.
414 2012-12-18 20:44:10 <donnchac> Unfortunatly bitcoind doesn't have the ability to make callbacks when new transactions are received?
415 2012-12-18 20:44:38 <jaromil> ah! now you posed The Question :)
416 2012-12-18 20:45:41 <gmaxwell> donnchac: it can tell you when new _blocks_ are recieved.
417 2012-12-18 20:45:56 <gmaxwell> Which is when the balance actually changes, assuming you're not doing something insecure.
418 2012-12-18 20:46:33 <gmaxwell> There is a proposed patch for callbacks on transactions too.. though .. uh. I'm not a big fan of it.
419 2012-12-18 20:46:44 <gmaxwell> (It has a lot of dos attack potential. :( )
420 2012-12-18 20:47:00 <jaromil> true. :/
421 2012-12-18 20:47:44 <gmaxwell> it'll probably get merged in any case, since the dos potential can be answered by "then don't use it" and there are some things that its pretty useful for.
422 2012-12-18 20:49:10 <sipa> gmaxwell: the idea someone proposed to keep a queue of callbacks to be performed, and handling those in N threads looks good, though
423 2012-12-18 20:49:49 <jaromil> sounds like a closure
424 2012-12-18 20:50:29 <donnchac> What does the timescale look like on that patch? I'm just wondering how most people manage payments interfacing with bitcoind at the moment?
425 2012-12-18 20:50:42 <gmaxwell> donnchac: they use the block callback.
426 2012-12-18 20:50:56 <gmaxwell> As mentioned, the balance doesn't change until the blockchain changes.
427 2012-12-18 20:51:11 <gmaxwell> If you just watch transactions you're also at risk of missing transactions falling out of the chain.
428 2012-12-18 20:51:44 <donnchac> Okay. I'll look into that. Okay, so I assume that only updates transaction when they are confirmed.
429 2012-12-18 20:51:51 <jaromil> donnchac: in web terms, you are not going to present people with a "thankyou" page, but with a "thankyou" email somehow later
430 2012-12-18 20:53:02 <donnchac> How does bitpay etc. do instant notification at the moment. I understand that the transaction would not yet be confirmed at that stage. Thanks for clarifications everyone. I just want to ensure I am sticking with best practice
431 2012-12-18 20:55:12 <sipa> donnchac: custom code, i expect
432 2012-12-18 20:56:55 <donnchac> Okay, the block callback looks like it will solve my requirements. I was not aware of it before which is why I wasen't sure of the best course of action.
433 2012-12-18 20:57:21 <Matt_von_Mises> I've done something to break my code and I'm wondering what. Are previous block hashes reversed in block headers? I can't remember. When reproducing block #1 I get a header that does not hash correctly. Can anyone find out what is wrong with this data? http://pastebin.com/yCAUeWG1
434 2012-12-18 21:05:05 <stealth222> donnchac: I have a database + callback solution that runs as a separate process
435 2012-12-18 21:05:23 <stealth222> I
436 2012-12-18 21:05:40 <stealth222> I'd like to eventually merge it with bitcoind
437 2012-12-18 21:06:08 <donnchac> stealth222: Okay that sounds like the option I will go with.
438 2012-12-18 21:07:09 <stealth222> block callbacks are sufficient if you only care about confirmed transactions
439 2012-12-18 21:11:12 <sipa> Matt_von_Mises: seems you have the merkle root backwards at least
440 2012-12-18 21:11:26 <sipa> and the prev block hash too
441 2012-12-18 21:11:29 <stealth222> doesn't the block header also need a transaction count?
442 2012-12-18 21:11:39 <sipa> stealth222: i wish it did
443 2012-12-18 21:11:42 <sipa> but it doesn't
444 2012-12-18 21:11:54 <Matt_von_Mises> sipa: Oh, I tried reversing them individually but not both at the same time. Should have tried that!
445 2012-12-18 21:11:58 <stealth222> oh, right - lol
446 2012-12-18 21:12:25 <Matt_von_Mises> I have no idea when I changed the code to not reverse the hashes, as it was working before.
447 2012-12-18 21:12:30 <sipa> Matt_von_Mises: you shouldn't need to reverse anything at all
448 2012-12-18 21:12:41 <sipa> Matt_von_Mises: it's byte-for-byte the output of the hash function
449 2012-12-18 21:12:45 <Matt_von_Mises> sipa: When hashing a block the zeros come out on the end...
450 2012-12-18 21:13:48 <sipa> Matt_von_Mises: hmm, you confuse me... i'm not sure anymore
451 2012-12-18 21:14:05 <stealth222> endianness is one of the most irritating aspects of the protocol
452 2012-12-18 21:14:26 <Matt_von_Mises> sipa: Bitcoin has this problem everywhere. Trying to remember which way things should go is a nightmare.
453 2012-12-18 21:14:45 <stealth222> why can't bitcoin just use protocol buffers?
454 2012-12-18 21:15:07 <sipa> stealth222: satoshi didn't know about that at the time he created the system; he later admitted he wished he did
455 2012-12-18 21:16:02 <gmaxwell> meh. most of the datastructures are quite simple.
456 2012-12-18 21:16:30 <stealth222> it's still irritating that there's lack of consistency in endianness
457 2012-12-18 21:16:58 <sipa> actually, it is consistent: it's always little-endian, except for crypto stuff which happens inside openssl
458 2012-12-18 21:17:37 <sipa> but then there is the weird side effect that hashes (which consists of big-endian serialized numbers) get interpreted as little-endian in the bitcoin side
459 2012-12-18 21:18:30 <gmaxwell> stealth222: I don't think its unusually infuriating. Then again, maybe my brain is damaged from spending time working with devices that have mixtures of ppc and x86 embedded hardware.
460 2012-12-18 21:18:46 <Matt_von_Mises> No matter what I reverse, I can't get it to hash right. Hmm.
461 2012-12-18 21:19:06 <sipa> Matt_von_Mises: i think you're right; the zeroes are at the end, and that's also where they should be
462 2012-12-18 21:19:17 <sipa> so there's probably something else wrong in your header
463 2012-12-18 21:19:30 <stealth222> gmaxwell: believe me, I've seen far more maddening issues in other projects - bitcoin is pretty clean by comparison to most
464 2012-12-18 21:19:54 <stealth222> it's still in my nature to try to find even better solutions to problems that have already been solved :p
465 2012-12-18 21:21:26 <gmaxwell> I think any complex system with more than 100 parts will inevitability have a least one mandatory part that is somewhat braindamaged no matter how hard people work on it. I'm just glad that most of bitcoin's braindamage is nitpicky stuff like byte ordering.
466 2012-12-18 21:22:17 <Matt_von_Mises> Going by the working genesis block, the merkle root is OK as it is also.
467 2012-12-18 21:23:37 <sipa> Matt_von_Mises: and time and nonce are also ok
468 2012-12-18 21:23:44 <etotheipi_> sipa, how many UTXOs are there currently? (do you know?)
469 2012-12-18 21:23:52 <etotheipi_> order of magnitude would work
470 2012-12-18 21:24:16 <gmaxwell> there is an rpc for this. :P
471 2012-12-18 21:24:37 <sipa> $ ./bitcoind gettxoutsetinfo
472 2012-12-18 21:24:45 <sipa> "transactions" : 1381563,
473 2012-12-18 21:24:46 <gmaxwell> "transactions" : 1381563,
474 2012-12-18 21:25:00 <gmaxwell> I like that we both thought to cut off the bestblock.
475 2012-12-18 21:25:18 <etotheipi_> heh
476 2012-12-18 21:25:45 <sipa> gmaxwell: seems our systems are well-synchronized :)
477 2012-12-18 21:25:58 <sipa> (my bytes_serialized is also the same)
478 2012-12-18 21:26:44 <stealth222> hmmm, I'm getting 3130985 - but I guess you're only counting outputs in blocks
479 2012-12-18 21:27:05 <stealth222> outputs in blocks in the main chain, yes?
480 2012-12-18 21:28:33 <sipa> where else?
481 2012-12-18 21:28:37 <stealth222> mempool
482 2012-12-18 21:28:47 <sipa> ah, yes, no only block chain
483 2012-12-18 21:28:48 <gmaxwell> stealth222: why do you do that?
484 2012-12-18 21:28:56 <stealth222> do what?
485 2012-12-18 21:29:06 <gmaxwell> Include the mempool in your txout set count?
486 2012-12-18 21:29:19 <sipa> well you have to while checking incoming mempool transactions
487 2012-12-18 21:29:36 <sipa> it makes sense, blockchain + mempool define a consistent UTXO set
488 2012-12-18 21:29:59 <gmaxwell> ::nods:: if you're actually combining them though that sounds messy when block knocks some out of your mempool as conflicts.
489 2012-12-18 21:30:26 <stealth222> that could happen with reorgs, too
490 2012-12-18 21:30:33 <sipa> well the gettxout RPC call allows you to select whether you include mempool txn
491 2012-12-18 21:30:43 <sipa> which removes things spent by the mempool
492 2012-12-18 21:30:53 <TD> the endianness was a huge PITA when i was first writing bitcoinj, but once it's done, it's done
493 2012-12-18 21:31:16 <TD> from that point on things are pretty much plain sailing. i mean the algorithms are complex.
494 2012-12-18 21:31:20 <TD> but the protocol itself isn't an issue
495 2012-12-18 21:32:00 <etotheipi_> sipa: do you know how many tx are completely pruneable?
496 2012-12-18 21:32:18 <etotheipi_> i.e. it sounds like if all outputs of a tx aren't entirely spent, then you have to keep the full tx
497 2012-12-18 21:33:18 <sipa> etotheipi_: 1381563 is the number of not-completely-spent transactions
498 2012-12-18 21:33:26 <etotheipi_> sipa: oh
499 2012-12-18 21:33:31 <etotheipi_> perfect
500 2012-12-18 21:33:45 <sipa> it's 9-10 million in total
501 2012-12-18 21:33:48 <etotheipi_> and if i'm not mistaken, there's about 10mil tx in the history?
502 2012-12-18 21:33:51 <TD> actually re: protocol buffers
503 2012-12-18 21:34:00 <etotheipi_> okay... that's a pretty good ration
504 2012-12-18 21:34:01 <TD> by the time they were open sourced satoshi had written the serialization code
505 2012-12-18 21:34:02 <etotheipi_> *ratio
506 2012-12-18 21:34:20 <TD> also he wanted the simplest protocol possible to minimize angles of attack
507 2012-12-18 21:34:21 <etotheipi_> though I imagine the bigger tx are the ones not getting pruned and taking up more space
508 2012-12-18 21:34:34 <etotheipi_> ("bigger" = more UTXOs)
509 2012-12-18 21:35:01 <sipa> 10051038 transactions in total now
510 2012-12-18 21:37:51 <gmaxwell> Two of the major pieces of "protocol significant" third party code we've been using??? openssl and BDB??? have both created problems for us. I'd only expect that to be true with a serialization library too.
511 2012-12-18 21:39:02 <stealth222> are you talking about security issues, gmaxwell? side channel attacks and stuff?
512 2012-12-18 21:39:10 <MC1984> any plans to dump openssl?
513 2012-12-18 21:39:23 <Luke-Jr> MC1984: not really practical, that's the problem :P
514 2012-12-18 21:39:34 <Luke-Jr> OpenSSL is effectively part of the Bitcoin network rules
515 2012-12-18 21:39:50 <stealth222> it is?
516 2012-12-18 21:40:01 <gmaxwell> It is. Though it can be immitated easily enough.
517 2012-12-18 21:40:12 <Luke-Jr> gmaxwell: as far as we know :/
518 2012-12-18 21:40:18 <gmaxwell> stealth222: implicit behavior creating effective rules that we don't know about.
519 2012-12-18 21:40:30 <MC1984> what do you use openssl for
520 2012-12-18 21:40:40 <Luke-Jr> MC1984: ECDSA signature verification
521 2012-12-18 21:40:46 <gmaxwell> Luke-Jr: well the testing sipa did on the encoding normalization is fairly persuasive.
522 2012-12-18 21:40:50 <Luke-Jr> MC1984: and it supports non-standard signatures!
523 2012-12-18 21:40:52 <stealth222> but there could be other implementations of bitcoind that use a different crypto library that still comply with the network rules
524 2012-12-18 21:41:07 <stealth222> not sure I understand what you mean by "network rules"
525 2012-12-18 21:41:19 <Luke-Jr> stealth222: they would need to implement OpenSSL's non-standard signature formats to be compliant
526 2012-12-18 21:41:19 <MC1984> i didnt know openssl did the elliptic curve stuff
527 2012-12-18 21:41:22 <gmaxwell> There could be. Maybe.
528 2012-12-18 21:41:26 <MC1984> i thought that was exotic and non mainstream
529 2012-12-18 21:41:50 <stealth222> aren't the formats independent of implementation?
530 2012-12-18 21:42:06 <gmaxwell> stealth222: the problem is that these libraries have undisclosed, undocumented, and probably unknown to everyone on earth behavior that when used by bitcoin became mandatory parts of the system.
531 2012-12-18 21:42:24 <gmaxwell> stealth222: we have serialized signatures and keys in transactions.
532 2012-12-18 21:43:08 <gmaxwell> If you send me a signature with an invalid encoding that openssl happens to accept, I'll accept it. And the bitcoin participants must be consistent, so once its in the chain everyone else must accept it too or the world ends.
533 2012-12-18 21:43:48 <stealth222> I understand that issues pertaining to randomization and such might be implementation-specific. but the actual ECDSA algorithm and signature formats are public knowledge. I guess this would be an issue if OpenSSL was the only way to verify the signature
534 2012-12-18 21:44:05 <gmaxwell> worse, if openssl were to have some subtle changes and accept more invalid encodings or reject some it used to accept??? something that might not even get a major version bump or a release note, ... that may cause a network split.
535 2012-12-18 21:44:34 <helo> terrifying :/
536 2012-12-18 21:44:52 <stealth222> are there any known inputs for which OpenSSL gives an output inconsistent with the published algorithms and formats?
537 2012-12-18 21:44:57 <gmaxwell> stealth222: okay, I'm not sure whats clear. The encoding is well know. Openssl's implementation will accept invalid encodings as valid. Do you follow that much?
538 2012-12-18 21:45:02 <gmaxwell> _yes_
539 2012-12-18 21:45:09 <stealth222> ok, invalid encodings
540 2012-12-18 21:45:18 <Luke-Jr> stealth222: many
541 2012-12-18 21:45:18 <stealth222> so it gives false positives
542 2012-12-18 21:45:37 <stealth222> does it give any false negatives?
543 2012-12-18 21:45:57 <gmaxwell> It lets you encode a valid ecdsa signature in an invalid way... and then successfully decodes it and allows it to pass.
544 2012-12-18 21:46:18 <stealth222> is there an error-correcting algorithm possible?
545 2012-12-18 21:46:24 <gmaxwell> and so that behavior means that an alternative correct implementation would give false negatives relative to openssl.
546 2012-12-18 21:46:56 <stealth222> is it possible to take any encoding that OpenSSL accepts and transform it into a cannonical one?
547 2012-12-18 21:46:57 <gmaxwell> I have no idea what you're thinking there.
548 2012-12-18 21:46:58 <Luke-Jr> http://finance.fortune.cnn.com/2012/12/18/bitcoin-money-laundering/ <-- someone smack this guy :\\
549 2012-12-18 21:47:33 <gmaxwell> stealth222: yes, by copying the openssl code and using it to extract the signatures.
550 2012-12-18 21:48:01 <stealth222> ok - so no information is lost in the "invalid encoding"
551 2012-12-18 21:48:04 <gmaxwell> stealth222: you could do less than that... but you'd have at least some risk that your reimplementation of openssl's implicit behavior was not faithful.
552 2012-12-18 21:48:34 <gmaxwell> (because none of that behavior was really intended it may not be especially obvious)
553 2012-12-18 21:49:15 <stealth222> couldn't a later version of bitcoin require valid encodings for all transactions signed past a certain block?
554 2012-12-18 21:49:27 <gmaxwell> stealth222: it just means that bitcoin nodes must accept the exact set (no more, no less) of invalid encodings that openssl accepts.
555 2012-12-18 21:49:42 <gmaxwell> Yes though thats a risky soft forking upgrade. We may perform it at some point.
556 2012-12-18 21:50:02 <weex> so is this a suggestion to add more tests to openssl?
557 2012-12-18 21:50:30 <gmaxwell> weex: I don't know that I'd say openssl is "at fault" here, we're using it for something it was not really intended for.
558 2012-12-18 21:51:20 <weex> sure, then in the bitcoin build process
559 2012-12-18 21:51:22 <stealth222> well, cannonical encodings are sort of necessary if you want to be able to make data searchable by hash
560 2012-12-18 21:51:26 <gmaxwell> For many things "be liberal in what you accept" works out okay, ??? it used to even be considered a good practice for network protocols, but its utterly incompatible with distributed consensus.
561 2012-12-18 21:51:58 <gmaxwell> weex: it's a dynamic library, it would need to be a runtime test.
562 2012-12-18 21:52:09 <sipa> stealth222: there were canonical encodings proposed, and checks (which don't depend on OpenSSL themselves!) are implemented
563 2012-12-18 21:52:10 <gmaxwell> since the behavior could change out from under us. :(
564 2012-12-18 21:52:24 <sipa> stealth222: they are not enforced on the network though (yet)
565 2012-12-18 21:52:43 <weex> this is a cool story
566 2012-12-18 21:53:08 <gmaxwell> stealth222: in the case of bdb, we had different fun??? for example old bitcoin nodes couldn't do reorgs that moved too many transactions because of some mostly invisible locking limits.
567 2012-12-18 21:53:11 <Luke-Jr> it would be nice to divide "this should be illegal" transactions from non-standard ones <.<
568 2012-12-18 21:53:33 <sipa> Luke-Jr: have you seen the implementation, even?
569 2012-12-18 21:53:51 <Luke-Jr> not yet :P
570 2012-12-18 21:53:58 <sipa> it's in HEAD
571 2012-12-18 21:54:18 <gmaxwell> there is still the mod-field issue.
572 2012-12-18 21:54:20 <sipa> pass a flag to VerifySignature and it will enforce canonical encodings
573 2012-12-18 21:54:44 <sipa> has nothing to do with IsStandard
574 2012-12-18 21:55:14 <sipa> gmaxwell: yes, but i fear rolling that out will take even longer (and even more custom crypto code)
575 2012-12-18 21:55:34 <sipa> in all clients
576 2012-12-18 21:56:13 <gmaxwell> Does the even/odd test just happen to work with (most) signatures already being produced?
577 2012-12-18 21:56:17 <jgarzik> Luke-Jr: (RE URL) SIGH
578 2012-12-18 21:56:35 <sipa> gmaxwell: don't think so
579 2012-12-18 21:56:43 <gmaxwell> :-/
580 2012-12-18 21:57:42 <gmaxwell> converting isn't so hard, assuming you can do an addition mod p, I guess. e.g. do the test, then do the addition if it fails.
581 2012-12-18 21:58:50 <gmaxwell> at least its easily tested code too.
582 2012-12-18 21:59:08 <sipa> sure, it's easy code, but it requires low-level EC calls instead of just ECDSA api calls
583 2012-12-18 21:59:49 <gmaxwell> I wonder if there is a more complex test that existing transactions do match.
584 2012-12-18 22:00:02 <sipa> very doubtful
585 2012-12-18 22:00:29 <gmaxwell> ACTION shakes his fist at randomness.
586 2012-12-18 22:01:04 <sipa> ifbthere was such a criterion, i'd consider that a bug in openssl
587 2012-12-18 22:02:32 <stealth222> were these OpenSSL issues known at the beginning?
588 2012-12-18 22:02:51 <gmaxwell> well, you could argue it another way: it's leaking a bit from the random value when you do repeated signatures, no? sounds a little scary.
589 2012-12-18 22:02:54 <gmaxwell> stealth222: of course not.
590 2012-12-18 22:05:04 <gmaxwell> aand again, I don't really mean to single out openssl ... some other crypto library might have its own implicit overly accepting behavior.
591 2012-12-18 22:08:34 <stealth222> do all implementations of OpenSSL share the same behavior? and is there a risk of future versions "fixing" them and therefore breaking bitcoin?
592 2012-12-18 22:09:51 <sipa> we've used this "flexibility" to our advantage as well already: compressed pubkeys would have required a hardfork otherwise
593 2012-12-18 22:10:32 <stealth222> not sure I follow, sipa
594 2012-12-18 22:11:24 <sipa> conpressed pubkeys were accepted by openssl, so we could just start using them, and old bitcoin nodes on the network would validate them as valid
595 2012-12-18 22:11:35 <stealth222> a compressed pubkey just means one of the two curve coordinates are passed, right?
596 2012-12-18 22:11:45 <sipa> yes
597 2012-12-18 22:11:53 <sipa> plus one bit to denote the oddness of Y
598 2012-12-18 22:13:24 <stealth222> but this is a well-known format that can be rigorously defined - sounds like a far cry from what you were saying earlier regarding mysterious behavior
599 2012-12-18 22:13:44 <sipa> sure
600 2012-12-18 22:14:02 <sipa> but it also allows a non-standard hybrid format for ec points
601 2012-12-18 22:14:50 <stealth222> nonstandard-but-rigorously-defined is not nearly as scary to me as standard-but-impossible-to-reproduce
602 2012-12-18 22:14:51 <sipa> and for signatures, it allows incorrectly padded numbers and negative numbers (as long as they're correct when interpreted as unsigned number)
603 2012-12-18 22:15:14 <sipa> and incorrect values in the headers
604 2012-12-18 22:16:18 <sipa> which are all fine... trying to accept as many inputs is meaningful
605 2012-12-18 22:16:21 <gmaxwell> sipa: it's computers, it's never impossible to reproduce! :)
606 2012-12-18 22:16:28 <gmaxwell> er stealth222
607 2012-12-18 22:16:36 <stealth222> well, not impossible - but often impractical
608 2012-12-18 22:16:42 <sipa> but we depend on the exact casesbwhuch are or are not allowed
609 2012-12-18 22:17:00 <stealth222> I mean, I'm talking about defining algorithmically what's going on independent from specific implementation
610 2012-12-18 22:17:35 <stealth222> a flexible format isn't too much of a problem as long as equivalence classes are well-defined
611 2012-12-18 22:18:09 <gmaxwell> stealth222: it is, however, impossible when you _don't_ know about the issue because the behavior is surprising and not documented.
612 2012-12-18 22:18:29 <stealth222> OpenSSL's documentation does leave a LOT to be desired - that's for sure
613 2012-12-18 22:19:04 <stealth222> gmaxwell: so are you saying all these things were intended as features? because it sounds like some of them are bugs.
614 2012-12-18 22:19:37 <gmaxwell> who knows, not documented. Some could be bugs, some could be intended.. for most of what openssl is used for being permissive is beneficial.
615 2012-12-18 22:20:58 <stealth222> so you know of instances where OpenSSL validates a signature and you can't figure out the pattern in the input that maps it to a standard format?
616 2012-12-18 22:21:24 <stealth222> regardless of how nonstandard that mapping is
617 2012-12-18 22:21:56 <Luke-Jr> stealth222: you can't correct the signatures without changing the txid
618 2012-12-18 22:21:59 <gmaxwell> No, thats not the issue. But for example because it will accept nonstandard forms you can modify other people's transactions before they're mined.
619 2012-12-18 22:23:09 <stealth222> you can modify the signature, which would modify the tx hash - but wouldn't the outputs still be the same?
620 2012-12-18 22:23:20 <stealth222> err, I mean
621 2012-12-18 22:23:28 <stealth222> wouldn't it still be possible for the recipient to claim it?
622 2012-12-18 22:23:29 <Luke-Jr> stealth222: yes, but if any of those outputs are spent, the transaction spending them is invalid if the txid changed
623 2012-12-18 22:23:51 <stealth222> I understand that part, Luke-Jr
624 2012-12-18 22:24:22 <gmaxwell> it also means you can parasitically stuff data into the blockchain on other people's txns.
625 2012-12-18 22:24:58 <stealth222> so the workaround to what Luke-Jr is saying is to monitor for this and resend using the new txid
626 2012-12-18 22:25:14 <stealth222> but it would break the entire transaction chain following it
627 2012-12-18 22:25:18 <Luke-Jr> stealth222: you're assuming the person resending isn't the hostile part
628 2012-12-18 22:25:20 <Luke-Jr> party*
629 2012-12-18 22:25:52 <sipa> stealth222: the problem is not making other implementations tolerant in their decodinh
630 2012-12-18 22:26:06 <sipa> the problem is making them exactly as tolerant as openssl is
631 2012-12-18 22:26:40 <sipa> and modifying transactions in-flight is a very bad idea
632 2012-12-18 22:26:51 <gmaxwell> no more, no less. Either spells disaster. And how tolterant it is isn't well defined.
633 2012-12-18 22:27:15 <gmaxwell> I mean, there is the existing code but they have not sworn to a suicide pact to never change it.
634 2012-12-18 22:27:40 <gmaxwell> This is by no means insoluable.. I brought it up as an example of how using third party code for encoding has not worked out so well for us.
635 2012-12-18 22:28:03 <phantomcircuit> which sort of begs the question
636 2012-12-18 22:28:18 <phantomcircuit> has anybody tested different versions of openssl?
637 2012-12-18 22:29:07 <stealth222> could someone use modification of transactions in flight for DoS?
638 2012-12-18 22:29:18 <stealth222> or splitting the network?
639 2012-12-18 22:29:46 <gmaxwell> stealth222: dos against end user applications that make some assumptions about how txn work, yes.. against the network itself, no not really.
640 2012-12-18 22:30:02 <gmaxwell> And it can't split the network so long as everything pretends to be the current version of openssl.
641 2012-12-18 22:30:21 <gmaxwell> phantomcircuit: I diff openssl when updating the rpms I host, and I might notice this code changing.
642 2012-12-18 22:31:03 <stealth222> the attack that Luke-Jr was referring to would almost appear like a double-spend, no?
643 2012-12-18 22:31:14 <gmaxwell> it would.
644 2012-12-18 22:32:42 <stealth222> except the outputs would be the same
645 2012-12-18 22:32:49 <stealth222> in both transactions
646 2012-12-18 22:32:59 <gmaxwell> Yes. it's isomorphic but you have to be sufficiently smart.
647 2012-12-18 22:33:55 <gmaxwell> I could also see thing being used to rob a bank like service. It pays you, you mutate the transaction.. and the clone confirms... but the site sees that its payment to you hasn't actually confirmed and you get it to reissue (perhaps with a bigger fee to make it confirm)...
648 2012-12-18 22:34:16 <gmaxwell> I don't believe anything is currently vulnerable to that, but I could easily see it happening in the future.
649 2012-12-18 22:34:27 <Luke-Jr> gmaxwell: I don't believe anyone has tried it :o
650 2012-12-18 22:34:59 <gmaxwell> well I don't know of a site that will do an automatic reissue with more fees.
651 2012-12-18 22:35:24 <gmaxwell> might work against joe-average OTCer.. "your payment didn't go through! send again!"
652 2012-12-18 22:35:25 <Luke-Jr> MtGox
653 2012-12-18 22:35:34 <stealth222> interesting...so you could do something akin to double-spend detection (i.e. check to see if any other transactions claim the same input) but on your own transactions
654 2012-12-18 22:35:39 <gmaxwell> oops I didn't actually intend to disclose a potential vulnerability.
655 2012-12-18 22:35:59 <Luke-Jr> MagicalTux:
656 2012-12-18 22:36:20 <Luke-Jr> gmaxwell: otoh, MagicalTux probably knows enough to ensure he's reusing an input
657 2012-12-18 22:36:51 <stealth222> right, to resend you have to sign from the same inputs to make sure you don't get ripped off
658 2012-12-18 22:36:54 <stealth222> err
659 2012-12-18 22:36:57 <stealth222> claim the same outputs
660 2012-12-18 22:37:42 <stealth222> or just rebroadcast the same transaction if you're not sure
661 2012-12-18 22:37:59 <gmaxwell> if you're changing the fees you'll want to change inputs and not send the same txn.
662 2012-12-18 22:38:14 <MagicalTux> when we re-issue with more fees, we use the same inputs
663 2012-12-18 22:38:24 <gmaxwell> but yes, this can be dealt with??? if you know. I think its likely that I would have gotten it right, but I suspect some people wouldn't.
664 2012-12-18 22:38:35 <gmaxwell> MagicalTux: thats good!
665 2012-12-18 22:48:44 <stealth222> if there were a field in the transaction which stored the type of encoding used in the signature (which was part of the structure that is signed) it would get rid of this issue while still affording flexibility in encoding
666 2012-12-18 22:49:07 <stealth222> but perhaps it isn't really a serious problem
667 2012-12-18 22:49:18 <stealth222> and such a solution requires a hardfork
668 2012-12-18 22:49:31 <sipa> stealth222: it wouldn't solve a thing
669 2012-12-18 22:49:44 <sipa> because every implementation would still be required to support every encoding
670 2012-12-18 22:50:08 <stealth222> it wouldn't solve the OpenSSL issues - but it would solve the issue Luke-Jr brought up
671 2012-12-18 22:50:15 <sipa> and the solution is easy: just allow one canonical encoding
672 2012-12-18 22:50:21 <sipa> and that doesn't require a hardfork
673 2012-12-18 22:50:29 <gmaxwell> yea, thats a nutty solution. :P
674 2012-12-18 22:50:55 <gmaxwell> (allowing _more_ encodings.. erk, that just means more code that has to be exactly right which people won't adequately test)
675 2012-12-18 22:52:01 <phantomcircuit> gmaxwell, clearly there should be some sort of exhaustive testing
676 2012-12-18 22:52:09 <stealth222> sipa: one canonical encoding was where we started - but then you talked about how you liked the flexibility that allowed for things like compressed pubkeys
677 2012-12-18 22:52:14 <phantomcircuit> say a bruteforce search of the key space to verify correctness!
678 2012-12-18 22:52:16 <phantomcircuit> ACTION runs
679 2012-12-18 22:52:31 <sipa> stealth222: we didn't start with one canonical encoding
680 2012-12-18 22:52:33 <stealth222> I guess multiple pubkey formats are ok...just not multiple signature formats
681 2012-12-18 22:52:46 <sipa> stealth222: we started with code that produced one type of serialized data, but accepted many
682 2012-12-18 22:53:04 <sipa> stealth222: i want code that produces just one type, and accepts just one type
683 2012-12-18 22:53:25 <stealth222> it was a suggestion I made a lot earlier "stealth222: is it possible to take any encoding that OpenSSL accepts and transform it into a cannonical one?"
684 2012-12-18 22:53:42 <sipa> that's not what i mean
685 2012-12-18 22:53:52 <gmaxwell> And it turned out to be 'fortunate' in that case. But the fact that sometimes a broken fork makes a better spoon does not mean you should wish your forks all be broken.
686 2012-12-18 22:54:00 <sipa> i mean making it simply illegal on the network to use any but the canonical encoding
687 2012-12-18 22:54:16 <stealth222> right, that's how it should be ideally
688 2012-12-18 22:54:33 <sipa> so no need to transform anything - wallet software would simply not be allowed to produce weird stuff
689 2012-12-18 22:55:18 <stealth222> I guess the pubkey issue is separate - that cannot be modified except by the person who originally signed the transaction
690 2012-12-18 22:55:24 <sipa> stealth222: see https://github.com/bitcoin/bitcoin/blob/master/src/script.cpp#L271
691 2012-12-18 22:55:49 <stealth222> right, I see
692 2012-12-18 22:56:02 <sipa> that code is not active right now
693 2012-12-18 22:56:08 <gmaxwell> stealth222: it can still be problematic, just not as problematic.
694 2012-12-18 22:56:26 <gmaxwell> E.g. if there were valdating nodes that didn't accept compressed public keys, someone could have forked the network by writing one.
695 2012-12-18 22:56:42 <stealth222> yes, I see what you're saying
696 2012-12-18 22:57:17 <sipa> at least now it is known, and people can test their software for the ability to verify compressed pbukesy
697 2012-12-18 22:57:36 <stealth222> my software got tripped up initially by that :)
698 2012-12-18 22:57:39 <stealth222> but I did fix it
699 2012-12-18 22:57:41 <gmaxwell> (more likely than you might guess??? there are patents related to compressed public keys (which are, fortunately only applicable to other curves and probably invalid) so some people have a reflexive "don't use them at all" plus they take more code to implement.)
700 2012-12-18 22:58:25 <gmaxwell> (okay, I guess you'd guess it pretty likely!)
701 2012-12-18 22:58:59 <gmaxwell> and really the concern there was that for a long time we just didn't know about it.. because it was behavior hidden inside openssl.
702 2012-12-18 22:59:29 <stealth222> so there's something to be said for not trusting third-party libraries to validate inputs :)
703 2012-12-18 23:00:17 <gmaxwell> Right. They're just often written with different goals in mind, even if they are excellent software. ... er not that I'm implying anything about openssl. :P
704 2012-12-18 23:00:31 <sipa> if bitcoin would have defined its own format for pubkeys and signatures, and transformed that from/to ssl when creating/checking, the code would be a lot easier, actually
705 2012-12-18 23:00:48 <sipa> that's something that would be considered a bad idea in most software, but here it'd make sense
706 2012-12-18 23:02:55 <stealth222> or at least decided upon a single standard format and enforced it
707 2012-12-18 23:03:00 <gavinandresen> we're about to bite off another huge pail of worms with X.509 certificates in the payment protocol proposal, by the way....
708 2012-12-18 23:04:16 <gmaxwell> stealth222: we have a single standard format, and openssl looks like it enforces it. Writing your own enforcement would fall into sipa's "something that would be considered a bad idea in most software" bin. Also, it's a little harder to get surely correct behavior from enforcement rather than implementation.
709 2012-12-18 23:05:00 <gmaxwell> e.g. if you define what you do, and you all use that defintion its automatically the same everywhere. If you instead write a filter you may be vulnerable to what you didn't think to filter.
710 2012-12-18 23:05:23 <sipa> gavinandresen: at least those are not bound by the pitfalls of distributed consensus
711 2012-12-18 23:05:41 <sipa> gavinandresen: if the other partner doesn't support the encoding, the transaction will just fail
712 2012-12-18 23:05:54 <gavinandresen> sipa: true, only merchant and customer need to agree on validity.
713 2012-12-18 23:05:59 <gmaxwell> I wonder if I could generally say that any time you MOST tight-bounded behavior from software you should not use third party code.
714 2012-12-18 23:06:05 <gmaxwell> er MUST
715 2012-12-18 23:06:16 <sipa> does not compute
716 2012-12-18 23:07:09 <gavinandresen> meh. some things are straightforward enough that third party code is fine (e.g. standard hashing algorithms, AES encryption....)
717 2012-12-18 23:08:47 <sipa> i must say i'm beginning to like the idea of having PKI certificates that can specify a base pubkey, and allow payment requests to use a pubkey that is provably derived from it
718 2012-12-18 23:11:32 <gavinandresen> sipa: mmm. we could encode the master public key in one of the Subject Alternative Names in the certificate....
719 2012-12-18 23:12:25 <donnchac> Sorry to bother you guys again, I'm just trying to find more information about the "block callback" function. I can't find any references to it in the standard Satoshi client.
720 2012-12-18 23:12:33 <sipa> donnchac: -blocknotify
721 2012-12-18 23:12:41 <donnchac> Thanks :)
722 2012-12-18 23:12:47 <sipa> gavinandresen: it would mean metadata attached to every output, though
723 2012-12-18 23:13:13 <gavinandresen> sipa: ?
724 2012-12-18 23:13:13 <sipa> gavinandresen: so even if v1 of the payment protocol doesn't support it, maybe keep that open for future extensibility
725 2012-12-18 23:13:33 <sipa> as you'll need a proof for every output that it uses a key that is derived from the base pubkey
726 2012-12-18 23:14:09 <gavinandresen> sipa: you mean the use-the-blockchain-as-the-payment-communication-channel case?
727 2012-12-18 23:14:15 <sipa> gavinandresen: no
728 2012-12-18 23:14:23 <gavinandresen> ... then I'm not following.
729 2012-12-18 23:14:28 <sipa> gavinandresen: i mean the outputs in the payment request message
730 2012-12-18 23:14:34 <sipa> *checks terminology*
731 2012-12-18 23:15:24 <gavinandresen> oh, I assume that the Output.script 's would be left out, and derived by the customer from the base key and hash(payment data)
732 2012-12-18 23:15:54 <sipa> oh, i see
733 2012-12-18 23:15:57 <donnchac> sipa: using the blocknotify would I just use this to call a standalone script in my webapp which would retrieve recent transactions and use that to update live bitcoin tally in my webapp's DB?
734 2012-12-18 23:16:01 <gavinandresen> I see, you're thinking of the case where the merchant includes multiple Outputs for some reason
735 2012-12-18 23:16:31 <sipa> gavinandresen: yes
736 2012-12-18 23:17:18 <sipa> gavinandresen: and specifies himself which multiplier to use (which is more sane, imho, for example he can make sure they're from a determinstic chain he chooses)
737 2012-12-18 23:17:19 <gavinandresen> sipa: yeah. I STILL waffle on whether allowing multiple Outputs is worth the extra complexity.
738 2012-12-18 23:17:36 <sipa> though i like the simplicity of having the sender derive it
739 2012-12-18 23:18:22 <gavinandresen> donnchac: if it is a webapp, you might -blocknotify 'curl http://localhost:8888/newblock?arguments_to_ping_your_web_app'
740 2012-12-18 23:31:48 <sipa> gavinandresen: i think we need to add a bytecode script language to the certificate to enumerate the valid output scripts, and script data in the paymentrequest.output that is passed as input to the txout-script-generating-script
741 2012-12-18 23:32:01 <sipa> that'd at least by gangnam style
742 2012-12-18 23:32:07 <sipa> i mean satoshi style
743 2012-12-18 23:32:12 <sipa> *be
744 2012-12-18 23:34:58 <gavinandresen> sure. We just need OP_ECCMULTIPLY and a pseudo-op OP_PUSHPAYMENTHASH....
745 2012-12-18 23:35:57 <gavinandresen> Actually, I guess OP_ECCMULTIPLY could be a pseudo-op just interpreted in payment request Outputs, too
746 2012-12-18 23:36:09 <sipa> haha
747 2012-12-18 23:36:34 <gavinandresen> :)
748 2012-12-18 23:36:47 <MC1984> OP_GANGNAM_STYLE
749 2012-12-18 23:37:24 <sipa> should be OPAN_GANGNAM_STYLE (
750 2012-12-18 23:39:51 <sipa> ^ remember that for april first
751 2012-12-18 23:41:08 <donnchac> gavinanderson: I assummed just make a curl callback to the webapp, can I provide data with the callback from bitcoind, I assume not. So the webapp just gets notified there is a new block and it is then up to the webapp to get the info it needs about new transactions by querying bitcoind via the JSON-RPC method
752 2012-12-18 23:41:12 <donnchac> .
753 2012-12-18 23:41:55 <sipa> donnchac: %s gets replaced by the block's hash
754 2012-12-18 23:44:13 <donnchac> Okay perfect, and the webapp just needs to ensure it doesn't miss any transactions itself?
755 2012-12-18 23:44:38 <donnchac> Sorry about all the spoon feeding :)
756 2012-12-18 23:50:40 <gavinandresen> somebody should publish some example code of keeping a transaction database up-to-date using -blocknotify. It is not trivial because of block-chain re-orgs
757 2012-12-18 23:51:41 <midnightmagic> lol, and if they do, it would be really nice if the orphans and/or expired branched stayed around for querying purposes. :)
758 2012-12-18 23:51:59 <gavinandresen> midnightmagic: sure.
759 2012-12-18 23:52:01 <donnchac> Anyone want to take up that offer, or would anyone like to release an excerpt of their code which deals with this kind of functionality.
760 2012-12-18 23:52:21 <donnchac> I don't want to reinvent the wheel, especially if I would do it incorrectly.
761 2012-12-18 23:53:11 <gmaxwell> gavinandresen: it's really hard to get right and handle all the corner cases.
762 2012-12-18 23:54:10 <gmaxwell> I think you have to keep a table of height->block hashes so you can tell how far back you need to process for reorgs.
763 2012-12-18 23:54:28 <gavinandresen> gmaxwell: shouldn't be THAT hard... just keep track of the block chain, if you get notified of a block who's parent is not your current tip, walk back to where it forked, mark all transactions on old chain as 0-confirmation, then walk forward on new chain adding transactions....
764 2012-12-18 23:55:12 <gmaxwell> right, thats what I was thinking??? but it means you need a pretty solid grasp of how the chain works. Which I'd classify as hard compared to everything else that person would need to know.
765 2012-12-18 23:55:19 <gavinandresen> agreed
766 2012-12-18 23:56:05 <donnchac> The whole -blocknotify schema for tracking payments seems to be less than ideal. Maybe it would be good if implementing more transaction callbacks in bitcoind should be prioritised?
767 2012-12-18 23:56:15 <gavinandresen> the easy way out is to just record transactions when they are 6-confirmations deep, and assume that you'll never get a 6-deep re-org on the main network.
768 2012-12-18 23:56:25 <gmaxwell> donnchac: transaction callbacks _do not solve this_
769 2012-12-18 23:56:48 <gmaxwell> gavinandresen: block explorer does something like that.
770 2012-12-18 23:57:19 <gmaxwell> I think for every new block it checks the last 14 blocks or so.. if there is a reorg longer than that the world ends.
771 2012-12-18 23:57:42 <gmaxwell> (I know this because I broke it on testnet several times)