1 2012-11-01 00:00:56 <D34TH> just got spammed with "Relaying wtx : BLAHBLAHHBLAH"
2 2012-11-01 00:01:11 <D34TH> ill assume im relaying transactions
3 2012-11-01 00:01:13 <D34TH> :D
4 2012-11-01 00:01:37 <sipa> no, you're transmitting
5 2012-11-01 00:01:41 <sipa> wtx == wallet transaction
6 2012-11-01 00:01:59 <D34TH> weird, i havent sent any txes
7 2012-11-01 00:02:16 <D34TH> probably because im just downloading the chain
8 2012-11-01 00:02:22 <D34TH> ill chalk it up to that
9 2012-11-01 00:11:03 <sipa> gmaxwell: fixed
10 2012-11-01 00:11:10 <sipa> ACTION zZzZ
11 2012-11-01 00:54:32 <jgarzik> sipa: actually, I've been meaning to report the same
12 2012-11-01 00:55:01 <jgarzik> sipa: I'll be sync'ing, from ~5 days ago blockchain to current, and get spammed with a -bunch- of "relaying wtx"
13 2012-11-01 00:55:12 <jgarzik> sipa: listtransactions reports no recent traffic
14 2012-11-01 01:00:40 <jgarzik> sipa: to be specific,
15 2012-11-01 01:00:41 <jgarzik> [jgarzik@bd tmp]$ grep -c 'Relaying wtx' debug.log
16 2012-11-01 01:00:52 <jgarzik> that's just one ResendWalletTransactions() iteration
17 2012-11-01 01:09:11 <jgarzik> I'm pretty sure it resending 100% of my wallet transactions
18 2012-11-01 01:09:15 <jgarzik> *it is
19 2012-11-01 01:09:29 <gmaxwell> jgarzik: sure, they're not confirmed! you want them confirmed don't you? :P
20 2012-11-01 01:10:08 <jgarzik> gmaxwell: they're all confirmed?
21 2012-11-01 01:10:24 <gmaxwell> not as far as your node is concerned though.
22 2012-11-01 01:12:06 <jgarzik> gmaxwell: ?
23 2012-11-01 01:12:19 <jgarzik> gmaxwell: it is 98% sync'd at the time of the messages
24 2012-11-01 01:12:35 <jgarzik> and no new transactions appear post-sync
25 2012-11-01 01:14:40 <gmaxwell> jgarzik: hm! I suppose they were queued for retransmit earlier? odd.
26 2012-11-01 01:41:06 <Garr255> hey, anyone here ever run into this error? :-1: error: D8021 : invalid numeric argument '/Wl,-z,relro'
27 2012-11-01 01:41:07 <Garr255> https://dl.dropbox.com/u/9542654/qt%20error.png
28 2012-11-01 01:41:32 <Garr255> Trying to compile qt in windows...
29 2012-11-01 11:39:46 <rdponticelli_> <jgarzik> I'm pretty sure it resending 100% of my wallet transactions << I've noticed the same behavior on my node
30 2012-11-01 11:45:49 <sipa> rdponticelli: which version?
31 2012-11-01 11:47:33 <rdponticelli> sipa: it started on the first ultraprune commit
32 2012-11-01 11:47:53 <rdponticelli> It was hit by the zero balance bug
33 2012-11-01 11:48:36 <rdponticelli> Recovered after the patch, by a wallet backup, but it kept resending the transactions now and then...
34 2012-11-01 11:55:47 <rdponticelli> Now it was on 578fc80, I'm updating...
35 2012-11-01 11:56:47 <rdponticelli> Anyway, they're only some changes to the gui
36 2012-11-01 11:58:34 <sipa> rdponticelli: will look into it
37 2012-11-01 12:01:42 <rdponticelli> sipa: If I can do anything else to help, just tell me...
38 2012-11-01 12:03:46 <TD> wiki is still broken :(
39 2012-11-01 12:03:52 <TD> definitely time to move at least the FAQ into the main website
40 2012-11-01 13:47:35 <jeremias> I have a another bitocin installation, which I trust. I want to copy the block chain to other place. Which files should I copy?
41 2012-11-01 13:48:01 <jeremias> Or does it speed up the process? What is the bottleneck?
42 2012-11-01 13:49:08 <sipa> jeremias: run the trusted instance with -detachdb, shut it down cleanly, and copy blkindex.dat and blk0000?.dat over
43 2012-11-01 13:49:11 <rdponticelli> jeremias: which version?
44 2012-11-01 13:49:31 <sipa> af of 0.8, it will be the blocks/, blktree/ and coins/ directories
45 2012-11-01 13:49:59 <jeremias> 0.7.1
46 2012-11-01 13:50:35 <jeremias> thanks
47 2012-11-01 13:50:56 <sipa> but please make sure you actually trust the other place (i.e., you control it yourself)
48 2012-11-01 13:52:03 <jeremias> yes, it is my other server and I'm setupping bitcoind to other
49 2012-11-01 13:52:24 <jeremias> however, has there been cases where this kind of situation has been used to hack?
50 2012-11-01 13:52:33 <jeremias> eg. someone providing fake block chain
51 2012-11-01 13:52:50 <rdponticelli> Even if you control it yourself, your node could have been theoretically isolated...
52 2012-11-01 13:53:01 <rdponticelli> But chances are pretty low
53 2012-11-01 13:53:14 <gmaxwell> rdponticelli: isolated isn't a problem
54 2012-11-01 13:53:25 <gmaxwell> rdponticelli: so long as the new node isn't.
55 2012-11-01 13:53:43 <rdponticelli> gmaxwell: It isn't?
56 2012-11-01 13:54:00 <gmaxwell> rdponticelli: No of course not.
57 2012-11-01 13:54:07 <rdponticelli> Ok, yeah...
58 2012-11-01 13:54:21 <rdponticelli> The new node would eventually find out the longer chain
59 2012-11-01 13:55:06 <gmaxwell> jeremias: people have had incorrect chains, but it's hard to tell if it was random corruption (most likely) or malicious without doing extensive research which no one has botered doing, esp since a latent attack is indistinguishable from corruption.
60 2012-11-01 13:59:04 <jeremias> hmm, is -detachdb supposed to take like ages?
61 2012-11-01 14:01:59 <rdponticelli> jeremias: -detachdb will start the node normally, it will make the detaching when you close it
62 2012-11-01 14:02:48 <jeremias> all right
63 2012-11-01 14:18:22 <TD> jeremias: detaching bdb on shutdown can be slow yes
64 2012-11-01 14:31:10 <TD> gavinandresen: good morning
65 2012-11-01 14:32:20 <gavinandresen> liar, it's not morning where you are, is it?
66 2012-11-01 14:32:32 <gavinandresen> I mean, "good morning!"
67 2012-11-01 14:34:13 <TD> it's afternoon indeed
68 2012-11-01 14:34:24 <TD> all of BlueMatts full verification work is now merged into bitcoinj
69 2012-11-01 14:34:39 <TD> so i think bcj has joined the ranks of fully verifying implementations! \\o/
70 2012-11-01 14:37:49 <gavinandresen> Nice!
71 2012-11-01 14:38:43 <gavinandresen> Anybody running a "dual stack" verifier yet to make sure they always agree on longest chain?
72 2012-11-01 14:38:54 <TD> matt is, i think. but more than one would be good
73 2012-11-01 14:39:14 <gavinandresen> running on testnet would probably be very good, testnet gets a lot of weirdness that main net doesn't.
74 2012-11-01 14:39:19 <TD> right
75 2012-11-01 14:41:15 <TD> gavinandresen: atm you are working on json apis for multi-sig txns?
76 2012-11-01 14:42:09 <gavinandresen> right this very moment I'm working on a batch email to all foundation members to announce member-only Foundation forums....
77 2012-11-01 14:42:52 <gavinandresen> But working out what needs to be done for multi-sig txn coordination is at the top of my TODO
78 2012-11-01 14:43:53 <TD> cool
79 2012-11-01 14:44:40 <gavinandresen> TD: were you part of the discussion a few days ago about messaging infrastructure for coordinating txn signing ?
80 2012-11-01 14:45:32 <TD> probably. i was the one suggesting usage of smartphone cloud messaging
81 2012-11-01 14:45:33 <helo> will the foundation forums be for foundation-specific discussions only, or will they include general, development, trading, and other sections?
82 2012-11-01 14:45:39 <TD> (for having phones be second factors)
83 2012-11-01 14:46:33 <gavinandresen> TD: right.... thanks for those pointers. Those APIs only guarantee sending the last message to the phone from any particular service, so they don't solve the whole problem.
84 2012-11-01 14:48:02 <helo> do we want to support offline-only auth factors?
85 2012-11-01 14:48:55 <gavinandresen> helo: foundation forums will be for whatever foundation members want to discuss...
86 2012-11-01 14:49:14 <gavinandresen> helo: offline-only auth factors??
87 2012-11-01 14:49:19 <gavinandresen> helo: how would that work?
88 2012-11-01 14:49:19 <TD> really? i thought GCM would deliver all messages, it just does not guarantee ordering
89 2012-11-01 14:49:32 <TD> it's designed to be used for things like IM apps so not delivering all messages would be ???.. suboptimal :)
90 2012-11-01 14:49:34 <TD> there is a collapse key system
91 2012-11-01 14:49:57 <gavinandresen> TD: I could be misremembering, it might have been Apple's system that only guaranteed delivery of the last message.
92 2012-11-01 14:50:00 <TD> ah, yes
93 2012-11-01 14:50:02 <TD> Apples system sucks
94 2012-11-01 14:50:18 <TD> but then again, they also have a habit of randomly banning bitcoin-related apps. so I basically assume iPhones don't exist.
95 2012-11-01 14:50:42 <helo> gavinandresen: qr-code, sound, NFC, other non-IP-based data transmission
96 2012-11-01 14:51:09 <TD> helo: that isn't a second factor. it's data that has to be loaded into the first factor.
97 2012-11-01 14:51:26 <TD> helo: the point of 2-factor auth is that multiple devices work together independently to sign. that way, if one is compromised, the wallet can't be stolen.
98 2012-11-01 14:51:30 <sipa> the message can be "last message received at timestamp" and then have a callback to the server for "read messages between tinestamp X and Y"
99 2012-11-01 14:51:32 <TD> (the devices double check each others requests)
100 2012-11-01 14:51:47 <sipa> then you only need the last message to be delivered to the mobile
101 2012-11-01 14:51:48 <gavinandresen> helo: oh... ummm, sure, I suppose, the protocol won't care HOW the signature gets generated, if it sent to an app on a phone or there's some complicated QR-code exchange that only geeks will use....
102 2012-11-01 14:51:51 <TD> sipa: APNS is really limited. the message has to be delivered to a human. iOS is a waste of time
103 2012-11-01 14:52:18 <helo> NFC is probably the only actually viable one...
104 2012-11-01 14:52:20 <TD> sipa: the messages aren't actually delivered to the apps, iirc. only the OS handles them, by popping an on-screen message, playing a sound or putting a badge onto the app icon
105 2012-11-01 14:52:30 <sipa> oh
106 2012-11-01 14:52:34 <sipa> really?
107 2012-11-01 14:52:37 <TD> on android it works as you'd expect
108 2012-11-01 14:52:46 <sipa> how does mail push work then?
109 2012-11-01 14:52:59 <sipa> it needs to trigger a call to the mail app
110 2012-11-01 14:53:49 <sipa> or does that get special treatment?
111 2012-11-01 14:53:54 <TD> http://developer.apple.com/library/mac/#documentation/NetworkingInternet/Conceptual/RemoteNotificationsPG/ApplePushService/ApplePushService.html
112 2012-11-01 14:53:59 <TD> apple doesn't have mail push
113 2012-11-01 14:54:09 <TD> this is why the google login service processes about a bajillion logins per second
114 2012-11-01 14:54:29 <TD> iOS is really a wasteland of opportunity, it's sad really.
115 2012-11-01 14:55:17 <TD> anyway, the android equivalent is powerful enough you can actually make a "peer to peer" network of apps that send each other messages, all routed via the GCM network
116 2012-11-01 14:56:00 <TD> sipa: more specifically, you can do mail push on iOS using ActiveSync, or something like that. it's weird.
117 2012-11-01 14:56:59 <BlueMatt> TD: without the "upgrade to lang3" commit, IIRC there is no dep on commons in pom.xml
118 2012-11-01 14:57:15 <TD> BlueMatt: yeah but i replaced the code that was using apache commons
119 2012-11-01 14:57:21 <BlueMatt> ah, ok
120 2012-11-01 14:57:26 <TD> BlueMatt: thanks again.
121 2012-11-01 14:57:33 <BlueMatt> sorry, just started reading the comments
122 2012-11-01 14:57:36 <TD> BlueMatt: given the huge amount of code involved it was a very easy set of merges
123 2012-11-01 14:57:50 <gavinandresen> TD: it wasn't clear to me how push authorization would work with GCM... e.g. how does a user running Bitcoin-Qt tell the GCM network it is OK for messages to go to the app running on their phone
124 2012-11-01 14:57:53 <TD> one or two i noticed you addressed in later commits (i was reading through and merging them in order)
125 2012-11-01 14:58:13 <BlueMatt> TD: (which is largely a testament to how cleanly structured bitcoinj is)
126 2012-11-01 14:58:31 <gavinandresen> TD: (it is clear if there is a central server acting as an intermediary, which is probably how it would have to work)
127 2012-11-01 14:58:47 <TD> gavinandresen: the app on the phone gets given a code. anything that has that code can send. so you still need to "rendezvous" with the phone somehow. if i was writing it i'd use wifi service discovery aka bonjour, as it's multi-platform, fairly well supported and from a users POV is simple - just make sure the devices are on the same LAN
128 2012-11-01 14:58:52 <TD> s/LAN/wifi network/
129 2012-11-01 14:59:05 <TD> gavinandresen: alternatively the code may even be typable, i don't know, never looked.
130 2012-11-01 15:00:37 <gavinandresen> TD: cool. My current plan is to do some prototyping using the raw transactions API, and then later take what I learn and maybe bake it into Bitcoin-Qt
131 2012-11-01 15:02:01 <TD> yeah. so for bitcoin-qt you could as well just have users directly type in the IP addresses of the other peers they want to connect to. you can then provide a label/button that copy/pastes some HTML containing another magic URL, so users can just copy/paste something into an email. the other side clicks it, now they are set up to do multi-sig transactions with each other
132 2012-11-01 15:02:13 <TD> s/email/email or chat/
133 2012-11-01 15:02:37 <TD> the downside is obv that if your IP address isn't stable you have to do it each time you want to sign.
134 2012-11-01 15:02:56 <TD> this is assuming you are aiming for the "multiple independent people controlling the same funds" use case
135 2012-11-01 15:03:14 <gavinandresen> TD: mmm. I was thinking that everybody involved would rendezvous with the same STOMP server that would relay messages for them.
136 2012-11-01 15:03:42 <helo> none of this would preclude users from using other messaging systems, like email, right?
137 2012-11-01 15:04:14 <helo> i.e. simple text payloads for copy/paste
138 2012-11-01 15:04:38 <sipa> i don't hardxoding the transport sysyem now... as a first step define a protocol to negotiate multisigs, perhaps with a no-trust coordinarion server first
139 2012-11-01 15:05:04 <TD> the downside of random third party servers is the loss of privacy that results - the server now knows which computers control what funds. it could be a Tor hidden service, i guess. i wonder how hard it is to link in a "libtor" and use it behind the scenes, or something.
140 2012-11-01 15:05:14 <TD> alternatively, satoshi did implement a pubsub system into the core network, didn't he
141 2012-11-01 15:05:18 <TD> i guess we removed it
142 2012-11-01 15:05:33 <sipa> yes, it was unfinished and removed
143 2012-11-01 15:05:34 <gavinandresen> TD: he half-implemented it...
144 2012-11-01 15:05:51 <gavinandresen> ... and didn't leave any comments about how he was going to solve potential spam/DoS problems
145 2012-11-01 15:06:59 <TD> one thing i wondered is if addr messages could contain pubkeys in future. it would have some advantages, like being able to use malicious wifi to connect to the network, if you have a list of nodes obtained from a trusted connection/seed list. it would also help solve the rendezvous problem, assuming you can get a public IP via UPNP
146 2012-11-01 15:07:00 <gavinandresen> sipa: RE: starting with defining a protocol: I like learning by doing, so I'd like to write code tied to a particular transport system along with the protocol spec.
147 2012-11-01 15:07:20 <gavinandresen> sipa: I trust myself enough to make it trivially generalizable to other transport systems....
148 2012-11-01 15:07:31 <TD> then a user could say "here's my nodes address <abcdefgh>" and your own node could find it by checking for a pubkey hash in the addr broadcasts
149 2012-11-01 15:08:24 <sipa> TD: gmaxwell and i have been talking about that
150 2012-11-01 15:08:34 <sipa> it would have large advantages
151 2012-11-01 15:09:00 <gavinandresen> what problem does that solve?
152 2012-11-01 15:09:05 <TD> see above
153 2012-11-01 15:09:22 <TD> 1) allows users to more easily rendezvous without needing a third party server
154 2012-11-01 15:09:40 <gavinandresen> ... if they're online at the same time....
155 2012-11-01 15:09:45 <sipa> you could have the addr uodate messages be signed by the host key, so others cannot maliciousky make the network think you still exist, or claim you have services yiu don't
156 2012-11-01 15:09:48 <TD> 2) allows you to use malicious internet connections to get onto the network, making zero-confirmed transactions more useful for SPV clients in this use case (no sybil)
157 2012-11-01 15:09:55 <jgarzik> sipa: yah, ditto rdponticelli. Let me know if you need test data or whatnot.
158 2012-11-01 15:10:05 <jgarzik> sipa: ultraprune is definitely resending 100% of my wallet transactions
159 2012-11-01 15:10:13 <jgarzik> sipa: and they are old, long confirmed
160 2012-11-01 15:10:16 <sipa> jgarzik: found the problem allready; pull req comig up tonight
161 2012-11-01 15:10:27 <jgarzik> great
162 2012-11-01 15:10:35 <sipa> thanks for noticing :)
163 2012-11-01 15:10:41 <TD> gavinandresen: yes, if they're online at the same time. the user would need to actively confirm the transaction they want to take place anyway, right?
164 2012-11-01 15:10:50 <jgarzik> sipa: It's not testing, until it's real money testing :)
165 2012-11-01 15:11:02 <jgarzik> sipa: I'm testing with my real, live bitcoin wallet under ultraprune
166 2012-11-01 15:11:09 <sipa> gavinandresen: i sent a mail about some attavks some time ago, which also suggested host keys
167 2012-11-01 15:11:11 <TD> gavinandresen: if you want to allow participants to ack asynchronously then you need somewhere to act as a store/forward. the network could do that, with some bounds of course. making the system centralized doesn't magically solve DoS after all
168 2012-11-01 15:11:53 <gavinandresen> TD: I think the common case will be asynchronous transaction confirmation.
169 2012-11-01 15:12:18 <TD> what use cases do you have in mind as first base? multiple people controlling an organizations funds (like the foundation) ?
170 2012-11-01 15:12:37 <gavinandresen> TD: e.g. Lindsay has to pay a big Foundation bill, so she creates and signs a transaction and then forwards it to the board members who can finalize the payment...
171 2012-11-01 15:13:01 <TD> bear in mind, without identity, it's not safe to ACK a transaction you just received randomly without any prior knowledge. Lindsay has to tell you she's going to pay the bill, and tell you what address and the amount she's going to pay
172 2012-11-01 15:13:12 <TD> if she's telling you that, it's probably via email or some other communication system that lets you attach files.
173 2012-11-01 15:13:23 <sipa> imho, you want somethig e-mail like for that, where everyone has some server acting as a messagebox for the user
174 2012-11-01 15:13:40 <gavinandresen> TD: sure, but we've been training users not to open email attachments for years. And email is insecure....
175 2012-11-01 15:13:41 <sipa> you can run your own, or use a third party
176 2012-11-01 15:13:45 <TD> email is not insecure
177 2012-11-01 15:13:51 <TD> (any more)
178 2012-11-01 15:14:11 <TD> eg, if you send a message from gmail to hotmail, it's encrypted from you to gmail, from gmail to hotmail (SMTP-TLS) and from hotmail back down to the user
179 2012-11-01 15:14:24 <TD> also, the message itself will be signed on your behalf by the service providers using DKIM
180 2012-11-01 15:14:38 <TD> so the only way to break this is to compromise either your computer, or the email providers, or SSL.
181 2012-11-01 15:15:03 <TD> anyway, for the foundation, maybe you're using some web forum software that is SSLd for all users.
182 2012-11-01 15:16:52 <gavinandresen> TD: re: identity: I agree that is a huge piece of the puzzle. If I receive a payment request, I need to make sure it is going to the person/organization that I expect and not Joe Random Hacker
183 2012-11-01 15:17:43 <gavinandresen> That is a different piece from the "how are payment requests stored and forwarded", though.
184 2012-11-01 15:18:10 <gavinandresen> (and I still think email attachments are the wrong way to go, I don't want to get them in the context of an email, I want them to pop up in whatever Bitcoin client I'm using)
185 2012-11-01 15:19:13 <TD> well, then you're going to end up reinventing a lot of wheels. email is old but it's a remarkably good store and forward mechanism. everyone has one, they check their inboxes regularly, and the email system does *not* lose mail because the specs are so thorough about MTAs checkpointing to disk and trying redelivery in case of failure
186 2012-11-01 15:19:32 <sipa> gavinandresen: sure, doesn't orevent you from using an architecture that uses message boxes to deal with the problem of offline clients
187 2012-11-01 15:19:43 <TD> and of course, if somebody doesn't want to use email, nothing stops them getting the files to you some other way.
188 2012-11-01 15:20:12 <TD> for example, a separate program implementing a separate P2P network that moves payment requests around via some as-yet-undetermined mechanism and then sends them to bitcoin-qt via an RPC
189 2012-11-01 15:20:15 <gavinandresen> Right, using a separate 'gavin+btc@gmail.com' mailbox for bitcoin messaging might be the right way to go.
190 2012-11-01 15:20:56 <gavinandresen> ... as long as I don't have to copy and paste anything or click on attachments.
191 2012-11-01 15:21:26 <Luke-Jr> TD: sorry, I disagree email doesn't lose mail. Absurdly aggressive "spam filters" are common.
192 2012-11-01 15:21:27 <TD> if you start by defining some files and file-based protocols that do require using attachments (in email or forums or share dropbox folders or whatever) then you can always layer on some other mechanism later
193 2012-11-01 15:22:05 <TD> Luke-Jr: spam filters don't lose mail, ever (unless they are amateur and buggy). either they store the mail and give it to you in some kind of spam folder, or they bounce it, in which case the sender will get a bounce message and know the message didn't get through
194 2012-11-01 15:22:15 <TD> spam filters that silently eat mail don't exist at any competent mail provider
195 2012-11-01 15:22:27 <Luke-Jr> TD: call them amateur or buggy or whatever you want, those exist :P
196 2012-11-01 15:22:27 <TD> now, you might not SEE the mail, if you don't check your spamfolder
197 2012-11-01 15:22:29 <TD> but that's different
198 2012-11-01 15:22:54 <TD> alright. in the universe of mail systems that follow the RFCs and are run by people who know what they're doing, mail does not get silently eaten. and any other system you come up with can suffer buggy/bad implementations
199 2012-11-01 15:23:10 <Luke-Jr> TD: also, getting a bit off-topic, wizkid057 seems to be having lost email issues with Gmail :P
200 2012-11-01 15:23:18 <TD> i haven't heard about that
201 2012-11-01 15:23:32 <TD> who is wizkid057? if he has actual evidence of lost messages i can investigate that.
202 2012-11-01 15:24:14 <TD> anyway, gavinandresen - i don't personally share your aversion to email attachments. so if you start with moving the messages around with files, at least i can start using it right away, whilst you go ahead and come up with some alternative scheme
203 2012-11-01 15:24:17 <TD> we both win :)
204 2012-11-01 15:25:13 <gavinandresen> TD: ok. You won't like the file format I'm going to choose,though....
205 2012-11-01 15:25:21 <Luke-Jr> ACTION wonders how a Bitcoin client would get at the email box
206 2012-11-01 15:25:23 <TD> i assume you're going to encode binary data as text
207 2012-11-01 15:25:28 <gavinandresen> yeah, JSON
208 2012-11-01 15:25:32 <TD> why?
209 2012-11-01 15:25:44 <gavinandresen> everything speaks JSON these days
210 2012-11-01 15:25:58 <TD> software that isn't written yet can speak whatever it wants
211 2012-11-01 15:26:09 <gavinandresen> minimal dependencies, and efficiency doesn't matter so there's no real reason to go for a binary encoding
212 2012-11-01 15:26:34 <TD> minimal dependencies for you - not for me. bitcoinj, for instance, does not depend on a json parser today :)
213 2012-11-01 15:26:50 <TD> protobufs aren't just about efficiency, btw. there are many other advantages. like an actual schema language.
214 2012-11-01 15:27:02 <TD> also, it generates all the wrapper code for you, so there's no ugly code to deal with JSONObjects
215 2012-11-01 15:27:08 <TD> you just write code like:
216 2012-11-01 15:27:10 <Luke-Jr> protobufs are more work than JSON???
217 2012-11-01 15:27:14 <TD> PaymentRequestMessage req;
218 2012-11-01 15:27:20 <TD> req.set_output_script(script_bytes);
219 2012-11-01 15:27:27 <TD> req.set_some_bool(true);
220 2012-11-01 15:27:28 <jgarzik> TD: that's overselling, a bit :) You still have plenty of validation to do, under protobufs
221 2012-11-01 15:27:36 <jgarzik> I know; I wrote lots of code for both.
222 2012-11-01 15:27:57 <TD> it takes care of type checking and parsing into actual language-specific data structures for you, which you have to do by hand for JSON
223 2012-11-01 15:28:30 <gavinandresen> In my experience, simple text encodings win standards wars.
224 2012-11-01 15:28:40 <jgarzik> protobufs does low-level type checking. JSON libs do low level type checking. protobufs supports more types, so it does a bit more.
225 2012-11-01 15:28:48 <jgarzik> in BOTH cases, you still must validate the contents of each field.
226 2012-11-01 15:28:48 <TD> well, except for TCP, IP, JPEG, PNG, etc :)
227 2012-11-01 15:29:06 <Luke-Jr> gavinandresen: I hope there's no standards war here :/
228 2012-11-01 15:29:36 <jgarzik> so the perspective of the typical program, for JSON or protobufs, remains: (1) lib reads raw data into structured form, (2) program verifies data inside structured form
229 2012-11-01 15:29:36 <TD> jgarzik: specifically you can encode binary data with protobufs, and you get them back out of the generated objects as an array of bytes.
230 2012-11-01 15:29:57 <TD> JSON requires you to encode binary in some basically random format, there's no real standardization over it either. so that's even more work
231 2012-11-01 15:30:12 <jgarzik> a little bit more work, yes
232 2012-11-01 15:30:35 <TD> jgarzik: no, with JSON, you read a bunch of generic JSON objects. then you have to check all the data you expect exists. protobufs does that for you. then you have to do type conversions for anything that isn't actually a string.
233 2012-11-01 15:30:46 <TD> jgarzik: then you insert the data into some language specific structure
234 2012-11-01 15:30:50 <TD> which you also have to write by hand
235 2012-11-01 15:31:08 <jgarzik> yes (in both cases)
236 2012-11-01 15:31:13 <TD> basically, when i look at JSON, i see a format with no advantages over anything. text isn't a help when the data you're serializing is actually binary.
237 2012-11-01 15:31:14 <Luke-Jr> ACTION notes Bitcoin already has a standard binary serialization protocol
238 2012-11-01 15:31:50 <jgarzik> It is certainly true that bitcoin text formats are really TWO formats... JSON, then binary bitcoin
239 2012-11-01 15:32:01 <jgarzik> protobufs would be... protobufs + binary bitcoin
240 2012-11-01 15:32:22 <jgarzik> because generally you are transiting a bitcoin structure under all this
241 2012-11-01 15:32:27 <TD> also, JSON can't represent ints, iirc?
242 2012-11-01 15:33:00 <wumpus> of course it can, you just interpret numbers without a point as ints
243 2012-11-01 15:33:13 <wumpus> that's the JS way isn't it *ducks*
244 2012-11-01 15:33:21 <sipa> haha
245 2012-11-01 15:33:31 <TD> great
246 2012-11-01 15:33:55 <TD> also, the only schema language it has actually represents the schema as JSON itself, making the schemas almost unreadable
247 2012-11-01 15:34:13 <wumpus> just like we're now handling the json floats basically as 'decimal'
248 2012-11-01 15:34:30 <jeremias> how resource-insentive operation is sweeping private key?
249 2012-11-01 15:34:47 <sipa> jeremias: right now: rescan entire history
250 2012-11-01 15:34:57 <wumpus> hey, it's just text, you can interpret it any way you want :-)
251 2012-11-01 15:35:01 <sipa> later maybe with an address-to-txid index, relatively fast
252 2012-11-01 15:35:02 <gmaxwell> ugh.
253 2012-11-01 15:35:02 <jeremias> seems to take ages to import one key
254 2012-11-01 15:35:53 <Luke-Jr> perhaps before arguing over the transport format, it would make sense to figure out what fields and data are needed exactly? :P
255 2012-11-01 15:35:56 <gmaxwell> I'd be strongly opposed to JSON _and_ protobuts for anything where normative behavior is required. (e.g. as a serialization for transactions in the blockchain, were we designing bitcoin from scratch.).. but since it's not required here, whatever people want to implement is probably fine.
256 2012-11-01 15:36:02 <gavinandresen> TD: I knew you wouldn't like it.... not sure if there is any point to arguing,though. I don't really care about protocol buffers versus JSON, except that using protocol buffers means pulling in another dependency for bitcoind/Bitcoin-Qt and pretty much any other project besides yours
257 2012-11-01 15:36:35 <wumpus> does protobuf require another dependency? or is the generated code self-sufficent?
258 2012-11-01 15:36:49 <sipa> wumpus: generated code is enough
259 2012-11-01 15:37:05 <sipa> though you want to keep the ability to update the format with new optional fields
260 2012-11-01 15:37:09 <TD> depends on the language. for java there's a lib, but you just add it in maven and you're done. java has a kind of apt-get/gitian cross thing
261 2012-11-01 15:37:32 <BlueMatt> TD: merges look good to me, nice work
262 2012-11-01 15:37:41 <sipa> gmaxwell: well, bitcoin transactions need to be encoded as bitcoin serialized transactions... that can be embedded into something else in any way though
263 2012-11-01 15:38:14 <sipa> any software dealing with bitcoin transaction will need to be able to calculate its hash, and that already implies being able to serialize it anyway
264 2012-11-01 15:38:25 <wumpus> anyway, I'm in favor of any serialization scheme that isn't hand-rolled
265 2012-11-01 15:38:28 <gmaxwell> sipa: sure sure. I just meant for normative usage pulling in a mystery meat encoding system that might change out from under you would be very ill advised (as we've expirenced with the openssl DER stuff!).
266 2012-11-01 15:38:41 <TD> protobufs have not changed their binary format for >10 years
267 2012-11-01 15:38:43 <jgarzik> hrm?
268 2012-11-01 15:38:50 <TD> it's basically as stable as TCP, as far as i'm concerned.
269 2012-11-01 15:38:56 <jgarzik> AFAIK, all protobufs require an additional lib
270 2012-11-01 15:39:00 <gmaxwell> For non normative stuff people are going to do moronic things but there nothing that can be done about that.
271 2012-11-01 15:39:00 <jgarzik> C, C++, python, perl, ...
272 2012-11-01 15:39:56 <wumpus> yeah for python/perl it's no big deal to require an additional lib
273 2012-11-01 15:40:07 <wumpus> for c++ it'd be sad
274 2012-11-01 15:40:13 <gmaxwell> TD: and it's not possible to make varrient encodings which read differently under different implementations? and not possible to make varrients which read the same but have a different serialization?
275 2012-11-01 15:40:19 <Luke-Jr> wumpus: does that include using the serialization Bitcoin itself uses? :P
276 2012-11-01 15:40:27 <TD> https://developers.google.com/protocol-buffers/docs/encoding
277 2012-11-01 15:40:33 <wumpus> Luke-Jr: yes, it does, unfortunately
278 2012-11-01 15:40:39 <wumpus> Luke-Jr: but we cannot change that can we :P
279 2012-11-01 15:40:41 <sipa> gmaxwell: not sure what you mean?
280 2012-11-01 15:40:51 <Luke-Jr> wumpus: I mean, using that same serialization for new stuff like this
281 2012-11-01 15:41:00 <TD> so, no, i don't think it's possible. i never checked in depth though.
282 2012-11-01 15:41:06 <wumpus> Luke-Jr: the block chain protocol is "sacred", but there's no need to use it anywhere outside of that
283 2012-11-01 15:41:25 <Luke-Jr> wumpus: except to avoid more serialization code???
284 2012-11-01 15:41:47 <wumpus> no
285 2012-11-01 15:42:00 <TD> well, bitcoin serialization achieves its goal of being simple and auditable (mostly), but it fails at almost everything else
286 2012-11-01 15:42:04 <wumpus> imo the block chain serialization code should be isolated and only used for the block chain, because it may never change etc
287 2012-11-01 15:42:05 <TD> in particular it's hard to extend
288 2012-11-01 15:42:21 <TD> i guess even with JSON you can just add new members and properties at will
289 2012-11-01 15:42:21 <wumpus> it should not be extended except when the block chain requires it etc
290 2012-11-01 15:42:36 <gavinandresen> TD: yes, I wish Satoshi had used protobufs for the on-the-wire serialization
291 2012-11-01 15:42:38 <wumpus> it's just a frozen protocol for one purpose
292 2012-11-01 15:43:17 <TD> gmaxwell: field ordering is free in protobufs but the reference implementations all store them in order, so parsing code can be optimized. however you're allowed to re-arrange them if you want.
293 2012-11-01 15:43:21 <wumpus> yes with json you can stuff into it what you want (this can be an disadvantage as well :p)
294 2012-11-01 15:43:24 <gavinandresen> TD: I'm still waiting for somebody to broadcast blocks/transactions across a different p2p network... that'd be great for robustness
295 2012-11-01 15:43:30 <TD> hmm
296 2012-11-01 15:43:32 <TD> interesting idea
297 2012-11-01 15:43:47 <gmaxwell> gavinandresen: done.
298 2012-11-01 15:43:50 <gmaxwell> :P
299 2012-11-01 15:43:57 <gmaxwell> gavinandresen: p2pool now fits that criteria.
300 2012-11-01 15:43:59 <wumpus> what p2p networks support broadcasting?
301 2012-11-01 15:44:09 <gavinandresen> gmaxwell: cool. Did they use protobufs?
302 2012-11-01 15:44:12 <TD> the dependencies issue doesn't bother me a whole lot. it's added once, and you're done. in future there may well be many protocols with complex schemas
303 2012-11-01 15:44:18 <TD> (like the payment protocol)
304 2012-11-01 15:44:24 <gmaxwell> No.
305 2012-11-01 15:44:40 <TD> i think embedding binary data like signatures, certificates, keys, hashes, even images (for logos and things) would be quite common
306 2012-11-01 15:44:47 <sipa> p2pool pretty much uses satoshi-serialization, iirc?
307 2012-11-01 15:45:45 <gmaxwell> TD: so looking at the example C code for protobufs, I see that you'd get multiple encodings possible for anything using strings, if applications use them like the C example code does.
308 2012-11-01 15:45:50 <TD> there is i guess one advantage of JSON over protobufs, which is that protobuf message fields are identified by tag number. so if you want to allow random third parties to add extensions without any kind of co-ordination (no process to reserve number ranges), you need to add something like repeated ExtensionMessage extensions = 10; where ExtensionMessage has string key/bytes value
309 2012-11-01 15:46:01 <gmaxwell> Because Foo, Foo\\0, and Foo\\0
310 2012-11-01 15:46:26 <gmaxwell> \\0 are different messages with unique serializations, but the C example code would treat them identically.
311 2012-11-01 15:46:38 <TD> i guess that's true. strings are defined to be UTF-8 and there are multiple ways to encode that. OTOH, satoshis format doesn't even define string encodings at all, does it
312 2012-11-01 15:46:53 <gmaxwell> Well I wasn't even talking about UTF-8, but there too.
313 2012-11-01 15:47:00 <wumpus> because we don't want strings in the block chain in the first place :P
314 2012-11-01 15:47:08 <Luke-Jr> TD: huh? UTF-8 *is* an encoding
315 2012-11-01 15:47:10 <TD> strings are used in a few places
316 2012-11-01 15:47:14 <wumpus> you can encode them as a word of 0 bits
317 2012-11-01 15:47:16 <gmaxwell> TD: sure sure, I'm just pointing out that adopting some code base has implications with hidden implicit behavior that you must be careful with.
318 2012-11-01 15:47:23 <TD> Luke-Jr: right but there are multiple ways to encode the same string to different byte sequences
319 2012-11-01 15:47:35 <Luke-Jr> TD: not with UTF-8, no
320 2012-11-01 15:47:39 <jgarzik> IIRC, in protobufs... "bytes" != "string"
321 2012-11-01 15:47:40 <TD> yeah. i'm not sure i'd use protobufs where everyone must arrive at exactly the same byte-for-byte serialization with no leeway allowed.
322 2012-11-01 15:47:44 <jgarzik> we would want "bytes"
323 2012-11-01 15:47:48 <jgarzik> TD: ^^
324 2012-11-01 15:47:49 <wumpus> jgarzik: which is exactly how it should be
325 2012-11-01 15:48:34 <jgarzik> protobufs would define a byte-for-byte field as "required bytes my_field"
326 2012-11-01 15:48:35 <gmaxwell> TD: yea, thats all I'm saying. I don't see anything wrong with it for IPC for other applications where that isn't a consideration.
327 2012-11-01 15:48:39 <TD> Luke-Jr: unfortunately yes. you can encode some characters with one or more code points. the text ends up looking and being semantically the smae
328 2012-11-01 15:48:59 <Luke-Jr> TD: well, those are different characters!
329 2012-11-01 15:49:02 <TD> ah ha :)
330 2012-11-01 15:49:09 <wumpus> yes like the ohm sign
331 2012-11-01 15:49:14 <TD> not to people or IMEs they aren't :)
332 2012-11-01 15:49:19 <wumpus> which is simply an omega
333 2012-11-01 15:49:22 <jgarzik> ACTION used "bytes" in pagedb's protobufs: https://github.com/jgarzik/pagedb/blob/master/PDcodec.proto
334 2012-11-01 15:49:25 <TD> ACTION has spent a while investigating a bug triggered by this
335 2012-11-01 15:50:02 <jgarzik> ACTION copied TD's string in a few places for smartcoin (formerly pybond): https://github.com/jgarzik/smartcoin/blob/master/codec.proto
336 2012-11-01 15:50:06 <TD> anyway, whatever. if we go with JSON i'm sure we can find some sane schema languages and compilers for it
337 2012-11-01 15:50:15 <TD> iirc there are json2protobuf adapters out there also
338 2012-11-01 15:50:44 <Luke-Jr> btw, one thing to note: protobufs require build-time dependencies for C++, you can't pre-generate the .cpp files and ship those
339 2012-11-01 15:50:57 <wumpus> Luke-Jr: crap
340 2012-11-01 15:51:00 <TD> you can actually. i do this in bitcoinj
341 2012-11-01 15:51:11 <TD> only people who want to edit the schema need the protobuf compiler
342 2012-11-01 15:51:12 <wumpus> then again, spritjson is also a huge beast
343 2012-11-01 15:51:15 <TD> you can just check in the generated code
344 2012-11-01 15:51:17 <Luke-Jr> TD: the .cpp files aren't forward or backward compatible with different versions
345 2012-11-01 15:51:37 <TD> right, you have to specify the version of the library to depend on.
346 2012-11-01 15:51:45 <TD> that said, new versions aren't very common
347 2012-11-01 15:52:00 <TD> the last release was april 2011
348 2012-11-01 15:52:18 <jgarzik> like autoconf/configure, you may pre-generate my_protobufs.cpp and ship that, or require the protobufs compiler at build time.
349 2012-11-01 15:52:19 <Luke-Jr> that's < 2 years ago :P
350 2012-11-01 15:52:26 <jgarzik> regardless, the protobufs lib is required for all cases.
351 2012-11-01 15:52:29 <jgarzik> you cannot skip a new dep.
352 2012-11-01 15:52:48 <gavinandresen> Lots of people here right now... would this be a good time/day of the week to have regular, scheduled developer chats ?
353 2012-11-01 15:52:49 <Luke-Jr> jgarzik: except if you pre-generate my_protobufs.cpp, it won't work if someone has a different version of protobuf installed
354 2012-11-01 15:53:03 <TD> actually there's a new version coming up, haha :) http://protobuf.googlecode.com/svn/trunk/CHANGES.txt
355 2012-11-01 15:53:08 <jgarzik> Luke-Jr: mostly true
356 2012-11-01 15:53:17 <wumpus> if it's not too bug you could embed it into the git repo, I mean that would aid stability as well...
357 2012-11-01 15:53:19 <jgarzik> Luke-Jr: but as TD indicates, the protobufs libs do not change much anymore
358 2012-11-01 15:53:23 <Luke-Jr> gavinandresen: can't hurt to try it
359 2012-11-01 15:53:25 <wumpus> gavinandresen: fine with me
360 2012-11-01 15:53:46 <TD> * Comments in proto files are now collected and put into generated code as
361 2012-11-01 15:53:47 <TD> nice
362 2012-11-01 15:53:50 <TD> looking forward to that one
363 2012-11-01 15:54:54 <jgarzik> TD: the only major annoyance I found with protobufs is an annoyingly odd way to add objects to a "repeated" (a list/array).... you request a new child object from the parent object, then fill that in
364 2012-11-01 15:55:00 <jgarzik> rather than normal create-and-add pattern
365 2012-11-01 15:56:09 <sipa> jgarzik: efficiency++ !
366 2012-11-01 15:56:21 <TD> jgarzik: see the first item in the C++ section here: http://protobuf.googlecode.com/svn/trunk/CHANGES.txt
367 2012-11-01 15:56:21 <wumpus> well inplace writing is quite common for serialization formats
368 2012-11-01 15:56:37 <TD> jgarzik: apparently the next version provides an API that lets you give messages stuff to take ownership of
369 2012-11-01 15:56:50 <TD> that said, i never found the request-and-modify pattern particularly an issue
370 2012-11-01 15:57:35 <jgarzik> TD: it is useful to store/generate child messages separately from parent creation
371 2012-11-01 15:57:47 <jgarzik> it leads to a lot of boring copying
372 2012-11-01 15:58:20 <jgarzik> message RootEnt {
373 2012-11-01 15:58:21 <jgarzik> required bytes key = 1;
374 2012-11-01 15:58:21 <jgarzik> required uint64 file_id = 2;
375 2012-11-01 15:58:23 <jgarzik> }
376 2012-11-01 15:59:02 <jgarzik> TD: in pagedb, the lifetime of "RootEnt" is very long -- entire program runtime. RootIdx's lifetime is very short: the length of a file open-write-close.
377 2012-11-01 15:59:25 <TD> oh, i see
378 2012-11-01 15:59:39 <jgarzik> TD: with such a lifetime mismatch, protobufs dictates (until 2.5.0?), request-and-modify requires copying all those RootEnt we already store.
379 2012-11-01 15:59:43 <TD> i'm not sure the new API would help you then, as the new API takes ownership
380 2012-11-01 16:00:49 <jgarzik> ACTION should look into the API... surely there is a CopyMessage helper that would at least eliminate the burden of field-by-field copying
381 2012-11-01 16:01:32 <TD> oh
382 2012-11-01 16:01:35 <TD> you can do something like this
383 2012-11-01 16:01:47 <TD> RootEnt *entry = idx.add_entry();
384 2012-11-01 16:01:55 <TD> entry->CopyFrom(some_ent);
385 2012-11-01 16:02:03 <jgarzik> had the same issue with smartcoin (aka pybond), with message Address: https://github.com/jgarzik/smartcoin/blob/master/codec.proto
386 2012-11-01 16:02:05 <TD> or entry->MergeFrom
387 2012-11-01 16:02:10 <jgarzik> Address is long-lived, but Addresses is very short term
388 2012-11-01 16:02:25 <jgarzik> TD: yep, something like that would help
389 2012-11-01 16:03:23 <jgarzik> TD: You can probably see some of your own code in that link :) smartcoin uses protobufs for all message encoding, including the "bond for sale" stuff, based on your distributed bonds post
390 2012-11-01 16:03:44 <TD> https://developers.google.com/protocol-buffers/docs/reference/cpp/google.protobuf.message#Message.CopyFrom.details
391 2012-11-01 16:03:49 <TD> yes, i saw that :)
392 2012-11-01 16:03:58 <TD> complete with bug fixes to my schema....
393 2012-11-01 16:04:38 <jgarzik> TD: yeah you should update your post ;p
394 2012-11-01 16:04:46 <TD> i want to move it to the wiki
395 2012-11-01 16:04:52 <sipa> jgarzik: please test #1971
396 2012-11-01 16:04:53 <TD> oh yeah. one final advantage for protobufs over json
397 2012-11-01 16:04:55 <gmaxwell> gavinandresen: basically p2pool now does the latency-reductions for block transmitting we talked about when people were worrying about block propagation delays. Nodes also pretransmit the txn they are mining. It does this so that p2pool _shares_ can also include the hash tree, so that shares don't take less bandwidth than blocks. The idea being that p2pool was previously giving too much reward to overly slow miners that were causing orphan bl
398 2012-11-01 16:05:26 <TD> if you read a protobuf and it contains unknown fields (like, data from a future version), and then write it out again, the system automatically keeps the unknown data and ensures it gets written out too
399 2012-11-01 16:05:33 <TD> so you don't need to do any code to prevent downgrade bugs
400 2012-11-01 16:05:55 <jgarzik> sipa: will do
401 2012-11-01 16:06:10 <jgarzik> sipa: immediate nit: your commit message should have included the discussion included in the pull req
402 2012-11-01 16:06:10 <sipa> TD: no need to convince me about the advantages of protobuf, but i'm not convinced this is one: in json you typically don't touch things you don't know either
403 2012-11-01 16:06:15 <TD> because you're going to convert JSON data into some custom hand written class, you'd have to do that by hand with JSON (or just accept that load/save sequences will destroy data, no good for a relay protocol)
404 2012-11-01 16:06:22 <sipa> jgarzik: ok
405 2012-11-01 16:06:24 <jgarzik> sipa: github will pull that from the commit message into the pull req automatically
406 2012-11-01 16:06:31 <jgarzik> (in the future)
407 2012-11-01 16:07:38 <sipa> jgarzik: no, once the pull req is sent, its description will not be changed anymore
408 2012-11-01 16:07:44 <sipa> unless you do so manually
409 2012-11-01 16:07:48 <sipa> fixed, by the way
410 2012-11-01 16:08:32 <jgarzik> sipa: yes, hence "in the future" addendum ;p
411 2012-11-01 16:08:38 <jgarzik> sipa: thanks for fixing
412 2012-11-01 16:08:47 <sipa> not sure what you mean, nothing changes in the future
413 2012-11-01 16:09:08 <sipa> or do you just mean that a pullreq by default gets its message from the commit message? i know that
414 2012-11-01 16:09:30 <sipa> i only realized that some explanation may have been usable when the commit was already made
415 2012-11-01 16:09:42 <sipa> s/usable/useful/
416 2012-11-01 16:14:29 <jgarzik> gavinandresen: for me, the preference is >= noon EST
417 2012-11-01 16:15:03 <jgarzik> gavinandresen: but I think my availability is less important than you / sipa / gmaxwell
418 2012-11-01 16:15:37 <gavinandresen> jgarzik: Lets try 1pm EST (are we EST right now or EDT?) Thursdays
419 2012-11-01 16:15:48 <jgarzik> Eastern Time
420 2012-11-01 16:15:49 <jgarzik> ;p
421 2012-11-01 16:15:59 <jgarzik> I never know EST or EDT either ;p
422 2012-11-01 16:16:04 <gavinandresen> "15 minutes ago every thursday"
423 2012-11-01 16:16:21 <TD> i have to go now, it's 6pm and my gf is waiting
424 2012-11-01 16:16:27 <gavinandresen> whatever that is un UCT/GMT/pick-your-favorite-time-standard
425 2012-11-01 16:16:27 <TD> but let me know what happens :)
426 2012-11-01 16:16:30 <jgarzik> gavinandresen: ACK
427 2012-11-01 16:16:44 <jgarzik> gavinandresen: but I'm OK with another choice, if you/others prefer
428 2012-11-01 16:17:54 <gavinandresen> TD: we'll let you know, I added rewriting all of the bitcoin serialization to use protobufs to my TODO list. (KIDDING!)
429 2012-11-01 16:19:16 <sipa> gavinandresen: idea for april 1st: "We're going to switch all bitcoin protocol serializations to DER"
430 2012-11-01 16:19:27 <gavinandresen> sipa: ooh, good idea!
431 2012-11-01 16:19:45 <TD> haha
432 2012-11-01 16:20:02 <gavinandresen> I vote for XML, though.
433 2012-11-01 16:20:11 <gavinandresen> More buzzwordy.
434 2012-11-01 16:20:26 <Luke-Jr> XML++
435 2012-11-01 16:20:29 <jgarzik> I remember when XML was so awesome, so much better than what had come before.
436 2012-11-01 16:20:31 <jgarzik> Those were the days.
437 2012-11-01 16:20:39 <sipa> ooh yes, XML!
438 2012-11-01 16:20:45 <Luke-Jr> let's use <byte digit="01000110"/> for binary
439 2012-11-01 16:20:46 <daybyter> I like xml...
440 2012-11-01 16:20:59 <gavinandresen> We could have the link to the DTD be a rickroll
441 2012-11-01 16:21:18 <midnightmagic> +1 rickroll idea
442 2012-11-01 16:21:22 <sipa> ACK!
443 2012-11-01 16:21:40 <TD> yaml?
444 2012-11-01 16:21:43 <TD> i have no idea what that is
445 2012-11-01 16:21:53 <sipa> gitian uses yaml
446 2012-11-01 16:22:08 <sipa> it's too compact and readable to be a useful standard
447 2012-11-01 16:22:12 <Luke-Jr> YAML is a bigger JSON\\
448 2012-11-01 16:22:31 <gavinandresen> I thought it was a smaller JSON... anyway, they're almost the same.
449 2012-11-01 16:22:53 <sipa> but but... it has relevant whitespace!
450 2012-11-01 16:23:11 <Luke-Jr> sipa: did you just suggest Whitespace?
451 2012-11-01 16:24:25 <sipa> whitespace is cool!
452 2012-11-01 16:24:41 <sipa> and i met the guy who invented it once at a conference
453 2012-11-01 16:25:09 <jgarzik> some days, I love my bitcoin email inbox
454 2012-11-01 16:25:16 <jgarzik> "Road map for the transition to a single world currency" <delete unread>
455 2012-11-01 16:26:13 <djoot> Luke-Jr: your ljrbot quit
456 2012-11-01 16:27:56 <wumpus> jgarzik: I've also not opened it, it has some strange attachments
457 2012-11-01 16:38:58 <gmaxwell> jgarzik: it was in russian anyways.
458 2012-11-01 16:41:32 <midnightmagic> google's weird serialized data structures have been "tested" extensively and they want everyone to use them.
459 2012-11-01 17:08:41 <phantomcircuit> jgarzik, was step one "get everybody arrested?"
460 2012-11-01 17:08:48 <phantomcircuit> move the ? one over
461 2012-11-01 17:09:41 <gmaxwell> phantomcircuit: it was in russian, do you read russian?
462 2012-11-01 17:10:01 <phantomcircuit> nope
463 2012-11-01 17:10:31 <phantomcircuit> most of the proposals i've seen for bitcoins success would invariably result in anybody involved in the plan getting arrested
464 2012-11-01 17:10:43 <phantomcircuit> genius at work
465 2012-11-01 17:10:49 <jgarzik> for each tx in block
466 2012-11-01 17:10:49 <jgarzik> wallet_key_set = set of pubkeys + pubkey hashes
467 2012-11-01 17:10:59 <jgarzik> is this the correct method for finding payments-to-me ?
468 2012-11-01 17:12:12 <phantomcircuit> jgarzik, for standard transactions yes
469 2012-11-01 17:12:22 <gmaxwell> jgarzik: that is a functional method. Though perhaps you should write your SPV software assuming the bloom stuff? (though you still have to filter)
470 2012-11-01 17:12:30 <jgarzik> yes, that's just searching known standard transactions
471 2012-11-01 17:12:36 <phantomcircuit> you could optimize that by having an index of unspent transaction outputs which would significantly improve performance though
472 2012-11-01 17:13:23 <jgarzik> gmaxwell: currently set_intersection() is testing two hashtables, but yes, a bloom filter is definitely applicable, and has been considered
473 2012-11-01 17:14:14 <jgarzik> gmaxwell: given that network is separated from $everything_else in picocoin, one improvement considered was having the main process pass a bloom filter over the OS pipe(2) to the network process, and the network process (which manages blockchain db) returns matching TX's
474 2012-11-01 17:14:21 <jgarzik> then, the main process filters from there
475 2012-11-01 17:15:10 <jgarzik> bloom filters dictate an exact match be performed, so exact match must happen regardless
476 2012-11-01 17:16:16 <jgarzik> phantomcircuit: this is an SPV client. Trying to keep just the block headers + merkle tx's, if at all possible.
477 2012-11-01 17:16:36 <jgarzik> phantomcircuit: maintaining the UTXO set would require significantly more resources
478 2012-11-01 17:16:56 <phantomcircuit> jgarzik, in that case yes that's the correct algorithm for identifying payments-to-me
479 2012-11-01 17:17:10 <phantomcircuit> or more accurately transaction outputs I can spend
480 2012-11-01 17:17:15 <jgarzik> thanks
481 2012-11-01 17:17:35 <jgarzik> yeah, just the basic, well known transactions. complicated transactions you will have to manually specify as "mine" somehow
482 2012-11-01 17:17:53 <phantomcircuit> i dont know if anybody here really cares but gribble seems to be having some troubles with gpg auth
483 2012-11-01 17:18:05 <phantomcircuit> i wouldn't rely on it being correct for now
484 2012-11-01 17:18:11 <jgarzik> phantomcircuit: poke nanotube
485 2012-11-01 17:18:20 <phantomcircuit> i tried he doesn't seem to be around
486 2012-11-01 17:19:41 <gmaxwell> jgarzik: right, they're doing some normalization first so "exact" isn't quite exact.
487 2012-11-01 17:21:28 <daybyter> bitparking changed it's trade format?
488 2012-11-01 17:23:09 <daybyter> no type field in the trade anymore?
489 2012-11-01 17:38:59 <JDuke128> hi , this must be stopped or bitcoin will die : https://www.privateinternetaccess.com/blog/2012/03/bitcoin-war-the-first-real-threat-to-bitcoin/
490 2012-11-01 17:39:48 <JDuke128> someone captured %15 of bitcoin network
491 2012-11-01 17:39:51 <JDuke128> or more
492 2012-11-01 17:42:06 <copumpkin> o.O
493 2012-11-01 17:42:17 <copumpkin> JDuke128: that sounds a little sensationalist
494 2012-11-01 17:42:38 <gmaxwell> JDuke128: that is complete bullshit written by clueless people who don't even read the docs.
495 2012-11-01 17:43:01 <copumpkin> bitcoin won't "die"
496 2012-11-01 17:43:06 <JDuke128> hope that
497 2012-11-01 17:43:19 <copumpkin> JDuke128: no need to hope
498 2012-11-01 17:43:24 <copumpkin> at least not for this reason
499 2012-11-01 17:43:32 <copumpkin> there might be other issues that will emerge, but this isn't one of them
500 2012-11-01 17:44:00 <JDuke128> if many fake servers will join on p2p network to allow all transactions ?
501 2012-11-01 17:45:58 <gmaxwell> JDuke128: are you the author of that blog message?
502 2012-11-01 17:46:05 <JDuke128> no
503 2012-11-01 17:46:14 <JDuke128> i m reader
504 2012-11-01 17:46:21 <gmaxwell> JDuke128: Okay, I've emailed and asked them to withdraw it.
505 2012-11-01 17:48:37 <JDuke128> i've installed bitcoind on my server , i can connect it from my client
506 2012-11-01 17:48:46 <JDuke128> i got some accepts from client to server
507 2012-11-01 17:48:53 <JDuke128> but i still got 0 bit coin on server
508 2012-11-01 17:48:54 <gmaxwell> JDuke128: There is no way to reliably identify the origin of blocks. What blockchain.info does is reports the first peer to relay a block to them, and they only talk to a small number of total nodes. This is a very lossy metric for many reasons.
509 2012-11-01 17:48:55 <JDuke128> why ?
510 2012-11-01 17:49:28 <helo> no good reason
511 2012-11-01 17:49:45 <helo> pretty much meaningless afact
512 2012-11-01 17:49:53 <gmaxwell> JDuke128: Huh. I'm not clear what you're talking about.
513 2012-11-01 17:50:27 <gmaxwell> JDuke128: are you trying to solo mine? Unless you have dozens and dozens of gpus you shouldn't expect to find a block on your own quickly.
514 2012-11-01 17:50:28 <JDuke128> i ve run this command on my server : "bitcoind -daemon"
515 2012-11-01 17:51:13 <sipa> JDuke128: has it synchronized with the block chain yet?
516 2012-11-01 17:51:31 <JDuke128> bash DiabloMiner-OSX.sh -u xxx -p xxx -o xxx.xxx.com -r 8332 -g 5 -w 256
517 2012-11-01 17:51:35 <JDuke128> i did this on client
518 2012-11-01 17:51:42 <sipa> so you are solo mining?
519 2012-11-01 17:51:51 <JDuke128> its solo mining ?
520 2012-11-01 17:51:56 <JDuke128> i ve 1 server and 1 pc
521 2012-11-01 17:52:00 <JDuke128> my pc has gpus
522 2012-11-01 17:52:02 <sipa> is xxx.xxx.com a pool, or your server?
523 2012-11-01 17:52:05 <JDuke128> server has only 2 cpu
524 2012-11-01 17:52:10 <JDuke128> server
525 2012-11-01 17:52:13 <JDuke128> my server
526 2012-11-01 17:52:21 <sipa> how much hash rate do you have?
527 2012-11-01 17:52:28 <sipa> total
528 2012-11-01 17:52:46 <JDuke128> ~300m on pc
529 2012-11-01 17:52:58 <gribble> The average time to generate a block at 300000 Khps, given current difficulty of 2864140.507811 , is 1 year, 15 weeks, 4 days, 14 hours, 10 minutes, and 32 seconds
530 2012-11-01 17:52:58 <sipa> ;;bc,calc 300000
531 2012-11-01 17:53:12 <sipa> you won't - most likely - never see any block
532 2012-11-01 17:53:18 <sipa> *ever
533 2012-11-01 17:53:41 <JDuke128> server needs gpu ?
534 2012-11-01 17:53:47 <sipa> no
535 2012-11-01 17:53:54 <sipa> blocks generate 50 BTC at a time, and with your hash rate it would take on average a year to create a block
536 2012-11-01 17:54:00 <sipa> it can be a month
537 2012-11-01 17:54:03 <sipa> it can be 3 years
538 2012-11-01 17:54:09 <sipa> but on average 1 year + 15 weeks
539 2012-11-01 17:54:17 <sipa> you don't want to wait that long, i suppose
540 2012-11-01 17:54:22 <sipa> it's a huge risk
541 2012-11-01 17:54:36 <JDuke128> for server?
542 2012-11-01 17:54:42 <sipa> for your income
543 2012-11-01 17:54:55 <JDuke128> okay i will use deep bit so
544 2012-11-01 17:55:10 <gmaxwell> JDuke128: why deepbit? ... you have bitcoind running. Use p2pool.
545 2012-11-01 17:55:11 <sipa> so i advize you to run p2pool or join a centralized pool, which splits the income among many contributors
546 2012-11-01 17:55:34 <JDuke128> p2pool ?
547 2012-11-01 17:55:48 <gmaxwell> It blows my mind that you'd show up flaming about mining centeralization and then decide to join one of the bigger centeralized pools.
548 2012-11-01 17:55:55 <JDuke128> whats the difference between mine and p2pool ?
549 2012-11-01 17:56:02 <gmaxwell> https://en.bitcoin.it/wiki/P2Pool
550 2012-11-01 17:57:32 <JDuke128> as i understand , to get bitcoin balance on server , i need to feed my server with many gpus with many clients
551 2012-11-01 17:57:33 <gmaxwell> JDuke128: what you do now has a very low odds of getting 50 BTC with every hash. When you mine with a pool you have a much higher odds of getting a much smaller amount with every hash.
552 2012-11-01 17:57:34 <JDuke128> right ?
553 2012-11-01 17:57:39 <D34TH> oh wow, just compiled latest bitcoind on mingw came out to like 65 mb
554 2012-11-01 17:57:44 <D34TH> ran strip, 5 mb
555 2012-11-01 17:58:57 <BlueMatt> D34TH: been that way forever
556 2012-11-01 17:59:02 <gmaxwell> JDuke128: minging is a randomized process. The move of it you do the more often you will solve a block. The difficulty changes over time so that the whole network makes about 2016 blocks in two weeks (one per ten minutes)
557 2012-11-01 17:59:59 <JDuke128> okay , how many gpu i need to make my server as big ?
558 2012-11-01 18:00:08 <JDuke128> 1000 + ?
559 2012-11-01 18:00:38 <gmaxwell> as big as what?
560 2012-11-01 18:00:49 <gmaxwell> The whole network?
561 2012-11-01 18:01:23 <JDuke128> yea
562 2012-11-01 18:01:44 <gmaxwell> About 100,000.
563 2012-11-01 18:04:12 <JDuke128> so , as i understand bitcoind is for people who is opening service to public like deepbit.net and many people join their service to synchronize big network in time will be much smaller time
564 2012-11-01 18:04:14 <JDuke128> right?
565 2012-11-01 18:04:35 <JDuke128> bitcoind for single network impossible
566 2012-11-01 18:06:18 <JDuke128> gmaxwell , right ?
567 2012-11-01 18:07:17 <sipa> JDuke128: bitcoin is not just mining, it is a currency
568 2012-11-01 18:07:31 <sipa> mining is a part of it, and its economics will drive people towards higher efficiency
569 2012-11-01 18:07:55 <sipa> that means better hardware, more specialized hardware, and also infrastructure to deal with it
570 2012-11-01 18:08:41 <sipa> one of those is centralized pools, but it has the disadvantage of people giving up their "voting power" in the network to some central instance
571 2012-11-01 18:09:07 <sipa> something like p2pool or some more modern pool setups give you the advantage of decreasing variance without giving up your rights
572 2012-11-01 18:09:41 <sipa> those are somewhat more complex to setup though
573 2012-11-01 18:11:50 <JDuke128> p2pool is complex ?
574 2012-11-01 18:12:15 <sipa> not really, but it is harder to setup than just pointing your miner to some central server
575 2012-11-01 18:12:31 <jgarzik> http://www.nytimes.com/2012/10/30/science/rethinking-the-computer-at-80.html?pagewanted=all
576 2012-11-01 18:16:08 <gmaxwell> What does bitcoind for single network impossible mean?
577 2012-11-01 18:16:44 <gmaxwell> JDuke128: bitcoind is just the reference bitcoin node software with the bitcoin-qt gui turned off.
578 2012-11-01 18:19:48 <helo> JDuke128: unless you're going to really expand, buy fpga/asic, etc., with 300mh you might as well just put as little time/effort into it as possible
579 2012-11-01 18:20:15 <helo> JDuke128: just point it at a non-registration-required pool, and let it run
580 2012-11-01 18:21:59 <helo> if you're interested in doing the work for the learning experience, then p2pool
581 2012-11-01 18:22:39 <jgarzik> looks like picocoin could probably steal BlueMatt's CBloomFilter with only minor modifications
582 2012-11-01 18:22:53 <BlueMatt> nice!
583 2012-11-01 18:23:17 <jgarzik> BlueMatt: is the serialization locked in stone yet?
584 2012-11-01 18:23:24 <jgarzik> ACTION hopes "yes"
585 2012-11-01 18:23:28 <sipa> jgarzik: close
586 2012-11-01 18:23:35 <BlueMatt> aside from the merkle tree stuff, yes
587 2012-11-01 18:23:46 <sipa> oh right, bloom filters, sure
588 2012-11-01 18:23:57 <BlueMatt> sipa: sorry I was so out of it yesterday when I tried to go over your stuff, I hope to be able to pull it into my pull request for bloom filter stuff and pull it into my bitcoinj branch over the next day
589 2012-11-01 18:24:00 <jgarzik> just the bloom filter serialization itself
590 2012-11-01 18:24:07 <jgarzik> + hash algo, etc.
591 2012-11-01 18:24:11 <BlueMatt> yea, thats pretty much set in stone
592 2012-11-01 18:24:18 <sipa> BlueMatt: not sure you saw it, but the rule nBits == nHash*2-1, is not exactly true
593 2012-11-01 18:24:36 <sipa> BlueMatt: so i'm not entirely sure I don't want to store the number of bits specifically
594 2012-11-01 18:25:00 <sipa> (as i like some sanity check "walking the tree consumed exactly as many bits as was claimed")
595 2012-11-01 18:26:18 <MC1984> How does that blend tec guy get away with blending phones full of lithium batteries without casuing a massive fire, or getting brain cancer from breathing the resulting surely toxic dust cloud
596 2012-11-01 18:26:52 <BlueMatt> jgarzik: gmaxwell: had one question re: non-cryptographically-strong hash function, but I dont think that was a blocker
597 2012-11-01 18:27:24 <sipa> well, i would prefer a cryptographic one, but i haven't thought about it enough to know whether it would actually matter
598 2012-11-01 18:27:35 <jgarzik> no clue how difficult it would be to change the hash, but my unthinking default is "prefer crypto hash"
599 2012-11-01 18:27:43 <jgarzik> double-SHA256 would be nice symmetry with the rest of the code
600 2012-11-01 18:28:30 <BlueMatt> well the other goal was to make sure bloom filters require 0 additional load on the serving nodes
601 2012-11-01 18:28:45 <BlueMatt> (whereas sha256 hashing everything does not fulfill that goal)
602 2012-11-01 18:28:59 <sipa> yes, eventually you're not going to use more than 20 bits or so of the hash function's output
603 2012-11-01 18:29:15 <sipa> which allows trivial collisions even with double-sha256
604 2012-11-01 18:29:40 <jgarzik> <shrug>
605 2012-11-01 18:29:51 <jgarzik> network messages use 32-bit truncated version of dbl-sha256
606 2012-11-01 18:29:59 <sipa> yes
607 2012-11-01 18:30:43 <jgarzik> CBloomFilter::Hash() appears to return 32-bit
608 2012-11-01 18:31:16 <BlueMatt> yes, and on big backbone nodes dont need more sha256 hash (its already a reasonable amount of cpu time in sha256 itself)
609 2012-11-01 18:31:18 <sipa> well yes, but the bloom filter itself has only at most 2^17 buckets or so?
610 2012-11-01 18:31:57 <BlueMatt> (well, it depends on what you are doing, but my dnsseed eats much of its time in sha256 hashing messages)
611 2012-11-01 18:32:13 <jgarzik> sounds like we should import sha256 mining cores for generic sha256 hashing!
612 2012-11-01 18:33:01 <sipa> 288000 max buckets in the bloom filter, according to the current BIP37 text
613 2012-11-01 18:33:18 <sipa> so 18 bits of the hash function are used
614 2012-11-01 18:33:29 <sipa> so finding a collision takes 512 double-SHA256 steps :)
615 2012-11-01 18:33:56 <sipa> oooh, i know, let's use scrypt as the bloom filter hash functioN!
616 2012-11-01 18:34:02 <jgarzik> still, seems like yet another hash function is a pain
617 2012-11-01 18:34:07 <jgarzik> compared to reusing an existing one
618 2012-11-01 18:34:19 <sipa> meh, i think here it is worth the cpu usage consideration
619 2012-11-01 18:34:26 <BlueMatt> well the attack is a bit more complicated than just finding collisions...you need to find a hash which is statistically the most likely to have collisions
620 2012-11-01 18:35:07 <BlueMatt> (which is obviously not possible with a theoretically perfect crypto hash, but I doubt its a serious attack against any reasonable hash function)
621 2012-11-01 18:35:37 <sipa> right, the problem is preimages, not collisions
622 2012-11-01 18:36:01 <BlueMatt> yep
623 2012-11-01 18:36:08 <sipa> still, 2^18 SHA256 steps is still trivial
624 2012-11-01 18:36:31 <sipa> ooh!
625 2012-11-01 18:36:41 <sipa> how about we include a salt?
626 2012-11-01 18:37:05 <sipa> which is included in the hash function, and chosen by the SPV node at connect time
627 2012-11-01 18:37:23 <sipa> if an attacker doesn't know the salt, he cannot produce preimages
628 2012-11-01 18:37:42 <helo> sounds legit
629 2012-11-01 18:37:42 <jgarzik> std::vector<unsigned char> vData;
630 2012-11-01 18:37:54 <sipa> jgarzik: ?
631 2012-11-01 18:37:56 <jgarzik> ACTION wonders how the magic serializes vector<uc>
632 2012-11-01 18:38:00 <jgarzik> varlen + bytes, I guess?
633 2012-11-01 18:38:03 <sipa> correct
634 2012-11-01 18:39:05 <BlueMatt> sipa: your goal is already to find a preimage which matches as many of the set of pubkeys as possible (which is what most keys in filters would be) so adding a salt wont help much
635 2012-11-01 18:39:17 <BlueMatt> s/preimage/item/
636 2012-11-01 18:39:28 <sipa> heh?
637 2012-11-01 18:40:16 <sipa> so the hash function would be H(item) but H(salt + item), basically
638 2012-11-01 18:40:22 <sipa> *not
639 2012-11-01 18:40:34 <BlueMatt> my point is that item is already random data, adding more random data wont help much
640 2012-11-01 18:40:54 <BlueMatt> (unless there is some attack which breaks on changes in length but not on changes in item)
641 2012-11-01 18:41:20 <jgarzik> ACTION likes djb2_hash
642 2012-11-01 18:41:43 <sipa> BlueMatt: but if i know an address that is yours, i know what its hash will be, and i can try to create for example a txout script that contains something that matches that hash
643 2012-11-01 18:42:07 <sipa> if i don't even know what its hash is, how do i go about looking for preimages?
644 2012-11-01 18:42:27 <BlueMatt> sipa: you can do that in any case, just add the address that is mine to the txout script
645 2012-11-01 18:42:37 <sipa> doh!
646 2012-11-01 18:42:43 <sipa> of course
647 2012-11-01 18:42:54 <BlueMatt> :)
648 2012-11-01 18:48:33 <BlueMatt> jgarzik: there are plenty of simple hash functions that could have been chosen...murmur seemed to fit the bill well (it is build on a hash test suite and is used in cassandra in their bloom filters, not that cassandra necessarily made a good choice, but...)
649 2012-11-01 18:48:57 <BlueMatt> jgarzik: if there is something that you prefer for some solid reason, Id be glad to change it
650 2012-11-01 18:49:11 <jgarzik> vs djb2_hash nah don't care
651 2012-11-01 18:49:32 <jgarzik> ACTION uses djb2_hash as the go-to hash in personal projects
652 2012-11-01 18:49:49 <jgarzik> djb did his usual good work on it
653 2012-11-01 18:50:07 <JDuke128> as you told me to use p2pool
654 2012-11-01 18:50:11 <JDuke128> i got it
655 2012-11-01 18:50:19 <JDuke128> python run_p2pool.py --p2pool-node 173.167.113.73 --bitcoind-rpc-port 8332 --bitcoind-p2p-port 9332 xxx xxx
656 2012-11-01 18:50:41 <jgarzik> for the purposes of picocoin, the path of least effort is simply copying BlueMatt's CBloomFilter(). That's the most-tested solution present, and most likely to align with a P2P extension for bloomers.
657 2012-11-01 18:50:43 <JDuke128> i got this error :Check failed! Make sure that you're connected to the right bitcoind with --bitcoind-rpc-port!
658 2012-11-01 18:50:47 <JDuke128> someone know ?
659 2012-11-01 18:51:02 <gmaxwell> BlueMatt: my concern was just a vague one like ... e.g. could I create a small number of addresses that all users matched completely... then I just do a bunch of transactions to flood them. I admit its not so worrysome.. but if a minor change makes it harder than it would be good.
660 2012-11-01 18:51:37 <gmaxwell> BlueMatt: I mostly brought it up with the hope that you/td would either reply "Oh yea, we solved that" or "Oh we can just twiddle this and it's no issue anymore"
661 2012-11-01 18:51:40 <BlueMatt> gmaxwell: yep (guess I restated that poorly...) anyway, I think the cpu-usage argument outweighs that concern here
662 2012-11-01 18:52:54 <gmaxwell> BlueMatt: well my obvious thought was to just stir the hash function for different users. E.g. when you submit a filter you also submit an unknown to the attacker secret which is used to permute the filter.
663 2012-11-01 18:53:13 <gmaxwell> This also means that if you find a filter is matching too much unwanted stuff you could try twiddling the knob.
664 2012-11-01 18:53:27 <gmaxwell> But if the hash function is sufficiently weak then I guess that wouldn't help.
665 2012-11-01 18:54:14 <jgarzik> ACTION is skeptical about CPU usage being noticeable with a ProcessMessages()-like re-used solution, but whatever
666 2012-11-01 18:54:22 <BlueMatt> gmaxwell: as stated above, I think the only case that would help is in a case where you provided a salt of nondeterministic length and the weakness is such that the attack only matches a number of items which have a set length, but does not apply when you twiddle the length
667 2012-11-01 18:54:59 <BlueMatt> jgarzik: its not huge, but for each block, if you are sha-hashing every script data element once per node on top of all the other hashing done, it gets pretty expensive
668 2012-11-01 18:55:43 <sipa> i assume you could get close to a few 1000 to 10000 hashes per block
669 2012-11-01 18:55:55 <JDuke128> what does this mean ? Check failed! Make sure that you're connected to the right bitcoind with --bitcoind-rpc-port!
670 2012-11-01 18:55:56 <sipa> that he server needs to do
671 2012-11-01 18:56:01 <BlueMatt> yea, for a cheap little arm board, that gets expensive
672 2012-11-01 18:56:08 <gmaxwell> BlueMatt: I don't understand why you're saying a salt won't help.
673 2012-11-01 18:56:31 <gmaxwell> The item is random data but it's globally known public data. The salt is not.
674 2012-11-01 18:56:33 <BlueMatt> gmaxwell: because the data in the filter is always random data (for the 99% use-case)
675 2012-11-01 18:56:47 <BlueMatt> gmaxwell: not neccessarily
676 2012-11-01 18:56:49 <jgarzik> any cheap little arm board is most likely a client, not a server
677 2012-11-01 18:57:05 <BlueMatt> jgarzik: meh, I disagree, but...oh well
678 2012-11-01 18:57:06 <jgarzik> client-supplied salt seems reasonable
679 2012-11-01 18:57:39 <gmaxwell> BlueMatt: for example, I can take a list of all txouts in the blockchain, and then search for a set of addresses that hit the hash values for most of them. I can't do that if there is a salt.
680 2012-11-01 18:58:38 <BlueMatt> gmaxwell: yes, but my point is that addresses are rotating fairly constantly, and the total number of addresses is growing very significantly and will be for the foreseeable future (unless, of course, bitcoin just stops growing...)
681 2012-11-01 18:58:59 <sipa> rotating?
682 2012-11-01 18:59:06 <sipa> the list just grows
683 2012-11-01 18:59:17 <BlueMatt> well, wallets come and go
684 2012-11-01 18:59:23 <sipa> at least for clients that request the entire history
685 2012-11-01 18:59:25 <BlueMatt> the useful-attackable addresses
686 2012-11-01 19:00:25 <BlueMatt> (and clients upgrade to full nodes, addresses are on a merchant running a full node, etc, etc)
687 2012-11-01 19:00:39 <gmaxwell> BlueMatt: I can still just attack the last, say, 100,000 most recently used.
688 2012-11-01 19:05:14 <sipa> well, adding a random salt (or a DH negotiated one, when we have host keys!), does make me feel safer
689 2012-11-01 19:05:59 <BlueMatt> what about: instead of a random salt, another parameter that twiddles the seed value to murmurhash?
690 2012-11-01 19:06:48 <sipa> how is that different?
691 2012-11-01 19:07:09 <BlueMatt> less data, less hash time, same effect
692 2012-11-01 19:08:13 <sipa> like a 32-bit value that gets added to the initial seed? fine by me
693 2012-11-01 19:08:31 <BlueMatt> that was the idea
694 2012-11-01 19:08:54 <BlueMatt> gmaxwell/jgarzik: concept ack?
695 2012-11-01 19:09:04 <BlueMatt> Ill throw that in when I redo the pull over the next ~day
696 2012-11-01 19:43:17 <gmaxwell> yea I was only assuming the 'salt' would be some tiny random value. There may be anonymity benefits in this too... you won't as easily identify a repeat client by an identical or nearly identical filter.
697 2012-11-01 19:43:41 <gmaxwell> (the filter will match most of the same stuff, ??? thin protection indeed??? but hey)
698 2012-11-01 19:45:15 <jgarzik> seems reasonable
699 2012-11-01 20:37:11 <Luke-Jr> oh awesome
700 2012-11-01 20:37:21 <Luke-Jr> +1 on the "kill bitcointalk" step :p
701 2012-11-01 20:38:45 <Luke-Jr> ACTION ponders trolling Gavin's self-contradiction in the forum rules :P
702 2012-11-01 20:43:15 <TD> Luke-Jr: out of curiousity, is Dashjr your real last name?
703 2012-11-01 20:43:23 <Luke-Jr> TD: yep
704 2012-11-01 20:43:31 <TD> what part of the world is it from?
705 2012-11-01 20:44:10 <[Tycho]> India, may be ?
706 2012-11-01 20:44:31 <Luke-Jr> TD: America
707 2012-11-01 20:45:14 <TD> how is it pronounced? dasher?
708 2012-11-01 20:45:46 <Luke-Jr> TD: some people pronounce it that way, but there's a 'j' sound before 'er'
709 2012-11-01 20:46:00 <gmaxwell> "and if it is funny, email it to me" lol.
710 2012-11-01 20:46:08 <Luke-Jr> not a strong 'j' so people often don't even notice it
711 2012-11-01 20:46:44 <sipa> Luke-Jr: +- "dashjur" ?
712 2012-11-01 20:47:14 <Luke-Jr> sipa: closer to "dasher" IMO
713 2012-11-01 20:47:19 <sipa> ok
714 2012-11-01 20:49:04 <gmaxwell> perhaps a little more like you might expect "dashger" (with a soft g, of course)
715 2012-11-01 20:49:08 <gmaxwell> ?
716 2012-11-01 20:49:53 <Luke-Jr> gmaxwell: maybe, I'm no expert on writing pronounciations out ???
717 2012-11-01 20:50:16 <sipa> Luke-Jr: but you do seem to use the 'jr' part separately (in your nick)
718 2012-11-01 20:50:31 <Luke-Jr> sipa: it's a play on my name; I replaced "dash" with a dash ;)
719 2012-11-01 20:50:44 <sipa> ha! never realized that
720 2012-11-01 20:51:30 <TD> i thought it might be something like "Luke Junior" and then the irc nick got turned into a sort of pseudonym. interesting name though
721 2012-11-01 20:51:40 <TD> so the new forum is just for discussing the foundation?
722 2012-11-01 20:51:44 <TD> ACTION has little to say on the topic
723 2012-11-01 20:52:18 <Luke-Jr> TD: I got the impression for discussing Bitcoin
724 2012-11-01 20:52:29 <Luke-Jr> "all things bitcoin" is what the email said
725 2012-11-01 20:52:43 <TD> yeah it seems inconsistent. the email says anything, but the site itself says "General discussion of anything related to the Foundation"
726 2012-11-01 20:52:49 <sipa> too bad IRC doesn't allow ???? as a nickname
727 2012-11-01 20:52:57 <Luke-Jr> lol
728 2012-11-01 20:53:06 <Luke-Jr> anyone have access to fix things? :p
729 2012-11-01 20:53:11 <Luke-Jr> can't view my own profile
730 2012-11-01 20:53:16 <sipa> same
731 2012-11-01 20:53:18 <gmaxwell> Luke-Jr: thats ... complicated.
732 2012-11-01 20:53:23 <Luke-Jr> gmaxwell: ?
733 2012-11-01 20:54:01 <gmaxwell> Thought not having access to your own I think wasn't anticipated. The way the access control works on the forum is that being able to access other profiles meant being able to see the list of all foundation members.
734 2012-11-01 20:54:02 <JDuke128> when will bitcoin exchange start ?
735 2012-11-01 20:54:13 <JDuke128> there is no good bit coin exchange yet
736 2012-11-01 20:54:25 <JDuke128> all people just want to buy bit coin but no one selling
737 2012-11-01 20:54:26 <forsetifox> Mt Gox and BTC-E are good. O.o
738 2012-11-01 20:54:26 <Luke-Jr> JDuke128: MtGox
739 2012-11-01 20:54:28 <sipa> JDuke128: start one!
740 2012-11-01 20:54:32 <Luke-Jr> forsetifox: not BTC-E!
741 2012-11-01 20:54:51 <JDuke128> i can't bit coin on mtgox
742 2012-11-01 20:54:57 <gmaxwell> So gavin had a choice to make between allowing profile access and making that list public ( / and hacking up the software, I suppose). I expect that that particular decision was temporarily deferred.
743 2012-11-01 20:55:01 <sipa> JDuke128: if all people wanted to buy and no one wanted to sell, the BTC/USD exchange rate would be infinite
744 2012-11-01 20:55:03 <forrestv> JDuke128, re: your p2pool issue, are you trying to mine for litecoins?
745 2012-11-01 20:55:31 <JDuke128> i don't see any future on litecoin
746 2012-11-01 20:55:35 <JDuke128> it will fail
747 2012-11-01 20:55:40 <JDuke128> bit coin better
748 2012-11-01 20:56:26 <forrestv> ah. hm, not sure why that error was popping up, then. it usually happens when you're connecting to litecoind instead of bitcoind... or a bitcoind in testnet mode
749 2012-11-01 20:57:43 <TD> hey forrest
750 2012-11-01 20:58:12 <TD> forrestv: what do you think the current bottleneck for p2pool adoption is?
751 2012-11-01 20:59:12 <forsetifox_> What's wrong with BTC-E Luke?
752 2012-11-01 20:59:25 <Luke-Jr> forsetifox_: they are known scammers
753 2012-11-01 20:59:40 <forsetifox_> First time I've heard that. I only use them for LTC -> BTC anyways.
754 2012-11-01 21:00:12 <Luke-Jr> any idea how to add friends? :p
755 2012-11-01 21:00:27 <forrestv> TD, by the time p2pool was stable everyone already was satisfied with whatever pool they were using, and there wasn't much of an influx of new miners. and just apathy about bitcoin's health
756 2012-11-01 21:00:59 <Luke-Jr> TD: p2pool is already a decent size
757 2012-11-01 21:01:03 <forrestv> and higher variance for small miners
758 2012-11-01 21:01:10 <TD> overall hash rate is growing though. you think it's existing miners upgrading their operations rather than number of miners growing?
759 2012-11-01 21:01:35 <gmaxwell> P2pool also simply takes more effort to setup. You need to have a healty running bitcoin node. There is a p2pool regular who often complains that p2pool complains that his bitcoin has gone unresponsive for minutes at a time.. due to his abacus overheating. :P
760 2012-11-01 21:01:43 <TD> i'm wondering if difficulty of setup/installation/running a full node is possibly a contributor
761 2012-11-01 21:01:44 <TD> yeah
762 2012-11-01 21:01:45 <forrestv> probably ... i don't think that people who are just beginning to get familiar with bitcoin are going to invest in expensive ASICs
763 2012-11-01 21:01:55 <gmaxwell> Ultraprune should help p2pool adoption somewhat.
764 2012-11-01 21:02:03 <gmaxwell> Oh also. BFL fpgas are p2pool incompatible.
765 2012-11-01 21:02:07 <TD> oh
766 2012-11-01 21:02:12 <TD> ah yeah. right. i remember this now.
767 2012-11-01 21:02:15 <TD> they can't stop hashing
768 2012-11-01 21:02:18 <gmaxwell> So a chunk of additional hashrate is those, and none of them use p2pool.
769 2012-11-01 21:02:26 <gmaxwell> well. s/none/few/ I guess.
770 2012-11-01 21:02:39 <Luke-Jr> TD: there's no reason to generally prefer p2pool over other centralized pools
771 2012-11-01 21:02:45 <Luke-Jr> other decentralized*
772 2012-11-01 21:03:10 <gmaxwell> I'd kinda hoped that most pools would sleep up to the introduction of asics and p2pool would win more asic miners by default. Alas.
773 2012-11-01 21:03:18 <Luke-Jr> TD: they can stop hashing, but the real problem is you don't get results until they finish
774 2012-11-01 21:03:34 <sipa> why can't BFL asics do p2pool?
775 2012-11-01 21:03:54 <gmaxwell> not asics. FPGAs.
776 2012-11-01 21:03:54 <sipa> (not "because they don't exist", please)
777 2012-11-01 21:03:55 <Luke-Jr> sipa: who says they can't? :p
778 2012-11-01 21:03:55 <TD> because of the 10 second share period
779 2012-11-01 21:04:05 <sipa> oh, right
780 2012-11-01 21:04:08 <TD> sounds like luke understands it better
781 2012-11-01 21:04:23 <TD> my understanding was that you send them a block header and they iterate through every nonce and let you know if they found a match
782 2012-11-01 21:04:35 <TD> it takes a few seconds to do that iteration. if the header you want to hash changes whilst it's iterating, you're out of luck
783 2012-11-01 21:04:37 <gmaxwell> Luke wrote the cgminer drivers for the thing, he ought to. :P
784 2012-11-01 21:04:38 <TD> so you get a lot of stales
785 2012-11-01 21:04:51 <TD> it's not that it doesn't work at all. it's that it's very inefficient
786 2012-11-01 21:05:04 <gmaxwell> I'm sure none of the asic devices will have that issue.
787 2012-11-01 21:05:05 <Luke-Jr> just slow to provide shares
788 2012-11-01 21:05:17 <sipa> sure, as p2pool has a 10s share interval, a 10s delay is dramatical
789 2012-11-01 21:05:21 <Luke-Jr> it produces stales on normal pools too, just much less at Bitcoin's 10 min per block
790 2012-11-01 21:05:37 <gmaxwell> (god knows, they'll likely have _other_ issues.. just not that one :P )
791 2012-11-01 21:06:00 <Luke-Jr> gmaxwell: cablepair had me write out the requirements for bASIC
792 2012-11-01 21:06:15 <TD> yeah BFL understand the issue. i don't know if they'll fix it. something to do with having to upgrade the "bitstream"
793 2012-11-01 21:06:24 <Luke-Jr> so I covered pretty much every known issue learned from FPGAs :P
794 2012-11-01 21:06:46 <Luke-Jr> TD: they tried to fix it with the minirigs, but that turned out sucky
795 2012-11-01 21:07:03 <Luke-Jr> but ASICs are much faster period
796 2012-11-01 21:07:11 <Luke-Jr> so this problem *should* simply not exist
797 2012-11-01 21:07:25 <gmaxwell> right even with the same design it wouldn't be a problem.
798 2012-11-01 21:07:26 <sipa> they'll do a 32-bit noncerange in under 1s, at the claimed specs
799 2012-11-01 21:07:53 <gmaxwell> TD: the pr BFL devices are FPGAs... they could fix this, but apparently it hasn't been a priority. Yet again something else technical that sucks in the bitcoin universe, what a surprise. :P
800 2012-11-01 21:09:07 <Luke-Jr> gmaxwell: I think changign their MCU firmware requires wiring to the unconnected JTAG :/
801 2012-11-01 21:09:39 <TD> heh
802 2012-11-01 21:10:36 <gmaxwell> Luke-Jr: but at least no one can figure out what chips they're using!
803 2012-11-01 21:10:42 <gmaxwell> (oh wait???)
804 2012-11-01 21:11:16 <TD> ACTION reads about getblocktemplate
805 2012-11-01 21:11:50 <Luke-Jr> I get the feeling this one-subforum will become a bit crowded fast
806 2012-11-01 21:12:01 <Luke-Jr> which might scare people out
807 2012-11-01 21:12:18 <sipa> TD: what kind of negative tests are you thinking about for the partial merkle tree? the current tests verify that a) decode(encode(x)) == x for some random x's and b) that the decoded merkle root matches the merkle root as calculated by bitcoin's regular algorithm
808 2012-11-01 21:12:30 <TD> gotta go
809 2012-11-01 21:12:33 <TD> we can talk about it tomorrow
810 2012-11-01 21:12:36 <sipa> ok
811 2012-11-01 21:12:40 <TD> basically checking that the tests fail if the data is bad
812 2012-11-01 21:21:47 <rdponticelli> sipa: Looks like 1971 has indeed solved the relaying bug, but that node keeps orphaning some transactions that are happily accepted by a 0.7.1
813 2012-11-01 21:22:55 <sipa> rdponticelli: were they started at the same time?
814 2012-11-01 21:23:18 <sipa> because it's always possible to not have a parent transaction that the claimed orphan depends on
815 2012-11-01 21:23:58 <sipa> if after a significant time running you notice one significantly has more orphans than the other, there is a problem
816 2012-11-01 21:27:46 <rdponticelli> No, they weren't started at the same time, but the number of orphaned on ultraprune seems too high, anyway
817 2012-11-01 21:29:12 <rdponticelli> sipa: Looks like a consistent behavior, sustained in time
818 2012-11-01 21:29:25 <sipa> you are running a recent head?
819 2012-11-01 21:29:41 <sipa> because there was a bug some time ago that caused high number of orphans
820 2012-11-01 21:30:26 <sipa> https://github.com/sipa/bitcoin/commit/4afc0b54112bd990217a4a6fea145574e5947f09
821 2012-11-01 21:31:53 <rdponticelli> Yeah, it's after that bug was solved, now I'm running your pull request 1971
822 2012-11-01 21:42:30 <senseless> Is anyone aware of an attempt to make a bitcoin based ssl certificate authority system (similar to namecoin, to use in conjunction to have a CA for .BIT)?