1 2013-01-06 01:36:30 <MC1984> 270mb logfile muh balls
2 2013-01-06 01:36:56 <etotheipi_> MC1984: mine is 5.5 GB
3 2013-01-06 01:37:29 <etotheipi_> not sure why it doesn't get cleared out
4 2013-01-06 01:37:33 <MC1984> open it lol
5 2013-01-06 01:37:43 <Diapolo> 5.5GB Log? holy shit ^^
6 2013-01-06 01:37:55 <etotheipi_> yeah... we never figured out why it's doing that
7 2013-01-06 01:38:28 <etotheipi_> actually, I deleted it about a week ago... and now it's already 117 MB again
8 2013-01-06 01:39:35 <Diapolo> you are talking about debug.log?
9 2013-01-06 01:40:35 <etotheipi_> yeah
10 2013-01-06 01:43:36 <Diapolo> with -debug switch?
11 2013-01-06 01:45:55 <etotheipi_> oh, yeah
12 2013-01-06 01:46:00 <etotheipi_> i do have that switch on
13 2013-01-06 01:46:10 <etotheipi_> I didn't realize my panel button was using that
14 2013-01-06 01:48:30 <Diapolo> with that switch the log can grow rather large indeed
15 2013-01-06 01:48:58 <etotheipi_> well I feel silly
16 2013-01-06 01:49:06 <etotheipi_> I didn't realize I ever added that to my launcher
17 2013-01-06 01:55:00 <MC1984> Connect 593 transactions: 110438.80ms (186.237ms/tx, 54.727ms/txin)
18 2013-01-06 01:55:01 <MC1984> 2013-01-06 02:41:33 - Verify 2018 txins: 194109.11ms (96.189ms/txin)
19 2013-01-06 01:55:09 <MC1984> guys......
20 2013-01-06 02:02:33 <gmaxwell> MC1984: how'd you manage that? 110 second for a lbock there? pretty cool.
21 2013-01-06 02:03:02 <MC1984> thats how i roll
22 2013-01-06 02:03:09 <gmaxwell> (and there are people who argue that 10 minute blocks is too long! :P)
23 2013-01-06 02:03:28 <gmaxwell> MC1984: is that unusual for the number of transactions or are you running this on a TI-85?
24 2013-01-06 02:03:39 <MC1984> this machine is 1.6ghz 1gb rams
25 2013-01-06 02:03:43 <MC1984> normal hdd
26 2013-01-06 02:04:15 <gmaxwell> kinda disappointing. then again.. 2018 txn is that testnet3 or did someone mine a max sized block on mainnet?
27 2013-01-06 02:04:26 <MC1984> main
28 2013-01-06 02:04:43 <MC1984> i think this machine was running AV scan at the same time
29 2013-01-06 02:05:26 <gmaxwell> on 2018 txins.. not so bad.
30 2013-01-06 02:05:31 <gmaxwell> s/on/oh/
31 2013-01-06 02:05:49 <gmaxwell> MC1984: it would be interesting to see some more numbers.. if the AV was doing something, well??? can't help that.
32 2013-01-06 02:06:21 <MC1984> its not now but its currently choking on dicks trying to start bitcoin again with a nearly omplete chain
33 2013-01-06 02:07:43 <MC1984> i hope killing the process doesnt mess up lbd
34 2013-01-06 02:08:33 <MC1984> ok its finally loaded
35 2013-01-06 02:10:16 <MC1984> ok this logfile has got to go
36 2013-01-06 02:11:53 <MC1984> i wish the splash screen wasnt so broken in qt
37 2013-01-06 02:12:21 <MC1984> i dont know what its doing now, i just have a suare with an image of what was infront of it
38 2013-01-06 02:12:59 <gmaxwell> MC1984: when did it break?
39 2013-01-06 02:13:13 <MC1984> its alwaus been like that
40 2013-01-06 02:13:22 <Diapolo> What OS do you have?
41 2013-01-06 02:13:32 <MC1984> xp
42 2013-01-06 02:13:36 <MC1984> i beleive its the same on 7
43 2013-01-06 02:13:52 <Diapolo> then it should not mess up... can you explain a little more in detail
44 2013-01-06 02:13:57 <gmaxwell> Hm. I've never noticed it being broken. ... but it does generally seem kinda dumb to me.. (gets in the way when loading is taking manny seconds)
45 2013-01-06 02:14:44 <MC1984> it always stays behind all windows
46 2013-01-06 02:14:53 <Diapolo> that splash screen for sure needs to die ^^
47 2013-01-06 02:15:10 <Diapolo> I just had not the motivation to look into how this can be done :D
48 2013-01-06 02:15:10 <MC1984> no its cool just fix it
49 2013-01-06 02:15:35 <Diapolo> so you suggest it should be a topmost window all the time?
50 2013-01-06 02:16:40 <MC1984> it should just act like any other just launched program prob
51 2013-01-06 02:17:26 <Diapolo> what do you mean it always stays behind all windows then?
52 2013-01-06 02:17:43 <MC1984> wowwwww i would do you a screenshot if i could open my browser
53 2013-01-06 02:18:05 <Diapolo> you start bitcoin-qt and just wait ... you should see that screen. You start your browser for example and it's window will hide the splash screen.
54 2013-01-06 02:18:32 <MC1984> not only does it hide it, its bugged too
55 2013-01-06 02:19:06 <MC1984> it copies whatever was infront of it
56 2013-01-06 02:19:22 <Diapolo> you should create an issue on Github and add some screenies IMO
57 2013-01-06 02:20:39 <MC1984> github makes me scared and angry
58 2013-01-06 02:21:02 <Diapolo> I really can't reproduce your findings
59 2013-01-06 02:21:20 <Diapolo> I always can read what Qt is doing and writing in the splash screen
60 2013-01-06 02:22:34 <MC1984> http://imgur.com/f8P7I
61 2013-01-06 02:22:44 <MC1984> i dont have findings, its just always been like that
62 2013-01-06 02:23:08 <MC1984> its not drawing properly, mirroring irc infront of it
63 2013-01-06 02:23:32 <Diapolo> do you have a recent graphics driver installed?
64 2013-01-06 02:23:58 <MC1984> this machine is six years old lol
65 2013-01-06 02:24:29 <Diapolo> what prevents you from giving it a recent driver?
66 2013-01-06 02:24:58 <Diapolo> either it is to slow (I/O) or it has some faulty graphics driver
67 2013-01-06 02:25:00 <Diapolo> too
68 2013-01-06 02:25:22 <Diapolo> what you observe is NOT standard or default ;)
69 2013-01-06 02:25:51 <MC1984> Plug and Play Monitor (1024x768@60Hz)
70 2013-01-06 02:26:54 <MC1984> Memory\t64 MB
71 2013-01-06 02:27:03 <MC1984> driver is whatever windows installed.....
72 2013-01-06 02:27:32 <Diapolo> goto intel.com support and auto driver update, perhaps that helsp
73 2013-01-06 02:30:07 <Diapolo> I'm out for now ^^
74 2013-01-06 02:31:16 <MC1984> playing with dbcache makes bitcoin take a long time to start
75 2013-01-06 09:13:15 <molecular> I'm having problems connection to network using v0.7.1-289-g1f4fdb7-beta. all connections either time out (~90%) or fail with "connect() failed: 101". Did something change?
76 2013-01-06 09:23:18 <vazakl> daleks in manhattan
77 2013-01-06 09:23:25 <vazakl> oops wrong win
78 2013-01-06 09:24:26 <sipa> molecular: nothing changed
79 2013-01-06 09:25:39 <gladoscc> Could a light client that does not keep older blockchain (only the last 6 for example) work?
80 2013-01-06 09:25:50 <gladoscc> It would verify transactions when there is a new block, and see if the transaction is there.
81 2013-01-06 09:25:57 <molecular> sipa: strange, what could cause the timeouts? I can ping the hosts it's trying to connect to.
82 2013-01-06 09:26:16 <gladoscc> It will keep a full history of it's own addresses for sending.
83 2013-01-06 09:26:45 <gladoscc> The light client would be useful for running on phones, etc
84 2013-01-06 09:26:54 <gladoscc> while still being decentralized
85 2013-01-06 09:27:26 <molecular> sipa: ok, it seems to be something with my network... tried nmap -p 8333 from other host: open. from here: filtered
86 2013-01-06 09:28:53 <gladoscc> thoughts?
87 2013-01-06 09:29:54 <sipa> gladoscc: perfectly possible
88 2013-01-06 09:30:46 <sipa> it doesn't even need to be a real light node
89 2013-01-06 09:31:12 <sipa> i mean, it is what SPV clients already do
90 2013-01-06 09:31:54 <sipa> but it's also possible to have a fully validating node that doesn't unlimited history
91 2013-01-06 09:32:12 <sipa> +keep
92 2013-01-06 09:33:05 <gladoscc> I think most would prefer that as it doesn't require large space, doesn't require you to download the entire blockchain, while you can verify everything is true
93 2013-01-06 09:33:15 <gladoscc> (obviously except 51% doublespends)
94 2013-01-06 09:38:54 <cardpuncher> Hi, does "Imports blocks from external blk000??.dat file" mean import all the blocks or import some blocks? I need this information for a translation.
95 2013-01-06 09:38:57 <xHire> hello, I just found a research on double-spending attacks and wanted to ask if you know about it? http://www.syssec.ethz.ch/research/Bitcoin ??? here is some basic info + patch to bitcoin-qt implementing detection mechanism
96 2013-01-06 09:41:33 <Dr-G> can anyone help me?
97 2013-01-06 09:41:45 <Dr-G> two transactions are somehow stuck
98 2013-01-06 09:45:40 <gladoscc> Dr-G: txids, how are they stuck, etc?
99 2013-01-06 09:46:32 <Dr-G> I don't know
100 2013-01-06 09:46:59 <Dr-G> I'm restarting my client
101 2013-01-06 09:47:06 <gladoscc> xHire: isn't it already implemented in 0.7.2?
102 2013-01-06 09:47:36 <gladoscc> Dr-G: they are not stuck really
103 2013-01-06 09:47:39 <gladoscc> satoshidice haven't set them out
104 2013-01-06 09:47:45 <Dr-G> they don't show up
105 2013-01-06 09:47:48 <gladoscc> yes
106 2013-01-06 09:47:52 <Dr-G> oh ok
107 2013-01-06 09:47:56 <Dr-G> hm
108 2013-01-06 09:48:05 <gladoscc> just wait a bit
109 2013-01-06 09:48:18 <gladoscc> 15 mins, 3 hours, a day.. nobody knows for sure but you will get your payment :3
110 2013-01-06 09:48:20 <Dr-G> ok, but normaly it shows up instant
111 2013-01-06 09:48:25 <gladoscc> yup
112 2013-01-06 09:48:31 <gladoscc> sometimes it doesn't.
113 2013-01-06 09:48:36 <Dr-G> and I'm freaking out :P
114 2013-01-06 09:48:59 <Dr-G> because It said something about double spending in blockchain
115 2013-01-06 09:49:12 <gladoscc> wait what?
116 2013-01-06 09:49:53 <gladoscc> I see
117 2013-01-06 09:49:59 <gladoscc> It looks like SatoshiDICE sent you coins which were double spent.
118 2013-01-06 09:50:05 <Dr-G> something I have to worry about?
119 2013-01-06 09:50:09 <gladoscc> yes
120 2013-01-06 09:50:13 <gladoscc> you probably will not get the coins
121 2013-01-06 09:50:18 <Dr-G> wut
122 2013-01-06 09:50:24 <gladoscc> post it on the forums
123 2013-01-06 09:50:56 <gladoscc> doublespends are where someone, makes two TXes with the same input coins
124 2013-01-06 09:51:05 <tonikt> hi guys. what's this leveldb about and why i cannot build the head anymore?
125 2013-01-06 09:51:33 <tonikt> not even bitcoind
126 2013-01-06 09:51:34 <gladoscc> both of them are unconfirmed so YOUR payment might get in the next block
127 2013-01-06 09:51:43 <gladoscc> tonikt: did you install all the dependicies?
128 2013-01-06 09:51:49 <Dr-G> so just wait a few blocks?
129 2013-01-06 09:52:01 <tonikt> gladoscc, which dependencies?
130 2013-01-06 09:52:09 <tonikt> I have all the old dependencies
131 2013-01-06 09:52:22 <gladoscc> Dr-G: just wait for the next block, and then if you don't get your coins post on the forums.
132 2013-01-06 09:52:30 <tonikt> boost, db-4.30, openssl and upnp
133 2013-01-06 09:52:49 <Dr-G> my client is not the newest version, is that a problem? 0.7 something
134 2013-01-06 09:53:10 <petertodd> leveldb is the new database backend for storing the block chain
135 2013-01-06 09:53:23 <petertodd> specifically the block chain index
136 2013-01-06 09:53:29 <gladoscc> yep, install it and then build
137 2013-01-06 09:53:30 <tonikt> petertodd, does it mena that I can disable barkley now?
138 2013-01-06 09:53:43 <petertodd> no, berkelydb is still used for your wallet
139 2013-01-06 09:53:44 <gladoscc> Dr-G: that's not a problem, althrough you should always be on the latest version
140 2013-01-06 09:53:47 <tonikt> ...and if so: how do i do it?
141 2013-01-06 09:53:55 <tonikt> oh, ok
142 2013-01-06 09:54:21 <gladoscc> tonikt: sudo apt-get install leveldb
143 2013-01-06 09:54:34 <tonikt> anyway, I get some linker errors trying to build bitcoind: ./build/main.o:main.cpp:(.text+0x13a2b): undefined reference to `leveldb::WriteBatch::WriteBatch()'
144 2013-01-06 09:54:52 <gladoscc> that is because you need to install leveldb.
145 2013-01-06 09:54:58 <tonikt> ok - i will try
146 2013-01-06 09:55:05 <Dr-G> thank you gladoscc
147 2013-01-06 09:55:18 <tonikt> thanks for your help so far - I will play with it a bit more
148 2013-01-06 09:55:19 <petertodd> apt-get install libleveldb-dev to be exact
149 2013-01-06 09:55:27 <Dr-G> I would reward you with my fancy internet moneyz, but it's all stuck
150 2013-01-06 09:56:11 <gladoscc> Dr-G: do you have a bitcointalk account?
151 2013-01-06 09:56:16 <Dr-G> yes
152 2013-01-06 09:56:36 <gladoscc> I suggest posting this because it means Satoshidice is still vulnerable to double spending attacks.
153 2013-01-06 09:56:52 <Dr-G> give me your adress, if I get money back I will give you something for your help ;)
154 2013-01-06 09:56:54 <xHire> gladoscc: I've gone through changelog but I didn't find any mention of double-spending attack detection
155 2013-01-06 09:57:34 <gladoscc> if you really want to, my address is 1GLadosEkeAsLReqS3yQ51E1R3wVtbJCDF
156 2013-01-06 09:57:49 <gladoscc> thanks :)
157 2013-01-06 09:58:17 <Dr-G> you helped, you get something :P
158 2013-01-06 09:58:30 <gladoscc> xHire: it would be stupid to accept transactions with 0 or few confirms
159 2013-01-06 09:59:55 <sipa> gladoscc: oh, it would still require you to download and process the entire history, just not store it
160 2013-01-06 10:00:40 <gladoscc> sipa: is that because the miner might include TXes with inputs that don't exist?
161 2013-01-06 10:00:54 <sipa> cardpuncher: all blocks in that file
162 2013-01-06 10:01:23 <Dr-G> does satoshi have an irc channel?
163 2013-01-06 10:01:32 <gladoscc> Dr-G: satoshidice? no
164 2013-01-06 10:01:51 <gladoscc> satoshi is the nym of who started bitcoin, btw
165 2013-01-06 10:01:54 <gladoscc> satoshidice is unrelated
166 2013-01-06 10:03:00 <gladoscc> sipa: I was thinking of ways that don't require a client to download the entire blockchain, and don't require trust
167 2013-01-06 10:03:24 <sipa> gladoscc: there is just no way to verify transactions without history
168 2013-01-06 10:04:10 <cardpuncher> Thanks sipa.
169 2013-01-06 10:04:27 <sipa> gladoscc: what you describe is an SPV client, that already exists
170 2013-01-06 10:04:34 <gladoscc> sipa: I think there would
171 2013-01-06 10:04:42 <gladoscc> the current block reward is 25 BTC
172 2013-01-06 10:04:49 <sipa> gladoscc: it needs a limited amount of trust
173 2013-01-06 10:05:02 <gladoscc> if someone includes a fake TX for <25 BTC, it's not going to go in the main blockchain
174 2013-01-06 10:05:08 <sipa> but it works without full history
175 2013-01-06 10:05:35 <gladoscc> thus, blocks with less than 25BTC (other than coinbase) wouldn't need to be verified.
176 2013-01-06 10:05:35 <sipa> ?
177 2013-01-06 10:05:45 <sipa> heh?
178 2013-01-06 10:05:56 <sipa> i don't follow that reasoning
179 2013-01-06 10:06:03 <gladoscc> the attacker is throwing away the block reward for a lesser gain
180 2013-01-06 10:06:17 <sipa> ???
181 2013-01-06 10:06:48 <gladoscc> ok I think we are getting more and more confused
182 2013-01-06 10:07:19 <gladoscc> first of all, is the problem that a block could contain a nonexistant input?
183 2013-01-06 10:07:24 <gladoscc> or are there other problems?
184 2013-01-06 10:07:35 <sipa> or an invalid signature
185 2013-01-06 10:07:54 <sipa> or spend an already spent input
186 2013-01-06 10:08:00 <stealth222> the block might be over the maximum size, could contain invalid signature, could have a number of other problems
187 2013-01-06 10:08:13 <stealth222> might not hash to target
188 2013-01-06 10:08:13 <Trixlert> Hey people
189 2013-01-06 10:08:15 <sipa> or have a transaction with output value > input value
190 2013-01-06 10:08:25 <gladoscc> OK
191 2013-01-06 10:08:33 <gladoscc> But, full nodes will not accept the block right?
192 2013-01-06 10:08:33 <sipa> or have a coinbase that claims more fees than valid
193 2013-01-06 10:08:38 <sipa> yes
194 2013-01-06 10:08:49 <gladoscc> So this means that the miner who creates fake blocks is losing the coinbase
195 2013-01-06 10:08:50 <stealth222> https://en.bitcoin.it/wiki/Protocol_rules#Blocks
196 2013-01-06 10:08:55 <Trixlert> http://btcdevlogs.ww7.be/bitcoin-dev..php
197 2013-01-06 10:09:24 <gladoscc> right?
198 2013-01-06 10:09:25 <Trixlert> That is some good stats.
199 2013-01-06 10:09:32 <gladoscc> forfeit the coinbase
200 2013-01-06 10:09:34 <sipa> but as soon as you rely on full nodes rejecting invalid blocks, you are trusting the presence of full nodes
201 2013-01-06 10:09:53 <sipa> which is reasonable
202 2013-01-06 10:10:03 <sipa> and it is what SPV nodes do
203 2013-01-06 10:10:18 <stealth222> if you connect to multiple full nodes that you're pretty sure are not in collusion with one another, you can get a pretty good sense for things
204 2013-01-06 10:10:22 <sipa> but you can't replace the full nodes themaelf with that
205 2013-01-06 10:10:36 <stealth222> furthermore, a client could download random blocks and test their validity
206 2013-01-06 10:10:52 <stealth222> without having to download the entirety of all blocks
207 2013-01-06 10:11:08 <stealth222> and be pretty confident that none of the full nodes are cheating
208 2013-01-06 10:16:43 <petertodd> gladoscc: their losing the coinbase in the sense that for the same effort they could have mined a real block
209 2013-01-06 10:16:51 <stealth222> specifically, if a node cheats with probability p, n random queries will find at least one failure with probability (1-p)^n
210 2013-01-06 10:16:52 <sipa> gladoscc: if you're arguing that the network can sustain light nodes: sure, satoshi even described that in the initial paper
211 2013-01-06 10:17:28 <sipa> gladoscc: and for example multibit and bitcoin wallet for android implement that trust model
212 2013-01-06 10:17:28 <stealth222> err, 1-(1-p)^n
213 2013-01-06 10:17:41 <gladoscc> ok, I see
214 2013-01-06 10:17:58 <gladoscc> also, do faster block times matter?
215 2013-01-06 10:18:59 <gladoscc> or is it time based
216 2013-01-06 10:24:10 <gladoscc> ACTION pokes
217 2013-01-06 10:33:17 <sipa> petertodd: leveldb is included in the bitcoin sources...
218 2013-01-06 10:33:30 <petertodd> ha, never mind
219 2013-01-06 10:33:43 <petertodd> interesting that the advice fixed that guys problems
220 2013-01-06 10:34:31 <gladoscc> huh
221 2013-01-06 10:34:37 <gladoscc> then leveldb is not properly included
222 2013-01-06 10:34:50 <gladoscc> because last time I tried to make bitcoin I had to get libleveldb-dev
223 2013-01-06 12:44:26 <TD> sipa: you there?
224 2013-01-06 16:09:00 <MC1984> Connect 615 transactions: 5638.11ms (9.168ms/tx, 6.400ms/txin)
225 2013-01-06 16:09:04 <MC1984> is that good
226 2013-01-06 16:10:20 <MC1984> this is well after block 193k
227 2013-01-06 16:11:40 <MC1984> Connect 165 transactions: 110.16ms (0.668ms/tx, 0.287ms/txin)
228 2013-01-06 16:12:00 <MC1984> height=215412 work=704918185256847701311 tx=10793393 date=2013-01-06 13:17:46
229 2013-01-06 16:12:22 <MC1984> sipas latest build
230 2013-01-06 17:26:46 <MC1984> what is libssp-0.dll and why is it not found :(
231 2013-01-06 17:48:08 <MC1984> sipa your latest build for windows is broken
232 2013-01-06 17:48:15 <MC1984> libssp-0.dll not found
233 2013-01-06 18:21:46 <MC1984> -reindex is a good way to simulate a sync where the network conenction is not a factor right?
234 2013-01-06 18:22:09 <MC1984> like the same as doing a IBD from a lan node right
235 2013-01-06 18:22:21 <jgarzik> MC1984: not quite
236 2013-01-06 18:22:31 <jgarzik> MC1984: you're not writing block data, so it's significantly cheaper
237 2013-01-06 18:22:51 <jgarzik> MC1984: -loadblock from a RAM device is ideal (though that takes hella lots of RAM)
238 2013-01-06 18:23:02 <jgarzik> -loadblock from SSD etc.
239 2013-01-06 18:24:22 <MC1984> so a good lan or good internet node is the way to go?
240 2013-01-06 18:24:36 <MC1984> apart from ramdisks and shit
241 2013-01-06 18:25:27 <MC1984> -connect= is the arg that forces it to connect to a single node isnt it
242 2013-01-06 18:51:24 <MC1984> wow lan sync is significantly faster
243 2013-01-06 18:51:48 <MC1984> first 10k blocks less than 30 seconds
244 2013-01-06 19:01:25 <BlueMatt> TD: ping
245 2013-01-06 19:01:48 <TD> pong
246 2013-01-06 19:01:50 <TD> how goes it?
247 2013-01-06 19:02:01 <BlueMatt> sitting in phoenix on layover
248 2013-01-06 19:02:11 <MC1984> is there a new checkpoint at 210k?
249 2013-01-06 19:03:11 <gmaxwell> MC1984: yes.
250 2013-01-06 19:03:43 <BlueMatt> TD: so im finally getting around to looking through the bloom filter stuff again and updating it (sorry for the delay...holidays and all)
251 2013-01-06 19:03:47 <TD> coo
252 2013-01-06 19:03:50 <TD> cool
253 2013-01-06 19:03:57 <TD> yeah no worries, i didn't get much done either :)
254 2013-01-06 19:04:27 <MC1984> ahh
255 2013-01-06 19:04:47 <MC1984> estimated blocks is stuckon 210000 when i connect to this lan node that is up to date
256 2013-01-06 19:12:17 <BlueMatt> TD: not sure if i forgot to push it or what, but an old version of the "make ...Store upgrade-friendly" got merged: https://code.google.com/r/bluemattme-bitcoinj/source/detail?r=ad1e1ed1f98b1d7609d19240bf67cdcd4bb6563c&name=bloomfilter
257 2013-01-06 19:12:37 <TD> sorry. must be my mistake.
258 2013-01-06 19:14:29 <TD> given that i pushed it now, could you patch it up?
259 2013-01-06 19:14:46 <BlueMatt> that commit should be the correct amend
260 2013-01-06 19:17:23 <TD> hearn-macbookpro:src hearn$ msg=hello
261 2013-01-06 19:17:26 <TD> what am i doing wrong here?
262 2013-01-06 19:17:30 <TD> does this functionality actually work?
263 2013-01-06 19:18:35 <gmaxwell> sure' works fine, but I expect your inner invocation is getting an extra \\n in it
264 2013-01-06 19:19:41 <gmaxwell> (or the inner one is failing because you're using wallet crypto?)
265 2013-01-06 19:20:53 <TD> it appears the gui won't let me verify my own signatures. nice.
266 2013-01-06 19:21:16 <TD> it's base64 - whitespace isn't meant to make any difference
267 2013-01-06 19:21:21 <TD> i'll double check that anyway
268 2013-01-06 19:21:54 <TD> hearn-macbookpro:src hearn$ ./bitcoind signmessage 14YPSNPi6NSXnUxtPAsyJSuw3pv7AU3Cag hello
269 2013-01-06 19:21:55 <TD> hearn-macbookpro:src hearn$ ./bitcoind verifymessage 14YPSNPi6NSXnUxtPAsyJSuw3pv7AU3Cag ILYcEhiO3oEekBespX8POM6OTGRQRxBmzavD4ri2DhctEKhP+GG26boJQCUDt/yYFOPs+yvKkA4DWy9857AkrP0= hello
270 2013-01-06 19:21:58 <TD> i'm not convinced this works ...
271 2013-01-06 19:23:48 <sipa> MC1984: the best pure block verification benchmark is starting from a synced node, stopping, deleting coins/, and restarting; that will rebuild just the coindb and verify
272 2013-01-06 19:24:47 <TD> could somebody send me a signed message i can try?
273 2013-01-06 19:24:50 <MC1984> isnt that a bit artificial?
274 2013-01-06 19:25:02 <MC1984> im syncing normally from a lan node and its pleasingly fast
275 2013-01-06 19:25:03 <sipa> yes
276 2013-01-06 19:25:57 <andytoshi> [username@titanic src]$ ./bitcoind signmessage 1AndrewgVPAnreTyFtdU7nxcSqyMvdTiHm "Heya"
277 2013-01-06 19:26:00 <andytoshi> G4ZVzlpzJls+RLQfQX05soRyhVBtqiBpFG/CUgAmSBQ3oSZMHP0Ab7/O2rvWsK/CLTpS0Rp/AAWkeLYz9HGnrHg=
278 2013-01-06 19:26:18 <andytoshi> I get "true" when i verify that
279 2013-01-06 19:26:26 <andytoshi> but false when i verify yours, TD
280 2013-01-06 19:26:29 <andytoshi> something weird is up
281 2013-01-06 19:26:32 <TD> oh, wait
282 2013-01-06 19:26:35 <TD> i think i see what i did
283 2013-01-06 19:27:57 <TD> never mind. i broke something adding a debug print
284 2013-01-06 19:37:06 <TD> BlueMatt: which is the new commit with the fixes?
285 2013-01-06 19:37:28 <BlueMatt> TD: the link (ad1e1ed1f98b1d7609d19240bf67cdcd4bb6563c)
286 2013-01-06 19:37:55 <TD> sorry, i meant the one that fixes the bloom filtering bugs you mentioned
287 2013-01-06 19:38:03 <TD> or rather that you mentioned you fixed in a new commit
288 2013-01-06 19:38:59 <BlueMatt> ahh, should just be the head of the bloomfilter branch
289 2013-01-06 19:39:10 <BlueMatt> or the same commit but in the new rebase
290 2013-01-06 19:40:28 <TD> oh ok
291 2013-01-06 19:40:35 <TD> great. i'll try soon
292 2013-01-06 19:41:31 <BlueMatt> well, i havent gotten that far yet (well, not to pmt/filteredblock yet) but more to come
293 2013-01-06 19:42:23 <TD> alright then, i'll wait - i just want to be able to merge both branches and see it sync correctly. don't care much right now about whether we can also serve filtered blocks
294 2013-01-06 19:43:20 <BlueMatt> yea, that may take until after the second plane...
295 2013-01-06 19:43:54 <TD> ok
296 2013-01-06 19:43:56 <TD> hehe
297 2013-01-06 19:44:00 <BlueMatt> Ill push it when i get it done (ip-over-dns to get free wifi may be slow, but its fast enough to do git and irc)
298 2013-01-06 19:44:16 <TD> there's no hurry
299 2013-01-06 19:44:46 <petertodd> has anyone else ever looked at p2pool's block timestamp validity rules? seems that it has something called "timestamp_cutoff" which I think restricts timestamps to be accurate to within an hour, just wondering if there is somethign subtle I'm missing
300 2013-01-06 19:46:36 <gmaxwell> bitcoind verifymessage 1GMaxweLLbo8mdXvnnC19Wt2wigiYUKgEB HK0723fozAJTOIEHzTpToSzgzUh3VuXRYNOpnOcQtUcdvN+6a5T9Mntl5pvo/X8ZOCietUXeObQaLg5Ge0t8PkM= hello
301 2013-01-06 19:46:39 <gmaxwell> true
302 2013-01-06 19:46:57 <TD> yeah, sorry, i discovered i was calling ss.GetHash() twice due to an added debug
303 2013-01-06 19:47:04 <TD> ACTION is nearly there with bitcoinj sign/verifymessage
304 2013-01-06 19:47:11 <MC1984> what blockheight did the SD bullshit start?
305 2013-01-06 19:47:15 <MC1984> 190 or so right
306 2013-01-06 19:47:55 <TD> 180k or so i think
307 2013-01-06 19:49:44 <MC1984> im at 190 and still screaming along at 10 ish blocks/second
308 2013-01-06 19:50:05 <MC1984> impressive
309 2013-01-06 19:50:31 <Diapolo> MC1984: did you investigate further because of that splash screen stuff?
310 2013-01-06 19:51:20 <MC1984> ill check that out later on my other machines
311 2013-01-06 20:11:01 <grau> sipa: does the newest bitcoind do full verification of the chain from block 0 or from checkpoint ? if checkpoint what is the highest in 0.8 ?
312 2013-01-06 20:12:47 <grau> TD: Mike?
313 2013-01-06 20:12:54 <TD> hello
314 2013-01-06 20:13:11 <TD> it does signature checking after the latest checkpoint
315 2013-01-06 20:13:23 <Diapolo> grau: https://github.com/bitcoin/bitcoin/blob/master/src/checkpoints.cpp
316 2013-01-06 20:13:31 <grau> thanks
317 2013-01-06 20:13:39 <grau> I will do the same
318 2013-01-06 20:13:39 <TD> i think the latest checkpoint is 210,000
319 2013-01-06 20:15:06 <grau> I recently joined irc and scanned previous log for bitsofproof. I saw you mentioning it would have bitcoinj bits in it. The only few lines I am aware are in calculation of the merkle root. Is this what you mean? Do yu miss attribution for that ?
320 2013-01-06 20:16:16 <TD> yeah that's what i was referring to. no, don't worry about it. i wondered if there might be more, but i indeed never saw any when reading the code.
321 2013-01-06 20:18:54 <grau> That was the only piece I think where I was lazy and too obvious to do it any differently. Thanks a lot.
322 2013-01-06 20:19:19 <TD> yeah, no problems.
323 2013-01-06 20:23:22 <grau> I now have jbehave embedded. Two instances of bitsofproof networking instantiate and you can drive with a plain text story file what should be feeded to one to the other.
324 2013-01-06 20:23:50 <grau> The story can also let api calls executed and check results from that plain text.
325 2013-01-06 20:24:00 <grau> I will convert the block chain tests into this
326 2013-01-06 20:24:19 <sipa> grau: from latest checkpoint, which is 210000
327 2013-01-06 20:26:11 <grau> TD: It would be great if I could easily create a dump of what the bitcoinj blocktester feeds
328 2013-01-06 20:27:13 <sipa> grau: i think it's more useful if you could try to get bitcoinj blocktester test your code
329 2013-01-06 20:27:25 <TD> yes, unfortunately i don't think i ever fully merged that. i don't recall why. maybe BlueMatt knows
330 2013-01-06 20:27:31 <TD> it's in one of his branches
331 2013-01-06 20:27:41 <TD> it shouldn't be too hard to link them together seeing as they're both java apps
332 2013-01-06 20:27:43 <sipa> grau: otherwise you'll need to maintain a mirror of them in your system anyway
333 2013-01-06 20:28:10 <grau> sipa: I plan to do that but not by integrating the code, but integrating the stories it runs
334 2013-01-06 20:28:36 <sipa> there's not code to integrate, just a patch to make your node run the alternative chain. afaik?
335 2013-01-06 20:28:58 <sipa> perhaps it uses RPC calls though... unsure
336 2013-01-06 20:29:02 <TD> i think it talks p2p
337 2013-01-06 20:29:10 <sipa> yes, it talks p2p
338 2013-01-06 20:29:19 <sipa> but i'm not sure whether it only talks p2p
339 2013-01-06 20:29:25 <TD> i think it only talks p2p
340 2013-01-06 20:29:29 <grau> sipa: check this https://github.com/bitsofproof/supernode/blob/master/src/test/resources/connect.story
341 2013-01-06 20:29:44 <grau> this is the first story I wrote.
342 2013-01-06 20:30:02 <TD> ACTION hasn't seen jbehave before
343 2013-01-06 20:30:36 <TD> it's a way to write unit tests in english?
344 2013-01-06 20:30:39 <grau> It looks obvious, it drives a generic driver that executes the steps here https://github.com/bitsofproof/supernode/blob/master/src/test/java/com/bitsofproof/supernode/test/StoriesRunner.java
345 2013-01-06 20:30:45 <upb> its fucking AGILE
346 2013-01-06 20:31:08 <grau> Yes that it!
347 2013-01-06 20:31:13 <sipa> grau: i'm sure things like that are useful for tests, but if i understand you correctly, you plan to translate the bitcoinj nodetester scenarios to that system?
348 2013-01-06 20:31:36 <TD> that could be a lot of work
349 2013-01-06 20:31:52 <grau> sipa: this is the next level after unit tests its test of behaviour
350 2013-01-06 20:32:10 <grau> I let the blocktester generate messages, record them and feed them
351 2013-01-06 20:32:11 <gmaxwell> grau: instead of converting it??? why not just run it against your software. Its an external program and speaks over the network. When you convert the code you run the risk of any lurking misunderstanding or typo causing a non-duplication of the tests.
352 2013-01-06 20:32:14 <sipa> grau: if that's the case, i think it is much more useful to make your system be testable by the bitcoinj directly, as it doesn't require you to maintain any parallel set of the tests
353 2013-01-06 20:32:21 <grau> while I use the API to check that it does what it should
354 2013-01-06 20:32:46 <grau> I do not want testing depending on an other tool
355 2013-01-06 20:32:48 <sipa> grau: that can be done in addition to any tests you write yourself in your own system, of course
356 2013-01-06 20:32:56 <grau> I want testing repeatable at every build
357 2013-01-06 20:32:57 <sipa> why not?
358 2013-01-06 20:33:10 <upb> Scenario: Since genesis can not be spent this balance is 0
359 2013-01-06 20:33:19 <upb> im pretty sure thats not english
360 2013-01-06 20:33:25 <grau> I instantiate two bitsofproof on the fly from maven and jbehave
361 2013-01-06 20:33:25 <sipa> the only way to test whether your P2P-visible interaction works, is by something that connects to P2P...
362 2013-01-06 20:33:57 <grau> sipa: Yes I connect two bitsofproofs
363 2013-01-06 20:34:13 <sipa> it still requires you to maintain your own set of tests
364 2013-01-06 20:34:17 <grau> sipa: it already works
365 2013-01-06 20:34:28 <TD> well
366 2013-01-06 20:34:35 <TD> the point of matts testing tool is it compares two implementations
367 2013-01-06 20:34:41 <TD> and walks them through blocks side by side
368 2013-01-06 20:34:43 <sipa> and if there is a bug in bitsofproof, it may consider the other node functional, even if they are both doing it wrong
369 2013-01-06 20:35:01 <TD> so you can't really re-implement it using something that's bitsofproof specific. that's just not how the tool works
370 2013-01-06 20:35:05 <gmaxwell> Yes, you ought to have your own tets??? but this doesn't replace the value of having an external test that can be run on many implementations, and which was created by someone else so it doesn't reaffirm mitaken assumptions.
371 2013-01-06 20:35:32 <grau> It is an external test, since I let the blocktester create the test cases
372 2013-01-06 20:35:43 <grau> I just save them into story files so I can reuse them
373 2013-01-06 20:35:48 <grau> without running that
374 2013-01-06 20:36:19 <grau> The goal is to validate behaviour for network messages
375 2013-01-06 20:36:19 <sipa> how can you know what bitcoinj is verifying in your output?
376 2013-01-06 20:36:42 <sipa> you can save what it is sending you, but you can't reverse engineer automatically what tests it applies to your responses
377 2013-01-06 20:37:06 <gmaxwell> ACTION muses about cobol never seeming to die
378 2013-01-06 20:37:37 <grau> The story describes what inputs are and what checks are. Just like you created json for scripts I do that for higher level of network interaction
379 2013-01-06 20:38:39 <grau> Hey guys, you deal with hight tech. How can you be so conservative :)))
380 2013-01-06 20:38:53 <upb> yes, we know what a story is
381 2013-01-06 20:39:13 <upb> seems youre have a hard case of NIH syndrom
382 2013-01-06 20:39:15 <sipa> so how will you 1) make sure your story files remain up-to-date with what bitcoinj tester verifies 2) make sure you catch errors in your tester node itself?
383 2013-01-06 20:39:25 <sipa> grau: i love your story-based testing
384 2013-01-06 20:39:34 <gmaxwell> grau: In any case, this does not replace the simple effort of making bluematt's tester run against your code for the purpose of gaining confidence that your code is correct. It just adds a level of uncertanty, and additional work because now when tests are added to matt's implementation neutral test someone has to remember to remind you to add the test to your tool.
385 2013-01-06 20:39:44 <sipa> grau: but it's not the same thing as testing with a standard tool that's already proven to be able to catch errors in implementations
386 2013-01-06 20:39:55 <gmaxwell> ha. What sipa said.
387 2013-01-06 20:41:02 <sipa> grau: from my point of view, i consider bitsofproof completely untrusted (not that i don't trust you, but i don't know it enough to make any judgement, so my conservative assumption is that is faulty) - if you're saying that it validates itself perfectly, i know nothing
388 2013-01-06 20:41:09 <grau> I will do the level of test with it that is possible on network and API level. Deeper than that is bitcoinj (implementation) specific
389 2013-01-06 20:41:46 <grau> BTW. What is NIH ?
390 2013-01-06 20:41:52 <gmaxwell> grau: there is substantial research that english like languages lead to devistating software bugs. English is not adequate for programming, so what you're really using is specialized english which is too easy to misread. But I really don't mind what tools you use, perhaps it's absolutely great... but there should be implementation generic network tests. (and implementation-specific unit tests too!)
391 2013-01-06 20:41:53 <sipa> Not Invented Here
392 2013-01-06 20:42:16 <sipa> and i don't see what the difficulty is; i think you need to change a few lines only to implement the alternative chain used by bitcoinj tester, and you're *done*
393 2013-01-06 20:43:04 <grau> I think, I could not tell what I do. Let me finish and then revisit.
394 2013-01-06 20:44:36 <grau> gmaxwell: You really took this from the wrong angle. This is nothing more than unit tests taken to a higher level with a convinient tool.
395 2013-01-06 20:44:46 <TD> yeah
396 2013-01-06 20:44:53 <TD> matts testing tool isn't really a unit test
397 2013-01-06 20:45:00 <TD> it's something different. maybe that's where the confusion/disagreement stems from
398 2013-01-06 20:45:21 <sipa> grau: and that's a great idea - you need unit tests, and this seems a very nice way to do it
399 2013-01-06 20:45:24 <grau> Yes, his testing describes a story with java code. I try to do the same cleaner
400 2013-01-06 20:45:51 <grau> sipa: I do have unit tests. Actually covering jsons form bitcoind.
401 2013-01-06 20:46:08 <sipa> grau: and even if you completely translate everything the bitcoinj tester does, it is not the same thing as a using it as an external validator
402 2013-01-06 20:46:10 <grau> I do this because I am done with unit tests
403 2013-01-06 20:46:24 <gmaxwell> 13:45 < grau> gmaxwell: You really took this from the wrong angle. This is nothing more than unit tests taken to a higher level with a convinient tool.
404 2013-01-06 20:46:41 <gmaxwell> ^ but that does not replace running an implementation neutral system test.
405 2013-01-06 20:46:45 <TD> grawell, his tool actually compares two different implementations to ensure they proceed in lockstep. yes, you can use it as a source of unit tests, but it's actually meant as a kind of blockchan-diff tool
406 2013-01-06 20:47:56 <gmaxwell> And my comment on stories was an aside, ??? I made it so you'd understand my cobol comment??? but its not relevant to here... I'll let you discover the the benefits of your approach on your own. Sorry for mentioning it.
407 2013-01-06 20:48:00 <grau> What I write is the ultimate neutral system tests. Since I extract the expected behaviour into s plain text and validate against.
408 2013-01-06 20:48:33 <sipa> it is not system neutral as it requires the system being tested to implement the tester itself!
409 2013-01-06 20:49:03 <gmaxwell> ^ which is exactly what we do not want, becuase the implementor will likely repeat his misunderstandings in the test.
410 2013-01-06 20:49:03 <grau> Let me explain the test setup in more detail:
411 2013-01-06 20:50:03 <grau> There is one full bitsofproof node using an in memory database. It seeks to connect to localhost:port x
412 2013-01-06 20:50:45 <grau> There is an other bitsofproof network module instanciated, no database, no logic just to provide socket for the first.
413 2013-01-06 20:51:15 <sipa> ok, now assume there is a bug in bitsofproof, which is active in both the tester node and the node being tested
414 2013-01-06 20:51:22 <grau> The second bitsofproof reads the story file to challange the first and used the RMI API of the first to validate it processes correctly.
415 2013-01-06 20:51:27 <sipa> some encoding/serialization bug, for example
416 2013-01-06 20:51:32 <grau> What correct is is again written in the story file.
417 2013-01-06 20:51:32 <sipa> it will not detect any error
418 2013-01-06 20:51:57 <grau> That low level bugs should be covered by unit tests.
419 2013-01-06 20:52:06 <sipa> should be
420 2013-01-06 20:52:24 <gmaxwell> Maybe! but maybe it isn't. You wrote your own unit tests. If the bug resulted from a misunderstanding you had the test would be wrong too.
421 2013-01-06 20:52:48 <sipa> but why not prove it, by showing that your system passes a standard test maintained by other people, which already is able to validate other node implementations?
422 2013-01-06 20:53:09 <sipa> eventually, your node needs to be able to interact with those other implementation anyway
423 2013-01-06 20:53:12 <grau> but that is what I do !
424 2013-01-06 20:53:36 <gmaxwell> I'm very concerned that you are unwilling to take a few minutes effort to run a preexisting implementation neutral external testing harness but willing to spend hours lossily reimplementing them specific to your system.
425 2013-01-06 20:53:36 <grau> I let that blockchaintester run, but record what it does
426 2013-01-06 20:53:40 <sipa> as long as all code running is bitsofproof, it's not a system-neutral validation
427 2013-01-06 20:53:56 <grau> I am very concerned that you are unwilling to listen
428 2013-01-06 20:54:10 <grau> I do buildsophisticated software for living
429 2013-01-06 20:54:11 <gmaxwell> grau: fine, if the chaintester passes your implementation then that is what I'd want. If you'd like to write additional tests, then great.
430 2013-01-06 20:54:44 <gmaxwell> grau: It sure doesn't sound like you do.
431 2013-01-06 20:55:19 <grau> I am honestly shocked by your resistance to try anything not invented here. Even though it is industry standard in environments that handle more money you imagimne
432 2013-01-06 20:55:30 <gmaxwell> grau: nor does the prior quality of your implementation and the bludgeoning I had to perform to get you to fix it support that.
433 2013-01-06 20:55:33 <sipa> grau: you expect us to trust you that you reimplement/replay matt's tests correctly
434 2013-01-06 20:56:05 <sipa> grau: i have no doubt you're a great programmer, and took your job very seriously when implementing bitcoin
435 2013-01-06 20:56:18 <gmaxwell> grau: I sure didn't invent matt's test??? they were part of his bitcoinj work which I had nothing to do with beyond sending him applause. You're making a logical fallcy of believing that I invented anything you didn't invent.
436 2013-01-06 20:57:02 <sipa> grau: but everyone makes mistakes (myself included, for sure), and if you misunderstood anything about bitcoin, it will likely be present in both your system and your unit tests
437 2013-01-06 20:57:13 <MC1984> if i change around wallet.dat files on a bitcoin installation, i also get the libssp-0.dll error
438 2013-01-06 20:57:31 <MC1984> an over the op reinstall from the installed fixes it
439 2013-01-06 20:57:39 <MC1984> this is with at least the last two sipa builds
440 2013-01-06 20:57:53 <gmaxwell> MC1984: you need that dll because the stack protector stuff got enabled for windows.
441 2013-01-06 20:57:56 <sipa> grau: and perhaps your story-based testing done by a bitsofproof node can become yet another system-neutral testing application for other systems
442 2013-01-06 20:58:20 <sipa> grau: that'd be great actually, if it's as flexible as you say, and it gets a great archive of scenarios to test
443 2013-01-06 20:58:22 <MC1984> gmaxwell i dont know what that means, im just saying what i see
444 2013-01-06 20:58:28 <grau> The problem is you claim to know the right way of testing. What I saw until now in all bitcoin projects does not support that hypothesis well. Yes, I have less tests at the moment but you are a few years ahead.
445 2013-01-06 20:58:45 <MC1984> i just swapped around a wallet.dat and got the error again and had to reinstall from the installer
446 2013-01-06 20:58:47 <sipa> grau: but that still isn't the same as testing whether your system interacts correctly with another system
447 2013-01-06 20:58:53 <sipa> grau: i will not dispute that
448 2013-01-06 20:59:13 <sipa> bitcoind is lacking unit tests, and improvement is certainly needed
449 2013-01-06 20:59:19 <gmaxwell> grau: Having tests isn't sufficient. The behavior of the bitcoin network is what matters, not the tests. The reference software (of the right versions at least) is _automatically correct_ no matter what the tests say.
450 2013-01-06 21:00:07 <sipa> grau: but you can't compare a unit tests to an interaction test with an independently-written implementation of the same rules
451 2013-01-06 21:00:26 <TD> mm, yes, unfortunately satoshi left us with a system which had no unit tests at all, and for which any deviation from its behavior can cause critical damage. which is why he wasn't a fan of the idea of reimplementations. unfortunately it also made evolving the code rather hard.
452 2013-01-06 21:00:30 <gmaxwell> This is, I agree, an unusual situation in terms of software testing??? but it's also not unique to bitcoin. This kind of thing can arise with any distributed system.
453 2013-01-06 21:00:36 <TD> we've been fixing that issue ever since
454 2013-01-06 21:00:37 <MC1984> also it has happened on two different xp machines
455 2013-01-06 21:00:47 <sipa> MC1984: yes, it seems you need that ddl
456 2013-01-06 21:00:49 <sipa> dll
457 2013-01-06 21:01:03 <gmaxwell> TD: even if he had great unit tests??? the implementation still wins. :) which is somewhat unusual for software. :P
458 2013-01-06 21:01:17 <TD> yeah
459 2013-01-06 21:01:26 <TD> but at least you could refactor with more confidence
460 2013-01-06 21:01:27 <MC1984> well yes, i suppose a bitcoin reinstall puts it back or something?
461 2013-01-06 21:01:38 <sipa> MC1984: no build of bitcoin ever required that dll
462 2013-01-06 21:01:41 <gmaxwell> MC1984: https://people.xiph.org/~greg/libssp-0.dll is probably what you need.
463 2013-01-06 21:01:42 <sipa> MC1984: my latest build do
464 2013-01-06 21:02:16 <MC1984> is it one of those ddls i dump into the windows sys folder?
465 2013-01-06 21:02:21 <TD> it should be included
466 2013-01-06 21:02:25 <TD> i guess
467 2013-01-06 21:02:35 <grau> Do not take that for granted. Every unit test we write and every nehaviour test we write cements what bitcoind will have to obey in the next release.
468 2013-01-06 21:02:36 <sipa> it should be linked statically, but apparently it isn't
469 2013-01-06 21:02:44 <MC1984> i wonder why i dont already have it
470 2013-01-06 21:02:50 <sipa> grau: ?
471 2013-01-06 21:02:55 <gmaxwell> grau: and a reimplementation of a test??? even a careful one??? is not the same as a test itself, unless there is some mechenism to show them to be mathmatically identical.
472 2013-01-06 21:03:02 <grau> At some points the tests will force compliance and not a particular implementation, and that is I am after.
473 2013-01-06 21:03:18 <grau> This is why I want tests self contained and plain text
474 2013-01-06 21:03:29 <gmaxwell> grau: The tests MAY NOT disagree with the exiting implementation or the test MUST be changed. Thats my point??? more tests are fantastic but they are not normative and can not be normative.
475 2013-01-06 21:04:12 <grau> I do not want to re-implement tests, only if they are not available in a self-contained format.
476 2013-01-06 21:04:29 <gmaxwell> please stop saying plain text. Your plain text tests are not plain text. They are in a specialzied dialect of english where words care more specific meanings than they do in english. An english reader who reads the test without a solid understanding of the specialized meaning may misunderstand them.
477 2013-01-06 21:04:46 <grau> An other imlementation s too wide.
478 2013-01-06 21:04:54 <grau> Json is also plain text.
479 2013-01-06 21:04:58 <grau> for me
480 2013-01-06 21:05:11 <sipa> well an x86 binary encoded in base64 is also plain test?
481 2013-01-06 21:05:39 <sipa> grau: good, imagine soon someone else starts a new bitcoin node implementation
482 2013-01-06 21:05:57 <sipa> grau: he is as smart as you, and spends as much effort writing unit tests, and verifying them
483 2013-01-06 21:06:17 <sipa> he actually comes up with a very similar system, and unfortunately he never heard about you and your code
484 2013-01-06 21:07:00 <sipa> however he doesn't understand one tiny subtle detail about bitcoin, and his unit tests fail to express this
485 2013-01-06 21:07:26 <sipa> wouldn't it be great if those two systems could test eachother, rather than just validating themself?
486 2013-01-06 21:07:38 <sipa> we are together working on software that needs to interact
487 2013-01-06 21:07:40 <grau> What I ask him to do is what I also do now. Take tests of other systems as captured in self contained manner like the json tests.
488 2013-01-06 21:08:03 <gmaxwell> ACTION hangs head
489 2013-01-06 21:08:32 <sipa> what if he makes a mistake translating the existing tests to his framework?
490 2013-01-06 21:08:34 <grau> gmaxwell: tests become normative
491 2013-01-06 21:08:37 <gmaxwell> grau: his misunderstanding of the system??? or bugs in his code??? may cause him translation of the other tests into actual behavior to be incorrect.
492 2013-01-06 21:09:39 <gmaxwell> grau: the tests are not normative unless they are the implementation people run. In fact, I can argue that we do have a fully normative test??? the reference software. You just need a harness to send conditions through both to verify that you match the fully normative 'test'.
493 2013-01-06 21:09:56 <grau> gmaxwell: it is not black or white
494 2013-01-06 21:10:00 <upb> what is this 'self contained' you keep blabbing about?
495 2013-01-06 21:10:27 <grau> we started with the implementation being the definition since there were no tests at all
496 2013-01-06 21:10:35 <gmaxwell> grau: in a distributed system like bitcoin the behaivor of your peers is what matters for normative behavior. Thats the only test that can really matter. If there is 99% of the network and it's "wrong" ... it's still right.
497 2013-01-06 21:10:41 <upb> the tests you showed are not self contained in anyway, there is a struture to those test 'sentences' and parts of those sentences are implemented in your java tester
498 2013-01-06 21:10:55 <sipa> and nothing will change that - we can add rules, rules which hopefully will be formally defined using tests
499 2013-01-06 21:11:05 <sipa> but nothing can break the rules of the original implementation
500 2013-01-06 21:11:11 <grau> then unit tests were added and I am sure now sipa does not release a code if they do not pass the tests
501 2013-01-06 21:11:19 <gmaxwell> The primary correctness criteria for bitcoin ??? above all other considerations??? is consistency.
502 2013-01-06 21:11:25 <grau> therefore the tests became normative (but not sufficient)
503 2013-01-06 21:11:51 <gmaxwell> (because or a technical level bitcoin is a distributed consensus system??? consistency is its purpose and reason for existing)
504 2013-01-06 21:11:51 <grau> the journey is to raise the bar
505 2013-01-06 21:12:21 <sipa> grau: my question is how you show that your unit tests are correct, except by showing they are at least observably identical to running connected to a reference implementation
506 2013-01-06 21:12:38 <gmaxwell> grau: say that a race condition makes the implementation sometimes fail a test but we release anyways because we don't know this. Then if most people are running that code.. the test is not normative, the behavior is.
507 2013-01-06 21:12:54 <sipa> if everyone has their own unit tests they consider normative, we are not building a single distributed system - we are building several
508 2013-01-06 21:13:26 <grau> I want to share the tests. That is why I separate them from implementation.
509 2013-01-06 21:13:37 <sipa> grau: *great*
510 2013-01-06 21:14:02 <gmaxwell> I couldn't be happier if you both showed your implementation passed bluematt's tester and created a similar tester of your own that bitcoinj and the reference implementation also pass.
511 2013-01-06 21:14:02 <grau> I make them data as far as possible instead of code.
512 2013-01-06 21:14:06 <sipa> that means your unit tests can hopefully at some point easily be used to test other implementation
513 2013-01-06 21:14:17 <sipa> which would be great
514 2013-01-06 21:14:26 <grau> YES
515 2013-01-06 21:14:28 <gmaxwell> (well, I could be happier: I'd be happier if you created your own without looking inside bluematt's and it was also as comprehensive)
516 2013-01-06 21:15:08 <sipa> but it is not, and never will be, normative for anyone
517 2013-01-06 21:15:24 <sipa> it's a compliance test to a part of the system you modeled in tests
518 2013-01-06 21:15:33 <gmaxwell> It can't be normative simply because no one can run it on the network and depend on it.
519 2013-01-06 21:15:41 <grau> we are raising the bar with every test that is portable to ALL implementations
520 2013-01-06 21:15:47 <sipa> grau: agree
521 2013-01-06 21:16:06 <sipa> grau: so why don't you verify your code against both your own tests, and already existing tests, which we know are good?
522 2013-01-06 21:16:22 <grau> See, and all I do with that jbehave is to create portable easy to repeat tests of a higher level than unit
523 2013-01-06 21:16:50 <grau> I do sipa
524 2013-01-06 21:16:53 <gmaxwell> A way of looking at it is that in a distributed consensus system only an implementation can be normative??? it its own tests??? and a "test" is really a harness that allows you to convert any implementation into a test of another implementation by challenging them with behavior you can compare.
525 2013-01-06 21:17:01 <grau> I did verify against bitcoind unit tests
526 2013-01-06 21:17:17 <sipa> grau: but didn't run it connected to that bitcoinj tester
527 2013-01-06 21:17:18 <gmaxwell> s/which we know are good/which we have time and evidence to believe are good but cannot know/ :P
528 2013-01-06 21:17:29 <grau> Not yet
529 2013-01-06 21:17:43 <sipa> if you do that, i have nothing to complain about anymore
530 2013-01-06 21:17:54 <sipa> but it's not the same as trying to capture its behaviour in your own tests
531 2013-01-06 21:18:41 <gmaxwell> grau: it would be more beneficial if you create your test without looking too much inside bluematt's at first??? maybe you will imagine some tests he forgot. You will test with his too so there is no risk of totally forgetting them. Then later pull in replications of his tests.
532 2013-01-06 21:20:57 <gmaxwell> sipa: we should probably merge the pulltester patch behind a hidden switch.
533 2013-01-06 21:21:29 <sipa> gmaxwell: i'd rather spend time on mining a chain for pulltester so it doesn't need a patch :)
534 2013-01-06 21:21:59 <sipa> that makes it more useful for other implementations as well
535 2013-01-06 21:22:02 <gmaxwell> yea, I'm willing to do that??? but I suspect there are still a bunch of missing tests.
536 2013-01-06 21:24:35 <MC1984> welp its slowed down significantly just after 210k
537 2013-01-06 21:24:49 <MC1984> also the sync bar on the bottom is gone even though its still syncing
538 2013-01-06 21:25:18 <gmaxwell> I know when I reviewed it I found a (two?) missing ones, so I might guess that there are a dozen more based on how well I feel I know the more obscure rules and how carefully I checked.
539 2013-01-06 21:26:55 <gmaxwell> I wonder of purecoin had been finished if here would be some crazy machine transformation of the code that could extract a testharness from the implementation.
540 2013-01-06 21:27:06 <grau> MC1984 : whats the speed now ?
541 2013-01-06 21:27:45 <grau> MC1984: please pick a block with a few hundred tx
542 2013-01-06 21:28:10 <grau> and tell me what timing is for input resolution, tx validation and storade
543 2013-01-06 21:28:15 <grau> I mean storage.
544 2013-01-06 21:28:25 <gmaxwell> grau: mc1984 is on a really slow atom, so his results probably don't reflect anything you've tested with.
545 2013-01-06 21:28:27 <MC1984> it was ~10 blocks per second just before 210k and probably avg less than 1 per seconds right after 210k
546 2013-01-06 21:28:39 <MC1984> when the full verify kicked in
547 2013-01-06 21:28:46 <grau> I am sure bitsofproof is slower, just curious how much
548 2013-01-06 21:28:57 <MC1984> gmaxwell no this is on my athlon 2.2ghz box
549 2013-01-06 21:28:59 <grau> yes after 210k please
550 2013-01-06 21:29:24 <MC1984> 1gb rams and a 7200rpm HDD or other
551 2013-01-06 21:29:38 <sipa> grau: on a hexacore machine, with parallel signature checking and hal's optimized ecdsa verification, i've seen close to 0.8ms/txin for validation
552 2013-01-06 21:29:43 <MC1984> its an athlon xp mind, the one before the athlon 64
553 2013-01-06 21:29:55 <MC1984> still i dont consider it slow
554 2013-01-06 21:30:25 <sipa> grau: wait, 0.08
555 2013-01-06 21:30:48 <sipa> that's 0.08ms/txin average clock time (so in practice close to 0.48ms cpu time per txin)
556 2013-01-06 21:31:52 <MC1984> Connect 194 transactions: 984.38ms (5.074ms/tx, 2.912ms/txin)
557 2013-01-06 21:31:58 <MC1984> hows that?
558 2013-01-06 21:32:13 <grau> Thanks, let me compare
559 2013-01-06 21:33:24 <sipa> ^ that's one a single-threaded system i suppose?
560 2013-01-06 21:33:31 <MC1984> athlon xbp barton 2.2ghz
561 2013-01-06 21:33:47 <sipa> signature checks are done inline during connection, so the connect and verify numbers are the same
562 2013-01-06 21:34:59 <grau> how many cores does that have?
563 2013-01-06 21:35:33 <gmaxwell> Thats a single core system.
564 2013-01-06 21:35:46 <gmaxwell> Which I assume is partially why it's so slow.
565 2013-01-06 21:36:40 <MC1984> jesus christ this CPU is ten years old
566 2013-01-06 21:37:10 <MC1984> athlon 54 came out in 2003
567 2013-01-06 21:37:13 <MC1984> 64
568 2013-01-06 21:39:09 <gmaxwell> I'm happy that you have old stuff. :P less things for me to keep. :P
569 2013-01-06 21:40:06 <MC1984> im poor
570 2013-01-06 21:40:24 <MC1984> well not that poor, im just too lazy to research and buy new stuff
571 2013-01-06 21:41:12 <MC1984> i do have acces to an I5 dual core so ill be trying that next
572 2013-01-06 21:42:38 <gmaxwell> the longer you wait the faster it gets.
573 2013-01-06 21:43:48 <MC1984> yeah
574 2013-01-06 21:44:00 <MC1984> ive wanted a new laptop for over two years
575 2013-01-06 21:44:27 <MC1984> theres a thing on the wiki about how the computer market proves bitcoin isnt doomed to a deflationary spiral
576 2013-01-06 21:44:32 <MC1984> im proof that it is lol
577 2013-01-06 21:45:36 <gmaxwell> haha. There is a cute paper showing that under a fixed budget you can complete some big computing tasks earlier by going to the beach for a year and then buying your supercomputer.
578 2013-01-06 21:45:37 <grau> Restarted from 210k to get to the same block. I run on a 4 core Xeon at 2GHz, with bare metal virtualization
579 2013-01-06 21:46:34 <MC1984> lol i cant beleive they got funding for that
580 2013-01-06 21:46:59 <gmaxwell> http://arxiv.org/pdf/astro-ph/9912202.pdf
581 2013-01-06 21:47:38 <MC1984> shit like that makes me think about how close to a singularity we might be
582 2013-01-06 21:47:52 <sipa> which height is it?
583 2013-01-06 21:48:12 <MC1984> The Effects of Moore?s Law and Slacking 1 on Large
584 2013-01-06 21:48:18 <sipa> 210433, it seems
585 2013-01-06 21:49:31 <gmaxwell> MC1984: the singularity will be prevented by slacking, of course. as soon as you predict the massive speed increases you should go to the beach, so tech will never get faster. :P
586 2013-01-06 21:50:29 <MC1984> if the singularity means we all chill out at the beach forever thats fine by me
587 2013-01-06 21:51:12 <MC1984> more likely there wont be any beaches left because it will all be turned into chips, perhaps by our robot overlords against our will
588 2013-01-06 21:53:51 <MC1984> chart y axis "optimal slackage"
589 2013-01-06 21:55:05 <sipa> grau: reindexing to try the same on my system (quad-core i7 laptop)
590 2013-01-06 22:12:18 <grau> 1806 ms total
591 2013-01-06 22:13:11 <grau> sipa: I could not beat that old CPU :)
592 2013-01-06 22:13:24 <grau> Your implementation is fast
593 2013-01-06 22:14:15 <grau> but beat 0.7 bitcoind I think
594 2013-01-06 22:14:16 <sipa> 2013-01-06 23:14:02 - Connect 194 transactions: 10.33ms (0.053ms/tx, 0.031ms/txin)
595 2013-01-06 22:14:19 <sipa> 2013-01-06 23:14:02 - Verify 338 txins: 59.00ms (0.175ms/txin)
596 2013-01-06 22:14:22 <sipa> 2013-01-06 23:14:02 - Connect: 68.36ms
597 2013-01-06 22:14:24 <sipa> 2013-01-06 23:14:02 - Flush 487 transactions: 1.28ms (0.0026ms/tx)
598 2013-01-06 22:14:27 <sipa> 2013-01-06 23:14:02 SetBestChain: new best=00000000000001591e48ca94c53c0e38bf034ca12c1a4a4b60b5e7239142b896 height=210433 work=635359256020621755616 tx=9450213 date=2012-12-01 12:57:05
599 2013-01-06 22:14:47 <sipa> so 70ms total
600 2013-01-06 22:14:59 <grau> impressing
601 2013-01-06 22:15:23 <sipa> well, the ECDSA stuff is someone unfair, as i think it's hard to get that kind of performance in native java code
602 2013-01-06 22:15:27 <sipa> *somewhat
603 2013-01-06 22:16:16 <sipa> BlueMatt spent already a lot of time trying to optimize bouncycastle's ecdsa code
604 2013-01-06 22:16:54 <grau> Yes, 87% of my time is in the verify
605 2013-01-06 22:17:26 <sipa> well, here it spent probably 200ms CPU time on ecdsa, and 12ms on the rest
606 2013-01-06 22:17:55 <sipa> well, 20-40ms of that 200ms is probably script processing outside of actual ecdsa
607 2013-01-06 22:20:02 <grau> Verify 338 txins: 59.00ms : Is this on the wall clock ?
608 2013-01-06 22:20:09 <sipa> yes
609 2013-01-06 22:20:14 <sipa> with 4 threads
610 2013-01-06 22:20:36 <sipa> well, 8 threads, but on a HT cpu
611 2013-01-06 22:21:18 <grau> 8 or 4 do you use?
612 2013-01-06 22:21:24 <sipa> 8
613 2013-01-06 22:21:39 <sipa> but the last 4 don't help much, as expected (but still a bit)
614 2013-01-06 22:21:59 <grau> ok, so its about 200-240ms cpu time, right ?
615 2013-01-06 22:22:23 <sipa> something like that, yes
616 2013-01-06 22:22:24 <grau> sorry, too tired
617 2013-01-06 22:22:35 <sipa> i can try single-threaded to compare
618 2013-01-06 22:22:40 <petertodd> I wonder how many of the trolls on the forums realize that the only reason I write such long, detailed technical replies is because I learn best by explaining things to people...
619 2013-01-06 22:23:23 <grau> its rather 400
620 2013-01-06 22:24:36 <sipa> grau: my current parallel script verification code is far from optimal, and has a lot of overhead and doesn't push all cores to 100%, though
621 2013-01-06 22:25:00 <sipa> a much better implementation is possible, but that requires more refactoring
622 2013-01-06 22:25:09 <grau> ok, I just try to estimate the magnitude of the difference
623 2013-01-06 22:25:35 <sipa> i'm now reindexing with single-threaded code
624 2013-01-06 22:26:02 <sipa> iirc, it's something like 0.5ms/txin on this CPU in single-threaded code
625 2013-01-06 22:28:06 <grau> my on the clock time for verification is 1579 ms, that is about 6000ms CPU (I press them quite high). That would mean that ecdsa code is about 30x faster.
626 2013-01-06 22:29:08 <grau> ok take away 2-3x for java, then about a magnitude
627 2013-01-06 22:29:11 <sipa> basic OpenSSL is around 20%-25% slower than what we have now
628 2013-01-06 22:29:22 <sipa> so most of it is really thanks to OpenSSL
629 2013-01-06 22:31:03 <grau> i will have to counter that with functionality :)
630 2013-01-06 22:31:06 <gmaxwell> It sounded like Jeff's in memory C implementation is much faster than bitcoin-qt/d at least with script validation disabled and without storing undo information.
631 2013-01-06 22:31:21 <grau> but now have to sleep.
632 2013-01-06 22:32:23 <sipa> gmaxwell: yes, it's tempting trying to get closer to that :)
633 2013-01-06 22:32:31 <MC1984> guys why dont you just write a validator in machine code
634 2013-01-06 22:32:34 <MC1984> should be pretty good
635 2013-01-06 22:33:11 <sipa> swainsson: hi there! did I meet you in Reykjavik and London?
636 2013-01-06 22:34:36 <MC1984> actually if bitcoin goes on long enough i bet someone would do that
637 2013-01-06 22:34:42 <MC1984> but it would be closed source
638 2013-01-06 22:34:54 <gmaxwell> sipa: it would at least be interesting to know where the difference comes from. If it's mostly leveldb reads, for example, then more caching might help. If its heap allocator pressure... welll.
639 2013-01-06 22:35:19 <gmaxwell> MC1984: usually writing in assembly/machine code results in slower code.
640 2013-01-06 22:35:33 <sipa> gmaxwell: with several gigabytes of cache, there shouldn't be any leveldb reads at all
641 2013-01-06 22:36:03 <MC1984> thats contrary to my (very limted) knowledge of programming
642 2013-01-06 22:36:16 <gmaxwell> MC1984: modern CPUs are so complex that humans tend to do a poor job of optimizing for them.. on very small segments of straighforward code a human can beat a compiler but it doesn't generalize.
643 2013-01-06 22:36:34 <sipa> gmaxwell: though the current cache implementation is stupid (=always assume all entries are dirty, and flush the entire set when it gets too big)
644 2013-01-06 22:37:09 <MC1984> wouldnt ecdsa verification count as simple enough to write machine code for? It just does the same smallt hing over and over again right
645 2013-01-06 22:37:17 <sipa> i've been wanting to replace it with a nice MRU cache
646 2013-01-06 22:37:24 <gmaxwell> MC1984: e.g. so a hand written ecdsa implementation may indeed be good (though not as good as algorithimic improvements) but a whole validation engine is less likely.
647 2013-01-06 22:37:44 <gmaxwell> MC1984: yea, ecdsa itself sure. I thought you meant the whole block validation code.
648 2013-01-06 22:37:57 <gmaxwell> MC1984: if someone wanted to spend a bunch of effort making ecdsa faster they'd write a gpu version.
649 2013-01-06 22:38:00 <sipa> gmaxwell: using a hashmap instead of std::map for the cache would also speed things up
650 2013-01-06 22:38:24 <sipa> gmaxwell: or an FPGA version... or wait, an ASIC!
651 2013-01-06 22:38:33 <MC1984> shit thats a good point, why not parralellise it on the GPU instead
652 2013-01-06 22:39:01 <swainsson> Sipa ... Yes this is Swain from Reykjavik
653 2013-01-06 22:39:03 <gmaxwell> sipa: could just make a fixed size hashtable and resolve collisions by overwriting.
654 2013-01-06 22:39:06 <sipa> MC1984: because making bitcoind do OpenCL stuff would be a dependency disaster :)
655 2013-01-06 22:39:25 <sipa> gmaxwell: nice idea
656 2013-01-06 22:39:29 <MC1984> thats a shame
657 2013-01-06 22:40:05 <gmaxwell> OpenCL seems to not be so bad dependency wise.. though if we had something like that I'd probably want to move the ecdsa checker to another process. Then you could even have an MPI one that shared it across multiple computers. :P
658 2013-01-06 22:40:33 <MC1984> oh well, im sure gpu compute support will be standard enough in future
659 2013-01-06 22:40:58 <sipa> swainsson: cool; how are you?
660 2013-01-06 22:41:01 <gmaxwell> MC1984: on modern cpus it's quite fast now without any fancy gpu stuff though, so.. I dunno if it matters much.
661 2013-01-06 22:41:25 <MC1984> yeah its an optimisation for the future for when the next dice comes around.......
662 2013-01-06 22:41:51 <MC1984> and the block limit is lifted i suppose, if ever
663 2013-01-06 22:41:53 <swainsson> Hey Sipa, pretty good, how are you?
664 2013-01-06 22:42:12 <sipa> good, but tired... just drove 650km from belgium to switserland :)
665 2013-01-06 22:42:51 <swainsson> Painful to drive, but at least there is no time difference!
666 2013-01-06 22:42:57 <sipa> yeah
667 2013-01-06 22:43:05 <swainsson> I'm curious about a comment I read somewhere about moving away from Berkeley database, didn't see any further remarks about it anywhere, wondering if anyone could explain briefly.
668 2013-01-06 22:43:23 <sipa> sure>
669 2013-01-06 22:43:33 <sipa> (if you can cite the comment...)
670 2013-01-06 22:44:38 <swainsson> https://github.com/gavinandresen/bitcointools "These tools are becoming obsolete as we move away from using Berkeley DB in Bitcoin-Qt/bitcoind."
671 2013-01-06 22:45:05 <sipa> yes, 0.8 will use a new database layout, built on LevelDB
672 2013-01-06 22:45:08 <sipa> instead of BDB
673 2013-01-06 22:45:40 <sipa> gmaxwell: what to do with a dirty cache entry that is collided with?
674 2013-01-06 22:46:25 <sipa> gmaxwell: you can't just write it to the backend database (as that must be done atomically with all changes together), and you can't just move it to a queue of future writes, as that will make re-reads returns the old value
675 2013-01-06 22:46:42 <sipa> so you need to deal with reading back collided entries anyway
676 2013-01-06 22:47:23 <swainsson> How soon is that may I ask? Just to get some dirt under the fingernails I was thinking of messing around with gavin's tools but if the levelDB is around the corner perhaps that' s not worth spending time on ..
677 2013-01-06 22:47:44 <sipa> a few weeks at least
678 2013-01-06 22:48:35 <sipa> swainsson: actually, i explained you part of the new database system in the Kex :)
679 2013-01-06 22:50:23 <cosurgi> ;;bc,stats
680 2013-01-06 22:50:25 <cosurgi> ;bc,stats
681 2013-01-06 22:50:25 <gribble> Current Blocks: 215478 | Current Difficulty: 2979636.6169381 | Next Difficulty At Block: 215711 | Next Difficulty In: 233 blocks | Next Difficulty In About: 1 day, 12 hours, 34 minutes, and 5 seconds | Next Difficulty Estimate: 3236645.20832965 | Estimated Percent Change: 8.62550117456
682 2013-01-06 22:50:37 <swainsson> Will the data schema (if that's the word for it) change materially ? You may have, but I was very new to it then and didn't know much c++ ... !!! It was like an ancient greek text to me, you gave me a synopsis and now I can at least spell some of the words!
683 2013-01-06 22:50:47 <gmaxwell> sipa: darn, didn't think about dirty entries.
684 2013-01-06 22:51:11 <sipa> swainsson: yes, very much
685 2013-01-06 22:51:28 <sipa> it's not comparable to what was stored in the BDB databases
686 2013-01-06 22:52:06 <swainsson> Ok, so I'll leave Gavin's tools alone ...
687 2013-01-06 22:54:25 <swainsson> Sipa and others: I'll leave you guys to your discussion for the time being. Thanks Sipa! Over and out!
688 2013-01-06 22:54:37 <sipa> no need to leave!
689 2013-01-06 22:54:39 <sipa> dang...
690 2013-01-06 22:56:42 <sipa> gmaxwell: i've pretty much thought it out, with a map whose values contain pointers to eachother to form 3 doubly-linked lists (one dirty, one nondirty, and one 'deleted')
691 2013-01-06 22:57:05 <sipa> gmaxwell: but haven't gotten to implementing it yet
692 2013-01-06 23:34:09 <BlueMatt> ok...so major issue with how the bloom filters are matched...we (well...ok I) hugely underestimated how quickly the bloom filter fp rate would balloon as soon as download starts and you get the first few false positives which add the tx hash to the filter....
693 2013-01-06 23:35:40 <sipa> that's bad news
694 2013-01-06 23:36:08 <sipa> so it means clients are essentially required to track the size of the remote filter, and reset the bloom filter occasionally
695 2013-01-06 23:36:55 <sipa> maybe clients can ask to receive a special message when a certain expect fp% is reached in the filter?
696 2013-01-06 23:37:31 <sipa> i hate adding even more p2p messages though
697 2013-01-06 23:51:39 <BlueMatt> I was thinking of re-sending the filter every N blocks (say, 2000)
698 2013-01-06 23:51:52 <BlueMatt> having the "server" keep track of # elements added to the filter since receiving it seems ok...though I agree adding yet another message is a pain
699 2013-01-06 23:55:07 <sipa> the client cannot be expected to make a good guess about the speed at which the server-side filter is degrading
700 2013-01-06 23:55:12 <sipa> while the server knows exactly
701 2013-01-06 23:55:19 <BlueMatt> the client does too
702 2013-01-06 23:55:31 <BlueMatt> for each txn it receives, it knows the server added one item
703 2013-01-06 23:55:33 <BlueMatt> (or is it 2?)
704 2013-01-06 23:55:44 <sipa> one, iirc
705 2013-01-06 23:55:53 <sipa> ok; that's true
706 2013-01-06 23:56:46 <sipa> the server knows slightly better (=the % of bits set in the filter), but assuming no collision attack against the hash functions, the client does indeed have a very good estimate
707 2013-01-06 23:58:01 <BlueMatt> not quite, the client actually knows better (you can only accurately calculate fp rate if you know the exact # elements)
708 2013-01-06 23:58:28 <BlueMatt> still...its kinda ugly to just re-send the filter every N blocks....
709 2013-01-06 23:59:05 <BlueMatt> i was hoping to point it out and have someone point out the obvious solution (tm) to which i would facepalm and go implement