1 2015-03-13 01:10:41 <kuthixa> Hi. I'm trying to build cpuminer from source on windows in MinGW and I'm getting an error "error: Missing required libcurl >= 7.15.2" that I can't figure out how to get by. I've downloaded Curl and tried using LDFLAGS="-L..." to point it at the correct directory but it's not working. I was wondering if anyone here might be able to help me out.
2 2015-03-13 01:13:45 <Luke-Jr> kuthixa: cpuminer is AFAIK not maintained in years; BFGMiner is the most current fork, and has windows build instructions (albeit not regularly tested)
3 2015-03-13 01:21:53 <kuthixa> Thank you Luke-Jr. I see your Github on BFGMiner and it looks very nice. It has good documentation right on the front page. I might play around with this also but at the moment I'm trying to get cpuminer building.
4 2015-03-13 01:22:23 <kuthixa> I'll look through this to see if anything can help. Any advice?
5 2015-03-13 01:22:23 <Luke-Jr> kuthixa: the same instructions may very well help with cpuminer
6 2015-03-13 01:22:29 <Luke-Jr> see windows-build.txt
7 2015-03-13 01:22:42 <Luke-Jr> it won't be exactly the same, but at least curl should fit
8 2015-03-13 01:23:11 <kuthixa> Yes, this is a big help. I think this will help me get by this issue. Thank you very much!
9 2015-03-13 01:23:24 <Luke-Jr> np, I'm just now left hoping you're not using it for malware :x
10 2015-03-13 01:23:55 <kuthixa> Lol. No, I'm not using it for anything illegal or deceptive. Though, I suppose that's what I would say either way.
11 2015-03-13 01:24:29 <Luke-Jr> yeah, just the only reason I could think of for trying to build cpuminer these days
12 2015-03-13 01:24:39 <Luke-Jr> my imagination sucks? :P
13 2015-03-13 01:26:18 <kuthixa> I think that's a very logical conclusion. I don't fault you for it.
14 2015-03-13 01:32:23 <kuthixa> Wow, BFGMiner looks really nice. Thanks for pointing me at this.
15 2015-03-13 05:47:47 <aburan28> are there any current proposed changes to the wire protocol or research that needs to be done into perfecting the networking architecture? I have time on my hand for the next few weeks and want to help out in anyway I can but don't know where to start
16 2015-03-13 05:49:45 <gmaxwell> aburan28: it's a bit difficult to dive in cold fast; how closely have you been following things? whats your background?
17 2015-03-13 05:58:56 <aburan28> I have read through most of the research papers on bitcoin. I have been following this currency since early 2011, and started writing some scripts in python to interact with the wire protocol successfully about a year ago and worked on a epoll networking server in python (I have since moved on to C++) . I have the p2p protocol memorized lets put it that way
18 2015-03-13 06:04:12 <gmaxwell> K. Well the most interesting things right now around the p2p protocol are probably eliminating varrious fingerprinting attacks, reducing sybil vulnerability in varrious ways, supporting authenticated connections, and creating p2p support for syncing up from partially pruned peers (e.g. communicating which ranges of a sparse block set the peers have). Seperately, there is a lot of oppturnity to wo
19 2015-03-13 06:04:18 <gmaxwell> rk on alternative transport protocols. Bluematt has created the relay network client, which is an alternative protocol which could be made more efficient/less buggy/more complete (you can't sync a new node over it). It would be interesting if work were done on implementations that used iblt or block network coding or just were connectionless. these alternative protocol projects have the benefit of
20 2015-03-13 06:04:24 <gmaxwell> being loosly coupled.
21 2015-03-13 06:11:24 <aburan28> i also have resources at my disposal such as 1 Gbps link and a few servers that I could put to good use
22 2015-03-13 06:23:39 <Farbdose> Hi everyone (i hope thats the right channel) - My homeserver(running bitcoind) just finished reindexing (i needed txindex=1 for electrum server) ... Took him about 5 days... Unfortunally i forgot to enable the swap file and the updateTip crashed with an out of memory at 99% and now the database is messed up and i want to ask you: is there another way to repair the database instead of reindexing again?
23 2015-03-13 06:24:28 <gmaxwell> no.. but 5 days to reindex? what hardware is this?! what os? Thats very slow.
24 2015-03-13 06:26:25 <wumpus> my ARM box takes about that long, on dbcache=4 :-)
25 2015-03-13 06:27:43 <Farbdose> Thats arch linux on a raspberry pi 2 xD
26 2015-03-13 06:27:57 <wumpus> no, you can't 'repair' the database meaningfully, you can copy it from another system or reindex
27 2015-03-13 06:32:14 <Farbdose> Hm ok thx, oh just found out that electrum is way to resource heavy -> i can use my txindex=0 backup
28 2015-03-13 06:53:11 <robbak> Luke-Jr: Thanks for the heads up: but regex is a write-only language -- what problems was that causing? The FreeBSD build isn't currently patching anything there.
29 2015-03-13 06:54:18 <robbak> Only patch we are doing now is adding "define __STDC_LIMIT_MACROS" so the compiler has access to -- LIMIT_MAX, if I recall correctly.
30 2015-03-13 08:18:39 <wumpus> cfields: hey, does statically linking libstdc++ means we can use c++11 now? :-)
31 2015-03-13 08:20:05 <gmaxwell> we'd probably want to advance the builder compiler, what are the gitian builds using right now?
32 2015-03-13 08:21:36 <sipa> gitian is ubunti 12.04 iirc?
33 2015-03-13 08:47:57 <m477> is someone familiar with writing trading bots?
34 2015-03-13 08:57:10 <wumpus> sipa: yes, current gitian image is 12.04 64-bit
35 2015-03-13 09:00:39 <wumpus> I don't know what subset of c++11 it supports
36 2015-03-13 09:01:09 <wumpus> at some point switching to 14.04 LTS makes sense
37 2015-03-13 09:02:43 <gmaxwell> getting onto a newer compiler will be good, GCC quality seems to be more or less monotonicly improving the last couple years.
38 2015-03-13 09:05:04 <wumpus> could do some experiments with gitian w/ 14.04. The only possible problem I see (now that libstdc++ is static) is GLIBC compatibility, but we may be able to work around that with more compatibility wrappers
39 2015-03-13 09:15:05 <wumpus> bleh, make-base-vm fails for trusty
40 2015-03-13 09:15:45 <Chillum> what linux flavor is the simplest to setup bitcoind on?
41 2015-03-13 09:17:41 <sipa> Chillum: ubuntu
42 2015-03-13 09:19:15 <wumpus> https://github.com/devrandom/gitian-builder/issues/83
43 2015-03-13 09:19:44 <wumpus> yes, most certainly ubuntu
44 2015-03-13 09:22:02 <m477> why trading bots are not popular?
45 2015-03-13 09:22:12 <sipa> m477: offtopic here
46 2015-03-13 09:22:25 <m477> sipa: ?
47 2015-03-13 09:22:46 <sipa> m477: this is about developing bitcoin's system itself, not trading systems
48 2015-03-13 09:23:32 <sipa> wumpus: how hard would it be to add a gcc/libstdc++/libc config to depends, even just for 64-bit linux now?
49 2015-03-13 09:23:47 <sipa> wumpus: so we don't need gitian anymore for that?
50 2015-03-13 09:24:33 <wumpus> sipa: building toolchains is something of a dark art, so I'd not recommend doing that directly. Using crosstool-ng it'd be pretty easy though.
51 2015-03-13 09:24:56 <aschildbach> Hmm, I have a bitcoind which is connect to ~10 nodes but doesn't fetch blocks. It's sitting at block 343405.
52 2015-03-13 09:25:05 <aschildbach> According to debug.log, it doesn't do anything
53 2015-03-13 09:25:15 <gmaxwell> hm? building GCC itself is a cinch, though the binutils and such for cross building can be a pain.
54 2015-03-13 09:25:15 <m477> sipa: "Development of Bitcoin; This is for discussion about the Bitcoin network and reference software between developers." I thought trading bots include, if not then where can talk about it?
55 2015-03-13 09:25:15 <sipa> aschildbach: does getpeerinfo list any infloght blocks?
56 2015-03-13 09:25:33 <aschildbach> getpeerinfo yields whitelisted:false for all of the nodes
57 2015-03-13 09:25:40 <wumpus> gmaxwell: it's a multi-stage pain
58 2015-03-13 09:25:42 <sipa> m477: sorry, no clue, but definitely not here
59 2015-03-13 09:26:00 <gmaxwell> aschildbach: how long has it been like that? and can you privately 0bin me your getpeer info?
60 2015-03-13 09:26:06 <gmaxwell> aschildbach: ans what version bitcoind is it?
61 2015-03-13 09:26:20 <aschildbach> sipa: the inflight arrays are empty for all of the peers
62 2015-03-13 09:26:34 <gmaxwell> okay so 0.10+
63 2015-03-13 09:26:39 <sipa> aschildbach: ok, can you paste the outout of getchaintips ?
64 2015-03-13 09:26:46 <aschildbach> gmaxwell: I think about 15 minutes now. It's 0.10 but bluematt's Ubuntu version.
65 2015-03-13 09:27:05 <sipa> there is a 20 minute block download timeout
66 2015-03-13 09:27:13 <gmaxwell> could just be stuck on a single broken peer and hasn't hit the timeout yet (it's 20 minutes)
67 2015-03-13 09:27:16 <sipa> though the usual syncup mechanism is mlrr aggressive
68 2015-03-13 09:27:20 <sipa> ;;blovks
69 2015-03-13 09:27:21 <gribble> Error: "blovks" is not a valid command.
70 2015-03-13 09:27:25 <sipa> ;;blocks
71 2015-03-13 09:27:27 <gribble> 347416
72 2015-03-13 09:27:29 <gmaxwell> "blocks" : 347416,
73 2015-03-13 09:28:58 <aschildbach> sipa: chaintips is like this: http://pastebin.com/3RMf7nHY Sorry it's not the full list
74 2015-03-13 09:29:25 <sipa> the top is the most interesting part :)
75 2015-03-13 09:29:46 <gmaxwell> "oops"
76 2015-03-13 09:32:23 <aschildbach> sipa: sorry, another try, this time top of list: http://pastebin.com/PmK0Hj5q
77 2015-03-13 09:33:43 <sipa> hmm you don't even have any better headers
78 2015-03-13 09:33:47 <sipa> that's strange
79 2015-03-13 09:34:03 <sipa> wait until the next block announcement?
80 2015-03-13 09:34:08 <sipa> ;;blocks
81 2015-03-13 09:34:10 <aschildbach> Ok I will wait
82 2015-03-13 09:34:10 <gribble> 347416
83 2015-03-13 09:34:59 <aschildbach> I get some errors in the log: 2015-03-13 09:34:56 ERROR: AcceptToMemoryPool : nonstandard transaction: non-final
84 2015-03-13 09:35:13 <wumpus> those are normal (and shouldnt be errors)
85 2015-03-13 09:35:16 <aschildbach> But that doesn't seem to be worrying to me
86 2015-03-13 09:35:30 <jcorgan> is there a way to cause bitcoind to close and reopen debug.log (as part of log rotation), without restarting it?
87 2015-03-13 09:35:40 <gmaxwell> jcorgan: kill -HUP
88 2015-03-13 09:35:52 <jcorgan> thanks
89 2015-03-13 09:36:08 <gmaxwell> (yea, amazing, it actually follows a few normal unix conventions now! :) )
90 2015-03-13 09:37:22 <aschildbach> Just to complete the picture: I started the node at 8:48 (server time) and it went up with block 343405 being most recent:
91 2015-03-13 09:37:24 <aschildbach> -13...2015-02-14)
92 2015-03-13 09:37:24 <aschildbach> 2015-03-13 08:48:35 LoadBlockIndexDB: last block file info: CBlockFileInfo(blocks=81, size=35499378, heights=343325...343405, time=2015-02
93 2015-03-13 09:37:48 <aschildbach> Ah now I got UpdateTip messages.
94 2015-03-13 09:38:20 <sipa> aschildbach: did it just complete syncing from scratch?
95 2015-03-13 09:38:25 <aschildbach> ;;blocks
96 2015-03-13 09:38:28 <gribble> 347418
97 2015-03-13 09:38:38 <aschildbach> No its an old install
98 2015-03-13 09:38:44 <aschildbach> Like 1 year old or so
99 2015-03-13 09:39:48 <aschildbach> Ok its downloading now. It just needed a block to arrive to get started.
100 2015-03-13 09:41:51 <Chillum> sipa: most recent LTS version I suppose?
101 2015-03-13 09:42:15 <jcorgan> i've re-setup my Tor-only node and after 2 days it is up to 70 connections. This is a lot higher than it was when I stopped it a few months ago after steady decline. But, those logs...:)
102 2015-03-13 09:42:18 <wumpus> Chillum: >=12.04
103 2015-03-13 09:42:24 <Chillum> thanks
104 2015-03-13 09:50:31 <bengt__> do you get less or more garbage nodes on tor?
105 2015-03-13 09:50:45 <bengt__> things like bitnodes, coinalyzer and whatever all the crap is
106 2015-03-13 09:54:53 <wumpus> much less
107 2015-03-13 09:55:11 <wumpus> holds for ipv6 too
108 2015-03-13 09:55:41 <bengt__> maybe I should just disable ipv4 ;P
109 2015-03-13 09:55:54 <sipa> -limitnet=iov4
110 2015-03-13 09:55:57 <sipa> iov4
111 2015-03-13 09:56:05 <sipa> iPv4
112 2015-03-13 10:52:04 <wallet42> what are the main problems/discussions of utxo commitments right now? getutxosetinfo can give me a hash⦠is there a github thread?
113 2015-03-13 10:54:23 <wumpus> without maintaining an updatable merkle-ized tree structure, it is expensive to compute, and needs a pass over the entire utxo set
114 2015-03-13 10:55:37 <wumpus> there have been discussions about it on the mailing list. Not on github, no one ever provided a concrete implementation yet
115 2015-03-13 12:00:40 <jtimon> wumpus any reason not to merge 5889 ?(your branch 2015_03_fix_base58_testss)
116 2015-03-13 12:06:09 <wumpus> jtimon: let's see
117 2015-03-13 12:09:32 <wumpus> should not be necessary anymore
118 2015-03-13 12:11:03 <wumpus> #5883 accomplishes the same thing, but cleaner and more thorough
119 2015-03-13 12:12:57 <wumpus> it also fixes the problem that some tests were writing to debug.log
120 2015-03-13 12:19:34 <jtimon> ok, great, let me check with 5696
121 2015-03-13 13:36:38 <jonasschnelli> sipa, "Segmentation fault" on ./dnsseed does this happen often? What approach is recommended to watch-and-restart? DNS dig check script and ./dnsseed restarting?
122 2015-03-13 13:37:03 <sipa> jonasschnelli: ugh, shouldn't happen
123 2015-03-13 13:37:08 <sipa> unless you have not enough memory
124 2015-03-13 13:37:19 <sipa> mine uses 675 MiB ram these days ;s
125 2015-03-13 13:37:51 <jonasschnelli> i have a total of 16GB on that machine, should be sufficient.
126 2015-03-13 13:38:04 <jonasschnelli> theres also a bitcoind running on the same machine
127 2015-03-13 13:38:05 <sipa> can you run in gdb and see where it crashes?
128 2015-03-13 13:38:18 <sipa> mine usually stays up for months at a time...
129 2015-03-13 13:38:25 <jonasschnelli> was crashing during the night... but yes. let me continue running it over gdb
130 2015-03-13 13:38:40 <jonasschnelli> but i assume you have kind of a check-and-restart script?
131 2015-03-13 13:39:14 <sipa> eh no :)
132 2015-03-13 13:41:41 <jonasschnelli> okay... then lets hope the seeds don't go down simultaneous... :)
133 2015-03-13 14:57:23 <Riiume> hello, i'm a guy who helps fund bitcoin projects (dark wallet and open bazaar to date)
134 2015-03-13 14:59:18 <Riiume> I am concerned about the latest "Sibil" attacks (https://bitcointalk.org/index.php?topic=978088.msg10756505#msg10756505). I would be willing to kick-in $75/month (along with any co-sponsors) to a credible effort to enhance privacy of BTC at protocol level, increase fungibility of BTC, and prevent such attacks in the future.
135 2015-03-13 15:04:37 <akstunt600> thx Riiume I think I will start whitelisting IPs for my node or something. I should make a tool that analyzes and allows approval of certain node types.
136 2015-03-13 15:05:11 <akstunt600> I think that would be really handy and useful for both increasing network reslience and efficiency for those that choose to "optimize"
137 2015-03-13 15:05:37 <akstunt600> Any thoughts? I have been thinking about this alot the past 2 months.
138 2015-03-13 15:07:28 <Riiume> akstunt600, my main objection to that idea is that reduces fungibility of the network (by classifying nodes)
139 2015-03-13 15:07:48 <akstunt600> Yes, but I would argue this would only help, and not hurt.
140 2015-03-13 15:08:05 <sipa> Only indirectly.
141 2015-03-13 15:08:14 <Riiume> do you think it could become a tool that many other node operators could put into operation?
142 2015-03-13 15:08:14 <sipa> There is no such thing as 'fungibility of the network'.
143 2015-03-13 15:08:19 <akstunt600> Clients like bitcoinj and stuff could use a tool like this too. For this reason tho i think keeping such a tool seperate from core would be a requirement
144 2015-03-13 15:08:35 <akstunt600> Thx just what i was thinking sipa
145 2015-03-13 15:08:36 <sipa> The property you want is fungible BTC (the currency).
146 2015-03-13 15:08:40 <Riiume> @sipa: concealment and obfuscation of IPs
147 2015-03-13 15:08:48 <akstunt600> meh
148 2015-03-13 15:08:50 <sipa> So use Tor for submitting transations.
149 2015-03-13 15:09:02 <Riiume> my understanding is that Tor is compromised
150 2015-03-13 15:09:18 <akstunt600> I am only interested in increasing efficiency and speed of network and this would help a tad i believe
151 2015-03-13 15:09:33 <sipa> aschildbach: efficiency and speed usually come at the cost of privacy
152 2015-03-13 15:09:36 <sipa> eh, akstunt600
153 2015-03-13 15:09:37 <Riiume> it's a nice project. Do you have a page for it or a Git repo?
154 2015-03-13 15:10:00 <akstunt600> No, just thought of it now
155 2015-03-13 15:10:21 <sipa> Riiume: well, then use I2P or cjdns or some other network that provides a form of sender obscurity
156 2015-03-13 15:10:33 <sipa> Riiume: the Bitcoin P2P network does not and should not
157 2015-03-13 15:10:59 <sipa> (doesn't mean we can't take other efforts to improve network security, but there is no full solution)
158 2015-03-13 15:11:13 <akstunt600> Should be super simple to make, defaults to 8333 and pulls getperrinfo, and provides allows a filter and add to .conf file for allowed or blocked ips or w/e
159 2015-03-13 15:12:23 <sipa> akstunt600: that's not a full solution; IP addresses are cheap
160 2015-03-13 15:12:24 <akstunt600> I think its that simple right? I guess i could make this in bash/bat or something and put on github and see what happens
161 2015-03-13 15:12:44 <sipa> if you start blocking based on IP address, such nodes will just start using others
162 2015-03-13 15:12:47 <akstunt600> Yeah true, just to get started, My hands are full right now with exchange engine stuffs.
163 2015-03-13 15:13:11 <akstunt600> Yup, yup, Wish i had time to make a robust solution
164 2015-03-13 15:14:22 <sipa> the only robust solution is not exposing your IP address when sending a transaction :)
165 2015-03-13 15:14:49 <akstunt600> hehehe, i mean just for detecting nodes. Im fine with current annonmity
166 2015-03-13 15:14:59 <sipa> 'detecting nodes' ?
167 2015-03-13 15:15:19 <akstunt600> Yeah, like i dont want some of the peers that connect to my client
168 2015-03-13 15:15:31 <akstunt600> They slow me down or try to submit bad transactions
169 2015-03-13 15:15:33 <Riiume> okay, maybe the IP address obfuscation isn't as important as i thought, but what about
170 2015-03-13 15:15:46 <Riiume> the issue of transaction anonymity?
171 2015-03-13 15:16:02 <akstunt600> Some just sit there and provide no value to me expcept slowing my fast node down
172 2015-03-13 15:17:07 <akstunt600> So i want to make a tool that makes it easy to blacklist or whitelist good/bad nodes
173 2015-03-13 15:17:42 <luke-jr_> robbak: just wanted to be sure BSD sed handled it correctly
174 2015-03-13 15:17:46 <akstunt600> I think with more companies making btc clients that dont relay or relay malformed shit this might be a pretty important thing..
175 2015-03-13 15:17:57 <sipa> Riiume: how do you expect to have transaction anonimity without obscuring your IP address?
176 2015-03-13 15:18:09 <akstunt600> Hehehe
177 2015-03-13 15:18:13 <sipa> please take this to #bitcoin
178 2015-03-13 15:19:54 <Riiume> not perfect anonymity, but some form of transaction mixing
179 2015-03-13 15:20:19 <Riiume> to make the cost of linking transactions and figuring out a person's behavior patterns is prohibitively high
180 2015-03-13 15:20:56 <akstunt600> Well i mean sounds like you are talking about "beyond reasonable doubt" scenarios
181 2015-03-13 15:21:23 <akstunt600> I mean you can not count on bitcoin for this, you must take at least some care to ensure levels of deniability
182 2015-03-13 15:29:09 <Riiume> anyway, won't clog up this channel, but if there are any devs serious about reforming BTC privacy and have a plan, I might be willing to send money your way; contact me @ saturnalia@bitmessage.ch, use my PGP key for all communications: http://pastebin.com/SQsmSTD8
183 2015-03-13 15:33:22 <wumpus> yes the two issues of mitigating sybil attacks, and increasing transaction privacy should be regarded separately, one won't help with the other
184 2015-03-13 15:59:20 <cfields> wumpus: fwiw, I tested a while back with gcc 4.9 (i believe) and added the necessary glibc compat
185 2015-03-13 15:59:58 <cfields> wumpus: also, i considered raising c++11 in the openssl/random replacement discussion, but i figured I'd be flamed to death :)
186 2015-03-13 16:00:59 <jonasschnelli> i'm facing new linker warnings: "ld: warning: direct access in QMetaTypeIdQObject<QWidget*, true>::qt_metatype_id()"
187 2015-03-13 16:01:04 <jonasschnelli> (and more)
188 2015-03-13 16:01:27 <cfields> jonasschnelli: since the qt5 change?
189 2015-03-13 16:18:20 <Chillum> I am working on a tool that keeps a tally of all balances by applying the deltas from every block to a mysql database. I just realized what I am doing does not take into account re-orgs. How does the client handle this? If I recorded a list of all the deltas from the last 100 block and then played them backwards on re-org to the fork point would that work?
190 2015-03-13 16:19:12 <sipa> yes, you need to be able to roll back on reorg
191 2015-03-13 16:21:21 <Chillum> Good, now I just need to figure out how to detect them
192 2015-03-13 16:21:29 <sipa> that's not how bitcoin core's wallet deals with it, though
193 2015-03-13 16:21:33 <sipa> but it should work
194 2015-03-13 16:21:46 <Chillum> re-orgs just turned my really simple idea into something much more complicated
195 2015-03-13 16:21:48 <sipa> (it computes balances on the fly)
196 2015-03-13 16:22:00 <Chillum> though it is mostly for stats, so I could just run it X blocks behind the chain to avoid re-orgs
197 2015-03-13 16:23:23 <Chillum> ACTION once again takes a naive approach
198 2015-03-13 16:25:23 <jonasschnelli> cfields, no just since today. Didn't had a close look at it...
199 2015-03-13 17:21:55 <maaku> gah, relative nLockTime requires access to the UTXO inside EvalScript. ugly...
200 2015-03-13 17:28:01 <Luke-Jr> indeed
201 2015-03-13 17:34:17 <ajweiss> cfields: did you find anything when you were playing with ebs full disk during sync stuff?
202 2015-03-13 17:34:39 <cfields> ajweiss: a whole bunch of rabbit holes...
203 2015-03-13 17:34:49 <cfields> spent all night chasing ghosts
204 2015-03-13 17:34:57 <ajweiss> oof
205 2015-03-13 17:36:16 <cfields> ajweiss: at one point I was convinced that nLastBlockFile was the problem, but it seems to work for twisty reasons
206 2015-03-13 17:37:15 <cfields> though it does appear as though it's possible to write to not-the-last block file after a re-index, which is not exactly obvious
207 2015-03-13 17:38:47 <ajweiss> the timestamps would bear that out though
208 2015-03-13 17:39:21 <cfields> yea. possible, but incredibly unlikely
209 2015-03-13 17:39:32 <cfields> and even so, from what i can see, it would be a valid write
210 2015-03-13 17:40:04 <ajweiss> i've been digging through LoadExternalBlockFile() looking for a way for it to skip ahead past valid blocks as if they were corruption and then index later valid blocks
211 2015-03-13 17:41:33 <cfields> well if that's what you're looking into, the out-of-order ordering would have something to do with it. the values would be too coincidental otherwise
212 2015-03-13 17:42:46 <ajweiss> yeah i thought about that, maybe magics being built on block boundaries now that they're randomized, but i don't think that's possible since every block starts with one
213 2015-03-13 17:44:42 <ajweiss> other thoughts were some kind of parse failure that would send the scan pointer into the body of a block, that may contain a magic and a length that would then advance the scan pointer a good bit
214 2015-03-13 17:45:10 <ajweiss> but afaik, the scan pointer doesn't advance by large chunks unless an actual block is deserialized afterwards
215 2015-03-13 17:46:30 <ajweiss> LoadExternalBlockFile() does play a little fast and loose though such that it may complain if things don't look right, but it does keep going
216 2015-03-13 17:47:27 <cfields> i don't think there are problems with the pointer advancing, or that it's necessarily re-writing block-data in the same run...
217 2015-03-13 17:48:24 <cfields> my current theory is that a file is written, bitcoind is stopped, the pointer is re-wound to over-write the data that didn't make it into the db, and somehow the boundaries don't align
218 2015-03-13 17:48:42 <cfields> in other words: i think the duplicated data is actually left-over from a previous run
219 2015-03-13 17:48:49 <ajweiss> no i think so too
220 2015-03-13 17:49:21 <ajweiss> but the missing blocks have pointers that are past the duplicated stuff, implying they were also part of the previous run
221 2015-03-13 17:49:58 <cfields> in that case, as a safety mechanism, it may make sense to truncate the last file before our first write
222 2015-03-13 17:50:06 <cfields> right
223 2015-03-13 17:50:13 <ajweiss> | good | duplicated unindexed cutoff | duplicated indexed and missing overlap | good
224 2015-03-13 17:51:45 <cfields> if we could prove that that's the source of the problem, i think truncating would fix it in all cases, without necessarily finding the cause of the overlap?
225 2015-03-13 17:52:34 <ajweiss> well i think your lagging write pointer fix probably fixes it anyway
226 2015-03-13 17:53:34 <ajweiss> but truncating the last block wouldn't hurt
227 2015-03-13 17:53:37 <cfields> so you think it just boils down to *some* garbage write (filesystem/driver/hardware issue) followed by a reindex in all cases?
228 2015-03-13 17:55:02 <ajweiss> that's tough to say, my view is largely constrained by focusing on the one datadir i have
229 2015-03-13 17:56:21 <maaku> BlueMatt: actually relative nLockTime has all sorts of oddities. for example, you don't know whether a tx is ready to go in the mempool until you know its utxos...
230 2015-03-13 17:56:39 <cfields> ok. I'm not quite ready to say that yet either. The fact that there are so many similar aws cases is pretty damning imo.
231 2015-03-13 17:58:44 <ajweiss> yeah i agree.
232 2015-03-13 17:59:35 <ajweiss> i tried banging on ebs with a stress script that loosely mirrors bitcoind's fs patterns during sync, but no dice.
233 2015-03-13 18:00:22 <ajweiss> i also wonder if maybe there's some common property with respect to network connectivity that might be triggering some yet unknown bug during sync.
234 2015-03-13 18:01:26 <ajweiss> but i ran a bunch a lot of full syncs on aws and none of them produced broken block files.
235 2015-03-13 18:15:17 <cfields> same
236 2015-03-13 18:16:47 <cfields> note that the only non-aws report in that ticket is also running virtualized
237 2015-03-13 18:17:07 <cfields> though afaik wumpus wasn't, i believe his was native arm
238 2015-03-13 18:19:09 <ajweiss> i think he said that he didn't reindex either as well
239 2015-03-13 18:19:37 <cfields> yea
240 2015-03-13 18:20:44 <cfields> his is different though, in that it looks like the corrupted block was not a part of the initial download
241 2015-03-13 18:22:53 <ajweiss> core doesn't automatically kick off reindexing, does it? maybe during initial sync?
242 2015-03-13 18:25:39 <cfields> i don't see any path to the file offset problems except though LoadExternalBlockFile
243 2015-03-13 18:37:11 <gmaxwell> I think crashes also could reveal the issue cfields fixed, if you crash between a block being written and it being added to the index.
244 2015-03-13 18:42:04 <cfields> gmaxwell: in that case though, upon restart, blocks would continue to be written from the index pointer, making everything that comes after it effectively null'd. index db dumps show that data being referenced somehow. any idea how the db could end with entries past what it thinks is the end of the blocks?
245 2015-03-13 18:42:24 <cfields> *could end up with
246 2015-03-13 18:43:04 <gmaxwell> cfields: we're supposted to flush the block files to disk before writing the index. So no.
247 2015-03-13 18:48:08 <ajweiss> gmaxwell: any ideas as to why modify timestamps on undo files wouldn't match their block files when a reindex isn't involved?
248 2015-03-13 18:53:08 <gmaxwell> ajweiss: hm. mine all match.
249 2015-03-13 18:53:21 <gmaxwell> at least to 1 min resolution.
250 2015-03-13 18:54:05 <ajweiss> hm
251 2015-03-13 18:54:12 <cfields> ajweiss: it looks like they can be touched with -checklevel=4
252 2015-03-13 19:41:28 <jonasschnelli> cfields, if you are interested, here are the endpart of my linker warnings: https://gist.github.com/jonasschnelli/9929f6e0d8435c883e85
253 2015-03-13 19:42:40 <jonasschnelli> dfletcher, travis tell me: "cc1plus: error: unrecognized command line option â-mavx2â"
254 2015-03-13 19:42:51 <cfields> jonasschnelli: is your boost built with hidden visibility?
255 2015-03-13 19:42:55 <jonasschnelli> maybe rebase and force push to see if it was a one-time-travis error
256 2015-03-13 19:43:30 <jonasschnelli> cfields, i use home-brews version...
257 2015-03-13 19:43:44 <jonasschnelli> cfields, but today i did a mac osx 10.0.3 beta update...
258 2015-03-13 19:44:08 <cfields> jonasschnelli: ah ok, that'll explain most of it. Our releases build boost with fvisibility=hidden
259 2015-03-13 19:44:10 <jonasschnelli> let me do a fresh and full autoreconf/configure/make
260 2015-03-13 19:44:45 <jonasschnelli> cfields, but yesterday i didn't saw the warnings,... you think it has something todo with the os update?
261 2015-03-13 19:45:12 <cfields> jonasschnelli: hmm, did clang/ld64 update?
262 2015-03-13 19:45:38 <jonasschnelli> (clang-600.0.57)
263 2015-03-13 19:45:48 <jonasschnelli> dunno if it was updated... could be...
264 2015-03-13 19:46:36 <dfletcher> jonasschnelli, hum rebase tells me "current branch master is up to date" and then there's nothing to commit or push.
265 2015-03-13 19:48:06 <jonasschnelli> dfletcher, yes. your on top... right.. maybe do a dummy-change (add newline, remove newline) amend commit force push to kick travis again?
266 2015-03-13 19:48:21 <dfletcher> hmm alright :)
267 2015-03-13 19:48:53 <dfletcher> oh hey I wanted to fix up this one comment anyway nice
268 2015-03-13 19:49:36 <dfletcher> alright thanks jonasschnelli guess we'll see in a bit :)
269 2015-03-13 19:50:28 <jonasschnelli> dfletcher, travis says "Project ERROR: mtdev development package not found"
270 2015-03-13 19:50:38 <jonasschnelli> (current state)
271 2015-03-13 19:51:09 <dfletcher> eh thats weird i didn't change anything in the build system
272 2015-03-13 19:51:11 <jonasschnelli> dfletcher, as well as "fatal error: X11/Xlib.h: No such file or directory"
273 2015-03-13 19:51:27 <jonasschnelli> so i assume this will be gone after your dummy-amend push
274 2015-03-13 19:56:06 <jonasschnelli> dfletcher, you did not amend you commit
275 2015-03-13 19:56:16 <jonasschnelli> now you have a new one and need to squash...
276 2015-03-13 19:56:40 <jonasschnelli> you need to squash them anyway
277 2015-03-13 19:57:12 <jonasschnelli> dfletcher, https://ariejan.net/2011/07/05/git-squash-your-latests-commits-into-one/
278 2015-03-13 20:01:19 <dfletcher> sorry im a github newb this is my first pull req :) ok so when I run rebase -i it only has "noop" but now it says I can push. should I ? and whoops can I --amend now? make another change just to --amend ?
279 2015-03-13 20:02:22 <dfletcher> in future i should --amend all changes in a pull req is that the pattern?
280 2015-03-13 20:03:35 <dfletcher> how do i fix my disaster area now lol i'm lost.
281 2015-03-13 20:07:38 <jonasschnelli> dfletcher, with git rebase -i you can squash commits into one. Check the link above how to do it or google for more infos...
282 2015-03-13 20:07:48 <dfletcher> right when I do that
283 2015-03-13 20:07:48 <jonasschnelli> --amend if only useful if you like to add/change the last commit
284 2015-03-13 20:07:59 <dfletcher> the only thing that comes up in the text file is: "noop"
285 2015-03-13 20:08:26 <dfletcher> oh wait ok got it
286 2015-03-13 20:08:36 <dfletcher> first time put "master" at the end, that's what gave me the noop
287 2015-03-13 20:11:12 <dfletcher> thanks for the help jonasschnelli, push'd
288 2015-03-13 20:11:38 <jonasschnelli> now you have 6 commits... :)
289 2015-03-13 20:11:45 <dfletcher> lol damnit
290 2015-03-13 20:12:10 <jonasschnelli> :) just don't close the PR and make a new one.. sort it out and you gain some git knowhow.
291 2015-03-13 20:13:24 <jonasschnelli> you can also take the hard way if you stuck,... save out your change (copy paste to a text file), "git reset --hard origin/master", copy back your changes and start with a fresh commit.
292 2015-03-13 20:13:46 <jonasschnelli> but you will gain more git knowhow by "git rebase -i"
293 2015-03-13 20:17:12 <BlueMatt> maaku: you never know if a tx is ready to go in the mempool until you know its utxos
294 2015-03-13 20:18:39 <BlueMatt> maaku: so you just do what everything else does and assume all utxos are confirmed at current_height + 1 and make sure there is no way for such a tx to become invalid later
295 2015-03-13 20:19:40 <dfletcher> huh well jonasschnelli looks right locally. "Your branch is ahead of 'upstream/master' by 1 commit" so that looks right. but it won't let me push. pulling is what gave me 6 commits last time heh.
296 2015-03-13 20:20:01 <jonasschnelli> also add "--force" to git push
297 2015-03-13 20:20:06 <dfletcher> ahh ok
298 2015-03-13 20:20:27 <jonasschnelli> or it will refuse by "fast forwarding" policies
299 2015-03-13 20:24:46 <dfletcher> jonasschnelli, genius thank you so much! feel like I just got a git level up haha. even managed to clean up the commit message.
300 2015-03-13 20:25:27 <jonasschnelli> nice. finally one commit. Now lets see what travis says about it...
301 2015-03-13 20:25:33 <dfletcher> also heh next time im doing this on a branch for sure. doing it on my master branch of my fork was a mistake.
302 2015-03-13 21:05:45 <dfletcher> huh jonasschnelli well looks like travis failed in the same way. I have to go run an errand soon back in about an hour and a half. figure this out when I get back.
303 2015-03-13 21:06:23 <jonasschnelli> dfletcher, okay. Yes. This maybe requires a window guy to have a look at
304 2015-03-13 21:07:12 <dfletcher> at least we can see with the patch all rolled up into one commit that its not messing with the build system at all heh
305 2015-03-13 21:14:34 <dfletcher> jonasschnelli, oh hey found this. https://travis-ci.org/bitcoin/bitcoin/jobs/54300397#L2607
306 2015-03-13 21:15:27 <jonasschnelli> for me this looks totally strange. But i'm not a windows/travis expert...
307 2015-03-13 21:15:35 <jonasschnelli> Maybe ask cfields...
308 2015-03-13 21:16:26 <dfletcher> oops train in 15 gotta run. back in a while. cfields if you feel like peeking ^^ and https://github.com/bitcoin/bitcoin/pull/5834
309 2015-03-13 21:16:57 <cfields> looks like a legit test failure to me
310 2015-03-13 21:17:55 <cfields> it's possible (likely) that those arguments are handled differently depending on the interpreter
311 2015-03-13 21:24:19 <cfields> dfletcher: the issue there may be that "echo" is a shell built-in on windows. I have no idea how those work
312 2015-03-13 21:24:57 <cfields> however, I would expect "echo foo >> bar" to work as an alertnotify
313 2015-03-13 21:40:28 <paveljanik> jonasschnelli, I can't find nRPCConsoleWindowHistory written in the settings after quit... Does persistence work for you on 10.10?
314 2015-03-13 21:45:11 <paveljanik> will check again with clean build...
315 2015-03-13 21:46:44 <phantomcircuit> a while ago i had a patch which attempted to mitigation inbound connection slots being congested by identifying the "least desirable" peer and disconnecting them
316 2015-03-13 21:46:54 <paveljanik> hmm, you can't test changes done in 2015/03/qt_console_update in the build with 2015/03/qt_debug_tab_order 8) sorry
317 2015-03-13 21:47:25 <phantomcircuit> the algorithm was pretty simple, more or less it would just disconnect anybody within the same ip group and then disconnect the last connection if there was no overlap
318 2015-03-13 21:47:34 <phantomcircuit> (obviously the policy there needs a bit of tuning)
319 2015-03-13 21:47:52 <phantomcircuit> would anybody be interested in having such a piece of code added back into the mix?
320 2015-03-13 21:50:19 <gmaxwell> phantomcircuit: the policy I've wanted but not had a chance to implement would work like this: There are several distinct desirability criteria, for each of them protect the best (say) 16 nodes by that criteria. That would cover about half the slots, then when a new connection comes in and we're full randomly select from the unprotected half (and the new node) and kick one. There is some extra c
321 2015-03-13 21:50:25 <gmaxwell> omplexity to deal with to make sure just rapidly reconnecting doesn't up your odds of making it in.
322 2015-03-13 21:51:01 <Luke-Jr> can we keep the political stuff off the dev ML? I agree legal action may be useful, but it's totally off-topic and doesn't help the ML noisiness problem.
323 2015-03-13 21:53:13 <gmaxwell> The critieria I'd considered interesting: Are they whitelisted? Are they on your local subnet? How recently did they relay a valid block to you?, How long has the node been connected? How long cumulative has the node ever been connected to you? How do they rank in a determinstic random preference on netblocks (e.g. hash(netgroup,nodekey),uptime), How do they rank in terns of lowest maximum pingti
324 2015-03-13 21:53:19 <gmaxwell> me (network distance)?
325 2015-03-13 21:54:24 <gmaxwell> some of these protected groups (e.g. all except the longest uptime) should probably have an extra constraint that you have to meet some other goodness criteria: e.g. must be a network node, must have relayed a valid transaction to us.
326 2015-03-13 21:57:41 <phantomcircuit> on an entirely unrelated note
327 2015-03-13 21:57:48 <phantomcircuit> is there a map of the headers?
328 2015-03-13 21:58:08 <phantomcircuit> (so i can check to see that a block being processed is actually in the best chain)
329 2015-03-13 21:58:48 <gmaxwell> it won't try to fetch a block that isn't in what it thinks is the best chain.
330 2015-03-13 21:59:05 <phantomcircuit> that doesn't stop someone from sending you one though
331 2015-03-13 21:59:31 <gmaxwell> true.
332 2015-03-13 22:00:04 <phantomcircuit> gmaxwell, im not really sure if this check is necessary in ConnectBlock but it should be relatively cheap
333 2015-03-13 22:00:07 <phantomcircuit> i think
334 2015-03-13 22:02:56 <gmaxwell> I think we'll never get to connectblock anymore unless it is.
335 2015-03-13 22:12:45 <phantomcircuit> gmaxwell, oh and i just realized there's a bug in my no_checkpoints match
336 2015-03-13 22:12:48 <phantomcircuit> patch*
337 2015-03-13 22:13:00 <phantomcircuit> im checking the height of the previous block not the one currently being checked
338 2015-03-13 22:13:13 <phantomcircuit> that probably doesn't matter though
339 2015-03-13 22:16:08 <dfletcher> cfields, thanks. the original version fell back to using ::system() if CreateProcess failed. Travis passed that one. guess that fallback might be necessary.
340 2015-03-13 22:16:48 <dfletcher> and heh I just tried running `ming32-make check` locally and it says "all 0 test passed". useful! ;>
341 2015-03-13 22:18:45 <phantomcircuit> oh that's fun
342 2015-03-13 22:19:01 <phantomcircuit> if you're reindexing you dont have the block headers and without checkpoints you run the scriptchecks on every block
343 2015-03-13 22:19:04 <phantomcircuit> nasty
344 2015-03-13 22:26:48 <gmaxwell> phantomcircuit: I'd suggest that it shoudl skip if the index says its already verified, except one reason to reindex is to correct a case where you previously accepted something you shouldn't have.
345 2015-03-13 22:27:15 <phantomcircuit> well maybe the reindex *should* check all the scripts
346 2015-03-13 22:27:32 <phantomcircuit> iirc it takes days though...
347 2015-03-13 22:27:55 <gmaxwell> it's not _that_ slow. well, I haven't run that test without libsecp256k1 in a while.
348 2015-03-13 22:46:57 <dfletcher> ahh yeah cfields, jonasschnelli. using CreateProcess() by itself breaks https://github.com/dfletcher/bitcoin/blob/master/src/test/alert_tests.cpp#L159
349 2015-03-13 22:47:11 <dfletcher> well local testing is telling me that. i expect travis to confirm it shortly.
350 2015-03-13 22:48:49 <dfletcher> and heh jonasschnelli now I see why one might want to make an actual separate commit as opposed to always --amend'ing. i can show success and fail in the thread with travis :)
351 2015-03-13 22:50:51 <phantomcircuit> gmaxwell, it's not that slow... on a powerful cpu
352 2015-03-13 22:50:59 <phantomcircuit> it is pretty damned slow on just about anything else
353 2015-03-13 23:05:59 <dfletcher> haha wtf. suddenly my msys terminal went whackadoo and started chanelling it's inner Billy Idol. no matter what I type into it it echos "More? More? More?"
354 2015-03-13 23:09:10 <dfletcher> ACTION kills the zombie terminal with a double tap
355 2015-03-13 23:13:11 <Dodger_> I have a question regarding how many hashes one needs to calculate in order to have a 50/50 chance of finding one lower than the target.
356 2015-03-13 23:15:56 <Dodger_> I'm finding x, such that (1-(current_target / 2^256))^x = 0.5
357 2015-03-13 23:16:45 <gmaxwell> 2^256/target of them.
358 2015-03-13 23:18:29 <Dodger_> But the solution I'm finding - 141196094759172878608 - is lower than the number quoted at http://blockexplorer.com/q/hashestowin - 203702905701946867973
359 2015-03-13 23:20:18 <gmaxwell> Dodger_: thats because you're computing it wrong. See my answer.
360 2015-03-13 23:21:04 <gmaxwell> The tests are independant. The number of hashes it's expected to take is independant of how many you've drawn so far.
361 2015-03-13 23:21:39 <Dodger_> Hrm. *spinning eggtimer*
362 2015-03-13 23:24:21 <gmaxwell> your ^ is like saying "and" whats the probablity that hte first fail sand second fails and third fails. But the distribution has an exponential shape. you have an exponential distribution with parameter lambda, the expectation is just 1/lambda.
363 2015-03-13 23:25:22 <Dodger_> Okay, so I was calculating the median instead of the average.
364 2015-03-13 23:31:04 <dgenr8> both your formulas are right :)
365 2015-03-13 23:31:23 <Dodger_> Yeah, I'm just calculating different things...
366 2015-03-13 23:32:48 <sipa> do you want to calculate the chance for exactly one solution
367 2015-03-13 23:32:53 <sipa> or at least one?
368 2015-03-13 23:33:39 <Dodger_> At least one, I guess.
369 2015-03-13 23:34:12 <Dodger_> Although, I guess that would be ( 2^256 - current_target )
370 2015-03-13 23:34:35 <sipa> your formula is right for that
371 2015-03-13 23:35:09 <sipa> but generally you don't care about that number
372 2015-03-13 23:35:23 <sipa> you care about how many solutions you find per time unit
373 2015-03-13 23:36:05 <Dodger_> Oh, this is for a presentation.
374 2015-03-13 23:36:52 <Dodger_> Think in terms of http://dilbert.com/strip/2003-08-07
375 2015-03-13 23:37:21 <sipa> haha
376 2015-03-13 23:37:43 <Dodger_> "How Bitcoin works"
377 2015-03-13 23:37:50 <Dodger_> o Bitcoin âminersâ collect new transactions, compile them into a block, and compete to add their block to the Blockchain
378 2015-03-13 23:38:02 <Dodger_> - Each block typically contains ~200-900 transactions
379 2015-03-13 23:38:14 <dgenr8> google petertodd's sudoku example
380 2015-03-13 23:38:29 <Dodger_> - Miners race to brute-force solve a computationally-intensive math problem (requires 141 quintillion calculations)
381 2015-03-13 23:38:33 <sipa> "compete" is not incorrect, but it may lead some people to think that it is a race
382 2015-03-13 23:38:37 <sipa> no, it is not a race
383 2015-03-13 23:38:42 <sipa> it os a lottery
384 2015-03-13 23:39:02 <sipa> if it was a race, the largest pool would win every blocl
385 2015-03-13 23:39:23 <Dodger_> Yeah, I know, but then we're back to the Dilbert cartoon.
386 2015-03-13 23:39:28 <gmaxwell> Dodger_: race implies progress. If one car is consistently faster than another, it always wins.
387 2015-03-13 23:39:38 <gmaxwell> The race misunderstanding is enormously important.
388 2015-03-13 23:39:48 <sipa> feel free to simplify
389 2015-03-13 23:39:57 <sipa> but don't mislead
390 2015-03-13 23:39:58 <gmaxwell> You may as well not present at all if you're going to misinform people like that.
391 2015-03-13 23:40:07 <Dodger_> Okay, I'll try to be less racist.
392 2015-03-13 23:44:35 <midnightmagic> ...
393 2015-03-13 23:52:12 <Dodger_> "Miners run their blocks through a computationally-intensive math equation, looking for a winning solution"
394 2015-03-13 23:53:39 <Dodger_> How's that? I can mention the lottery analogy verbally.
395 2015-03-13 23:55:38 <phantomcircuit> gmaxwell, thinking about it some more reindex probably should check all the script sigs
396 2015-03-13 23:56:44 <phantomcircuit> and has the happy coincidence that i dont have to write more clever logic...
397 2015-03-13 23:56:52 <Hasimir> more like they're all trying to prove they have the sharpest pseudo-intelligence by solving the problem first, forgetting that the more they hurl into it, the greater the barriers rise ...