1 2015-03-03 00:48:04 <Luke-Jr> cfields: I don't think you can eliminate all cases of Travis failures.. it affects more than just BCCore
2 2015-03-03 02:17:40 <cfields> Luke-Jr: i think this one in particular is timing-related, though
3 2015-03-03 02:18:10 <cfields> Luke-Jr: er.. i suppose i should've asked: what similar things have you seen elsewhere?
4 2015-03-03 02:20:25 <Diablo-D3> https://github.com/neovim/neovim/pull/2076
5 2015-03-03 02:20:26 <Diablo-D3> woah
6 2015-03-03 02:35:29 <Luke-Jr> cfields: for almost every build-set of BFGMiner (62 builds), Travis has at least 4 or 5 that fail
7 2015-03-03 02:35:37 <Luke-Jr> and I have to manually restart them if I care
8 2015-03-03 02:36:03 <Luke-Jr> I haven't looked lately, but some months ago when I checked, they were things like failing to lookup github.com
9 2015-03-03 02:36:16 <Luke-Jr> (so not something the scripts get a chance to retry)
10 2015-03-03 02:41:57 <Luke-Jr> hm, here's a recent one https://travis-ci.org/luke-jr/bfgminer/jobs/52313584 - looks like it's probably network still (but later on, so maybe autorestartable)
11 2015-03-03 04:23:56 <cfields> Luke-Jr: there's travis-retry for that
12 2015-03-03 04:23:58 <cfields> which we use
13 2015-03-03 04:24:47 <cfields> Luke-Jr: i understand/agree that there are network issues that could get in the way of build, but i don't think this one (in particular) is one of em
14 2015-03-03 06:09:52 <droark> Is it okay to ask questions about writing Gitian scripts in here or should I head elsewhere?
15 2015-03-03 07:04:06 <anon_> does anyone have a link to a tutorial on how i can start debugging bitcoin core?
16 2015-03-03 07:06:03 <justanotheruser> anon_: github.com/bitcoin/bitcoin
17 2015-03-03 07:08:10 <anon_> that just looks like development guidelines.
18 2015-03-03 07:08:22 <anon_> i am talking about getting the code running in an ide
19 2015-03-03 07:10:51 <sipa> anon_: we typically use gdb, but you're free to use any ide that supports c++
20 2015-03-03 07:12:06 <treehug88> I just submitted a PR with an init script for CentOS and documentation. Anyone want to take a look?
21 2015-03-03 07:12:37 <paveljanik> treehug88, is bitcoind installed into /usr/bin?
22 2015-03-03 07:13:00 <treehug88> paveljanik: not in my case
23 2015-03-03 07:13:01 <paveljanik> looks strange to me...
24 2015-03-03 07:13:27 <treehug88> what looks strace paveljanik
25 2015-03-03 07:13:32 <treehug88> strange I mean
26 2015-03-03 07:13:49 <paveljanik> +bitcoind=${BITCOIND-/usr/bin/bitcoind}
27 2015-03-03 07:16:34 <justanotheruser> anon_: oh, I thought you meant debugging the errors.
28 2015-03-03 07:16:45 <justanotheruser> in the user binary
29 2015-03-03 07:19:31 <treehug88> paveljanik: what would you suggest as the default. In my case Iâm using the binary /opt/bitcoin/bin/64/bitcoind
30 2015-03-03 07:19:55 <treehug88> also it can be overridden via the BITCOIND env var in /etc/sysconfig/bitcoind
31 2015-03-03 07:19:57 <paveljanik> treehug88, check where make install installs the binary
32 2015-03-03 07:23:10 <treehug88> paveljanik: I donât have access to a machine that can build the source easily. I presume the default it /usr/local/bin/bitcoind ?
33 2015-03-03 07:24:33 <paveljanik> the file should contain the path used when configured with --prefix=/usr.
34 2015-03-03 07:25:04 <paveljanik> I can't check right now, sorry
35 2015-03-03 07:25:53 <paveljanik> will check later today
36 2015-03-03 07:26:28 <treehug88> no problem, paveljanik . I agree itâd be nice to have that feature. Though this other init file contributed https://github.com/bitcoin/bitcoin/blob/master/contrib/init/bitcoind.openrc also has hard-coded defaults
37 2015-03-03 07:27:47 <paveljanik> maybe the default for bitcoind is really /usr/bin/
38 2015-03-03 07:27:52 <paveljanik> I'll check..
39 2015-03-03 07:28:26 <paveljanik> this is what I asked at the beginning.
40 2015-03-03 07:32:51 <treehug88> gotcha, paveljanik . I installed from the 0.9.3 linux tarball; it doesnât have a default install location for the binaries as far as I know.
41 2015-03-03 07:32:52 <treehug88> The current linux binary package at https://bitcoin.org/bin/bitcoin-core-0.10.0/bitcoin-0.10.0-linux64.tar.gz also unpacks bitcoind to bitcoin-0.10.0/bin/bitcoind
42 2015-03-03 07:36:39 <treehug88> Iâd expect the normal ./configure && make && make install to put them at /usr/local/bin/bitcoind
43 2015-03-03 07:36:45 <paveljanik> so we should probably change this first ;-)
44 2015-03-03 07:37:07 <paveljanik> I'd expect daemons to be installed into /sbin/...
45 2015-03-03 07:37:07 <treehug88> yes, Iâve updated the PR
46 2015-03-03 07:37:37 <sipa> paveljanik: sbin is for system critical stuff you need for system recovery
47 2015-03-03 07:39:20 <paveljanik> /usr/sbin/...
48 2015-03-03 07:39:24 <paveljanik> not /sbin
49 2015-03-03 07:39:53 <paveljanik> /usr/sbin/bitcoind
50 2015-03-03 07:40:02 <paveljanik> /usr/bin/bitcoin-cli
51 2015-03-03 07:41:38 <treehug88> thanks, paveljanik. Iâve updated the pr
52 2015-03-03 07:51:57 <paveljanik> treehug88, right now, make install installs into .../bin/bitcoind
53 2015-03-03 07:52:22 <paveljanik> hmm, bitcoind should go into .../sbin/bitcoind
54 2015-03-03 08:06:12 <paveljanik> cfields, do you agree that bitcoind should go into ${prefix}/sbin instead of ${prefix}/bin?
55 2015-03-03 08:06:42 <sipa> it's a user binary
56 2015-03-03 08:07:07 <sipa> as long as it's not structured in a way that it can run as a system-wide service, i'd sag no
57 2015-03-03 08:07:32 <paveljanik> that's true.
58 2015-03-03 08:07:42 <paveljanik> but then we do not need system init files at all...
59 2015-03-03 08:08:02 <paveljanik> these files assume, the system is signle-user only
60 2015-03-03 08:09:26 <paveljanik> this is probably the assumption, we should also do for the default location of bitcoind. You can always run system-wide bitcoind with a local config file/path.
61 2015-03-03 08:09:53 <paveljanik> or even on different network (testnet e.g.). but the binary could be common, and in /usr/sbin.
62 2015-03-03 08:12:30 <paveljanik> OTOH, bitcoind without wallet is a perfect candidate for running system-wide.
63 2015-03-03 08:13:04 <sipa> yup
64 2015-03-03 08:13:23 <sipa> especially if we'd have an spv wallet client that could connect to it
65 2015-03-03 08:13:32 <paveljanik> or other services.
66 2015-03-03 08:13:32 <paveljanik> yes
67 2015-03-03 08:13:40 <paveljanik> gateways and such
68 2015-03-03 08:18:17 <wumpus> <paveljanik> OTOH, bitcoind without wallet is a perfect candidate for running system-wide. <- YES
69 2015-03-03 08:21:29 <wumpus> does anyone by chance have a tool to dump the block index (esp. file, offset, flags) in human readable format?
70 2015-03-03 08:22:17 <justanotheruser> wumpus: never used this, but it may be what you want https://github.com/znort987/blockparser
71 2015-03-03 08:23:24 <wumpus> justanotheruser: I think that just parses the block files directly, which is useful, but in this case I need to sanity-check the database
72 2015-03-03 08:24:23 <justanotheruser> no idea, sorry
73 2015-03-03 08:24:49 <wumpus> I'll just hack bitcoind to do it, thanks for the suggestion anyhow
74 2015-03-03 08:27:01 <bedeho> If an SPV wallet imports a new private key, then it will have to redownload all blocks in the chain again in order to find any corresponding utxo? i.e., there is no information from the previous synchronization which can be reused for a new private key?
75 2015-03-03 08:27:44 <sipa> bedeho: solution: do not import keys, generate fresh ones
76 2015-03-03 08:27:50 <sipa> problem solved
77 2015-03-03 08:27:57 <bedeho> :D, fair enough
78 2015-03-03 08:47:53 <Luke-Jr> cfields: travis-retry?
79 2015-03-03 15:06:04 <ceedz> If I run bitcoin-cli getinfo, it shows "errors" : "EXCEPTION: St12out_of_range \nCInv::GetCommand() : type=9568257 unknown type \nbitcoin in ProcessMessages() \n"
80 2015-03-03 15:06:06 <ceedz> is that normal?
81 2015-03-03 15:06:20 <sipa> it means someone is sending weird messages to your node
82 2015-03-03 15:06:26 <sipa> it's harmless, they're ignored
83 2015-03-03 15:06:39 <ceedz> ah cool
84 2015-03-03 15:06:40 <sipa> it's probably a bug that we report them through the getinfo "errors" field
85 2015-03-03 15:12:08 <ajweiss> wumpus: dumb question: did you do the usual for the usb filesystem with the screwed up block file? (fsck, dmesg, etc)?
86 2015-03-03 15:20:30 <wumpus> ajweiss: no, no recovery tools run, also no fs-level errors
87 2015-03-03 15:22:39 <wumpus> all in all the device is rock stable, I've only had corruption issues with bitcoin, not with anything else (then again, bitcoin has a tendency to reveal these things)
88 2015-03-03 15:29:38 <ajweiss> and i assume it's a single core pi or smth?
89 2015-03-03 15:29:51 <wumpus> quad core cubox-i w/ 2GB memory
90 2015-03-03 15:30:00 <ajweiss> hrm
91 2015-03-03 15:30:16 <timothy> bitcoin core on arm?
92 2015-03-03 15:31:25 <wumpus> timothy: yes, works quite well on the higher end ARM boxes, rpi1 is too weak
93 2015-03-03 15:31:41 <timothy> I'll try it on rpi2 :P
94 2015-03-03 15:43:08 <treehug88> have I properly squashed commits on https://github.com/bitcoin/bitcoin/pull/5847 ( @ajweiss, this PR is what we talked about a few days ago )
95 2015-03-03 15:43:52 <ajweiss> ooh cool.
96 2015-03-03 15:44:14 <sipa> treehug88: no, there are still 8 commits there
97 2015-03-03 15:44:38 <treehug88> do I need to recut the branch? I tried rebasing as per http://stackoverflow.com/questions/5189560/squash-my-last-x-commits-together-using-git
98 2015-03-03 15:44:54 <Luke-Jr> treehug88: you may need to repush it
99 2015-03-03 15:44:56 <Luke-Jr> with push -f
100 2015-03-03 15:45:56 <ajweiss> here's a nice howto: http://push.cwcon.org/learn/squashing-commits.html
101 2015-03-03 15:46:43 <jgarzik> hrm
102 2015-03-03 15:48:37 <jgarzik> ACTION has forgotten - during sync, you might receive a transaction and discard it as non-final. however the rest of the network considers that TX final as it is fully sync'd up. will the sync'ing node reject future offers of that TX? certainly it will accept it in a block, that's not my concern.
103 2015-03-03 15:49:29 <wumpus> it won't reject it
104 2015-03-03 15:49:40 <sipa> it may not re-request it
105 2015-03-03 15:49:45 <sipa> as it's already in the knowninv
106 2015-03-03 15:49:49 <wumpus> currently there is no remembering of invalid transactions
107 2015-03-03 15:49:58 <wumpus> hm not sure about that
108 2015-03-03 15:50:58 <wumpus> it can e.g. re-request invalid transactions (e.g. those that have zero vins) many times, leading to a node to be banned. But of course only when it's offered again
109 2015-03-03 15:51:08 <Luke-Jr> treehug88: left a couple of comments for you
110 2015-03-03 15:51:40 <Luke-Jr> eh, non-IsFinal txs should never ban, should they? O.o
111 2015-03-03 15:52:04 <sipa> indeed
112 2015-03-03 15:53:00 <wumpus> Luke-Jr: right
113 2015-03-03 15:55:27 <jgarzik> ok
114 2015-03-03 15:55:52 <jgarzik> certainly the network is unlikely to offer it again after the first X seconds of bouncing around the network
115 2015-03-03 15:56:15 <wumpus> indeed
116 2015-03-03 15:56:27 <treehug88> thank Luke-Jr
117 2015-03-03 15:56:32 <jgarzik> just curious what would happen upon re-offer of same tx, at different sync states (resulting in different final vs non-final evaluation)
118 2015-03-03 15:56:32 <treehug88> (thanks)
119 2015-03-03 15:56:55 <sipa> i think it should accept it
120 2015-03-03 15:57:30 <jgarzik> if offered again via inv + in knownInv, will it request the tx ?
121 2015-03-03 15:57:51 <jgarzik> ACTION should test :)
122 2015-03-03 16:11:45 <ThomasV> aschildbach: http://i.imgur.com/gcVngeT.png
123 2015-03-03 16:12:10 <ThomasV> p2sh outputs are shown as "?"
124 2015-03-03 16:19:03 <Luke-Jr> I thought BW4A stopped showing "from address" nonsense already? :|
125 2015-03-03 16:21:08 <helo> still showing it in the latest official release, at least
126 2015-03-03 16:26:27 <treehug88> Luke-Jr: ajweiss: paveljanik : PR updated
127 2015-03-03 16:40:41 <rubensayshi> how do you pronounce secp256k1 xD
128 2015-03-03 16:41:15 <sipa> "secstuff"
129 2015-03-03 16:41:19 <Luke-Jr> "sep two five six kay one" for me
130 2015-03-03 16:41:33 <rubensayshi> what does the P stand for anyway
131 2015-03-03 16:41:42 <sipa> i say "sec pee twofiftysix kay one"
132 2015-03-03 16:41:47 <helo> ^
133 2015-03-03 16:41:47 <rubensayshi> ah Parameters
134 2015-03-03 16:41:51 <sipa> no, prime
135 2015-03-03 16:41:58 <rubensayshi> o
136 2015-03-03 16:42:29 <rubensayshi> thx :D
137 2015-03-03 16:42:41 <sipa> it is a curve by SEC, namely a 256-bit prime one, of the Koblitz-like type
138 2015-03-03 16:42:47 <sipa> the first of those (and only)
139 2015-03-03 16:43:26 <Luke-Jr> TIL the *why* behind it :P
140 2015-03-03 16:43:41 <sipa> why behind what?
141 2015-03-03 16:44:21 <Luke-Jr> reason for the name
142 2015-03-03 16:44:40 <Luke-Jr> (what you just explained)
143 2015-03-03 16:44:45 <sipa> the k is especially confusing, as it's not actually a Koblitz curve
144 2015-03-03 16:45:04 <sipa> just a prime curve with some properties shared by Koblitz curves (which never use a prime size)
145 2015-03-03 17:53:23 <cfields> Luke-Jr: sorry, travis_retry
146 2015-03-03 17:53:25 <cfields> Luke-Jr: http://blog.travis-ci.com/2013-05-20-network-timeouts-build-retries/
147 2015-03-03 17:54:28 <cfields> paveljanik: i agree with what sipa and wumpus said.
148 2015-03-03 17:57:15 <paveljanik> cfields, OK, thanks for your opinion.
149 2015-03-03 18:28:44 <morcos> sipa: Have a couple of questions about autoprune. mrbandrews has been working on a rewrite of the PR. were wondering if you had thoughts on a couple of the remaining obstacles
150 2015-03-03 18:29:10 <sipa> morcos: i left a bunch of comments on the pr recently
151 2015-03-03 18:29:18 <sipa> if you've seen those
152 2015-03-03 18:30:37 <morcos> I think we've read through all those, we were thinking about reorgs greater than the prune size and DOS attacks in startup/IBD
153 2015-03-03 18:31:10 <sipa> reorg greater than the prune size is a good question
154 2015-03-03 18:31:33 <morcos> It seems like keeping orphan blocks makes it a bit hard to know how much data you have to keep to have a minimum number of blocks on the main chain. In the steady state, this is likely small enought that a heuristic is good enough to estimate the min prune size.
155 2015-03-03 18:32:00 <morcos> But on startup, it seems relatively easy to feed big non-main chain blocks that fill up your files and make pruning hard
156 2015-03-03 18:32:16 <sipa> by startup you mean IBD?
157 2015-03-03 18:32:24 <sdaftuar> yes IBD
158 2015-03-03 18:32:30 <morcos> We had some thoughts on how to deal with that, such as potentially only storing blocks that are from the most work chain or on a chain that has a minimum amount of work
159 2015-03-03 18:32:49 <morcos> and feel like these might be some of the same ideas that you guys are discussing for removing checkpoints.
160 2015-03-03 18:33:10 <sipa> how does this relate to pruning?
161 2015-03-03 18:33:12 <morcos> Is it worth trying to address these from the beginning in auto prune and serve as a kind of beta test for how checkpoints might be removed
162 2015-03-03 18:33:22 <sdaftuar> only because DOS at IBD could be worse for an autoprune node
163 2015-03-03 18:33:34 <sipa> sdaftuar: in what way?
164 2015-03-03 18:33:50 <sdaftuar> if the DOS attack fills up your disk, so you have to again go into IBD over and over (because you keep reorging past your prune size)
165 2015-03-03 18:34:10 <morcos> s/disk/prune limit/
166 2015-03-03 18:34:46 <morcos> the same attack applies to regular nodes if you remove checkpoints right? isn't that one of the purposes?
167 2015-03-03 18:34:49 <sipa> i need to run some errands
168 2015-03-03 18:34:56 <morcos> convenient : )
169 2015-03-03 18:34:57 <sipa> i'll be back in an hour
170 2015-03-03 18:35:07 <morcos> sounds good
171 2015-03-03 20:57:57 <sipa> as ibd is not full operation, i consider failing to sync a bug, but not a security issue
172 2015-03-03 20:58:13 <morcos> so what should the failure mode be
173 2015-03-03 20:58:39 <sipa> reorg beyond prune limit: node dies, require manual intervention
174 2015-03-03 20:58:49 <morcos> i think its the same failure mode as a reorg longer than your prune depth, but maybe it coudl do something different in ibd
175 2015-03-03 20:59:17 <morcos> you just made ajweiss very happy...
176 2015-03-03 20:59:28 <sipa> ?
177 2015-03-03 21:00:08 <morcos> he was arguing for that, we were arguing that if pruning might be a basic feature of all nodes in the future, then they should preserve the ability to handle all reorgs
178 2015-03-03 21:01:13 <sipa> i don't think we can consider pruned nodes to be normal operation right now
179 2015-03-03 21:01:34 <sipa> maybe i the future, when there are better ways to discover and sync partially from pruned nodes
180 2015-03-03 21:02:42 <sipa> but as implemented in the current PR, there is no recovery from a deep reorg after pruning
181 2015-03-03 21:03:01 <sipa> when the undo data is gone, you have only one option, and that is starting over
182 2015-03-03 21:03:28 <sipa> which is such a long term procedure that i don't think it should be done transparently, even of it could be
183 2015-03-03 21:04:05 <morcos> yeah we were trying to fix that! yeah, thats what we were doing, and i thought what gmaxwell wanted in his PR comment once it got bumped to .11
184 2015-03-03 21:04:05 <sipa> so i think intervention from the node operator is better in that case
185 2015-03-03 21:04:26 <morcos> but i can see punting on that for now is a way to at least move forward
186 2015-03-03 21:04:28 <sdaftuar> basically ask the node operator to restart bitcoind -reindex?
187 2015-03-03 21:04:50 <sipa> well tje alternative is having a separate undo pruning depth and block pruning depth
188 2015-03-03 21:05:20 <sipa> but the current undo files depend on the block files
189 2015-03-03 21:05:33 <sipa> so you'd need some undo-only-merged-view files
190 2015-03-03 21:05:54 <sipa> eh, i'm blathering
191 2015-03-03 21:06:22 <sdaftuar> yeah i guess we were just thinking we'd keep undo depth the same as block depth, but figure out how to kick the node into initial download state upon a too-long-reorg
192 2015-03-03 21:06:23 <morcos> as for spiking disk usage during IBD, i would just suggest that if you get attacked in IBD and it causes you to reorg bigger than your prune limit that you just abort as well... instead of spiking disk usage
193 2015-03-03 21:06:31 <sipa> just keep undo up to 1000 deep, blocks up to 2000 deep; when you have a 1500 deep reorg, you can still redownload the blocks
194 2015-03-03 21:08:20 <sipa> sdaftuar: you'd wipe the chainstate and rebuild it
195 2015-03-03 21:08:28 <sdaftuar> right
196 2015-03-03 21:08:41 <sipa> no need to wioe block index and blocks
197 2015-03-03 21:09:06 <sdaftuar> well in this scenario it's likely none of the blocks you have are useful; they're on the wrong chain
198 2015-03-03 21:09:21 <sipa> you still have all headers
199 2015-03-03 21:09:29 <sdaftuar> yes agreed
200 2015-03-03 21:09:37 <sipa> which is valuable for guiding your sync
201 2015-03-03 21:09:53 <sipa> (and avoiding being attacked during the reentered ibd)
202 2015-03-03 21:10:14 <sdaftuar> yep agreed -- though the ibd logic would need to change a little to take advantage of that
203 2015-03-03 21:10:39 <sdaftuar> (at any rate we can punt on this til later...)
204 2015-03-03 21:10:46 <sipa> well there would need to be a "i need this block, and used to have it, but it was pruned, download it again
205 2015-03-03 21:11:02 <sipa> i think that's not.particularly hard anymore
206 2015-03-03 21:11:24 <morcos> whats the scenario you're referring to?
207 2015-03-03 21:11:35 <sdaftuar> ah you mean, instead of going back into the existing IBD logic, write some new logic for this case?
208 2015-03-03 21:11:53 <sipa> there really is no "ibd logic"
209 2015-03-03 21:12:14 <sipa> blocks are fetched as they are needed, using two independent mechanism
210 2015-03-03 21:13:09 <sdaftuar> right now though you download any block with a valid header that is further along than your tip, yes? even if it's not on the most-work chain
211 2015-03-03 21:13:24 <sipa> indeed
212 2015-03-03 21:14:36 <sdaftuar> so that seems like behavior we'd want to change for an autopruned node that was trying to recover from a big reorg...
213 2015-03-03 21:14:53 <sipa> random thought: what if the prune limit is not for total disk space, but per chain
214 2015-03-03 21:15:08 <sdaftuar> seems like that is subject to DOS if checkpoints go away?
215 2015-03-03 21:15:37 <sipa> i do not like changing that fetch-any-candidate-best
216 2015-03-03 21:15:38 <sdaftuar> per chain over X work maybe...
217 2015-03-03 21:16:03 <sdaftuar> hmm how else should a node behave then?
218 2015-03-03 21:16:09 <morcos> ok so to get back to what we're doing on the first iteration.. i'm not sure i understand why the minimum hard limit should be for 288 (or whatever #) of blocks, instead of only being for a minimum settable MB limit, that is very likely to provide at least 288 or whatever blocks of the main chain always on hand.
219 2015-03-03 21:17:01 <sipa> morcos: good question
220 2015-03-03 21:17:10 <morcos> but it seems preferable to me that we respect the invariant of the prune limit the user sets even if it means in unusual circumstances we might dip under 288 blocks... worst thing that happens is the reorg is bigger than the number of blocks we have left
221 2015-03-03 21:18:03 <sipa> i'm starting to feel that a user settable limit has attack potential
222 2015-03-03 21:18:14 <sipa> as it's a shared resource across all chains
223 2015-03-03 21:18:33 <sipa> allowing a forged chain to influence how the node deals with the right chain
224 2015-03-03 21:18:51 <sipa> of course, it is probably also what the user actually cares about
225 2015-03-03 21:19:00 <morcos> yeah well thats what we were worried about in IBD, when its easy to stuff a lot of extra data in junk chains
226 2015-03-03 21:19:12 <sipa> yeah, i see that now
227 2015-03-03 21:19:15 <morcos> but outside of IBD, its expensive
228 2015-03-03 21:19:55 <sipa> in any case, i think shutdown + manual interventin reorg beyond pruning limit happens is fine
229 2015-03-03 21:20:38 <sipa> and that immediately also covers the case of a stuffing attack during ibd
230 2015-03-03 21:21:13 <morcos> right
231 2015-03-03 21:22:00 <sipa> a next step could be preventing pruning unless sufficient work on top of a chain (which coincided with the checkpoint-reduction code)
232 2015-03-03 21:22:21 <morcos> are you ok losing the 288 hard minimum? suppose we set 500MB as the min prune limit or something.
233 2015-03-03 21:22:55 <morcos> yeah but to avoid data spiking then you have to delay downloading as well, not just pruning
234 2015-03-03 21:23:10 <morcos> but i think you might have to do that anyway when you get rid of checkpoints
235 2015-03-03 21:23:25 <morcos> otherwise might not be too hard to fill up disks of full nodes with garbage orphan chains
236 2015-03-03 21:23:33 <mrbandrews> so i've written code which uses some (not a lot, but some) of the code in pr 4701. i was thinking of submitting a new pr... is that the right way to go about it?
237 2015-03-03 21:23:37 <morcos> if you're fast and catch them at IBD
238 2015-03-03 21:23:51 <sipa> well you don't lose the 288 as block limit if the minimum size limit is 500
239 2015-03-03 21:23:57 <sipa> it just becomes implicit
240 2015-03-03 21:24:32 <morcos> right, but you could theoretically dip below it... even if there were lots of 1 block reorgs or whatever...
241 2015-03-03 21:24:38 <sipa> i do think some (explicit or implicit) limitation that avoids pruning beyond the ~days scale is necessary
242 2015-03-03 21:24:44 <morcos> agreed
243 2015-03-03 21:25:32 <sipa> mrbandrews: i think submitting a new pr is fine
244 2015-03-03 21:25:56 <mrbandrews> ok cool
245 2015-03-03 21:25:56 <sipa> i'm getting tired, sorry
246 2015-03-03 21:26:13 <sipa> thanks for working on this, though
247 2015-03-03 21:27:11 <mrbandrews> thx for all the guidance
248 2015-03-03 21:27:16 <mrbandrews> will get a pr submitted as soon as i/we can
249 2015-03-03 21:27:49 <sipa> feel free to ping me if you want to discuss
250 2015-03-03 21:27:56 <sipa> i'm back in europe now, though
251 2015-03-03 21:28:48 <mrbandrews> cool, thx again
252 2015-03-03 21:33:02 <gmaxwell> morcos: Really all that trouble above is coming from orphans, no? There is no reason to keep around any orphans at all... unlike pruning along the best chain, deleting an orphan does nothing irreversable.
253 2015-03-03 21:34:03 <morcos> isn't that difficult b/c of the first come first stored method of block files
254 2015-03-03 21:34:58 <sipa> not entirely
255 2015-03-03 21:35:11 <morcos> but yes not downloading in the first place blocks that we don't think are on that main chain seems a good idea
256 2015-03-03 21:35:12 <sipa> when there are multiple possible future chains
257 2015-03-03 21:35:22 <sipa> there is no privileged one
258 2015-03-03 21:36:04 <gmaxwell> morcos: we don't download blocks we don't think are in the main chain, AFAIR.
259 2015-03-03 21:36:28 <sipa> gmaxwell: we download blocks in any chain that extends beyond our current best blocl
260 2015-03-03 21:37:14 <sipa> otherwise you can a variant of a block witholding attack
261 2015-03-03 21:37:18 <sipa> i think
262 2015-03-03 21:37:19 <gmaxwell> under what case might we not think that was the best chain?
263 2015-03-03 21:37:41 <gmaxwell> sipa: no, otherwise you won't converge (quickly) in the event of an invalid block.
264 2015-03-03 21:38:04 <sipa> that too
265 2015-03-03 21:38:08 <gmaxwell> morcos: the append-only-mostly-ness of block files is why we don't already erase orphans.
266 2015-03-03 21:38:14 <morcos> during IBD someone can send you an early long chain that is not the main chain, with big blocks on it right?
267 2015-03-03 21:39:03 <gmaxwell> morcos: yes, thus the logic that pruning should not start until the node is past some work threshold. (and see my earlier comments about not thinking causing spiky disk use during IBD being that interesting)
268 2015-03-03 21:39:35 <morcos> i guess i don't understand why thats not intersting
269 2015-03-03 21:39:39 <morcos> i think it could spike a lot
270 2015-03-03 21:39:40 <gmaxwell> We could defer downloading any blocks until we've found a header chain past that point, I guess. probably would only add a minute to initial sync.
271 2015-03-03 21:39:43 <morcos> right
272 2015-03-03 21:40:45 <gmaxwell> morcos: because its a somewhat costly attack, that is just a DOS thats a one time exposure for a new node. Don't misunderstand, I'd rather it not be an issue. But I don't know that its worth compromising on basically any other desirable quality to do something about it.
273 2015-03-03 21:41:04 <gmaxwell> Maybe the additional minute of initial sync would be an acceptable fix.
274 2015-03-03 21:42:25 <morcos> seems like we all agree that we should start by not worrying about that attack, i just prefer that if you succumb to it your node aborts, rather than spikes in data usage
275 2015-03-03 21:42:32 <gmaxwell> e.g. you could mine a 100k block 1MB block chain to use a spike of 100GB at the cost of 100k diff 1 solutions, and the bandwidth to actually send those bloat blocks to your victim.
276 2015-03-03 21:43:08 <gmaxwell> morcos: ah. That doesn't sound unreasonable to me; again so long as it's limited to IBD.
277 2015-03-03 21:43:21 <morcos> hmm?
278 2015-03-03 21:43:27 <morcos> what happens after IBD
279 2015-03-03 21:44:49 <gmaxwell> morcos: I said that with the idea in mind that you were thinking of shutting down whenever the disk limit was exceeded. Okay, so then you'd also shut down if there were a lot of orphans. Then thats an attack that someone could use against the network (or in particular competing miners), which might not exist otherwise.
280 2015-03-03 21:44:57 <sipa> perhaps afterwards it should go into some safe mode, where it keeps functioning but complains loudly that it failed to do a deep reorg
281 2015-03-03 21:45:14 <sipa> then again, the best way to be sure that warning is seen, is by shutting doen
282 2015-03-03 21:45:34 <morcos> but its a very expensive attack after IBD....
283 2015-03-03 21:45:38 <gmaxwell> if it actually fails a reorg I think it should shut down.
284 2015-03-03 21:45:41 <morcos> too expensive
285 2015-03-03 21:46:19 <sipa> "after IBD" is essentially similar to saying "on a node not shbilled for a few minutes"
286 2015-03-03 21:46:21 <gmaxwell> morcos: but unlike IBD it's actually possible to make it profitable. I'm really uncomfortable with providing a reliably triggerable remote shutdown mechenism to Bitcoin miners.
287 2015-03-03 21:46:44 <sipa> anyway, afk
288 2015-03-03 21:46:58 <gmaxwell> sipa: he means it's expensive in that there would need to be many orphans that extend the current tip.
289 2015-03-03 21:46:58 <morcos> but don't you have to generate at least 288 orphans at current difficulty?
290 2015-03-03 21:47:30 <morcos> i'm not even sure how you could do it with less than 50% of the hashing power
291 2015-03-03 21:47:41 <gmaxwell> by partitioning the node.
292 2015-03-03 21:48:15 <gmaxwell> also, there are potentials for weird interactions. Withholding attacks allow you to produce orphans cheaply.
293 2015-03-03 21:48:56 <gmaxwell> In any case, I really wouldn't want to take a runtime attack vulnerability, even if it is a costly one just to avoid having code to garbage collect orphans.
294 2015-03-03 21:49:16 <gmaxwell> which is what that concern really is about, I think.
295 2015-03-03 21:50:00 <gmaxwell> There is, I think, no reason that if we hit the prune limit we couldn't halt operation for a moment and go and repack all the block files.
296 2015-03-03 21:50:26 <morcos> hmm... ok.. would it make sense to say suppose the prune min limit was 500MB which we thought would be enough for at least 300 blocks in the usual circumstances... that you respect the 500MB limit until you get to some lower number of main chain blocks, such as 100 and then let disk space grow
297 2015-03-03 21:50:59 <morcos> or even 100 could be 288 really
298 2015-03-03 21:51:23 <gmaxwell> yea, thats what I intended the functionality to be for now. And I think thats fine.
299 2015-03-03 21:51:57 <morcos> i just think the min prune limit (500MB or whatever) should be high enough that if you set it to that, you don't expect to ever exceed it because of some other hard coded limit such as 288 blocks
300 2015-03-03 21:52:14 <morcos> but i agree using more disk space under an attack is better than aborting
301 2015-03-03 21:53:29 <gmaxwell> That sounds reasonable to me too. Though I'm not sure if expressing the limit as size is helpful to the user. "I set it to 500, but bitcoin is using 1GB" <commence explaining chainstate to angry user>
302 2015-03-03 21:53:39 <morcos> ha
303 2015-03-03 21:54:07 <morcos> yeah well that was my next step once i dug into it more to see what was reasonable, but i thought it might make sense to factor all that in and make the minimum 1GB or 1.5GB or something
304 2015-03-03 21:54:45 <gmaxwell> A more complete solution is probably a total limit on bitcoin's space usage; with a minimum of 1-2gb, and the node yelling at you and then shutting down when it can't keep up. :(
305 2015-03-03 21:55:17 <morcos> give a buffer both for orphans and for growth of chainstate... right... but seems like you think it shouldn't shut down, and now i might agree
306 2015-03-03 21:55:57 <gmaxwell> Yes, I don't think shutting down is good. Hm. well okay: you can tell the user "sorry, network requires using more space than the limit you've set"
307 2015-03-03 21:56:48 <morcos> As long as that only happens under attack or if they've been using 0.11 for 2 years and the chainstate grew or something like that, that seems perfectly fine to me...
308 2015-03-03 21:56:54 <gmaxwell> perhaps best to call the option "target" not "limit". Seems silly but these things matter.
309 2015-03-03 21:58:13 <morcos> Sure, I just want it to be rare to exceed, even if its a target..
310 2015-03-03 21:58:39 <ajweiss> maybe borrow from unix and have hard and soft limits
311 2015-03-03 21:59:03 <gmaxwell> ajweiss: any hard limit is attack surface, esp as the minimum is an attractor that people will set at.
312 2015-03-03 22:02:01 <spadinskiz> Does anyone know of a guide to adding bitcoin to a python program? I looked at bitcoin-pythonlib, bitcoin-python and a few other libraries for it but they all seem pretty complicated. All I want to be able to do is add a bitcoin address to each entry in my mysql table of usernames and be able to send bitcoins from those addresses.
313 2015-03-03 22:02:10 <gmaxwell> morcos: since you're thinking about this... one of the motivations for 288 blocks beyond being a "maximum sane reorg interval times a safty factor" is that blocks near the tip are fetched by peers _far_ more often. Right now pruning makes the node a client only, but in the future I'd like to have it so that pruned nodes participate in serving at a very minimum these most-in-demand blocks... so we did
314 2015-03-03 22:02:16 <gmaxwell> n't want the limit so low as to undermine this.
315 2015-03-03 22:03:18 <gmaxwell> sipa collected data a while back about frequency of block fetches based on depth in chain, it was a nice 1/e^x sort of shape. We should probably collect data like that again.
316 2015-03-03 22:03:48 <morcos> Ok, that makes sense. An overall limit might have to be a next step though, because there are so many things that go into... You don't want somebody stuffing your debug.log with rejected messages, and that causing your pruning to hit the limit or something...
317 2015-03-03 22:03:52 <gmaxwell> (well 1/e^x+c .. there is a floor of uniformly fetched blocks from peers doing IBD)
318 2015-03-03 22:04:34 <gmaxwell> yea, ultimately I'd like to replace the debug log with a circular buffer in memory that gets dumped on crash and otherwise doesn't get logged except for a small number of very interesting things.
319 2015-03-03 22:05:01 <gmaxwell> (by default, I mean, if people want disk filling logging they can have it)
320 2015-03-03 22:05:19 <belcher> spadinskiz pybitcointools is pretty simple, though might not do what you want
321 2015-03-03 22:05:27 <gmaxwell> makes _no_ sense for the network to store fewer blocks just to have more logs.
322 2015-03-03 22:05:32 <morcos> I was just hoping thats what you meant... and it does fill up quickly... couple gigs a month
323 2015-03-03 22:05:41 <belcher> also you could just use bitcoin json-rpc for sending
324 2015-03-03 22:05:49 <spadinskiz> belcher do you think it'd be fairly easy to pick up for someone who's new to coding?
325 2015-03-03 22:06:05 <belcher> yes, definitely
326 2015-03-03 22:06:11 <gmaxwell> ACTION bbiab
327 2015-03-03 22:06:43 <belcher> i dont actually know, but the correct attitude is always "try it and maybe fail"
328 2015-03-03 22:06:54 <spadinskiz> belcher what do you mean when you say it might not do what I want?
329 2015-03-03 22:07:34 <belcher> it might not have the features you need, im not actually clear what you're trying to do so its hard to say
330 2015-03-03 22:09:15 <spadinskiz> belcher I'm making a gambling website, I'm about 80% done with the backend and html stuff all I really have left to do is add bitcoin. I just need people to be able to deposit coins, when they bet them I need to be able to send the coins to a holding address, then send most of them to the winner and some to me as a fee, and then people need to be able to withdraw coins.
331 2015-03-03 22:09:19 <spadinskiz> That's all I'm trying to do
332 2015-03-03 22:10:24 <belcher> given that you're new to coding, you dont really have the skills to write a site handling other people's money
333 2015-03-03 22:10:44 <spadinskiz> belcher I'm having my more experienced friends review the code before I let anyone use it
334 2015-03-03 22:10:52 <belcher> mind you dont end up like mark karpeles
335 2015-03-03 22:11:02 <belcher> well, whatever
336 2015-03-03 22:11:10 <belcher> you'd probably want to use json-rpc of bitcoin code
337 2015-03-03 22:11:13 <spadinskiz> I've managed other people's bitcoins plenty of times, the only difference now is that I'm writing the website instead of managing it
338 2015-03-03 22:11:16 <belcher> err, bitcoin core
339 2015-03-03 22:11:23 <spadinskiz> alright thanks for the info
340 2015-03-03 22:11:36 <belcher> where you send inter process communication queries to bitcoin core and it does stuff, like send bitcoins around
341 2015-03-03 22:12:15 <spadinskiz> do you think that when I move bitcoins from one account to another, but they stay in the website, that I should just modify database values or is it best to actually make a bitcoin transaction?
342 2015-03-03 22:13:19 <belcher> just update your internal database, moving btc around costs time and money
343 2015-03-03 22:19:37 <goatpig> hi
344 2015-03-03 22:31:17 <belcher> in bitcoin 0.10 importing lots of watchonly addresses is slow as hell, if i ran it with -txindex would it be faster? since it would have already scanned
345 2015-03-03 22:31:26 <belcher> would like to know before i put in the time to do -txindex
346 2015-03-03 22:32:29 <goatpig> just curious, is there an RPC command to ask bitcoind to shutdown?
347 2015-03-03 22:32:54 <ajweiss> stop
348 2015-03-03 22:34:56 <goatpig> ok im an idiot
349 2015-03-03 22:34:57 <goatpig> thanks
350 2015-03-03 22:40:26 <gmaxwell> belcher: no, txindex will not make it faster.
351 2015-03-03 22:40:55 <gmaxwell> belcher: you can tell it to not rescan and then restart once with rescan for an n-fold improvement. (or if they're not used yet, just never scan.)
352 2015-03-03 22:46:26 <phantomcircuit> belcher, the rescan has to actually rescan every single transaction output
353 2015-03-03 22:46:49 <phantomcircuit> also if you dont actually care about historical use you can simply not rescan
354 2015-03-03 22:47:12 <phantomcircuit> gmaxwell, actually im guessing that optimization isn't done
355 2015-03-03 22:47:20 <phantomcircuit> in theory we could just walk the utxo
356 2015-03-03 22:52:27 <gmaxwell> phantomcircuit: well we can't without changing what a wallet does.
357 2015-03-03 22:52:58 <gmaxwell> I mean, it could be quite problematic for someone when they _believe_ they've seen the whole history, but have not.
358 2015-03-03 22:53:27 <gmaxwell> Including causing them to rip people off in a way that is indistinguishable from fraud.
359 2015-03-03 22:55:05 <phantomcircuit> gmaxwell, sure we can scanning the utxo and then use the txindex to find the full transaction
360 2015-03-03 22:55:13 <phantomcircuit> oh
361 2015-03-03 22:55:20 <gmaxwell> hah. No.
362 2015-03-03 22:55:20 <phantomcircuit> yeah there's ui issues for sure
363 2015-03-03 22:55:42 <gmaxwell> I mean, if you have no more outputs from that transaction there is nothing to find in the utxo set.
364 2015-03-03 22:55:44 <phantomcircuit> gmaxwell, :P
365 2015-03-03 22:55:52 <phantomcircuit> yeah that's what i was saying
366 2015-03-03 22:56:15 <phantomcircuit> some people might actually not care about transaction without spendable outputs
367 2015-03-03 22:57:34 <gmaxwell> Say you and I have a deal to split funds paid to some address 50/50. Funds get moved around. And later I go to make good on our deal, I import that key; and then send you half of what I see. Later you realize there was more and claim I defrauded you. Indeed, there are many times you don't care; but sorting out when you care or not is tricky. One possibility would be having a mode that only shows
368 2015-03-03 22:57:40 <gmaxwell> a balance and transactions within some time window. In that mode it would be safe (only rescan that window) and would have a consistent UI.
369 2015-03-03 22:59:07 <phantomcircuit> gmaxwell, ok so we agree
370 2015-03-03 22:59:11 <phantomcircuit> :P
371 2015-03-03 22:59:42 <gmaxwell> Yea. kind of obnoxious... for a wallet that was around the whole time the windowed view is super pointless.
372 2015-03-03 23:13:55 <jgarzik> belcher, You can import your watchonly addresses with a recent timestamp, limiting the rescanning
373 2015-03-03 23:15:01 <belcher> how do you import with a timestamp?
374 2015-03-03 23:18:32 <belcher> it just has rescan