1 2014-07-31 01:07:54 <super3> asked this in wizards too, but does anyone have stats on the average bitcoin transaction size?
2 2014-07-31 01:08:23 <gmaxwell> super3: why don't you go measure it yourself? then you could control for or exclude things according to your interests.
3 2014-07-31 01:09:44 <super3> hmm that would be a fun project
4 2014-07-31 01:09:57 <super3> gmaxwell: do you recommend and tools or existing libs to make that easy?
5 2014-07-31 01:10:04 <super3> any*
6 2014-07-31 01:11:11 <gmaxwell> super3: I mean, it's a shell script oneliner... or use pybitcoinlib.
7 2014-07-31 01:11:34 <super3> wait. i could just take the blockchain size divided by the # of transactions right?
8 2014-07-31 01:11:40 <sipa> BLOCK=300000; ./bitcoin-cli getblock $(./bitcoin-cli getblockhash $BLOCK) >/tmp/a; echo $(fgrep size </tmp/a | cut -d ':' -f 2 | cut -d ',' -f 1) / $(fgrep ' ' </tmp/a | wc -l) | bc -ql
9 2014-07-31 01:12:28 <sipa> seems around 500-600 bytes
10 2014-07-31 01:13:21 <super3> ok, that give some validity to the wiki page
11 2014-07-31 02:01:40 <cfields> sipa: around?
12 2014-07-31 02:17:31 <cfields> sipa: just fyi: I'm very happy for reviews, but it's not cleaned up yet.. just stuff piled on as I solve problems. So i imagine it's probably pretty rough to read.
13 2014-07-31 02:17:52 <cfields> I think i've pretty much crossed everything off wrt deps now though, so the cleanup is up next
14 2014-07-31 05:00:37 <jgarzik> It is disappointing there appears to be no high speed way to get a "linearized" (per linearize.py) list of block hashes.
15 2014-07-31 05:00:41 <jgarzik> via RPC
16 2014-07-31 05:00:49 <jgarzik> need a getblockheaders RPC
17 2014-07-31 05:01:35 <Belxjander> Heya jgarzik
18 2014-07-31 05:01:47 <Belxjander> jgarzik: I'm looking again at the bitcoin-qt core code...
19 2014-07-31 05:03:50 <gmaxwell> jgarzik: iirc getblockhash at least got caching so it's only slow due to jsonrpc overhead.
20 2014-07-31 05:04:26 <Belxjander> gmaxwell: is there any particular location with an array or list of the RPC function call set?
21 2014-07-31 05:04:56 <jcorgan> rpcserver.cpp
22 2014-07-31 05:48:02 <jgarzik> gmaxwell, good point
23 2014-07-31 06:24:34 <wumpus> at least use batching to request a whole slew at once
24 2014-07-31 06:26:06 <wumpus> seems the current linearize.py doesn't do that
25 2014-07-31 07:20:15 <Questionin> Is there a discussion anywhere that explains the decision to make transaction values go from an input to an output as a discrete quantity (completely spending the entire input and returning it as change). Rather than simply summing or subtracting the total value and storing the aggregate in the address?
26 2014-07-31 07:21:23 <dsnrk> it's easier to scale with outputs.
27 2014-07-31 07:22:01 <Questionin> dsnrk: Not sure I follow.
28 2014-07-31 07:23:00 <dsnrk> well with an output you can use a pruned merkle tree to prove an output is in a particular block with no trust.
29 2014-07-31 07:23:10 <gmaxwell> Questionin: the notion of an address as an "account" is pretty inconsistent with how the system works. There are no addresses, per se, in the backendâ a scriptPubKey sets up conditions which much be satisfied to spend a coin. In the original vision of the system every single coin would have a distinct scriptPubkey. Beyond preserving privacy, this design permits compact (log() scaling) proofs for payments being processed so light ...
30 2014-07-31 07:23:17 <gmaxwell> ... clients can reliably track what they can spend. It also permits validating systems to efficiently track working data in a way that is robust with respect to reorginization.
31 2014-07-31 07:23:43 <gmaxwell> Questionin: consider what happens if you have a very simple balanced based system. Say you pay me. You have coins remaining. Now I rebroadcast your payment. Oops, you've now paid me twice.
32 2014-07-31 07:24:24 <dsnrk> not all transactions involve addresses at all either, some have scripts which don't fit into the "balance" model at all.
33 2014-07-31 07:24:30 <gmaxwell> Preventing this requires anti-replay tokens, but the straightforward way of implementing them requires a reciever of coins to be shown a non-existance proof for a replay token in a set of already existing tokens that grows forever and cannot be forgotten.
34 2014-07-31 07:25:25 <gmaxwell> Then also consider cases like, I pay alice, and I pay bob. Both are unconfirmed. Alice becomes mad that the payment hasn't gone through and asks me to reissue with more fees. I do, but alice-reissue and the payment to alice goes through, and bob's cant because there isn't adequate balance left.
35 2014-07-31 07:26:53 <gmaxwell> Resolving these issues, controlled conflicts and anti-replay, results in a system which is isomorphic to one which just tracks coins instead of balances in the first place... I suppose we could instead imagine a whole new language to explain what it already does in those terms... but we're still left with the design that really expects every scriptPubKey to be unique.
36 2014-07-31 07:28:42 <gmaxwell> dsnrk: yes, thats trueâ though you could just 'widen' your concept of an address and say that an 'address' is the complete scriptPubkey. E.g. A coin that can be spent by (A&&B)||C||D is part of the "(A&&B)||C||D" balance not the A, B, C, or D ones.
37 2014-07-31 07:29:30 <dsnrk> gmaxwell: I'm not even sure most people realise this is even a feature of bitcoin at all. the blockchain.info teaching is that every output has an address.
38 2014-07-31 07:30:13 <Belxjander> gmaxwell: any particular place to start for wallet specific data and functions?
39 2014-07-31 07:31:05 <gmaxwell> dsnrk: along with the broken notion that transactions have 'from's too.. alas.. I've had good luck picking transactions like https://blockchain.info/tx/54fabd73f1d20c980a0686bf0035078e07f69c58437e4d586fb29aa0bee9814f and challenging those folks to tell me where it came 'from'.
40 2014-07-31 07:31:30 <gmaxwell> Belxjander: wallet.h / wallet.cpp?
41 2014-07-31 07:32:02 <Belxjander> I was more thinking "which object?" instead of file...
42 2014-07-31 07:32:16 <Belxjander> but I'll do what I can in trying to decipher what is actually going on
43 2014-07-31 07:33:05 <Questionin> gmaxwell: I can see how a replay attack is a problem, but if the whole balance is spent amongst two outputs (one for the original sender and the recipient). So long as all parties have the whole blockchain. I don't see totaling the values to be of an issue (each individual transaction can always be deduced by subtracting the current from the last).
44 2014-07-31 07:34:00 <dsnrk> Questionin: you can't really make the assumption that all parties have the block chain. bitcoin is designed to not need that at all for SPV clients.
45 2014-07-31 07:34:12 <gmaxwell> "So long as all parties have the whole blockchain" â You don't see a problem there? You don't need the whole blockchain to run a wallet today, in fact you don't even need the whole blockchain to fully validate new blocks.
46 2014-07-31 07:34:30 <Questionin> gmaxwell: And that's a beautiful thing.
47 2014-07-31 07:35:05 <gmaxwell> Questionin: and yes, if you always the whole balance then you close off some of the problemsâ but you've just accomplished something that looks very much like the current design. :)
48 2014-07-31 07:35:22 <gmaxwell> (except because the current design guarentees it, we can do validation with very small amounts of state)
49 2014-07-31 07:36:14 <Questionin> Say a person has a lightweight client. Is there ever a chance a person can't send from an input that they might otherwise have, but if they had the full balance tracked they could?
50 2014-07-31 07:36:42 <gmaxwell> I don't follow your question.
51 2014-07-31 07:37:57 <Belxjander> Questionin: actually that won't happen as inputs can be used various ways for any given transaction to be "correct" without enforcing the use of the tracked balance instead of a comparison of the whole input vs an output and outputs allowing partials to sum for completion afaik
52 2014-07-31 07:38:03 <gmaxwell> Questionin: okay, I've read it a few times and I think you're askingâ can a lightweight client be mistaken about what it can spend?
53 2014-07-31 07:38:20 <Questionin> I'm saying since bitcoin tracks each coin as a "thing." If I have a lightweight client and my client doesn't see my full history because it doesn't have the full blockchain. In a model where the full the balance is tracked. Then I would be able to spend that unit.
54 2014-07-31 07:38:22 <gmaxwell> Or more specifically, can it think it can spend less than it really can.
55 2014-07-31 07:39:03 <gmaxwell> Questionin: depends on the nature of your lightweight client. A lightweight client can observe the whole history, but only remeber things relevent to itâ all the original lightweight clients worked this way.
56 2014-07-31 07:39:26 <gmaxwell> (not having to store everything else is still an pretty big improvement)
57 2014-07-31 07:40:12 <gmaxwell> Some modern existing lightweight wallets, like electrum query some publically accessible systems and could have some of their coins concealed from them, however. So they just wouldn't know they could spend those.
58 2014-07-31 07:41:39 <Luke-Jr> Questionin: addresses should only ever be used once ever, so this problem is imaginary ;)
59 2014-07-31 07:41:41 <gmaxwell> Some people have wanted to include special scriptPubKey indexed utxo commitments in blocks to make those kinds of lookups provable too... personally I think the added cost for verifying nodes isn't worth it, especially since wallets could query multiple peers and if all peers they could find wanted to DOS attack them, they could no matter what techniques were in use.
60 2014-07-31 07:41:54 <Questionin> Luke-Jr: =) I like the way you think.
61 2014-07-31 07:42:03 <gmaxwell> Luke-Jr: well not quite imaginary! I could just conceal a complete payment from, say, an electrum wallet.
62 2014-07-31 07:42:20 <gmaxwell> (though you're pretty likely to notice that!)
63 2014-07-31 07:42:25 <Questionin> gmaxwell: It's an interesting thing to think about.
64 2014-07-31 07:42:39 <Questionin> gmaxwell: At first my reaction was negative towards an aggregate.
65 2014-07-31 07:42:47 <Questionin> But I can see some uses.
66 2014-07-31 07:42:54 <dsnrk> wonder if Electrum makes a local cache of UTXO and diffs it with the remote server.
67 2014-07-31 07:42:54 <gmaxwell> Questionin: an aggregate doesn't solve this.
68 2014-07-31 07:43:05 <Luke-Jr> gmaxwell: I think we mean different problems :p
69 2014-07-31 07:43:36 <gmaxwell> dsnrk: hm? no it just tracks its own inputs.
70 2014-07-31 07:44:05 <dsnrk> gmaxwell: of it's wallets UTXO I mean. checking to see if any outputs don't get returned by a server that did before.
71 2014-07-31 07:44:24 <dsnrk> ie, basic censorship detection.
72 2014-07-31 07:44:38 <gmaxwell> dsnrk: really these queries should be made PIR, which would also make them censor proof... but AFAIK no one is working on that.
73 2014-07-31 07:44:50 <jcorgan> PIR?
74 2014-07-31 07:45:12 <dsnrk> good gives me Portland International Raceway, which can't be right.
75 2014-07-31 07:45:15 <dsnrk> *google
76 2014-07-31 07:45:29 <gmaxwell> http://en.wikipedia.org/wiki/Private_information_retrieval
77 2014-07-31 07:45:41 <dsnrk> oh, bloom filters?
78 2014-07-31 07:45:46 <gmaxwell> f-no.
79 2014-07-31 07:46:21 <gmaxwell> (or more practically, http://percy.sourceforge.net/ which is a realistic implementation of the underlying primitives)
80 2014-07-31 07:46:32 <dsnrk> doesn't a lossy bloom filter achieve that to some degree? if you set the filter too small you get junk back too.
81 2014-07-31 07:47:12 <dsnrk> looks like I've some reading to do.
82 2014-07-31 07:47:23 <gmaxwell> dsnrk: no, because you have no control at over the leakage amount, and it's almost always so leaky that if a single query doesn't deanonymize you, two or three over time do. :(
83 2014-07-31 07:47:29 <jcorgan> ACTION adds yet another gmaxwell topic to his queue of "things to go off and research when you want to avoid real work" :)
84 2014-07-31 07:48:50 <gmaxwell> PIR is fun, information theoretic PIR is practicalâ but requires an assumption of n servers that won't conspire against you. (computational PIR requires so much cpu on the server its usually cheaper to just send the whole database).
85 2014-07-31 07:49:04 <dsnrk> gmaxwell: you could build your filter from say, the first 500 keys of a BIP32 wallet and then the filter would be for all intents totally static. still easy to work out what's mess and what's not though, all of the ones you're interested in would probably interact as change addresses, versus ones which never merge being the chaff.
86 2014-07-31 07:50:09 <dsnrk> issue is even with perfect PIR, you've still got to broadcast transactions- which shows off what your addresses are anyway.
87 2014-07-31 07:51:26 <gmaxwell> dsnrk: but thats not inherently linking in the way that incoming payment lookups are, because the sends are rare events, while you need to poll or constantly scan on the incoming side.
88 2014-07-31 07:52:05 <dsnrk> true.
89 2014-07-31 07:52:25 <dsnrk> wonder how many of the electrum nodes are "evil" ones logging every request. you'd expect at least one of them is.
90 2014-07-31 07:53:34 <dsnrk> short of being blockchain.info that's the cheapest way I can think of to expose large numbers of people's addresses.
91 2014-07-31 08:00:16 <Belxjander> Would it be legitimate for a node implimenation to use any DB backend of choice as long as the clients and network facing connectivity was unchanged?
92 2014-07-31 08:00:32 <Belxjander> and only node local storage was affected where an "export" tool was also available for the same?
93 2014-07-31 08:02:39 <gmaxwell> Belxjander: if you're asking these questions you're probably not ready to make a node implementation... but if its externally invisible it doesn't matter. But there isn't a guarentee that the blockchain database implementation isn't leaking through a bit, and a node that falls out of consensus is a very sad node indeed.
94 2014-07-31 08:04:31 <Belxjander> gmaxwell: I'm more trying to just work through concept for building a node then adding a wallet on top...and connecting to ASIC mining hardware later if I can see a practical way for each step
95 2014-07-31 08:05:06 <Belxjander> but right now I have no sense of what practical steps are needed to just get the blockchain locally stored and accessible for anything at this point
96 2014-07-31 08:11:46 <stonecoldpat> gmaxwell: im looking at your wizardary transaction, it still requires a signature from a specified public key right? (pubkey is in the scriptpubkey) - so argubly that is still from a bitcoin address (just need to hash it up), its the same (needing more than just a signature) as needing a sig + pre-image of hash approach used in your coinswap protocol
97 2014-07-31 08:13:06 <gmaxwell> stonecoldpat: nah, it's not really a public key (I mean, I found that txout randomly on the blockchain, and solved it)
98 2014-07-31 08:13:16 <gmaxwell> stonecoldpat: obviously there are also txouts with no public keys at all.
99 2014-07-31 08:15:05 <gmaxwell> if thats causing a mental hangup for you: https://blockchain.info/tx/4f1433d6433d3ce8a877519ba9ddc310dbee96dba939aca0dbef0176a3563436
100 2014-07-31 08:16:05 <dsnrk> that's weird. Insight thinks that output is uncomfirmed (what).
101 2014-07-31 08:25:05 <stonecoldpat> thats very weird, initially i assumed OP_DROP would drop the rest of the script, but it would drop 04678afd04678afd from the stack, and then hash 04678afd04678a, but 04678afd04678a != sha256(894eeb82f9a851f5d1cb1be3249f58bc8d259963832c5e7474a76f7a859ee95c), the internet has beat me today
102 2014-07-31 08:26:41 <gmaxwell> stonecoldpat: stack machine, never used a RPN calculator?
103 2014-07-31 08:26:49 <gmaxwell> the operations are always operating on the stack.
104 2014-07-31 08:26:57 <gmaxwell> never on the tape
105 2014-07-31 08:29:08 <stonecoldpat> gmaxwell: i haven't (an area I need to practice for sure), I was using https://en.bitcoin.it/wiki/Script#Standard_Transaction_to_Bitcoin_address_.28pay-to-pubkey-hash.29 as reference
106 2014-07-31 08:30:06 <stonecoldpat> so you consider scriptsig the tape, and scriptpubkey the stack?
107 2014-07-31 08:31:33 <gmaxwell> no, both are tapes. the stack exists only during execution.
108 2014-07-31 08:32:06 <dsnrk> stonecoldpat: this shows it quite well. the stack is on the left, execution follows down the table through each operation. http://webbtc.com/script/5d445ddf7cad4bb79bb49e2c3695a1b73c7be3283ec87f2c454270b41418f41a:0
109 2014-07-31 08:32:13 <gmaxwell> the stack starts empty, step through the script sig tape and perform every action it specifies, updating the stack as you go, then the scriptpubkey.
110 2014-07-31 08:32:28 <gmaxwell> dsnrk: pft, cheater. :P
111 2014-07-31 08:41:02 <stonecoldpat> dsnrk: +1 for that link, being bookmarked, very nice.
112 2014-07-31 08:43:01 <stonecoldpat> gmaxwell: i understand how that transaction works now thanks guys :)
113 2014-07-31 09:49:35 <petertod1> sipa: yeah, separate eval was a hardfork due to the concatenation issue. i found thst issue a few months ago and posted it in -wizards (or was it -dev?)
114 2014-07-31 09:50:03 <petertod1> theymos: link to the stack overflow thing? i wonder if they beat me too it
115 2014-07-31 12:04:16 <zigz> hi
116 2014-07-31 12:04:20 <zigz> what does keystore.cpp do?
117 2014-07-31 12:06:13 <zigz> and alert.cpp
118 2014-07-31 12:06:26 <zigz> is there proper documentation somewhere for this?
119 2014-07-31 12:16:07 <sipa> zigz: keystore defines an interface (and some implementations) for something that stores key
120 2014-07-31 12:16:24 <sipa> zigz: so that script can depend on that interface, instead of needing to depend on the entire wallet code
121 2014-07-31 12:16:45 <sipa> alert.cpp deals with alerts
122 2014-07-31 14:05:18 <gribble> jgarzik was last seen in #bitcoin-dev 8 hours, 17 minutes, and 15 seconds ago: <jgarzik> gmaxwell, good point
123 2014-07-31 14:05:18 <michagogo> ;;seen jgarzik
124 2014-07-31 14:06:00 <michagogo> Was cbe39a38526a6c17619d02cc697b80ebfd57203b PR'd?
125 2014-07-31 14:06:33 <michagogo> Looks like it was merged in a715a643bcda579c324b2c1d2dac449a320951c9
126 2014-07-31 14:07:00 <gribble> home_jg was last seen in #bitcoin-dev 2 days, 21 hours, 9 minutes, and 3 seconds ago: <home_jg> jcorgan, +1 re discovering zmq port. please don't add it to "getinfo" kitchen sink though. I think there is a "getnetworkinfo" or somesuch.
127 2014-07-31 14:07:00 <michagogo> ;;seen home_jg
128 2014-07-31 14:08:39 <michagogo> home_jg: ping?
129 2014-07-31 14:10:21 <home_jg> michagogo, da?
130 2014-07-31 14:10:37 <michagogo> jgarzik: ^
131 2014-07-31 14:11:03 <michagogo> Was cbe39a38526a6c17619d02cc697b80ebfd57203b PR'd? I see you merging it with commit a715a643bcda579c324b2c1d2dac449a320951c9
132 2014-07-31 14:11:22 <michagogo> Also, it seems you forgot to make it optional at configure-time
133 2014-07-31 14:11:25 <jgarzik> michagogo, yes #4332
134 2014-07-31 14:11:34 <jgarzik> michagogo, no, not forgotten
135 2014-07-31 14:11:45 <michagogo> hm?
136 2014-07-31 14:12:07 <jgarzik> michagogo, intentional. that's a wrong practice to add --enable-just-one-prog for every tiny util or prog.
137 2014-07-31 14:12:32 <michagogo> ...we have --with-cli
138 2014-07-31 14:12:36 <michagogo> Is that not comparable?
139 2014-07-31 14:12:44 <jgarzik> michagogo, that needs to go
140 2014-07-31 14:13:07 <michagogo> Why?
141 2014-07-31 14:13:09 <zigz> <sipa> alert.cpp deals with alerts
142 2014-07-31 14:13:12 <jgarzik> michagogo, further, the logic must not be "is there an existing feature like this?" for that leads to blind additions.
143 2014-07-31 14:13:17 <zigz> i thought as much
144 2014-07-31 14:13:21 <jgarzik> michagogo, the logic must be, "are there users that need this?"
145 2014-07-31 14:13:26 <michagogo> What if someone wants one and not the other?
146 2014-07-31 14:13:27 <jgarzik> answer: not really. maybe 1.
147 2014-07-31 14:13:42 <jgarzik> michagogo, who cares? they can cp or rm as they need.
148 2014-07-31 14:13:54 <zigz> sipa: https://en.bitcoin.it/wiki/Bitcoin_Alert is about an android app
149 2014-07-31 14:14:07 <jgarzik> michagogo, the vast majority of builders build everything, then pick what to package.
150 2014-07-31 14:14:07 <michagogo> jgarzik: is it always compiled, or is it compiled iff one of the other things is compiled?
151 2014-07-31 14:14:11 <zigz> sipa: i assume alert.cpp is something else, but what?
152 2014-07-31 14:14:19 <jgarzik> michagogo, always compiled
153 2014-07-31 14:14:32 <michagogo> hm
154 2014-07-31 14:15:04 <kinlo> zigz: https://en.bitcoin.it/wiki/Alerts
155 2014-07-31 14:27:13 <jgarzik> About to update blockchain torrent. Anybody know of a good HTTP tracker or two? I only have UDP trackers listed.
156 2014-07-31 14:28:20 <michagogo> jgarzik: which blocks are you adding?
157 2014-07-31 14:28:28 <wumpus> michagogo: they could do make <executablename>
158 2014-07-31 14:28:41 <jgarzik> michagogo, up to 313,000
159 2014-07-31 14:28:58 <wumpus> michagogo: we do need to keep a way to enable/disable the large units, such as the qt gui
160 2014-07-31 14:29:06 <jgarzik> wumpus, +1
161 2014-07-31 14:29:53 <wumpus> something could be said for always building the tests
162 2014-07-31 14:29:58 <jgarzik> michagogo, I tend to update the torrent to almost-latest block, usually tip-288 or somesuch.
163 2014-07-31 14:30:38 <hearn> jgarzik: how do you think the torrent will compare to headers-first sync? it seems with headers first bandwidth is no longer the constraint on sync speed
164 2014-07-31 14:30:51 <jgarzik> michagogo, by the time the torrent has been created and beta-tested and uploaded, it is usually tip-(288*N) blocks
165 2014-07-31 14:31:07 <jgarzik> hearn, yes, headers-first should be faster than torrent in theory
166 2014-07-31 14:31:16 <jgarzik> hearn, we have far higher replication factor too
167 2014-07-31 14:31:21 <jgarzik> on bitcoin P2P
168 2014-07-31 14:31:44 <michagogo> https://www.irccloud.com/pastebin/rk6wWWSD
169 2014-07-31 14:33:44 <jgarzik> michagogo, beta @ http://gtf.org/garzik/bitcoin/bootstrap.dat.torrent
170 2014-07-31 14:33:52 <michagogo> Should just be able to run that, and then cat bootstrap.dat.295001-313000 >> bootstrap.dat, right?
171 2014-07-31 14:34:15 <jgarzik> michagogo, yes, though be careful of off-by-one
172 2014-07-31 14:34:31 <michagogo> jgarzik: hm?
173 2014-07-31 14:35:26 <michagogo> (by "run that" I mean run linearize.py with that config)
174 2014-07-31 14:35:36 <jgarzik> michagogo, yes
175 2014-07-31 14:35:56 <michagogo> What do you mean by "be careful of off-by-one"?
176 2014-07-31 14:36:48 <jgarzik> 08ec49237decea485b246ff7f29e8ec900ffcfef4388a53be92428e800317df6 bootstrap.dat
177 2014-07-31 14:36:48 <jgarzik> $ sha256sum bootstrap.dat
178 2014-07-31 14:37:20 <jgarzik> michagogo, I forget how the linearize.py loop is written. It might want height-1 as start or something. I ran into that a couple times when I had the first couple bootstrap.dat's.
179 2014-07-31 14:37:46 <michagogo> I actually tweaked linearize.py at some point
180 2014-07-31 14:38:05 <michagogo> (surprising that I managed to do it on the first try, considering I don't know Python)
181 2014-07-31 14:39:05 <michagogo> for height in xrange(settings['min_height'], settings['max_height']+1):
182 2014-07-31 14:39:19 <michagogo> ;;google python xrange
183 2014-07-31 14:39:20 <gribble> 2. Built-in Functions â Python v2.7.8 documentation: <https://docs.python.org/2/library/functions.html>; Difference between range and xrange ?? - Python.org mailing lists: <https://mail.python.org/pipermail/python-list/2012-November/634509.html>; python - What is the difference between range and xrange? - Stack ...: <http://stackoverflow.com/questions/94935/what-is-the-difference- (1 more message)
184 2014-07-31 14:40:12 <michagogo> [1, 2, 3, 4, 5, 6, 7, 8, 9, 10]
185 2014-07-31 14:40:12 <michagogo> >>> range(1, 11)
186 2014-07-31 14:40:32 <michagogo> So 295001 should be right
187 2014-07-31 14:41:47 <michagogo> I wish μTorrent supported updating of a torrent
188 2014-07-31 14:42:06 <michagogo> Instead, I need to remove the torrent and add the new one, discarding the stats :(
189 2014-07-31 14:43:00 <michagogo> jgarzik: http://exodus.desync.com:6969/announce ?
190 2014-07-31 14:43:20 <michagogo> (re: 17:27:11 <jgarzik> About to update blockchain torrent. Anybody know of a good HTTP tracker or two? I only have UDP trackers listed. )
191 2014-07-31 14:45:36 <michagogo> jgarzik: what's the size of the new torrent, btw?
192 2014-07-31 14:46:53 <jgarzik> michagogo, 21575802677 bytes
193 2014-07-31 14:47:03 <sipa> :o
194 2014-07-31 14:47:09 <hearn> gavinandresen: hey, around? do you know what kind of hardware/network bitcoincore.org has?
195 2014-07-31 14:47:44 <michagogo> Oh, and is the torrent you linked the exact one that'll be posted if nothing comes up?
196 2014-07-31 14:48:31 <gavinandresen> hearn: one sec, let me look
197 2014-07-31 14:51:51 <jgarzik> michagogo, yes
198 2014-07-31 14:52:12 <jgarzik> ACTION is currently fighting with $something in Ubuntu that keeps restarting transmission-daemon, even though I kill it.
199 2014-07-31 14:52:26 <jgarzik> Virus? Malware? Most likely..... systemd.
200 2014-07-31 14:53:05 <pigeons> systemd might be the straw that chases people to netbsd
201 2014-07-31 14:54:25 <jgarzik> pigeons, major Linux folks hate systemd... but it would take a lot more to chase folks to NetBSD I think ;p
202 2014-07-31 14:54:33 <michagogo> just 5k blocks to go...
203 2014-07-31 14:55:02 <michagogo> I wish that script could somehow just copy the blocks from disk
204 2014-07-31 14:55:40 <jgarzik> michagogo, yah, it needs a rewrite to (a) grab block hashes from RPC, then (b) scan blocks on disk
205 2014-07-31 14:55:43 <michagogo> (I wonder how much of this time is disk I/O and how much is assorted overhead)
206 2014-07-31 14:55:44 <jgarzik> michagogo, much much faster
207 2014-07-31 14:55:49 <jgarzik> michagogo, overhead
208 2014-07-31 14:56:04 <michagogo> jgarzik: I know, I'm sure there's a bunch of overhead
209 2014-07-31 14:56:09 <michagogo> I'm just wondering, how much
210 2014-07-31 14:56:38 <michagogo> Hm, I might see if I can hack it to add a timestamp to the messages it prints
211 2014-07-31 14:57:16 <michagogo> (again, I don't know Python...)
212 2014-07-31 14:57:31 <jgarzik> michagogo, >50% overhead. a major amount.
213 2014-07-31 14:58:33 <jgarzik> michagogo, Also, I have been thinking for a while that the torrent should be blocks/blk*.dat and then -reindex, for zero copying
214 2014-07-31 14:58:39 <jgarzik> michagogo, current method results in 21G of duplication
215 2014-07-31 14:59:05 <michagogo> Or change the imported to eat the file :P
216 2014-07-31 14:59:08 <michagogo> importer*
217 2014-07-31 14:59:52 <michagogo> The thing is, different people can end up with different blocks/blk*.dat
218 2014-07-31 15:00:05 <sipa> jgarzik: just rename bootstrap.dat to blocks/blk00000.dat, and reindex
219 2014-07-31 15:00:15 <sipa> jgarzik: there's no constraints on the blk file sizes afaik
220 2014-07-31 15:00:16 <michagogo> If they get onto a short fork at some point, or if they upgraded from an older version
221 2014-07-31 15:00:19 <sipa> oh, maybe 4G
222 2014-07-31 15:02:11 <michagogo> Wrote block 313000
223 2014-07-31 15:02:13 <michagogo> \o/
224 2014-07-31 15:02:50 <michagogo> Now as soon as I get the result of $ cat /cygdrive/c/Users/Micha/.../bootstrap.dat /cygdrive/c/Users/Micha/.../bootstrap.dat.295001-313000 | sha256sum
225 2014-07-31 15:02:55 <michagogo> I can update to the new torrent
226 2014-07-31 15:03:54 <jgarzik> sipa, actually there is
227 2014-07-31 15:03:58 <jgarzik> sipa, I tried that, and it failed
228 2014-07-31 15:04:19 <jgarzik> sipa, with near-HEAD master
229 2014-07-31 15:05:13 <jgarzik> michagogo, I have my home laptop seeding + cloudatcost.com slow server @ 0.1%, so more is welcome. :)
230 2014-07-31 15:05:26 <michagogo> hm
231 2014-07-31 15:05:36 <michagogo> I have an OpenVZ container with a lot of bandwidth
232 2014-07-31 15:05:44 <cdecker_> Will join the bittorrent seeds as soon as I am home
233 2014-07-31 15:05:45 <michagogo> ACTION checks the `df -h`
234 2014-07-31 15:06:10 <michagogo> 9.9G avail :/
235 2014-07-31 15:06:29 <cdecker_> Magnet link for the Torrent?
236 2014-07-31 15:07:09 <jgarzik> cdecker_, magnet:?xt=urn:btih:682759b001fac5d3b0d8517d2a179af2e65ebed6&dn=bootstrap.dat&tr=udp%3A%2F%2Ftracker.openbittorrent.com%3A80&tr=udp%3A%2F%2Ftracker.publicbt.com%3A80&tr=udp%3A%2F%2Ftracker.ccc.de%3A80&tr=udp%3A%2F%2Ftracker.istole.it%3A80
237 2014-07-31 15:07:22 <cdecker_> Thanks
238 2014-07-31 15:15:49 <sipa> jgarzik: yes, offset in block file is an unsigned int
239 2014-07-31 15:16:19 <sipa> jgarzik: it's stored as a varint, so we could just change it to int64_t
240 2014-07-31 15:16:43 <sipa> but that's an extra >1MiB of RAM
241 2014-07-31 15:17:06 <sipa> jgarzik: also, for pruning we really want smaller files
242 2014-07-31 15:17:22 <Guest86401> Guys, what's the best way to get a previous transaction hash for a new bitcoin address?
243 2014-07-31 15:17:23 <jgarzik> sipa, I think it's fine. It forces a split. :)
244 2014-07-31 15:17:35 <sipa> Guest86401: run a wallet? :)
245 2014-07-31 15:17:56 <sipa> you need to somehow know about transactions that pay you
246 2014-07-31 15:18:26 <sipa> jgarzik: it forces a split?
247 2014-07-31 15:19:23 <jgarzik> sipa, the offset being unsigned int implies a solution of "mv bootstrap.dat blocks/blk00000.dat" is impossible on current systems, forcing an alternative to be sought. Which is IMO good.
248 2014-07-31 15:19:51 <sipa> Guest86401: please stay in channel
249 2014-07-31 15:19:53 <jgarzik> sipa, or IOW, blkNNNNN.dat is therefore required to be <4G, which is fine
250 2014-07-31 15:20:08 <sipa> jgarzik: a torrent with blk*.dat files would be fine
251 2014-07-31 15:20:35 <jgarzik> sipa, yes, that is the probable plan for the next torrent.
252 2014-07-31 15:20:59 <sipa> i'd like to retry my effort of coming up with a compact block format sometime
253 2014-07-31 15:21:10 <sipa> using backreferences for txids, varint encodings for headers, ...
254 2014-07-31 15:21:16 <sipa> and piping it through zlib
255 2014-07-31 15:21:27 <jgarzik> Speaking of, people also call for a compressed torrent, which (a) shaves several GB, but (b) prevents re-use of old torrents.
256 2014-07-31 15:21:50 <jgarzik> and archive formats are not perfect across multiple OS's, vis a vis directories
257 2014-07-31 15:29:53 <michagogo> jgarzik: just don't use a directory and put "ZIP-BOMB" in the name :P
258 2014-07-31 15:32:49 <michagogo> Anyway, the thing is, if you make the torrent blk*.dat files, you make it harder for people to generate their own, don't you?
259 2014-07-31 15:34:47 <sipa> linearize.py could produce blk*.dat files
260 2014-07-31 15:35:22 <michagogo> But then you lose the advantage of not needing duplicate data
261 2014-07-31 15:37:11 <sipa> how so?
262 2014-07-31 15:37:15 <sipa> it can be deterministic
263 2014-07-31 15:38:09 <michagogo> Well, I guess if you then proceed to replace your current node's blocks and reindex, it could work
264 2014-07-31 15:38:49 <cdecker_> I wanted to ask: how do I register my GPG key for the gitian build system?
265 2014-07-31 15:39:02 <michagogo> cdr"register"?
266 2014-07-31 15:39:09 <michagogo> cdecker_: "register"?
267 2014-07-31 15:40:01 <cdecker_> Or do I simply build with any of my keys and then PR the signatures?
268 2014-07-31 15:40:05 <michagogo> sipa: only thing is you'd need to make sure not to include any blk*.dat files that aren't at max size
269 2014-07-31 15:40:34 <sipa> oh i see
270 2014-07-31 15:40:37 <sipa> well, that won't work
271 2014-07-31 15:40:46 <sipa> it'd be only good for replacing your current files
272 2014-07-31 15:41:20 <michagogo> And there, duplication can be solved by just having the import eat the file
273 2014-07-31 16:02:06 <jgarzik> michagogo, In theory, the "bootstrap.dat -> blkNNNNN.dat" process is deterministic. As sipa guessed, my plan by distributing blocks/blk*.dat would be to have linearize.py download block hashes via RPC, then produce a deterministic blocks/blk*.dat by scanning a local blocks/blk*.dat + block hash dataset.
274 2014-07-31 16:02:15 <jgarzik> It's still linearized, just split into multiple files.
275 2014-07-31 16:02:21 <jgarzik> ie. no forks stored
276 2014-07-31 16:03:00 <jgarzik> And blocks/blk*.dat still preserves the "append" property of torrent reuse.... presuming no compressing+archive is done
277 2014-07-31 16:03:12 <jgarzik> I think compression saves ~7G nowadays
278 2014-07-31 16:03:14 <jgarzik> via xz
279 2014-07-31 16:09:19 <michagogo> jgarzik: The question is, why do we want to distribute blk*.dat instead of bootstrap.dat
280 2014-07-31 16:09:49 <michagogo> The issue of it creating a duplicate when you use it to bootstrap can be solved by having the importer eat the file
281 2014-07-31 16:11:21 <jgarzik> michagogo, "import than delete" doesn't really work for lower resource machines (as I ran into with my own vps), as you still require $chainstorage*2.
282 2014-07-31 16:11:34 <jgarzik> michagogo, with -reindex, there is never any need for duplication, however temporary
283 2014-07-31 16:11:39 <michagogo> 19:09:46 <michagogo> The issue of it creating a duplicate when you use it to bootstrap can be solved by having the importer eat the file
284 2014-07-31 16:12:11 <michagogo> Btw, I got the storage upgraded on the container, so I can seed on there
285 2014-07-31 16:14:40 <jgarzik> michagogo, you cannot eat from the front to the back
286 2014-07-31 16:14:46 <jgarzik> michagogo, in a single 20G file
287 2014-07-31 16:14:54 <michagogo> You can't?
288 2014-07-31 16:15:13 <jgarzik> michagogo, no. filesystems can truncate (eat from back to front), but not the reverse.
289 2014-07-31 16:15:14 <michagogo> :-/
290 2014-07-31 16:15:27 <jgarzik> [well, not easily, and with boatloads of caveats]
291 2014-07-31 16:15:39 <michagogo> ...reverse the file? :P
292 2014-07-31 16:15:51 <jgarzik> michagogo, breaks existing bitcoind's
293 2014-07-31 16:16:04 <michagogo> er, right.
294 2014-07-31 16:16:26 <michagogo> hm
295 2014-07-31 16:16:31 <michagogo> ^
296 2014-07-31 16:16:31 <michagogo> File "linearize.py", line 5
297 2014-07-31 16:16:31 <michagogo> host=127.0.0.1
298 2014-07-31 16:16:31 <michagogo> micha@bitcoinnode:~/bootstrap$ python linearize.py example-linearize.cfg
299 2014-07-31 16:16:31 <michagogo> SyntaxError: invalid syntax
300 2014-07-31 16:16:50 <michagogo> Wait a minute
301 2014-07-31 16:16:53 <michagogo> WTF?
302 2014-07-31 16:17:08 <michagogo> How did I end up with linearize.py being the example conf?
303 2014-07-31 17:32:13 <denisx> with the switch from 0.8.x to 0.9 bitcoind generate 50% more disk io
304 2014-07-31 17:32:15 <denisx> is that ok?
305 2014-07-31 17:36:24 <sipa> increase your db cache :)
306 2014-07-31 17:36:29 <sipa> it's expected, yes
307 2014-07-31 17:36:34 <sipa> oh, 0.9?
308 2014-07-31 17:36:39 <sipa> no, it was expected from 0.7 to 0.8
309 2014-07-31 17:39:08 <hearn> sipa: yeah i was a bit mystified by that. i can't think of any changes that would have changed disk io a lot in 0.9
310 2014-07-31 17:39:14 <hearn> denisx: how are you measuring this?
311 2014-07-31 17:39:32 <denisx> hearn: my machine is a munin-node
312 2014-07-31 17:39:42 <denisx> Iâm collecting all the data
313 2014-07-31 17:39:49 <denisx> and disk IO is one of them
314 2014-07-31 17:39:51 <hearn> munin node?
315 2014-07-31 17:40:42 <denisx> is some sort of monitoring software
316 2014-07-31 17:41:00 <denisx> http://munin-monitoring.org
317 2014-07-31 17:43:11 <hearn> so you know for sure it comes from bitcoin? then i don't have any explanation ...
318 2014-07-31 17:43:23 <denisx> no, not 100% sure
319 2014-07-31 17:44:14 <denisx> the time it happened was exactly the time the freebsd port of bitcoin 0.9 was released
320 2014-07-31 17:45:55 <michagogo> Grr
321 2014-07-31 17:46:18 <michagogo> jgarzik: looks like aria2 doesn't support using an existing file to seed a torrent
322 2014-07-31 17:47:35 <jgarzik> michagogo, c'est la vie. That's why there is a pre-seed beta announce...
323 2014-07-31 17:47:52 <jgarzik> michagogo, My 2nd node is at ~25% for annoying reasons as well
324 2014-07-31 17:48:02 <michagogo> [#5b7a68 99MiB/20GiB(0%) CN:7 SD:1 DL:534KiB UL:563KiB(40MiB) ETA:10h54m14s]
325 2014-07-31 17:48:03 <michagogo> :-/
326 2014-07-31 17:50:39 <michagogo> maybe I should look for a different torrent client
327 2014-07-31 17:51:49 <denisx> try transmission
328 2014-07-31 18:29:15 <maaku> why is CFeeRate an int64, not a double?
329 2014-07-31 18:29:32 <maaku> the code it replaced at various points used both
330 2014-07-31 18:38:19 <michagogo> jgarzik: okay, seed is up
331 2014-07-31 18:43:14 <gdm85> did also other people here receive that mass mail (with recipients in clear Cc) about "louisdor project"
332 2014-07-31 19:19:57 <jcorgan> i did
333 2014-07-31 19:28:12 <wumpus> maaku: because we don't use doubles for monetary amounts
334 2014-07-31 19:34:21 <wumpus> or at least, shouldn't, there are still doubles used in the JSON parser and AmountFromValue/ValueFromAmount, hope we can get rid of those at some point too
335 2014-07-31 19:38:23 <michagogo> jgarzik: so I have it being seeded on a box with a bunch of bandwidth
336 2014-07-31 19:38:38 <michagogo> And then also on my laptop, when it's on
337 2014-07-31 19:50:53 <jgarzik> michagogo, yep, suddenly the torrent is healthy ;p
338 2014-07-31 19:51:13 <michagogo> Define "healthy"?
339 2014-07-31 19:51:39 <jgarzik> michagogo, several connected peers suddenly went to 100% in the past couple hours (since I left for late lunch)
340 2014-07-31 19:51:55 <jgarzik> versus a single seeder
341 2014-07-31 19:52:16 <michagogo> Are you 162.219.x.xx?
342 2014-07-31 19:54:13 <michagogo> I'm seeing 3 seeds
343 2014-07-31 19:54:22 <michagogo> I guess you have one?
344 2014-07-31 19:54:48 <michagogo> (or, you have two and the seed count doesn't include localhost)
345 2014-07-31 19:55:43 <jgarzik> michagogo, 24.98. and 162.219.
346 2014-07-31 19:55:54 <jgarzik> michagogo, transmission-daemon, both
347 2014-07-31 19:56:03 <maaku> wumpus: in consensus code sure, in the fee code doubles were used for good reason
348 2014-07-31 19:56:03 <michagogo> Guessing the former is a seed already?
349 2014-07-31 19:56:08 <jgarzik> michagogo, yes
350 2014-07-31 19:56:20 <michagogo> Both of my seeds are uploading to the latter
351 2014-07-31 19:56:40 <jgarzik> one quite rapidly ;p
352 2014-07-31 19:56:41 <michagogo> (albeit very, very slowly on my laptop :P )
353 2014-07-31 19:57:06 <michagogo> Yeah, 1.5-2 MB/s
354 2014-07-31 19:57:48 <michagogo> Wow, spiking up now
355 2014-07-31 19:59:13 <jgarzik> cloudatcost.com server is very bursty. sometimes it receives and sends at many mb/s, sometimes there are multi-second stalls.
356 2014-07-31 19:59:32 <michagogo> what's that?
357 2014-07-31 19:59:34 <michagogo> ACTION cliks
358 2014-07-31 19:59:37 <michagogo> clicks*
359 2014-07-31 20:00:59 <michagogo> Heh, cool
360 2014-07-31 20:01:18 <michagogo> If they took bitcoins, I might get one of the base ones
361 2014-07-31 20:01:28 <michagogo> Apparently they have half-off everything today
362 2014-07-31 20:01:45 <jgarzik> ACTION just started his eval. So far, looks overloaded but usable, so if you set expectations properly [low]...
363 2014-07-31 20:02:06 <michagogo> Right, I would assume as much
364 2014-07-31 20:33:57 <denisx> do you need help with seeding something?
365 2014-07-31 21:12:31 <sipa> ;;nethash
366 2014-07-31 21:12:32 <gribble> 149472406.697
367 2014-07-31 21:13:21 <sipa> 27 hours left until we hit 2^80 hashes of chainwork!
368 2014-07-31 21:15:36 <dsnrk> Total chain work: 2**79.98 hashes
369 2014-07-31 21:15:56 <dsnrk> like watching the odometer tick over, you just know you're going to miss it.
370 2014-07-31 21:16:56 <Emcy> little do we know
371 2014-07-31 21:17:12 <Emcy> sipa has a secret killswitch to stop hashing right on 2^80
372 2014-07-31 21:27:19 <michagogo> denisx: yes, the new bootstrap.dat
373 2014-07-31 21:27:38 <michagogo> magnet:?xt=urn:btih:682759b001fac5d3b0d8517d2a179af2e65ebed6&dn=bootstrap.dat&tr=udp%3A%2F%2Ftracker.openbittorrent.com%3A80&tr=udp%3A%2F%2Ftracker.publicbt.com%3A80&tr=udp%3A%2F%2Ftracker.ccc.de%3A80&tr=udp%3A%2F%2Ftracker.istole.it%3A80
374 2014-07-31 21:27:59 <denisx> ok, its running
375 2014-07-31 21:28:09 <denisx> even with ipv6
376 2014-07-31 21:29:04 <michagogo> I guess you're 195.160.xxx.x?
377 2014-07-31 21:29:50 <denisx> yep
378 2014-07-31 21:32:12 <denisx> michagogo: all peers have 100%
379 2014-07-31 21:32:39 <michagogo> denisx: yeah
380 2014-07-31 21:32:48 <michagogo> It's not announced (on the forum thread, etc) yet
381 2014-07-31 21:33:27 <denisx> I see, you are preparing?
382 2014-07-31 21:33:39 <michagogo> yep
383 2014-07-31 21:40:42 <sipa> jgarzik: the advantage of blk*.dat in torrents is that you could sym/hardlink the datadir files and the torrent serving together
384 2014-07-31 21:41:08 <jgarzik> sipa, yep. several advantages.
385 2014-07-31 21:43:24 <jgarzik> sipa, That requires programming and time, and people are getting impatient, so that will be in the next torrent release.
386 2014-07-31 21:43:47 <sipa> let's merge headersfirst instead :p
387 2014-07-31 21:45:26 <denisx> jgarzik: I see you use tracker.ccc.de, can you remove that. we donât operate that tracker anymore
388 2014-07-31 21:45:50 <jgarzik> sipa, sure :)
389 2014-07-31 21:46:55 <jgarzik> denisx, Any chance you can post on https://bitcointalk.org/index.php?topic=145386.msg8124601#msg8124601 saying same, for the record?
390 2014-07-31 21:47:44 <jcorgan> sipa: any progress on -reindex with headersfirst?
391 2014-07-31 21:47:55 <sipa> jcorgan: haven't bothered, no
392 2014-07-31 21:47:56 <denisx> jgarzik: no, never managed to get an account there
393 2014-07-31 21:48:08 <sipa> jcorgan: it won't get merged without pulltester support anyway
394 2014-07-31 21:48:16 <sipa> which will take a while
395 2014-07-31 21:48:22 <jcorgan> ah
396 2014-07-31 21:51:02 <jgarzik> ACTION tries to crowdsource the tracker question, http://www.reddit.com/r/Bitcoin/comments/2ca2uw/bitcoin_blockchain_torrent_call_for_trackers/
397 2014-07-31 21:51:43 <denisx> every open tracker will be used by fileshares after some time
398 2014-07-31 21:55:17 <michagogo> 00:40:40 <sipa> jgarzik: the advantage of blk*.dat in torrents is that you could sym/hardlink the datadir files and the torrent serving together
399 2014-07-31 21:55:46 <dsnrk> denisx: so don't make it open?
400 2014-07-31 21:55:49 <michagogo> Right, only thing is that you need to make sure that the torrent only contains max-size blk*.dat files
401 2014-07-31 21:56:11 <denisx> dsnrk: that would help, also it wouls not need much resources then
402 2014-07-31 21:56:11 <michagogo> Also, you need to reindex each time you want to update the torrent
403 2014-07-31 21:58:07 <sipa> we could have type of -reindex that only recreates the block index, and doesn't touch the chainstate
404 2014-07-31 21:58:16 <sipa> so it wouldn't require full validation again
405 2014-07-31 21:59:03 <michagogo> That would be nice
406 2014-07-31 21:59:55 <michagogo> Do chainstate entries include what block created an utxo?
407 2014-07-31 22:00:47 <Emcy> jgarzik
408 2014-07-31 22:00:52 <Emcy> udp://tracker.prq.to:80
409 2014-07-31 22:00:52 <Emcy> udp://tracker.publicbt.com:80
410 2014-07-31 22:00:53 <Emcy> udp://inferno.demonoid.com:3396
411 2014-07-31 22:00:53 <Emcy> udp://open.demonii.com:1337
412 2014-07-31 22:00:53 <Emcy> udp://tracker.istole.it:80
413 2014-07-31 22:00:53 <Emcy> udp://tracker.openbittorrent.com:80
414 2014-07-31 22:01:00 <Emcy> these are the ones i normally add
415 2014-07-31 22:01:09 <Emcy> more isnt necessarily better with trackers though
416 2014-07-31 22:01:55 <michagogo> (it'd need to make sure that the chain is intact)
417 2014-07-31 22:02:03 <Emcy> only incrementally more resilient really. Consider demonoid died for 2 and half years and the torrent didnt notice due to dht
418 2014-07-31 22:02:59 <sipa> michagogo: no
419 2014-07-31 22:03:14 <sipa> michagogo: chainstate = utxo data + block hash up to where synced
420 2014-07-31 22:03:24 <sipa> so as long as the block index does have that block hash, it's fine
421 2014-07-31 22:03:30 <sipa> otherwise they are completely separate
422 2014-07-31 22:03:33 <michagogo> Ah
423 2014-07-31 22:03:39 <denisx> is the start of bootstrap.dat the same like blk00000.dat?
424 2014-07-31 22:03:52 <jgarzik> denisx, yes
425 2014-07-31 22:04:13 <jgarzik> denisx, if it weren't for an internal software limitation, you could simply "mv bootstrap.dat blocks/blk00000.dat", as they are the same format
426 2014-07-31 22:04:25 <Emcy> hm what happened to exodus.desync.com
427 2014-07-31 22:04:37 <michagogo> Hm, I guess the ideal would be to have linearize look at your blocks/blk*.dat
428 2014-07-31 22:04:55 <michagogo> Detect which files are already in the "canonical" format
429 2014-07-31 22:05:01 <jgarzik> michagogo, ...after getting the list of hashes
430 2014-07-31 22:05:09 <jgarzik> michagogo, blk*.dat includes weak chains
431 2014-07-31 22:05:23 <michagogo> And then create the ones that need to be replaced
432 2014-07-31 22:05:34 <michagogo> jgarzik: I know
433 2014-07-31 22:05:49 <michagogo> Maybe '"canonical" format' was a bad way of putting it
434 2014-07-31 22:06:08 <michagogo> I mean, have linearize get the hashchain
435 2014-07-31 22:06:40 <michagogo> Look at your blocks/blk*.dat to see which files, if any, are in the right format (right size, right content)
436 2014-07-31 22:08:11 <michagogo> and then create the files that need to be changed, and then just tell you to shut down the node, `mv <linearize output dir>/* ~/.bitcoin/blocks`, and `bitcoind -litereindex`
437 2014-07-31 22:08:30 <michagogo> or whatever
438 2014-07-31 22:09:46 <Emcy> jgarzik i dont understand your comment about "associated with illegal activities"
439 2014-07-31 22:09:55 <jgarzik> michagogo, no need to shut down the node
440 2014-07-31 22:09:59 <Emcy> tpb hasnt run its tracker for like 4 years now
441 2014-07-31 22:10:11 <jgarzik> Emcy, tpb-associated
442 2014-07-31 22:10:17 <michagogo> jgarzik: hm?
443 2014-07-31 22:10:26 <Emcy> i dont think there are such a think any more
444 2014-07-31 22:10:32 <michagogo> If you don't, won't it crash if it's asked for a block and it's not where it's expected?
445 2014-07-31 22:10:49 <Emcy> the openbt and publicbt ones are completely open, they will track anything without prior arrangement
446 2014-07-31 22:11:00 <jgarzik> michagogo, oh, I wouldn't bother with trying to sync. a pointless exercise.
447 2014-07-31 22:11:20 <jgarzik> people will start with a "pretty" blk*.dat, then immediately diverge within 24 hours.
448 2014-07-31 22:11:33 <michagogo> https://www.irccloud.com/pastebin/lt5yw202
449 2014-07-31 22:11:39 <michagogo> Er
450 2014-07-31 22:11:49 <michagogo> that was in response to 00:58:05 <sipa> we could have type of -reindex that only recreates the block index, and doesn't touch the chainstate 00:58:13 <sipa> so it wouldn't require full validation again
451 2014-07-31 22:11:56 <jgarzik> you can torrent-host the existing files, as torrent will not write to the hosted files, and bitcoind only appends.
452 2014-07-31 22:12:00 <Emcy> "politicised trackers" is kind of a strange term anyway
453 2014-07-31 22:12:10 <michagogo> jgarzik: right
454 2014-07-31 22:12:14 <michagogo> well
455 2014-07-31 22:12:15 <jgarzik> You have to trick bitcoind into not appending to the last file, though
456 2014-07-31 22:12:18 <jgarzik> by creating a tiny one
457 2014-07-31 22:12:43 <michagogo> Right.
458 2014-07-31 22:12:50 <michagogo> Oh, interesting
459 2014-07-31 22:13:12 <michagogo> Right, of course
460 2014-07-31 22:13:40 <michagogo> You don't need to make them all max-size the way I was thinking, you just touch the next file
461 2014-07-31 22:14:09 <michagogo> jgarzik: also, I wasn't talking about syncing constantly
462 2014-07-31 22:14:20 <michagogo> Just periodically when you update the torrent
463 2014-07-31 22:15:19 <jgarzik> 2GB is the safety margin max-size. Sizes can be random. Torrent remains append-only. I was thinking about putting all TX from 2009 into blocks/blk00000.dat, 2010 in blk00001.dat, 2011 in blk00002.dat, ...
464 2014-07-31 22:15:27 <jgarzik> until it gets too large to do that anymore
465 2014-07-31 22:18:14 <helo> blk02009.dat etc
466 2014-07-31 22:18:29 <helo> oh, wrong channel... </bikeshedding>
467 2014-07-31 22:18:38 <jgarzik> heh, alas no
468 2014-07-31 22:20:08 <gmaxwell> They're in the current geometry to be pruning compatible, because rewriting a N gigabyte file to remove some of its blocks would be kinda lame.
469 2014-07-31 22:30:59 <jgarzik> so
470 2014-07-31 22:31:12 <jgarzik> let us see how long 313,000 RPC calls takes
471 2014-07-31 22:39:43 <jgarzik> 7 minutes. tolerable.