1 2012-10-12 00:04:55 <kjj_> very strange that there isn't a DHT proxy
2 2012-10-12 00:07:13 <jgarzik> a poor spread of DHT takes a little while to heal, as the DHT is coded to prefer longer lived nodes... and there are a few occasions where you would rather they swapped them out of buckets more rapidly
3 2012-10-12 00:13:08 <kjj_> no, I mean I'm surprised that there isn't a daemon I can run that talks to DHT, and then presents a pseudo-tracker on a local port
4 2012-10-12 00:15:35 <kjj_> oh god, the copyright blockchain thread managed to get even WORSE while I wasn't paying attention
5 2012-10-12 00:16:08 <jgarzik> kjj_: heh, yeah that kind of DHT proxy would be neat
6 2012-10-12 00:17:54 <Diablo-D3> lol
7 2012-10-12 00:25:47 <asciilifeform> Please comment: "Shitcoin: a Modest Proposal." (http://www.loper-os.org/?p=988)
8 2012-10-12 00:29:19 <kjj_> jgarzik: you going to post your torrent on the forums?
9 2012-10-12 00:29:49 <jgarzik> kjj_: I wanted to give it several hours to get healthy in the DHT
10 2012-10-12 00:29:57 <jgarzik> kjj_: only just finished uploading to my two seeds 29 minutes ago
11 2012-10-12 00:30:25 <jgarzik> ACTION was just running azureus/vuze, and poking its DHT
12 2012-10-12 00:30:30 <kjj_> heh. if you'd included a tracker, I'd have had two seeds for you ready as soon as you started
13 2012-10-12 00:30:42 <jgarzik> looks like azureus sees the magnet link and downloads the torrent very quickly
14 2012-10-12 00:31:07 <graingert> jgarzik: where did you put the torrent?
15 2012-10-12 00:31:43 <jgarzik> graingert: magnet:?xt=urn:btih:0bb0521942f586ed96203c6f4d136324756f8a9a&dn=bootstrap.dat or http://gtf.org/garzik/bitcoin/bootstrap.dat.torrent
16 2012-10-12 00:31:53 <jgarzik> with the proviso that it does not include trackers... requires DHT
17 2012-10-12 00:32:08 <graingert> no trackers?
18 2012-10-12 00:32:09 <graingert> y
19 2012-10-12 00:34:03 <graingert> jgarzik: adding trackers, it just works tm
20 2012-10-12 00:34:27 <kjj_> he was worried about it being a DOS target
21 2012-10-12 00:34:37 <graingert> kjj_: in what way?
22 2012-10-12 00:35:06 <kjj_> bitcoin attracts a lot of attention from pricks
23 2012-10-12 00:35:45 <graingert> so if you download the torrent then you get DDOS
24 2012-10-12 00:35:58 <graingert> bull
25 2012-10-12 00:36:20 <kjj_> no, if you host the tracker for it, someone might
26 2012-10-12 00:36:35 <graingert> udp://tracker.openbittorrent.com:80 has enough oomph
27 2012-10-12 00:36:55 <kjj_> heh. I built mine with three public trackers embedded in it
28 2012-10-12 00:37:03 <kjj_> http://www.jerviss.org/bitcointalk.org/bootstrap_193000.dat.torrent
29 2012-10-12 00:37:23 <kjj_> please don't link that in the forums. it'll be bad enough when the logbot finds it
30 2012-10-12 00:39:44 <BlueMatt> yay, a bitcoin seed torrent...and no one has bothered to fix any of the many ibd issues...
31 2012-10-12 00:40:24 <kjj_> ahh, shit. don't use my link
32 2012-10-12 00:40:36 <graingert> BlueMatt a bitcoin seed torrent?
33 2012-10-12 00:40:42 <graingert> BlueMatt what does it seed?
34 2012-10-12 00:40:52 <BlueMatt> block data torrent, I meant
35 2012-10-12 00:40:57 <graingert> IC
36 2012-10-12 00:41:17 <graingert> I always thought it would be handy to piggy back off of trackers to get peers
37 2012-10-12 00:41:27 <kjj_> I just noticed that my two seeds don't understand UDP either. haha
38 2012-10-12 00:41:39 <BlueMatt> ie my usual "lets fix the root of the problem instead of hacking on more crap to work around it" complaint
39 2012-10-12 00:41:41 <graingert> kjj_: UTP
40 2012-10-12 00:42:21 <kjj_> graingert: no, UDP
41 2012-10-12 00:42:48 <Diablo-D3> utp? wtf?
42 2012-10-12 00:42:49 <gmaxwell> BlueMatt: on the plus side the torrent doesn't appear to work for many (most?) people, without adding centeralization
43 2012-10-12 00:42:49 <graingert> uTP?
44 2012-10-12 00:42:58 <Diablo-D3> MICRO TOLIET PAPER
45 2012-10-12 00:43:03 <Diablo-D3> FOR UNUSUALLY SMALL ASSES
46 2012-10-12 00:43:07 <graingert> utorrent protocol
47 2012-10-12 00:43:47 <BlueMatt> gmaxwell: well, that wont stop people from using it...
48 2012-10-12 00:43:47 <Diablo-D3> graingert: er, which protocol is that?
49 2012-10-12 00:44:16 <kjj_> well fuck, I guess my job for tomorrow is to get rtorrent
50 2012-10-12 00:44:19 <gmaxwell> BlueMatt: I was being silly. The torrent is harmless other than distracting us from fixing our networking code
51 2012-10-12 00:44:21 <graingert> Micro Transport Protocol or ?TP (sometimes also uTP) is an open UDP-based variant of the BitTorrent peer-to-peer file sharing protocol intended to mitigate poor latency and other congestion control issues found in conventional BitTorrent over TCP, while providing reliable, ordered delivery.
52 2012-10-12 00:44:27 <gmaxwell> BlueMatt: it doesn't violate the security at all.
53 2012-10-12 00:44:39 <Diablo-D3> graingert: er
54 2012-10-12 00:44:51 <BlueMatt> gmaxwell: of course it doesnt violate bitcoin security, but it keeps us from having to fix the crap that is ibd logic...
55 2012-10-12 00:44:53 <Diablo-D3> I thought all clients already spoke that including the main torrent client?
56 2012-10-12 00:45:05 <gmaxwell> BlueMatt: it doesn't really, we're only seeding up to the highest checkpoint.
57 2012-10-12 00:45:05 <kjj_> graingert: I'm talking about the tracker connection being over UDP instead of http, not the bulk transfer
58 2012-10-12 00:45:16 <BlueMatt> ACTION feels like hes become the old curmudgeon even more than Diablo-D3 here...
59 2012-10-12 00:45:26 <gmaxwell> BlueMatt: people will still suffer from IBD but a little less.
60 2012-10-12 00:45:26 <kjj_> BlueMatt: I don't think it'll help performance much either way
61 2012-10-12 00:45:44 <BlueMatt> gmaxwell: it doesnt really what?
62 2012-10-12 00:45:44 <gmaxwell> (and we concurrently improved IBD a little)
63 2012-10-12 00:45:47 <Diablo-D3> bluematt: wat
64 2012-10-12 00:45:54 <gmaxwell> BlueMatt: remove the need to fix ibd.
65 2012-10-12 00:46:02 <Diablo-D3> no one is more curmugeony than me
66 2012-10-12 00:46:18 <BlueMatt> gmaxwell: no, it doesnt, but it eases the pressure on doing it a bit more...
67 2012-10-12 00:46:19 <graingert> ibd?
68 2012-10-12 00:46:24 <BlueMatt> initial block download
69 2012-10-12 00:46:32 <graingert> ah
70 2012-10-12 00:46:44 <graingert> yeah just torrent it it's fine LOLOLOLOL!!!11111oneone
71 2012-10-12 00:46:47 <BlueMatt> kjj_: it will help many people because of the crap logic we have now (got a slow peer? sucks for you!!!)
72 2012-10-12 00:46:53 <gmaxwell> BlueMatt: A little. besides there are people who are just fixated on doing the download externally no matter how fast it is
73 2012-10-12 00:47:14 <gmaxwell> this gives them a way that doesn't give them a @!#! insecure index, and is still ultraprune compatible.
74 2012-10-12 00:47:21 <BlueMatt> gmaxwell: yep, which is why I always complain when people bring up torrents for ibd...
75 2012-10-12 00:47:49 <gmaxwell> BlueMatt: and magically ibd is not fixed yet.
76 2012-10-12 00:47:58 <gmaxwell> Although it has improved a little.
77 2012-10-12 00:48:14 <BlueMatt> gmaxwell: I dont call shipping a bootstrap.dat improved a little
78 2012-10-12 00:48:21 <BlueMatt> I call it taking a step backwards...
79 2012-10-12 00:48:47 <gmaxwell> Though we're now seeing a fairly high rate of reports of windows users who can't manage to not press the @#$#@ reset button once an hour getting corrupted block indexes and having to try to sync several times in the 12 hours its taking them.
80 2012-10-12 00:49:00 <gmaxwell> BlueMatt: er. Thats _not_ what I'm talking about.
81 2012-10-12 00:49:18 <gmaxwell> Improved a little is that we don't pick braindead nodes as our pull point.
82 2012-10-12 00:49:24 <gmaxwell> (the height check)
83 2012-10-12 00:49:28 <BlueMatt> (sorry doing like 5 things at once, just got back in after cloning to a new drive, trying to test what was lost and copy it back off backups, etc, etc)
84 2012-10-12 00:49:39 <gmaxwell> s'okay.
85 2012-10-12 00:50:03 <gmaxwell> ultraprune also greatly improves IBD and will be pulled any day now.
86 2012-10-12 00:50:09 <BlueMatt> gmaxwell: oh, we actually do height check now? well thats somewhat better, but that doesnt really solve the incredibly slow ibd anyway...
87 2012-10-12 00:50:31 <gmaxwell> well it prevent the IBD for getting stuck over and over again because it's picked a stupid peer that can't give it blocks anyways.
88 2012-10-12 00:50:53 <BlueMatt> true
89 2012-10-12 00:51:41 <gmaxwell> actually fixing this stuff requires a big restructure for reverse header sync, and it doesn't make sense to even start on an implementation until ultraprune is merged
90 2012-10-12 00:51:47 <gmaxwell> because it needs structural changes.
91 2012-10-12 01:11:17 <BlueMatt> true...if only we had some kind of setup through which major refactors of one set of code didn't effect major refactors of other parts...
92 2012-10-12 01:12:02 <kjj_> ok, I'm out. long drive home
93 2012-10-12 01:12:16 <gmaxwell> er. block connection is as the heart of that kind of change.
94 2012-10-12 01:12:59 <BlueMatt> oh, well doing reverse headers sync sure, but doing other (simple) sane things like fetching blocks from multiple peers isnt
95 2012-10-12 01:13:30 <gmaxwell> our whole process for merging big changes is a bit broken, don't take it personally. :(
96 2012-10-12 01:16:19 <BlueMatt> meh, its not, but I still fear so much for minor bugs creeping in in anything...especially locking issues (which we already have many of)
97 2012-10-12 01:21:47 <jgarzik> [BETA] Bitcoin blockchain torrent - https://bitcointalk.org/index.php?topic=117982.0
98 2012-10-12 01:22:06 <jgarzik> yell if that post misses some of the "why"
99 2012-10-12 01:22:14 <jgarzik> will copy to bitcoin-devel in a bit
100 2012-10-12 01:22:59 <BlueMatt> :(
101 2012-10-12 01:23:13 <BlueMatt> "blockchain torrent project" oh god
102 2012-10-12 01:23:21 <BlueMatt> really???
103 2012-10-12 01:23:54 <BlueMatt> why not just tell people to -addnode=`good bw node`
104 2012-10-12 01:24:00 <BlueMatt> I know you have several
105 2012-10-12 01:24:05 <BlueMatt> also: dorm.bluematt.me
106 2012-10-12 01:28:33 <gmaxwell> BlueMatt: because they rapidly get overloaded.
107 2012-10-12 01:31:14 <BlueMatt> so keep a bigger list?
108 2012-10-12 01:33:32 <jgarzik> shoveling old, static data is not the best use of public P2P node resources
109 2012-10-12 01:33:49 <gmaxwell> I don't agree with that.
110 2012-10-12 01:34:11 <gmaxwell> But it seems I can't convince you otherwise.
111 2012-10-12 01:34:23 <BlueMatt> they dont currently do anything else (though high bw p2p node resources should...)
112 2012-10-12 01:34:56 <jgarzik> yes they do -- they validate and relay current txs and blocks
113 2012-10-12 01:35:15 <BlueMatt> low bw nodes can do that just about as much...
114 2012-10-12 01:35:18 <gmaxwell> And they have the data by virtue of their operation. Sure, it's not a good use of a resource constrained system or one busy doing other things... But nodes should not be resource constrained.
115 2012-10-12 01:36:17 <gmaxwell> You happily ignored all the bandwidth measurements I made, you've made no effort to make any of your own, but keep promoting this claim that it's a performance bottleneck. I am not impressed.
116 2012-10-12 01:36:21 <jgarzik> field translation: "nodes should have more than 3GB filesystem cache memory"
117 2012-10-12 01:36:28 <jgarzik> because that is the working set size
118 2012-10-12 01:36:31 <gmaxwell> jgarzik: Thats a loat of bullshit.
119 2012-10-12 01:36:34 <jgarzik> otherwise you're just punishing the disk
120 2012-10-12 01:37:07 <gmaxwell> Because the blocks are fairly large even a system with a slow disk can saturate a fairly ast network pipe _purely_ off disk.
121 2012-10-12 01:37:31 <jgarzik> yes -- at the cost of constantly running the disk
122 2012-10-12 01:37:47 <gmaxwell> And bittorrent has the same working set problems.
123 2012-10-12 01:38:34 <jgarzik> bittorrent offloads the work onto those most interested in the data
124 2012-10-12 01:38:41 <jgarzik> which frees public nodes from constantly running their disks
125 2012-10-12 01:38:49 <gmaxwell> 0_o
126 2012-10-12 01:38:55 <BlueMatt> thats my point here, too...
127 2012-10-12 01:39:00 <jgarzik> ultraprune doesn't fix that problem
128 2012-10-12 01:39:19 <BlueMatt> if you dont mind serving the data, serve the data...
129 2012-10-12 01:39:22 <jgarzik> moving serving of ancient data off P2P (for the most part) does fix that problem
130 2012-10-12 01:39:42 <gmaxwell> do you think magical pixies are going to run bittorrent? It's not only not reducing the work, its shifting it to a smaller set of candidtes??? and it takes more effort to become one??? even though many other systems have the data.