1 2012-11-15 00:06:24 <Luke-Jr> sipa: should I have blocks/rev00000.dat when I've only ever used testnet? :/
  2 2012-11-15 00:42:04 <cjd> I'm writing a difficulty calculator and I'm wondering if I should just reject the sign bit outright or try to handle it
  3 2012-11-15 00:42:34 <Diablo-D3> erm
  4 2012-11-15 00:42:42 <Diablo-D3> isnt it an unsigned number?
  5 2012-11-15 00:42:53 <cjd> I guess there's no way for the difficulty to end up in the negatives so it's just a latent bug that the retargetting calculation will work in negative
  6 2012-11-15 00:43:24 <copumpkin> sign bit?
  7 2012-11-15 00:43:38 <cjd> this bit should never be set in the diff in any block 0x00800000
  8 2012-11-15 00:43:58 <cjd> if the high bit is set then it's 0 padded by openssl
  9 2012-11-15 00:45:34 <cjd> but if you feed GetNextWorkRequired() a difficulty with that bit set, you get another number which will also have that bit set but all of the math is openssl is done using negative numbers so it's all wacky
 10 2012-11-15 00:46:56 <cjd> At first I thought it was somehow abusable but apparently there's no way to force a retarget into the negatives so it's just an annoying caviat, not a bug
 11 2012-11-15 00:59:07 <Luke-Jr> I don't see how a difficulty could ever be negative.
 12 2012-11-15 01:00:58 <cjd> it's not negative, it's just that openssl handles it as a negative
 13 2012-11-15 01:02:04 <cjd> bnNew.SetCompact(0x0ded7263).GetHex()  --> -6d726300000000000000000000
 14 2012-11-15 01:03:40 <cjd> this is why you see difficulties with a 0 byte like  0x12003456  even though there should be no high zeros in a mantissa.
 15 2012-11-15 01:04:26 <cjd> heh I need to scan the actual header chain to know if they really exist but according to my fuzzer there should be plenty of them
 16 2012-11-15 01:04:51 <Luke-Jr> difficulty 1 for example
 17 2012-11-15 01:04:59 <Luke-Jr> should be 50%
 18 2012-11-15 01:05:20 <rdponticelli> I know some chics who can handle to turn it's difficultness into the negatives...
 19 2012-11-15 01:05:30 <cjd> indeed, any difficulty with a high bit set will have a zero padding
 20 2012-11-15 01:06:07 <cjd> it's annoying because it's "part of the definition" but it never happens
 21 2012-11-15 01:06:43 <Luke-Jr> cjd: it's part of the format for the custom floating-point type. but it's not used in the context of targets/difficulties
 22 2012-11-15 01:06:57 <cjd> unless the last retarget was in the future
 23 2012-11-15 01:07:08 <Luke-Jr> &
 24 2012-11-15 01:07:17 <Luke-Jr> which is impossible
 25 2012-11-15 01:07:28 <cjd> mmm
 26 2012-11-15 01:08:21 <cjd> it doesn't look like there's any other way to trick it into setting that bit but if there was then the network would split instantly
 27 2012-11-15 01:08:30 <Luke-Jr> would it?
 28 2012-11-15 01:08:42 <cjd> unless all implementations have this weird negative computation behavior
 29 2012-11-15 01:08:51 <gmaxwell> 'all' lol
 30 2012-11-15 01:08:57 <Luke-Jr> cjd: they're required to
 31 2012-11-15 01:09:29 <cjd> meh it's hard to write and it can't happen in real life
 32 2012-11-15 01:09:37 <gmaxwell> An implementation isn't correct unless it reproduces the references undocumented normative bugs, including the ones that key imported from implicity behavior in system libraries.
 33 2012-11-15 01:09:59 <cjd> hehe do there's only one implementation :P
 34 2012-11-15 01:10:14 <cjd> one correct one
 35 2012-11-15 01:10:25 <gmaxwell> Cant happen != isn't expected, be very careful in assuming what can't happen.
 36 2012-11-15 01:11:39 <Luke-Jr> cjd: note this same sign bit thing also affects the height in blocks
 37 2012-11-15 01:12:11 <cjd> I'm trying to get a grasp on what would happen if it did, all the math would round "up" since it rounds to zero and the numbers would come out with 1 more byte of precision
 38 2012-11-15 02:25:51 <cjd> looks like I got it
 39 2012-11-15 02:25:53 <cjd> https://ezcrypt.it/nB5n#BvbyfcJ0bj3Axzp956gw381e
 40 2012-11-15 02:26:42 <Diablo-D3> got what?
 41 2012-11-15 02:27:04 <cjd> reimplemented the retarget function
 42 2012-11-15 02:27:08 <Diablo-D3> ahh
 43 2012-11-15 02:27:10 <cjd> it does not handle broken "negative" difficulty
 44 2012-11-15 02:27:20 <Diablo-D3> [08:42:53] <cjd> I guess there's no way for the difficulty to end up in the negatives so it's just a latent bug that the retargetting calculation will work in negative
 45 2012-11-15 02:27:23 <Diablo-D3> btw, yes
 46 2012-11-15 02:27:25 <Diablo-D3> it just goes below 0
 47 2012-11-15 02:27:28 <Diablo-D3> er
 48 2012-11-15 02:27:29 <Diablo-D3> below 1
 49 2012-11-15 02:27:46 <cjd> are we talking about the same negative?
 50 2012-11-15 02:27:52 <Diablo-D3> we might be?
 51 2012-11-15 02:28:17 <Diablo-D3> half of 1 difficulty is 0.5, twice 1 difficulty is 2
 52 2012-11-15 02:28:39 <cjd> I'm talking about a bug in the compaction scheme wherein openssl will do the multiplication and division asif the number was negative if bit number 24 is set
 53 2012-11-15 02:28:49 <Diablo-D3> ahh
 54 2012-11-15 02:28:53 <Diablo-D3> two different things then
 55 2012-11-15 02:29:38 <cjd> but the only way that can happen AFAICT is for the last retargetting timestamp to be in the future
 56 2012-11-15 02:30:06 <Diablo-D3> erm, it can be in the future.
 57 2012-11-15 02:30:27 <cjd> indeed, it appears that it can't happen
 58 2012-11-15 02:31:18 <cjd> but it's there and unless we (aka the devs) want to ban blocks where (block->bits & 0x00800000) != 0, the bug will just sit there
 59 2012-11-15 02:32:46 <cjd> If people like the idea, I'll even write the patch because I want bitcoin to be proven and standardized rather than magical and indeterminent
 60 2012-11-15 02:34:12 <Diablo-D3> I like magical code :<
 61 2012-11-15 02:34:57 <cjd> :)
 62 2012-11-15 02:35:35 <cjd> in cryptography, anything that makes the code hard to validate is IMO a bug
 63 2012-11-15 02:35:49 <Diablo-D3> yeah, but arguably thats a problem with the language
 64 2012-11-15 02:36:07 <Diablo-D3> some languages make writing magic code so much easier
 65 2012-11-15 02:36:40 <cjd> I can say I'm not too thrilled with C++ at the moment ;)
 66 2012-11-15 02:37:07 <Diablo-D3> I should try to make seaking as antimagic as possible
 67 2012-11-15 02:37:26 <cjd> seaking?
 68 2012-11-15 02:37:33 <Diablo-D3> yeah, Im working on a new language
 69 2012-11-15 02:37:37 <cjd> ahh
 70 2012-11-15 02:37:46 <Diablo-D3> one supreme language to end all other languages
 71 2012-11-15 02:37:52 <cjd> I had an idea for a language which is infinitely extensible
 72 2012-11-15 02:38:01 <Diablo-D3> lol, thats seaking
 73 2012-11-15 02:38:03 <Diablo-D3> @keywords
 74 2012-11-15 02:38:13 <cjd> my name for it is hashfilter because that's it's only command
 75 2012-11-15 02:38:15 <Diablo-D3> tells the compiler to use a plugin named keyword
 76 2012-11-15 02:38:27 <Diablo-D3> and you can do _whatever you want_ in the plugin
 77 2012-11-15 02:38:35 <Diablo-D3> no matter how weird or strange
 78 2012-11-15 02:38:53 <cjd> #filter <afilter.c>  causes afilter.c to be compiled and loaded and it preprocesses the rest of your code
 79 2012-11-15 02:39:10 <Diablo-D3> yeah, you'd like seaking I think
 80 2012-11-15 02:39:25 <cjd> I like it because it's just a few thousand lines of code to make it work
 81 2012-11-15 02:39:34 <cjd> and because I can use it to target any language I like
 82 2012-11-15 02:39:41 <Diablo-D3> yeah, see, Im not writing a vm
 83 2012-11-15 02:39:43 <cjd> I'll write C with my own filter stack
 84 2012-11-15 02:39:46 <Diablo-D3> or a compiler, to be honest
 85 2012-11-15 02:39:52 <Diablo-D3> Im just plugging it into llvm
 86 2012-11-15 02:39:56 <Diablo-D3> and seaking is C-like
 87 2012-11-15 02:40:05 <Diablo-D3> its going to end up as fast as C in the end since Im using a real compiler
 88 2012-11-15 02:40:18 <cjd> as fast as llvm c :P
 89 2012-11-15 02:40:22 <Diablo-D3> well, yeah
 90 2012-11-15 02:40:28 <Diablo-D3> although theres certain shit I wont allow
 91 2012-11-15 02:40:51 <Diablo-D3> like, you know the restrict keyword in c99?
 92 2012-11-15 02:41:03 <Diablo-D3> thats the default behavior in seaking
 93 2012-11-15 02:41:11 <cjd> I'm partial to GCC and I'd just like to replace the regular C preprocessor with my filter processor
 94 2012-11-15 02:41:20 <Diablo-D3> well, thats what I started with seaking
 95 2012-11-15 02:41:29 <Diablo-D3> but cpp just isnt strong enough
 96 2012-11-15 02:41:49 <cjd> I'd rather compile to source because I want to be able to disect the result of the compilation
 97 2012-11-15 02:41:57 <Diablo-D3> seaking without plugins technically could just be output to C
 98 2012-11-15 02:42:04 <Diablo-D3> but its a waste
 99 2012-11-15 02:42:19 <cjd> Also I'd like to target other languages
100 2012-11-15 02:42:19 <Diablo-D3> and it'd still only work on fully compliant c99 compilers, which is pretty much just gcc and clang
101 2012-11-15 02:42:35 <Diablo-D3> I dont want to target other languages that way
102 2012-11-15 02:42:37 <Diablo-D3> I dont see the point
103 2012-11-15 02:42:44 <Diablo-D3> all languages, in the end, do the same thing
104 2012-11-15 02:43:06 <Diablo-D3> thats why Im not even inventing new syntax for seaking, its all shit people have seen before
105 2012-11-15 02:43:09 <cjd> ahh, if you want to use MSVC, just load an MSVC filter which converts all of the normal syntax into wacky msvc syntax :)
106 2012-11-15 02:43:21 <Diablo-D3> yeah, but msvc is a shit compiler
107 2012-11-15 02:43:32 <Diablo-D3> llvm internally is very flexible
108 2012-11-15 02:43:40 <cjd> yeah but my point is filter stacks are extensible
109 2012-11-15 02:43:50 <Diablo-D3> you can implement any known language in it correctly, and usually better than the language's own compiler or vm
110 2012-11-15 02:43:54 <Diablo-D3> yeah they are
111 2012-11-15 02:43:56 <cjd> and I want people to be able to write filters easily
112 2012-11-15 02:44:02 <Diablo-D3> @keyword could be a filter
113 2012-11-15 02:44:12 <Diablo-D3> Im going more generic than you
114 2012-11-15 02:44:25 <cjd> because like hell I know what a good language is, I have some ideas but I'm not the master here
115 2012-11-15 02:44:37 <Diablo-D3> your plugin is given whatever it wants
116 2012-11-15 02:44:47 <cjd> sounds similar
117 2012-11-15 02:45:06 <cjd> The filters idea is just because I don't want to tie it down to anything
118 2012-11-15 02:45:18 <Diablo-D3> well, you know how lisp can DSL itself, right?
119 2012-11-15 02:45:30 <cjd> theoretically you could construct a whole compiler with a filter stack
120 2012-11-15 02:45:45 <cjd> hmm I'd heard something about DSLs in lisp but not much
121 2012-11-15 02:45:59 <Diablo-D3> a lisp vm's native IL is lisp.
122 2012-11-15 02:46:05 <cjd> ok
123 2012-11-15 02:46:20 <cjd> also compiling bitcoin script could be done with a filter stack
124 2012-11-15 02:46:23 <cjd> or decompiling it
125 2012-11-15 02:46:23 <Diablo-D3> lisp macros are everything cpp should be and isnt
126 2012-11-15 02:46:30 <cjd> heh
127 2012-11-15 02:46:45 <Diablo-D3> and lisp isnt limited to s-expressions either
128 2012-11-15 02:46:52 <Diablo-D3> someone made a json expression type
129 2012-11-15 02:47:40 <cjd> It would be interesting to write a filter which converted variables to stack space
130 2012-11-15 02:47:45 <cjd> and back
131 2012-11-15 02:47:55 <cjd> then you could compile something readable into bitcoin script
132 2012-11-15 02:48:02 <Diablo-D3> oh, thats another thing with seaking
133 2012-11-15 02:48:22 <Diablo-D3> stacks are segmented and parallel
134 2012-11-15 02:49:05 <Diablo-D3> anything thats in the heap is executed haskell style
135 2012-11-15 02:49:11 <Diablo-D3> all async futures
136 2012-11-15 02:49:15 <cjd> actually one reason I like targetting C is because you have asm keyword so you can pretty easily add actual keywords to the language
137 2012-11-15 02:49:53 <cjd> if you have filters :)
138 2012-11-15 02:49:55 <Diablo-D3> so programs that would be expensive to do in parallel would end up being very cheap
139 2012-11-15 02:50:15 <cjd> hmm
140 2012-11-15 02:50:49 <cjd> haskell is interesting since it kind of takes the constant folding optimizer to it's logical extreme
141 2012-11-15 02:51:00 <Diablo-D3> yeah, haskell is nuts that way
142 2012-11-15 02:51:39 <Diablo-D3> and erlang
143 2012-11-15 02:51:56 <Diablo-D3> erlang does parallelization insanely
144 2012-11-15 02:52:11 <Diablo-D3> everything IS an async message. everything.
145 2012-11-15 02:52:30 <cjd> benchmark time :)
146 2012-11-15 02:52:34 <Diablo-D3> haskell you still have to explicitly build an actors model
147 2012-11-15 02:52:36 <Diablo-D3> cjd: well
148 2012-11-15 02:52:43 <Diablo-D3> erlang's existing vms are kinda slow
149 2012-11-15 02:52:47 <Diablo-D3> but its not a reflection on erlang
150 2012-11-15 02:53:05 <Diablo-D3> ahh
151 2012-11-15 02:53:14 <cjd> (it's going to be a slaughter ofc)
152 2012-11-15 02:53:45 <cjd> did you check the paste above?
153 2012-11-15 02:56:05 <Diablo-D3> but yeah, I think erlang's execution model, lisp's self-DSLing, C type syntax
154 2012-11-15 02:56:09 <Diablo-D3> yeah
155 2012-11-15 02:56:14 <Diablo-D3> I saw the paste
156 2012-11-15 02:56:41 <cjd> annoying that it's as complex as it is but that's the standard...
157 2012-11-15 02:56:52 <Diablo-D3> it doesnt look too complex
158 2012-11-15 02:57:19 <cjd> the nice thing about it is it's code which anyone can port to any language, the openssl stuff is full of magic and fail
159 2012-11-15 02:57:21 <cjd> (IMO)
160 2012-11-15 02:57:28 <Diablo-D3> openssl is full of speed
161 2012-11-15 02:57:52 <Diablo-D3> and its not magiced, the problem is openssl devs are language lawyers
162 2012-11-15 02:58:39 <Diablo-D3> they know exactly what real hardware does and they know exactly what spec compliant compilers do
163 2012-11-15 02:58:47 <Diablo-D3> its deep magic wizardry
164 2012-11-15 02:58:48 <cjd> no, I blame satoshi for this one, he should not have used such a tall stack of code to handle something which is essentially a protocol definition
165 2012-11-15 02:58:59 <Diablo-D3> well
166 2012-11-15 02:59:02 <Diablo-D3> satoshi isnt a programmer
167 2012-11-15 02:59:16 <Diablo-D3> and its very clear in large parts of the code
168 2012-11-15 02:59:19 <Diablo-D3> hes an academic
169 2012-11-15 02:59:25 <Diablo-D3> the code works, but dear god is it ugly
170 2012-11-15 03:00:06 <cjd> real 5m24.051s openssl
171 2012-11-15 03:00:18 <cjd> real 0m9.774s me :)
172 2012-11-15 03:00:23 <Diablo-D3> wat
173 2012-11-15 03:00:27 <Diablo-D3> do the openssl one again
174 2012-11-15 03:00:31 <Diablo-D3> are you doing disk io?
175 2012-11-15 03:00:35 <cjd> no
176 2012-11-15 03:00:40 <Diablo-D3> its all in memory?
177 2012-11-15 03:00:47 <cjd> calculating difficulty
178 2012-11-15 03:00:55 <Diablo-D3> how many iterations?
179 2012-11-15 03:01:20 <Diablo-D3> because the compiler probably ate your loop
180 2012-11-15 03:01:28 <cjd> https://ezcrypt.it/oB5n#htpjFulWOS1DEYxI84zSL37q
181 2012-11-15 03:01:35 <cjd> there's the whole script
182 2012-11-15 03:01:57 <cjd> getNextWorkRequired() is the control
183 2012-11-15 03:02:10 <cjd> Difficulty_retarget() is my version
184 2012-11-15 03:02:35 <Diablo-D3> Im not seeing any problems with that
185 2012-11-15 03:02:50 <cjd> CBigNum bnNew;
186 2012-11-15 03:03:02 <cjd> you just don't see how much crap happens when you call that
187 2012-11-15 03:03:09 <Diablo-D3> oh?
188 2012-11-15 03:03:30 <cjd> calls into openssl, then openssl does it's computations and stuff
189 2012-11-15 03:03:43 <Diablo-D3> yeah, external library calls are expensive
190 2012-11-15 03:03:49 <Diablo-D3> but theres no way its 5x longer expensive
191 2012-11-15 03:04:16 <cjd> and it does division which openssl extends the number out to zero even if none of that is going to be used
192 2012-11-15 03:04:26 <cjd> it probably allocates and frees memory although I'm not sure
193 2012-11-15 03:04:42 <cjd> I can't figure out anything about how memory allocation works in C++ :|
194 2012-11-15 03:04:50 <Diablo-D3> new delete.
195 2012-11-15 03:05:02 <cjd> and if you don't use those then it uses the stack?
196 2012-11-15 03:05:08 <Diablo-D3> yes
197 2012-11-15 03:05:12 <cjd> ic
198 2012-11-15 03:05:28 <Diablo-D3> although c++ makes zero sense to me
199 2012-11-15 03:05:36 <Diablo-D3> stack allocation should be used as much as possible
200 2012-11-15 03:05:41 <cjd> yeah, it's annoying
201 2012-11-15 03:06:03 <cjd> you should look at cjdns allocator structure, it makes life a lot easier if you're writing in C
202 2012-11-15 03:06:19 <Diablo-D3> the only time you want heaps is objects surviving past scope exit
203 2012-11-15 03:06:27 <Diablo-D3> or you need to allocate large amounts of shit
204 2012-11-15 03:06:28 <cjd> mhm
205 2012-11-15 03:06:54 <Diablo-D3> well, your stack might be limited to only 2mb
206 2012-11-15 03:07:03 <cjd> heh ouch
207 2012-11-15 03:07:10 <Diablo-D3> although the C stack design kind of sucks
208 2012-11-15 03:07:43 <Diablo-D3> seaking segmented parallel stacks means stack allocation isnt a problem
209 2012-11-15 03:07:58 <cjd> /nod
210 2012-11-15 03:08:08 <Diablo-D3> although I may not even have real stacks
211 2012-11-15 03:08:14 <Diablo-D3> it could be multiple heap styles
212 2012-11-15 03:08:32 <Diablo-D3> stacks only being used for internal message passing shit and the actual function call stack
213 2012-11-15 03:09:08 <Diablo-D3> cjd: btw, you're 5x faster, I think you won
214 2012-11-15 03:09:25 <Diablo-D3> I mean, the code isnt exactly called millions of times a second here
215 2012-11-15 03:09:35 <cjd> 5 times?
216 2012-11-15 03:10:28 <cjd> ;;calc 9.7 / ((5*60)+24)
217 2012-11-15 03:10:28 <gribble> 0.0299382716049
218 2012-11-15 03:10:43 <cjd> 2.9% as long?
219 2012-11-15 03:10:53 <cjd> ;;calc ((5*60)+24) / 9.7
220 2012-11-15 03:10:53 <gribble> 33.4020618557
221 2012-11-15 03:11:03 <cjd> 33 times the speed? is that right?
222 2012-11-15 03:11:46 <cjd> anyway I'm going for brevity and understandability
223 2012-11-15 03:15:13 <Diablo-D3> wait lemme reread
224 2012-11-15 03:15:29 <Diablo-D3> dero
225 2012-11-15 03:15:31 <Diablo-D3> derp
226 2012-11-15 03:15:41 <Diablo-D3> 9 seconds
227 2012-11-15 03:15:45 <Diablo-D3> not 59
228 2012-11-15 03:15:54 <Diablo-D3> so yeah
229 2012-11-15 03:15:56 <Diablo-D3> I think you won
230 2012-11-15 03:16:26 <cjd> I just think there should be one very simple function which anyone can use and it will always work, it defined what bitcoin is.
231 2012-11-15 03:18:09 <Diablo-D3> yeah
232 2012-11-15 03:26:51 <cjd> https://github.com/bitcoin/bitcoin/pull/1823
233 2012-11-15 03:29:34 <cjd> that pull is way too scary
234 2012-11-15 03:30:03 <cjd> I'd favor leaving it as it is but invalidating blocks where 0x00800000 & bits is not 0
235 2012-11-15 03:31:21 <Diablo-D3> well for one
236 2012-11-15 03:31:24 <Diablo-D3> do any such blocks exist?
237 2012-11-15 03:31:38 <cjd> I've not checked but logically it is impossible
238 2012-11-15 03:32:04 <Diablo-D3> check.
239 2012-11-15 03:32:08 <Diablo-D3> because if they dont, its moot
240 2012-11-15 03:32:09 <cjd> it might be possible to trick a node into trying to validate a chain of fake blocks where one of them has the value
241 2012-11-15 03:32:15 <Diablo-D3> if they exist, they exist as a bug
242 2012-11-15 03:32:24 <Diablo-D3> imo anyways
243 2012-11-15 03:32:29 <Diablo-D3> I wonder what sipa and gmaxwell think
244 2012-11-15 03:32:59 <cjd> no it's not moot because there is a bug in the number handling and it should be fixed otherwise it will linger and if you don't fix these things, sooner or later one of them will bite you
245 2012-11-15 03:33:23 <Diablo-D3> Im saying any worrying about backwards compatibility is moot
246 2012-11-15 03:33:34 <cjd> oh I see
247 2012-11-15 03:33:55 <Diablo-D3> if the blocks dont exist and shouldnt exist, you're fine
248 2012-11-15 03:34:08 <cjd> it is critical that they do the silly zero padding because they are compared using ==
249 2012-11-15 03:34:09 <Diablo-D3> it should reject them if they do exist if they shouldnt, though
250 2012-11-15 03:34:17 <cjd> (continue to do the zero padding)
251 2012-11-15 03:39:35 <cjd> heh bitcoinj also handles negative numbers the same way
252 2012-11-15 03:40:55 <Diablo-D3> heh
253 2012-11-15 03:45:34 <Diablo-D3> welp, Im on my way to seaking glory
254 2012-11-15 03:45:48 <Diablo-D3> my parser properly understands both C and C++ comments.
255 2012-11-15 03:46:03 <Diablo-D3> http://pastebin.com/SrV5TyyB
256 2012-11-15 03:55:30 <jgarzik> huh
257 2012-11-15 03:55:37 <jgarzik> gnutella now using "semi reliable UDP"
258 2012-11-15 03:56:08 <jgarzik> some torrent extensions do same (_not_ referring to DHT or UDP trackers, that's something different)
259 2012-11-15 03:56:41 <jgarzik> seems to be more firewall friendly in some cases, and probably avoids DPI [for the moment]
260 2012-11-15 04:10:00 <cjd> I think almost all bittorrent these days uses uTP/LEDBAT which is a roll-your-own tcp stack based on UDP
261 2012-11-15 04:11:01 <cjd> What's interesting about it is it uses rtt to determine lag and uses that to back off
262 2012-11-15 04:11:23 <gmaxwell> cjd: so does every modern tcp stack.
263 2012-11-15 04:12:14 <cjd> *shrug* I guess this does it more because they talk a lot about it
264 2012-11-15 04:12:36 <gmaxwell> No, you're just aware of it because they talk a lot. :P
265 2012-11-15 04:13:25 <cjd> ahh, now I remember why they did it, bittorrent needs to keep open lots of connections and using a seperate tcp connection for each would not backoff properly
266 2012-11-15 04:13:53 <Diablo-D3> cjd: ooh
267 2012-11-15 04:14:12 <Diablo-D3> gnutella ported uTP over?
268 2012-11-15 04:14:24 <gmaxwell> Yea, thats an issue, though some modern TCP control will do cross connection state pooling.
269 2012-11-15 04:14:49 <gmaxwell> The roll your own reliable transport replaces one set of firewall problems for another, unfortunately. Trying to get UDP working is driving more realtime streaming people to do really scarry things like https://www.usenix.org/conference/nsdi12/minion-unordered-delivery-wire-compatible-tcp-and-tls
270 2012-11-15 04:16:36 <cjd> I've done a little bit of tcp and a little bit of udp, I favor udp by far, you only get 1024 bytes or so and you can't expect your message to get there but you don't have to deal with buffers
271 2012-11-15 04:17:07 <cjd> but then if you're downloading something, it won't work
272 2012-11-15 04:17:21 <cjd> only useful for idle chatter between nodes
273 2012-11-15 04:17:50 <Diablo-D3> cjd: udp isnt bufferbloat proof though
274 2012-11-15 04:18:12 <cjd> indeed
275 2012-11-15 04:18:31 <Diablo-D3> wondershapper seems to work pretty fucking well though
276 2012-11-15 04:18:37 <Diablo-D3> *shaper
277 2012-11-15 04:19:21 <cjd> if you've got lots of nodes and not much to say to eachother, sendto() and recvfrom() with one socket to poll on looks pretty nice
278 2012-11-15 04:20:41 <cjd> anyway it looks like I now have a fully compliant retargetting function
279 2012-11-15 04:20:42 <cjd> https://ezcrypt.it/tB5n#Ui1Uz4nNu1zpUaq7QAa2OgPG
280 2012-11-15 04:20:55 <cjd> just have to take the sign bit off and then put it back after
281 2012-11-15 04:21:16 <cjd> but I still think blocks with that bit should be invalidated
282 2012-11-15 06:22:21 <sirebral> Is this the proper channel for someone seeking a developer or team that can work with me on a Bitcoin web project?
283 2012-11-15 06:33:15 <amiller> feel free to talk a bit about your project, especially if it's quiet and late at night! sirebral
284 2012-11-15 06:34:42 <sirebral> Working on YABBS, yet another bitcoin wagering site ;)
285 2012-11-15 06:35:14 <sirebral> Yet it's not like anything I've seen out there as of yet..
286 2012-11-15 06:35:25 <sirebral> I've got a full RFP and specification..
287 2012-11-15 06:35:34 <sirebral> Having trouble finding talent..
288 2012-11-15 06:35:46 <sirebral> My talent is in Network and Server Engineering..
289 2012-11-15 06:35:55 <sirebral> Doesn't help me much with development work.
290 2012-11-15 06:37:02 <amiller> you could mention some highlights from the RFP or specification
291 2012-11-15 06:37:41 <sirebral> Hrm.
292 2012-11-15 06:41:13 <sirebral> The RFP seeks someone, or a team who can develop a site that accepts wagers using only bitcoin addresses, there are similar sites out there in that vain, yet the gameplay I'm proposing is totally original,..don't wanna let too much out of the bag in a chatroom...
293 2012-11-15 06:41:44 <sirebral> I've found a few developers on eLance, yet I've never had much luck with projects there..
294 2012-11-15 06:43:27 <Luke-Jr> sirebral: sounds like a copycat of the DDoS attack called SatoshiDice
295 2012-11-15 06:43:40 <sirebral> It's not..
296 2012-11-15 06:44:09 <sirebral> Copycat sites never go anywhere.
297 2012-11-15 06:44:15 <Luke-Jr> that's not the problem.
298 2012-11-15 06:44:49 <Luke-Jr> sirebral: Bitcoin is not meant to encode data of any form, and any attempt to do so is almost inevitably going to end up being a DDoS attack
299 2012-11-15 06:45:54 <Luke-Jr> (obviously not including the data that is financial transactions, which contain only a destination address)
300 2012-11-15 06:46:54 <phantomcircuit> there's only 2890 connectable peers :/
301 2012-11-15 06:46:56 <phantomcircuit> that's not a lot
302 2012-11-15 06:47:09 <sirebral> I understand the open nature of the system, yet I'm not sure I understand the DDoS aspect..
303 2012-11-15 06:47:41 <Luke-Jr> sirebral: trying to say "send this transaction to do this one action" is an abuse of the system
304 2012-11-15 06:47:58 <Luke-Jr> it's not designed for that, and cannot handle it
305 2012-11-15 06:48:00 <sirebral> Not how my site works...
306 2012-11-15 06:48:11 <Luke-Jr> perhaps I misunderstand your summary then
307 2012-11-15 06:48:12 <phantomcircuit> sirebral, you have a proper RFP?
308 2012-11-15 06:48:24 <phantomcircuit> have to say that would be the first proper RFP i've yet to see in the bitcoin world...
309 2012-11-15 06:48:27 <sirebral> Depends what you consider proper ;)
310 2012-11-15 06:48:53 <jgarzik> phantomcircuit: I imagine connectable peers changes over the course of a day
311 2012-11-15 06:49:03 <jgarzik> phantomcircuit: but yeah, need more
312 2012-11-15 06:49:15 <jgarzik> phantomcircuit: should include a full node secretly in an MMORPG game
313 2012-11-15 06:49:24 <Luke-Jr> lol
314 2012-11-15 06:49:34 <sirebral> It's doesn't go into the full core of the inter-workings, rather it discusses the team requirements, the platform and the game play...
315 2012-11-15 06:49:54 <jgarzik> make proof-of-work involve killing a high level WoW NPC
316 2012-11-15 06:50:12 <Luke-Jr> sirebral: might try posting on the Bitcoin Foundation forum about it
317 2012-11-15 06:50:15 <phantomcircuit> jgarzik, i'm currently scanning what is probably all of the available peers roughly once every ten minutes
318 2012-11-15 06:50:46 <amiller> jgarzik, i was thinking, you know how occasionally you want DoS resistance between p2p nodes
319 2012-11-15 06:50:49 <amiller> why not give them mining work to do?
320 2012-11-15 06:50:50 <jgarzik> phantomcircuit: us[24].exmulti.net on your list?
321 2012-11-15 06:50:58 <jgarzik> amiller: indeed
322 2012-11-15 06:51:30 <amiller> this is the ultimate rpow
323 2012-11-15 06:51:44 <phantomcircuit> jgarzik, yes
324 2012-11-15 06:51:49 <phantomcircuit> 69.64.34.118|8333|1352963268|1352963278
325 2012-11-15 06:51:57 <phantomcircuit> host port lasttry lastconnect
326 2012-11-15 06:52:09 <phantomcircuit> also have port 8334 for that ip
327 2012-11-15 06:52:50 <sirebral> Yeah, I just started a thread on BCT, had to stick in in the n00bs section, however... I've been more of a lurker in the past...
328 2012-11-15 06:53:57 <Luke-Jr> sirebral: I didn't mean BitcoinTalk.
329 2012-11-15 06:54:05 <Luke-Jr> mostly just trolls there
330 2012-11-15 06:54:14 <phantomcircuit> jgarzik, last connected to that peer roughyl 8 minutes ago
331 2012-11-15 06:54:17 <jgarzik> phantomcircuit: something's broken on your side WRT 8334...
332 2012-11-15 06:54:22 <jgarzik> phantomcircuit: clearly inaccessible
333 2012-11-15 06:54:39 <phantomcircuit> jgarzik, it's being broadcast on the network so im trying to connect to it
334 2012-11-15 06:54:49 <jgarzik> weird
335 2012-11-15 06:54:49 <phantomcircuit> but there's no record of every successfully doing so
336 2012-11-15 06:54:52 <phantomcircuit> so nothing broken
337 2012-11-15 06:55:00 <jgarzik> I'm not broadcasting that
338 2012-11-15 06:55:02 <jgarzik> wonder who is
339 2012-11-15 06:55:22 <doublec> namecoin is 8334 iirc
340 2012-11-15 06:55:28 <phantomcircuit> jgarzik, who knows
341 2012-11-15 06:55:42 <sirebral> Luke:Interesting, I've seen quite a few spammy posts there, yet also seems to be the most active community I've found thus far...
342 2012-11-15 06:56:25 <Luke-Jr> sirebral: not all activity is equal.
343 2012-11-15 06:56:28 <phantomcircuit> sirebral, if you want to actually get something done asking for help on bitcointalk is counter productive
344 2012-11-15 06:57:38 <sirebral> Hence... why I am posing my questions here..
345 2012-11-15 06:58:44 <Luke-Jr> sirebral: most people are asleep right now
346 2012-11-15 06:59:08 <sirebral> As I should be as well.
347 2012-11-15 06:59:13 <Luke-Jr> sirebral: Foundation is at https://bitcoinfoundation.org/
348 2012-11-15 06:59:25 <sirebral> Thanks for the assist..
349 2012-11-15 06:59:29 <phantomcircuit> jgarzik, i actually have code to ddos the entire network
350 2012-11-15 06:59:31 <phantomcircuit> it works
351 2012-11-15 06:59:44 <Luke-Jr> um
352 2012-11-15 06:59:44 <phantomcircuit> but i run out of local port numbers before i get past about 100 peers
353 2012-11-15 06:59:45 <phantomcircuit> :(
354 2012-11-15 06:59:55 <Luke-Jr> you can't know it works without having DDoS'd the network -.-
355 2012-11-15 07:00:44 <phantomcircuit> Luke-Jr, well it worked against the roughly dozen bitcoin nodes i run
356 2012-11-15 07:00:52 <phantomcircuit> scale to all connectable peers
357 2012-11-15 07:01:35 <Luke-Jr> & you just said it didn't scale <.<
358 2012-11-15 07:02:45 <phantomcircuit> Luke-Jr, no i just said i ran out of local port numbers
359 2012-11-15 07:02:59 <phantomcircuit> a few virtual machines and it would scale to the entire network easily
360 2012-11-15 07:05:52 <Diablo-D3> http://mjg59.dreamwidth.org/20187.html
361 2012-11-15 07:05:54 <Diablo-D3> bwhahahaha
362 2012-11-15 07:06:11 <OffTopic> wsup noob devs?
363 2012-11-15 09:29:46 <slush1> etotheipi_: ping?
364 2012-11-15 11:12:30 <BitDev> hi all
365 2012-11-15 11:13:33 <BitDev> i want to write my own bitcoin client
366 2012-11-15 11:13:39 <BitDev> i stuck at the start
367 2012-11-15 11:14:01 <BitDev> i want to send `version` packet
368 2012-11-15 11:15:04 <BitDev> there are some fields like `addr_recv` and `addr_from` - i must fill them?
369 2012-11-15 11:15:55 <BitDev> if i dont want to start local server or i can't - how i must fill `addr_from` field?
370 2012-11-15 11:19:05 <BitDev> where in sources of bitcoin i can find about it?
371 2012-11-15 11:57:19 <emeitner> Anybody here who is an admin on the bitcoin.it wiki?
372 2012-11-15 12:00:20 <sipa> BitDev: if no address is known, 0.0.0.0 or ::0 is often used
373 2012-11-15 12:00:30 <sipa> BitDev: the addresses in the version packet aren't that important
374 2012-11-15 12:13:18 <drizztbsd> :: is more used than ::0 :P
375 2012-11-15 14:33:18 <phantomcircuit> within the last hour i've connected to and done a version handshake with 2697 bitcoin nodes
376 2012-11-15 14:36:15 <MC1984> cool beans?
377 2012-11-15 14:47:49 <phantomcircuit> no not really but ok
378 2012-11-15 14:48:40 <cjd> heh careful what you send as lastblock
379 2012-11-15 14:49:15 <phantomcircuit> cjd, i just picked a recentish block height
380 2012-11-15 14:49:17 <phantomcircuit> 208042
381 2012-11-15 14:49:28 <cjd> /nod
382 2012-11-15 14:49:30 <phantomcircuit> if i get bored i'll have it update using other peers numbers
383 2012-11-15 14:49:51 <cjd> sending big numbers there can be problematic as I suppose you heard
384 2012-11-15 14:49:56 <phantomcircuit> yeah
385 2012-11-15 14:50:18 <phantomcircuit> im going for something *much* harder to sto
386 2012-11-15 14:50:19 <phantomcircuit> p
387 2012-11-15 14:50:47 <cjd> I was working on my halfnode and then looked at my satoshi node and ooops
388 2012-11-15 14:50:48 <cjd> he
389 2012-11-15 14:50:49 <cjd> h
390 2012-11-15 14:51:56 <phantomcircuit> cjd, maybe you know this
391 2012-11-15 14:52:06 <phantomcircuit> how do i set the connect timeout for a socket that's non blocking
392 2012-11-15 14:52:19 <cjd> nope, no idea :)
393 2012-11-15 14:52:25 <phantomcircuit> please tell me i dont have to write a custom timeout handler and then close() the socket
394 2012-11-15 14:52:26 <cjd> udp ftw :)
395 2012-11-15 14:52:35 <phantomcircuit> damn
396 2012-11-15 14:52:41 <phantomcircuit> i *think* RCVTIMEO
397 2012-11-15 14:52:47 <cjd> meh, don't use a custom timeout handler, just pull it in from cjdns
398 2012-11-15 14:52:50 <cjd> Timeout.h
399 2012-11-15 14:53:11 <cjd> Timeout_setTimeout(callback, context, milliseconds, eventBase, allocator)
400 2012-11-15 14:53:15 <jgarzik> phantomcircuit: no
401 2012-11-15 14:53:24 <jgarzik> phantomcircuit: gotta alarm() or whatever
402 2012-11-15 14:53:47 <phantomcircuit> jgarzik, sigh
403 2012-11-15 14:54:15 <phantomcircuit> it's already an epoll based program so i dont need to alarm
404 2012-11-15 14:54:21 <cjd> pfft @ alarm, just use libevent
405 2012-11-15 14:54:25 <jgarzik> gah!!!
406 2012-11-15 14:54:30 <phantomcircuit> just have to keep a list of outstanding connections and then kill them as they come in
407 2012-11-15 14:54:48 <jgarzik> matt_von_mises is soliciting for thousands of dollars in donations
408 2012-11-15 14:54:50 <jgarzik> for cbitcoin
409 2012-11-15 14:55:00 <jgarzik> but he is obviously ignorant of so many network rules
410 2012-11-15 14:55:01 <phantomcircuit> the fuck is cbitcoin
411 2012-11-15 14:55:18 <jeremias> alt client to my knowledge
412 2012-11-15 14:55:24 <jeremias> or alternative library
413 2012-11-15 14:55:32 <jgarzik> phantomcircuit: a lib, in C.   https://bitcointalk.org/index.php?topic=123488.0 - [ANN] cbitcoin 2.0 - The Route to Bitcoin's Future
414 2012-11-15 14:55:48 <jgarzik> just read the thread for idiocy
415 2012-11-15 14:56:02 <phantomcircuit> so he's just going to end up splitting anybody who uses it
416 2012-11-15 14:56:04 <phantomcircuit> great plan
417 2012-11-15 14:56:24 <jgarzik> it's competition for picocoin's libccoin, so I have a conflict
418 2012-11-15 14:56:43 <jeremias> what's the problem, the more there are implementations the better
419 2012-11-15 14:56:46 <jeremias> even crappy ones
420 2012-11-15 14:56:56 <phantomcircuit> what's really needed is a few libraries
421 2012-11-15 14:57:04 <jgarzik> jeremias: crappy ones that -- he admits -- do not follow network rules?  that is bad for users.
422 2012-11-15 14:57:10 <phantomcircuit> one that handles basic stuff like proper serialization and script execution
423 2012-11-15 14:57:17 <phantomcircuit> one that handles storing the blockchain
424 2012-11-15 14:57:22 <jgarzik> That he wants to be paid for shite is particularly galling.
425 2012-11-15 14:57:23 <phantomcircuit> one that handles network interaction
426 2012-11-15 14:57:24 <jeremias> don't use it then
427 2012-11-15 14:57:35 <jgarzik> phantomcircuit: picocoin's libccoin does that.
428 2012-11-15 14:57:47 <phantomcircuit> i think libbitcoin does it
429 2012-11-15 14:57:56 <phantomcircuit> although to be honest i haven't been paying attention for a while
430 2012-11-15 14:58:03 <jgarzik> phantomcircuit: see "include/ccoin" and "lib" in https://github.com/jgarzik/picocoin/ for that
431 2012-11-15 14:58:33 <jgarzik> phantomcircuit: yeah, genjix's lib is a refactoring, and should cover that
432 2012-11-15 14:58:55 <sipa> libbitcoin is not a refactoring
433 2012-11-15 14:59:02 <sipa> libcoin by michael gronager is a refactoring
434 2012-11-15 14:59:20 <jgarzik> jeremias: it's not an issue for me.  it is an issue for any bitcoin user who mistakenly uses Matt_von_Mises' cbitcoin.
435 2012-11-15 14:59:35 <jgarzik> jeremias: they can create invalid transactions with it.... that cbitcoin tells them are valid
436 2012-11-15 15:00:02 <sipa> jgarzik: why?
437 2012-11-15 15:00:03 <jgarzik> jeremias: at best that's dishonest.  at worst it weakens the network, or even produces DoS.
438 2012-11-15 15:00:29 <sipa> i mean, he's shown to be ignorant about some things, but i haven't seen him make actual mistakes
439 2012-11-15 15:00:47 <jgarzik> sipa: specifically excluding the 120-confirmation rule, for one
440 2012-11-15 15:01:34 <sipa> yeah, he should understand why that rule exists, but he won't create invalid transactions because of it
441 2012-11-15 15:02:05 <i18n> picocoin looks neat. going to see if i can get it running here
442 2012-11-15 15:03:02 <cjd> it seems like as long as the miners use a standards complient implementation, everyone else will be dragged along anyway
443 2012-11-15 15:03:40 <gmaxwell> cjd: not so. It's more subtle than that.
444 2012-11-15 15:03:58 <gmaxwell> cjd: it's quite easy to have bugs that make reorginization impossible.
445 2012-11-15 15:04:03 <cjd> and in their defense, some of the rules make no sense and only exist because of what was originally a bug (ie the negative difficulty issue)
446 2012-11-15 15:04:54 <cjd> if they can't reorganize then that's their bug right?
447 2012-11-15 15:05:06 <gmaxwell> That isn't a defense. Thats part of the challenge.  It's like saying "in the defense of the designers of rockets that explode, friction and gravity are REALLY hard to deal with"
448 2012-11-15 15:06:00 <cjd> You can take steps (I think it would be good to invalidate any header with the negative bit set as a way to simplify it for everyone else)
449 2012-11-15 15:06:28 <sipa> that'd be a good candidate for a new network rule, yes
450 2012-11-15 15:06:37 <gmaxwell> cjd: Thats arm waving. If you introduce a new negative rule you _increase_ the risk of non-convergence.
451 2012-11-15 15:06:55 <gmaxwell> We've done so but it's a pain and takes time because it takes phased rollouts.
452 2012-11-15 15:07:11 <sipa> but even those have to be made carefully... first probably through shunning, then a voting mechanism to be sure enough miners agree on updating, and then globally define a switchover point
453 2012-11-15 15:07:34 <cjd> And because if there actually was a way to drive diff into the negatives, it would bypass minimum difficulty since the number is always smaller...
454 2012-11-15 15:07:34 <sipa> together with things like the signature malleability problem, for example
455 2012-11-15 15:09:15 <gmaxwell> Any media codec implementor has to deal with the same crap inscrutible bit-exact normative behavior, some of which is the result of reference implementation mistakes or design flaws.  And in those cases the consequences of getting it wrong are much lower.  It's a pain, but it's life. And someone who isn't prepared to deal with it to look crazy behavior in the face and not flinch has no business writing node software. :P
456 2012-11-15 15:09:54 <cjd> ok but one of these bugs will come back to haunt you, mark my words
457 2012-11-15 15:11:03 <sipa> cjd: I don't think we're disagreeing... yes unneccessary weirdnesses in the protocol/validity rules should be dealt with, and possibly made invalid altogether
458 2012-11-15 15:11:12 <gmaxwell> huh, I'm not saying we shouldn't phase in a rule to deny it. But it's not just a wand wave. We can't just add a line of code and call it done.
459 2012-11-15 15:11:31 <sipa> we've done so in the past, and there are proposals for more such in the future
460 2012-11-15 15:11:37 <gmaxwell> ::nods::
461 2012-11-15 15:11:52 <sipa> but in the mean time, someone writing node software should be prepared to deal with reality
462 2012-11-15 15:12:05 <sipa> and probably afterwards, weirdnesses will remain that can't be easily fixed
463 2012-11-15 15:12:32 <gmaxwell> cjd: and thank you very much for bringing it up. One of the metrics I look for in new implementations is that I don't believe they are trustworthy unless they report some odd behavior of the reference which they must duplicate.
464 2012-11-15 15:13:16 <gmaxwell> (if they do not it's because they've either inadequately reviewed and tested, or because they've so throughly copied from the reference that they're really just a refactor in disguise)
465 2012-11-15 15:13:19 <cjd> heh I'm thankfully not writing a whole implementation, more of a clock :)
466 2012-11-15 15:14:24 <cjd> https://ezcrypt.it/3B5n#ALPyJ4ldZUpN3yxMh8itFvQa <-- here's my version, it does handle negatives now
467 2012-11-15 15:22:25 <phantomcircuit> well im now scanning the entire network about once every 10 minutes
468 2012-11-15 15:22:36 <phantomcircuit> think i got a pretty good idea of how many peers there are...
469 2012-11-15 15:24:23 <sipa> phantomcircuit: ever checked my seeder?
470 2012-11-15 15:24:46 <phantomcircuit> what's the ip?
471 2012-11-15 15:25:10 <sipa> eh... seed.bitcoin.sipa.be runs one... but i mean the software
472 2012-11-15 15:25:36 <phantomcircuit> oh
473 2012-11-15 15:25:38 <sipa> https://github.com/sipa/bitcoin-seeder
474 2012-11-15 15:25:39 <phantomcircuit> no
475 2012-11-15 15:25:42 <phantomcircuit> i haven't
476 2012-11-15 15:26:05 <sipa> it crawls the network, by doing version/verack/getaddr/addr exchange with peers, and exposes that via dns
477 2012-11-15 15:26:31 <gavinandresen> phantomcircuit: why are you scanning the entire network?
478 2012-11-15 15:27:13 <phantomcircuit> counting connectable nodes
479 2012-11-15 15:27:54 <phantomcircuit> if my day isn't wasted entirely doing laundry i might just fill up half the nodes available connection slots as a poc
480 2012-11-15 15:28:41 <phantomcircuit> sipa, lol 24 threads
481 2012-11-15 15:28:47 <gmaxwell> phantomcircuit: many (most?) nodes are already half full.
482 2012-11-15 15:28:49 <phantomcircuit> python script running epoll
483 2012-11-15 15:29:07 <gmaxwell> phantomcircuit: if you want to attack things, please run on a private network.
484 2012-11-15 15:29:21 <phantomcircuit> gmaxwell, yes im going to segment the connectable peers into two groups
485 2012-11-15 15:29:33 <phantomcircuit> not going to do something based on just the connection count
486 2012-11-15 15:29:35 <gavinandresen> yes.  Or even better, implement a fix first and submit a patch.
487 2012-11-15 15:29:36 <phantomcircuit> that would be silly
488 2012-11-15 15:30:09 <gavinandresen> if you just want to be an asshole....  umm.... what's the point?  You'll just make us pissed off at you.