1 2015-04-04 01:01:33 <cfields> jtimon: since you're working on the decoupling atm, I'll PR my changes for checkpoints to spare you the trouble
  2 2015-04-04 01:01:43 <cfields> probably won't get to it this weekend, but first thing next week
  3 2015-04-04 01:05:09 <jtimon> the only thing I want to do with the checkpoints is separate ContextualCheckBlockHeader in 2 functions, one that uses the checkpoints to check the index and the other that actually does the consensus validations in the CBlockHeader
  4 2015-04-04 01:05:55 <jtimon> I've found out that exposing VerifyBlockHeader will be much simpler than VerifyTx, and VerifyBlock is definitely the last
  5 2015-04-04 01:06:15 <cfields> ok
  6 2015-04-04 01:07:41 <jtimon> please tell me if you will conflict with #5968 or #5669
  7 2015-04-04 01:08:48 <jtimon> on chainparams related, I haven't touched net, checkpoints or base58, but I think I've figured it out for main, miner, init and alert
  8 2015-04-04 01:10:45 <cfields> jtimon: those should be no problem
  9 2015-04-04 01:12:00 <cfields> i think i have checkpoints done as you'd like. It becomes a class under chainparams
 10 2015-04-04 01:12:23 <cfields> as for net, i've been trying different approaches at starting to break that up logically. It's a beast, though
 11 2015-04-04 01:13:02 <sipa> why do you need to break up net?
 12 2015-04-04 01:13:14 <sipa> (i mean: i'd love to see it broken up, but what does it have to do with consensus)
 13 2015-04-04 01:13:43 <cfields> sipa: nothing to do with consensus. Just trying to get some of the more basic structures out of it and separated from app state
 14 2015-04-04 01:13:47 <jtimon> you can only const CChainParams params = Params() in the beginning of thread functions
 15 2015-04-04 01:13:49 <sipa> ok, cool
 16 2015-04-04 01:14:30 <sipa> cfields: i have some plans for refactoring net/main/protocol, but it'll be a while before i get to it
 17 2015-04-04 01:14:53 <jtimon> but I don't know how to pass the CCahinParams as paremeter to those functions, need to investigate that
 18 2015-04-04 01:15:15 <sipa> jtimon: as a const CChainParams& ?
 19 2015-04-04 01:15:23 <cfields> sipa: i'd love to hear em. I created a p2p dir with great ambitions this week.. then they slowly faded away as i started working on it :)
 20 2015-04-04 01:15:47 <sipa> cfields: i plan to do a very minimal version just for ping/pong handling or so
 21 2015-04-04 01:16:03 <jtimon> well, what I've done in #5970 is be as explicit as possible with Consensus::Params and CChainParams
 22 2015-04-04 01:16:18 <cfields> sipa: ah, just taking one nib at a time?
 23 2015-04-04 01:16:43 <sipa> cfields: yes
 24 2015-04-04 01:17:26 <cfields> sipa: makes sense.
 25 2015-04-04 01:17:28 <sipa> cfields: i've previously argued against "move all of the handling code elsewhere" approaches, as those imho do not fundamentally improve thing; you want to split the individual handling up
 26 2015-04-04 01:17:38 <jtimon> if you grep "Params()\." there's still ugly things in net and base58
 27 2015-04-04 01:17:58 <jtimon> after #5970
 28 2015-04-04 01:18:05 <cfields> jtimon: yes, i've come up with a few options for base58
 29 2015-04-04 01:18:11 <cfields> nothing's pretty, though
 30 2015-04-04 01:18:46 <jtimon> I was thinking that the params could return an vector with all the prefixes
 31 2015-04-04 01:19:22 <cfields> sipa: i'm not following. You're wanting to avoid a monolithic handler by having the app throw messages at small individual handlers?
 32 2015-04-04 01:19:29 <jtimon> well, a reference to it
 33 2015-04-04 01:20:08 <sipa> cfields: imho, you want separate modules to handle different pieces of the protocol, in a pluggable way
 34 2015-04-04 01:20:31 <sipa> cfields: the problem is that currently, handling state is spread over main and net, with different locks, and much uglyness
 35 2015-04-04 01:20:46 <sipa> cfields: the solution is to have separate handlers, that manage their own independent state
 36 2015-04-04 01:21:19 <jtimon> in any case, cfields, I'm happy leaving base58 to you, please ping me for review on that when you've decided which option you like more
 37 2015-04-04 01:21:52 <sipa> cfields: previously suggested solutions just move message handling to a different file, without splitting up or abstracting the state those handlers used, which is imho useless, as it doesn't need to decrease the amount of work left to solve the problem correctly
 38 2015-04-04 01:22:39 <sipa> -need
 39 2015-04-04 01:23:31 <sipa> cfields: what i would suggest is to first have net expose a nice api for writing a message handler (which is able to manage its own per-module and per-node state), and register it
 40 2015-04-04 01:23:51 <sipa> cfields: and then use that for moving different pieces of processing out one by one
 41 2015-04-04 01:24:49 <jtimon> there's much of "the state" which is really just config, like chainparams, policy or other options stuff
 42 2015-04-04 01:24:50 <cfields> sipa: i've seen some work (jgarzik's PR iirc) that moves things to a different file and adds a main handler, but i assumed that was a first step in what you were talking about
 43 2015-04-04 01:25:37 <sipa> cfields: yes, that's what i'm talking about; imho that helps 0 in achieving the goal, and makes things just temporarily more ugly
 44 2015-04-04 01:27:14 <jtimon> that first step was moving processmessage to another file and then dividing the huge elif list in different functions, I liked that a lot
 45 2015-04-04 01:27:40 <jtimon> and I would do the same with the script interpreter btw
 46 2015-04-04 01:28:16 <jtimon> and the build can be identical if you make the new functions static, I think
 47 2015-04-04 01:28:36 <cfields> sipa: mmm, i won't argue since you've obviously thought it through. I'd be happy to help out if you'd like to work out a more detailed plan though, that's the kind of refactor i like working on
 48 2015-04-04 01:29:01 <cfields> not now though, time to go be social :)
 49 2015-04-04 01:29:12 <jtimon> I mean, moving it out of main like that wasn't very useful, but dividing it in functions, I think so
 50 2015-04-04 01:29:59 <cfields> and jtimon we can discuss base58 next week too, maybe you've come up with something nicer than my ugliness
 51 2015-04-04 01:31:02 <sipa> jtimon: the point is that moving to another file is useless, if the code later needs to move to a bunch of separate files anyway
 52 2015-04-04 01:31:12 <jtimon> well, after this week focusing on chainparams I'll go back to consensus and policy, partly building on top of these refactotings
 53 2015-04-04 01:34:46 <jtimon> so getting review on 5968 and 5669 (for consensus) and #5595 (policy) is moore prioritary for me
 54 2015-04-04 01:35:20 <cfields> jtimon: hehe, message received
 55 2015-04-04 01:35:21 <jtimon> 5970 and 5969 are the other 2 about chainparams
 56 2015-04-04 01:35:57 <jtimon> 5970 may be hard to merge without breaking many people's PRs though...
 57 2015-04-04 01:39:02 <phantomcircuit> 4230 is way too complicated for what it's goal is
 58 2015-04-04 01:41:43 <phantomcircuit> and why create a new class CTimeoutCondition instead of just use the boost::condition_variable
 59 2015-04-04 01:42:40 <gmaxwell> yea, this should be possible with a very small patch. just add the condition, have a reasonable timeout. there should be no need to pepper the code with manual triggers, the timeout should prevent starvation from anything that needs to get updated as a side effect.
 60 2015-04-04 01:44:10 <phantomcircuit> the only place i think a wake up is necessary is for an incoming message being complete or for an outbound message added to the queue
 61 2015-04-04 01:44:20 <phantomcircuit> both of which should be ~1 loc
 62 2015-04-04 01:45:17 <gmaxwell> I vaguely recall that there was some other incidental processing we were doing in that loop and so there was some desire to add these wakeups all over the place to avoid stalling it.
 63 2015-04-04 01:46:57 <phantomcircuit> gmaxwell, the trickle logic
 64 2015-04-04 01:47:10 <phantomcircuit> doesn't matter if that runs every 100ms though...
 65 2015-04-04 01:47:10 <sipa> phantomcircuit: no, because message-to-send creation is done asynchronously
 66 2015-04-04 01:47:17 <gmaxwell> yea but who cares if thats not totally timely, a 1 second level timeout would be fine, even preferable.
 67 2015-04-04 01:47:23 <phantomcircuit> sipa, ?
 68 2015-04-04 01:47:25 <sipa> phantomcircuit: so the loop needs to be woken up when there are messages to send
 69 2015-04-04 01:47:36 <sipa> phantomcircuit: see the SendMessages function
 70 2015-04-04 01:48:05 <gmaxwell> sure, so thats a single wake point. that seems fine to me; having two wake points makes sense; having a dozen does not.
 71 2015-04-04 01:48:09 <sipa> reason is that ProcessMessages does not lock the send buffer, and SendMessages does not lock the receive buffer
 72 2015-04-04 01:49:13 <phantomcircuit> sipa, yeah it assumes that it's being called on a timer
 73 2015-04-04 01:49:49 <phantomcircuit> so you call it on a timer and put a condition wake up in places that seem reasonable
 74 2015-04-04 01:50:14 <phantomcircuit> but if you miss some it at least doesn't get worse
 75 2015-04-04 01:54:35 <gmaxwell> just please don't end up with wakes all over the code, thats not maintainable, and its a sign something is wrong... as inevitably one will get forgotten where its really needed.
 76 2015-04-04 02:00:06 <phantomcircuit> gmaxwell, the trickle logic might not work correctly if SendMessages is called rapidly
 77 2015-04-04 02:00:27 <phantomcircuit> i dont see any timestamps or anything
 78 2015-04-04 02:01:05 <phantomcircuit> but actually i suspect that logic is generally broken probably
 79 2015-04-04 06:39:05 <fanquake> ;;blocks
 80 2015-04-04 06:39:06 <gribble> 350651
 81 2015-04-04 09:58:36 <DoughGaDeeDos> Accidently the whole Bitcoin-Dev Community Distributed Denial of Effectless debaucheries
 82 2015-04-04 09:58:37 <DoughGaDeeDos> Accidently the whole Bitcoin-Dev Community Distributed Denial of Effectless debaucheries
 83 2015-04-04 09:58:38 <DoughGaDeeDos> Accidently the whole Bitcoin-Dev Community Distributed Denial of Effectless debaucheries
 84 2015-04-04 09:58:39 <DoughGaDeeDos> Accidently the whole Bitcoin-Dev Community Distributed Denial of Effectless debaucheries
 85 2015-04-04 09:58:40 <DoughGaDeeDos> Accidently the whole Bitcoin-Dev Community Distributed Denial of Effectless debaucheries
 86 2015-04-04 09:58:41 <DoughGaDeeDos> Accidently the whole Bitcoin-Dev Community Distributed Denial of Effectless debaucheries
 87 2015-04-04 09:58:42 <DoughGaDeeDos> Accidently the whole Bitcoin-Dev Community Distributed Denial of Effectless debaucheries
 88 2015-04-04 09:58:43 <DoughGaDeeDos> Accidently the whole Bitcoin-Dev Community Distributed Denial of Effectless debaucheries
 89 2015-04-04 09:58:44 <DoughGaDeeDos> Accidently the whole Bitcoin-Dev Community Distributed Denial of Effectless debaucheries
 90 2015-04-04 10:48:30 <_1_rafael555> hdjsndj
 91 2015-04-04 10:48:32 <_1_rafael555> jdjdhjdbd
 92 2015-04-04 12:39:19 <bwen> is there a way to listen to new transactions going on the block chain without having to host the whole blockchain?
 93 2015-04-04 12:42:21 <leakypat> Ask chain analysis
 94 2015-04-04 12:42:32 <leakypat> They are the experts at this
 95 2015-04-04 12:44:12 <belcher> bwen what are you trying to do?
 96 2015-04-04 12:46:15 <belcher> perhaps using the websocket api of certain blockchain explorer websites
 97 2015-04-04 14:12:33 <treehug88> bwen yeah it's possible
 98 2015-04-04 23:16:30 <xhoch3> hey, anyone knows why bitcoin core uses Qt as well as boost at the same time?
 99 2015-04-04 23:17:00 <phantomcircuit> xhoch3, because those are for different things?
100 2015-04-04 23:17:14 <xhoch3> it's duplicate functionality
101 2015-04-04 23:17:20 <phantomcircuit> no it's not
102 2015-04-04 23:17:20 <xhoch3> Qt is an application framework..
103 2015-04-04 23:17:25 <phantomcircuit> one is a framework with a bunch of nice tools and the other is a gui framework
104 2015-04-04 23:17:35 <xhoch3> you're wrong
105 2015-04-04 23:17:39 <justanotheruser> xhoch3: I don't think Qt is used anywhere other than the GUI and there wasn't GUI functionality when Boost was originally used to make bitcoind
106 2015-04-04 23:17:45 <xhoch3> Qt is *not* a gui framework
107 2015-04-04 23:17:59 <phantomcircuit> justanotheruser, the original client was windows/gui only....
108 2015-04-04 23:18:00 <justanotheruser> phantomcircuit: umm, Qt isn't exclusively a GUI framework
109 2015-04-04 23:18:41 <xhoch3> ok, so for historical reasons
110 2015-04-04 23:19:07 <justanotheruser> I thought it was windows/cli only...
111 2015-04-04 23:19:21 <phantomcircuit> justanotheruser, nope the original client was windows/gui only
112 2015-04-04 23:19:44 <justanotheruser> hmm, the fact that it was called bitcoind confused me
113 2015-04-04 23:19:56 <justanotheruser> since until recently, bitcoind was the CLI
114 2015-04-04 23:20:01 <phantomcircuit> justanotheruser, it was a bitcoin daemon... with a gui
115 2015-04-04 23:22:18 <xhoch3> I see whether I can polish the UI a bit, need to build from source anyways
116 2015-04-04 23:24:29 <gmaxwell> xhoch3: they are largely orthorgonal, esp in the few parts of boost we use.
117 2015-04-04 23:24:45 <gmaxwell> xhoch3: much of boost is just "future C++" and those are the parts we use.
118 2015-04-04 23:24:54 <Luke-Jr> xhoch3: we're getting rid of boost in parts too
119 2015-04-04 23:25:13 <gmaxwell> And most of Bitcoin-qt is not the gui, the qt parts are well isolated and used only for the wallet.
120 2015-04-04 23:25:32 <Luke-Jr> justanotheruser: it wasn't called bitcoind until the RPC server was added later
121 2015-04-04 23:25:55 <xhoch3> just out of curiosity: why are the Qt parts well-isolated?
122 2015-04-04 23:26:02 <xhoch3> sounds like they were evil or sth :P
123 2015-04-04 23:26:10 <Luke-Jr> xhoch3: the eventual goal is to have it entirely separate
124 2015-04-04 23:27:03 <xhoch3> I understand the idea, but still I don't see why it is necessary, isn't Qt free software?
125 2015-04-04 23:27:33 <lewellyn> it's necessary for maintainability and ensuring decoupling.
126 2015-04-04 23:27:48 <lewellyn> software freedom has nothing to do with good practice.
127 2015-04-04 23:27:53 <lewellyn> (well, not directly.)
128 2015-04-04 23:28:21 <xhoch3> well, right now you stiff an application framework with a language into a small box and call it encapsulated
129 2015-04-04 23:28:30 <Luke-Jr> xhoch3: Qt is a lot of code. That tends to be bad for security.
130 2015-04-04 23:28:35 <xhoch3> language extension*
131 2015-04-04 23:28:53 <Luke-Jr> and horrendous for consensus systems
132 2015-04-04 23:29:39 <gmaxwell> (and bitcoind doesn't link qt at all)
133 2015-04-04 23:29:51 <xhoch3> I don't agree, and I think what there is currently is not beneficial, but ok, I don't want to discuss until the end
134 2015-04-04 23:30:03 <xhoch3> I live with it
135 2015-04-04 23:30:45 <xhoch3> hm, build fails here
136 2015-04-04 23:31:10 <xhoch3> make[2]: *** No rule to make target 'qt/test/moc_uritests.cpp', needed by 'qt/test/qt_test_test_bitcoin_qt-moc_uritests.o'.
137 2015-04-04 23:31:39 <xhoch3> I retry in non-parallel mode
138 2015-04-04 23:36:11 <gmaxwell> xhoch3: you need to run autogen again since your latest checkout, most likely.
139 2015-04-04 23:36:25 <xhoch3> I cloned today for the first time
140 2015-04-04 23:36:41 <gmaxwell> xhoch3: whatever you're not agreeing with; you likely lack the domain specific technical background to make that judgement, you should probably trust what luke says there.
141 2015-04-04 23:37:25 <xhoch3> gmaxwell, I am dev full-time
142 2015-04-04 23:37:27 <gmaxwell> qt/test/moc_uritests.cpp is generated at build time, how did you build.
143 2015-04-04 23:37:50 <xhoch3> I don't lack technical background
144 2015-04-04 23:38:02 <xhoch3> I have seen this approach you guys have fail badly many times
145 2015-04-04 23:38:09 <gmaxwell> xhoch3: Yes, I didn't say I doubted your general software expertise. We have very particular domain specific requirements in Bitcoin related to the required determinism of consensus software which are generally unlike other areas of software.
146 2015-04-04 23:38:17 <xhoch3> and seen how it (can) work well
147 2015-04-04 23:38:57 <xhoch3> now I'm interested :)
148 2015-04-04 23:39:22 <xhoch3> tell me more
149 2015-04-04 23:41:49 <gmaxwell> A substantial (and critically important) part of bitcoin is the implementation of the Bitcoin cryptosystem-- this is all that which must be absolutely and completely _bit_ identical in its operation on every system in the network.  The behavior requirement is similar to that in state of the art compression algorithims; where the encoder and decoder must stay in perfect sync. You can't use (convention
150 2015-04-04 23:41:51 <xhoch3> gmaxwell, as usual, ./autogen.sh, ./configure, make, but now I get a different error
151 2015-04-04 23:41:55 <gmaxwell> al) floating point in that software; because it will behave differently on different systems, you can't call out to other libraries (generally) because they're also not guarenteed to behave exactly the same forever.
152 2015-04-04 23:43:33 <gmaxwell> So it's the case that most observable behaviors in any consensus critical code become completely normative, if there is a bug you usually cannot "just" fix it.  Most software is not maintained with any thought to preserving the exact behavior-- quite the opposite: you want to fix bugs.  In bitcoin a behavior discrepency means consensus splits and nodes fragment into seperate networks.
153 2015-04-04 23:43:50 <xhoch3> gmaxwell, ok, now I get what you mean. You're absolutely right. But I was rather talking about it would be wise to have QThreads instead of boost threads, having the command line arguments better parsed with QCommandLineParser and so on
154 2015-04-04 23:44:19 <xhoch3> so you minimize dependencies
155 2015-04-04 23:44:39 <xhoch3> and actually use Qt like it was designed, that doesn't contradict with what you're saying
156 2015-04-04 23:44:44 <gmaxwell> Yea, we could do that-- QT is reasonable in non-consensus parts, no problem; but then it would add a gratitious qt dependency to bitcoind which otherwise could be built without QT at all.
157 2015-04-04 23:45:12 <phantomcircuit> (and is built without qt at all)
158 2015-04-04 23:45:36 <gmaxwell> yes, and, indeed, is built without QT.
159 2015-04-04 23:45:59 <xhoch3> I think Qt can be regarded as a more or less fixed part on many systems these days. There is a reason the Ubuntu Phones completely rely on Qts ecosystem
160 2015-04-04 23:46:32 <xhoch3> what you say is absolutely true and depends on how conservative you are
161 2015-04-04 23:47:07 <xhoch3> if you want to run bitcoind on a machine that's not seen updates for five years, I completely agree
162 2015-04-04 23:47:16 <xhoch3> but things have changed drastically
163 2015-04-04 23:47:19 <gmaxwell> Bitcoin Core is often run on server that have on QT; libconsensus is also intended to eventually run on small embeded devices where QT would be a significant overhead.  I think our general direction will be to eliminate boost too.
164 2015-04-04 23:47:22 <phantomcircuit> xhoch3, think firmware for artificial heart conservative
165 2015-04-04 23:47:51 <xhoch3> gmaxwell, I can oppose that with practical experience
166 2015-04-04 23:47:53 <xhoch3> :)
167 2015-04-04 23:48:26 <gmaxwell> xhoch3: how so?
168 2015-04-04 23:48:55 <xhoch3> at work I have some Olimex boards (A20, less than 5W power consumption), 8 GB sd card and Qt is not much of an overhead there. I write small IoT appliances with this approach
169 2015-04-04 23:48:59 <phantomcircuit> then he was wrong about <gmaxwell> xhoch3: Yes, I didn't say I doubted your general software expertise.
170 2015-04-04 23:49:04 <gmaxwell> Right now from boost mostly we use things that are in C++11, plus boost asio (ugh; which we'll on track replace with libevent), plus boost threads, plus the commandline stuff.
171 2015-04-04 23:49:05 <xhoch3> having said that, Qt is very popular in the embedded world
172 2015-04-04 23:49:33 <phantomcircuit> gmaxwell, the boost asio stuff is all non consensus and rpc stuff right
173 2015-04-04 23:50:04 <gmaxwell> xhoch3: there are orders of magnitude, think more about 100mhz devices with 128k of ram, no mmu, that draw microwatts in sleep-- which is far more reflective of "IoT" appliances in general. :)
174 2015-04-04 23:50:27 <gmaxwell> phantomcircuit: yes, the only consensus use of boost is locking constructs and things like FOR_EACH which are in C++11.
175 2015-04-04 23:50:36 <xhoch3> I don't always accept challenges, but when I do..
176 2015-04-04 23:51:14 <xhoch3> I see your point, but why would anyone run bitcoind on such a device
177 2015-04-04 23:52:29 <xhoch3> except for the reason he is able to so so
178 2015-04-04 23:52:35 <xhoch3> do*
179 2015-04-04 23:52:42 <gmaxwell> xhoch3: as a SPV client, for things like smart property.   In any case, general point is that such dependencies are precluded in consensus code for good reason, and if we don't have it in consensus code; there is _relatively_ little gain in bitcoind, and reviewability and such still prefers we minimize dependencies and complexity even where its not strictly needed.
180 2015-04-04 23:53:30 <gmaxwell> Also, running the consensus code inside zero knoweldge proofs will be important for scalability in the future; and there the execution enviroment is pretty similar to those kinds of very limited microcontrollers.
181 2015-04-04 23:54:46 <xhoch3> I doubt you can run it if it's not simple C at all, let me have a look
182 2015-04-04 23:54:54 <gmaxwell> xhoch3: e.g. I do substantial fraction of the review work on Bitcoin Core non-QT and I know virtually nothing about QT and don't need to learn it. Using QT there would be a substantial burden on me if I were to retain the same level of thoroughness on review.
183 2015-04-04 23:55:23 <gmaxwell> xhoch3: Mind restating that?
184 2015-04-04 23:55:25 <kanzure> hey you know what's a fun way to blow a few hundred hours? reading qt source code :\
185 2015-04-04 23:55:25 <Luke-Jr> gmaxwell: QtCore is probably lighter than the whole QtGui etc; but since we want the GUI independent anyway, no reason to prefer Qt for other stuff
186 2015-04-04 23:55:44 <gmaxwell> kanzure: it can't be worse than boost.
187 2015-04-04 23:55:52 <kanzure> i made the mistake of reading webkit source code
188 2015-04-04 23:55:59 <gmaxwell> okay, maybe it could be.
189 2015-04-04 23:56:21 <kanzure> "i am going to need how many arrows and boxes??"
190 2015-04-04 23:56:29 <kanzure> er, *how*
191 2015-04-04 23:56:31 <Luke-Jr> QtWebKit isn't even part of QtGui :P
192 2015-04-04 23:56:52 <kanzure> whoops yes i do mean qtwebkit in this case
193 2015-04-04 23:58:19 <xhoch3> can't find paymentrequest.pb.h
194 2015-04-04 23:59:12 <gmaxwell> xhoch3: configure should have disabled building any of the qt stuff if you do not have the protobuff librarles/tools. Does for me.