1 2015-09-18 00:10:35 <wumpus> god damnit ,zillions of new issues for zmq
  2 2015-09-18 00:10:38 <wumpus> who can I assign them to?
  3 2015-09-18 00:11:21 <wumpus> also the documentation about dependencies has not been updated
  4 2015-09-18 00:11:31 <wumpus> and as said, the release notes
  5 2015-09-18 00:12:25 <wumpus> who is going to do that?
  6 2015-09-18 00:15:21 <sipa> wumpus: wait a few days, if not fix, revert?
  7 2015-09-18 00:16:13 <wumpus> agreed
  8 2015-09-18 01:53:07 <jcorgan> with the current zmq build, not finding the dependencies just results in the feature being disabled, and the rest of the configure/compile succeeding.  where is that failing for people?
  9 2015-09-18 02:05:29 <wumpus> that sounds good to me
 10 2015-09-18 02:20:06 <jcorgan> i'll add the dependencies to the readme, but it should have been a "nop" merge unless you specifically ask for the feature.  if that's not the case, i need to understand what is breaking.
 11 2015-09-18 02:21:39 <jgarzik> jcorgan, that changed in the interim based on feedback
 12 2015-09-18 02:22:44 <jcorgan> the feature is "on" by default, but configure will disable it if the deps aren't found
 13 2015-09-18 02:22:50 <jgarzik> A relevant general principle is that we want to maximize the amount of code built by default.  Code that isn't built by default breaks often, since other programmers don't see it.
 14 2015-09-18 02:22:58 <jcorgan> right
 15 2015-09-18 02:23:14 <jgarzik> (Obviously within that general principle you exercise judgement)
 16 2015-09-18 02:23:29 <jcorgan> i just tested all combinations of enable/disable and deps installed/not installed and it did the right thing each time
 17 2015-09-18 02:23:42 <ludbb> is the idea to publish every transaction (not just those related to the wallet) through zmq?
 18 2015-09-18 02:24:06 <jcorgan> yes if configured to do so
 19 2015-09-18 02:24:15 <ludbb> nice
 20 2015-09-18 02:24:26 <jcorgan> blocks tooo
 21 2015-09-18 02:24:34 <ludbb> the full block?
 22 2015-09-18 02:24:48 <jcorgan> in all its raw glory.  but only if configured to do so.
 23 2015-09-18 02:25:18 <ludbb> I guess someone could use that, what are you using this specific feature for?
 24 2015-09-18 02:25:55 <ludbb> publishing hashes seemed fine already
 25 2015-09-18 02:26:04 <jcorgan> the origal use case was to let bitcoind do all the network and consensus stuff, and be able to notify listening zmq clients of incoming blocks/txes that passed.  that way, other software didn't have to know anything about p2p or consensus but could "trust" bitcoind
 26 2015-09-18 02:26:36 <ludbb> except in all those cases where one piece of that system goes down, yes? because you will need to get in sync when it comes back
 27 2015-09-18 02:27:03 <ludbb> or is it possible to re-sync through zmq too? like "the last block I know is x, give me everything after that"
 28 2015-09-18 02:27:20 <jcorgan> zmq is a notification only system, not a query interface
 29 2015-09-18 02:27:39 <ludbb> well, I guess the way zmq is being used is as a notification system
 30 2015-09-18 02:27:43 <jcorgan> it's just an unreliable "network interface", like p2p
 31 2015-09-18 02:27:45 <ludbb> it clearly is not limited to that
 32 2015-09-18 02:28:02 <ludbb> sure
 33 2015-09-18 02:28:10 <jcorgan> yes, zmq can do much more; this feature was designed to make very minimal use of it
 34 2015-09-18 02:28:42 <ludbb> so it doesn't seem the promise of "not know anything about p2p or consensus" can be met as easy as that
 35 2015-09-18 02:28:59 <jcorgan> well, ok
 36 2015-09-18 02:29:12 <ludbb> but it's a nice step forward, in any case
 37 2015-09-18 02:29:16 <jcorgan> i meant, "not having to know how to do p2p or how to validate blocks/tx"
 38 2015-09-18 02:29:31 <ludbb> there was a different implementation that is/was using zmq, right? libbitcoin?
 39 2015-09-18 02:29:35 <jcorgan> because bitcoind will do that for you and let you know what passes the filter
 40 2015-09-18 02:30:03 <jcorgan> not sure what it was, but yes, there have been other, grander users of zmq by others in the past
 41 2015-09-18 02:30:20 <ludbb> that's the only I remember
 42 2015-09-18 02:30:55 <jcorgan> this is designed to be very lightweight and limited in what it can do
 43 2015-09-18 02:31:37 <ludbb> I see the changes now
 44 2015-09-18 02:32:04 <ludbb> is this sponsored by bitreserve?
 45 2015-09-18 02:32:42 <jcorgan> no
 46 2015-09-18 02:33:16 <jcorgan> i did the original work and PR a long time ago, then had to step away, it was taken over by jgarzik and then jonasschnelli
 47 2015-09-18 02:33:20 <jcorgan> and recently merged
 48 2015-09-18 02:33:37 <ludbb> but it started at bitreserve?
 49 2015-09-18 02:33:42 <jcorgan> no
 50 2015-09-18 02:33:49 <ludbb> so much confusion :)
 51 2015-09-18 02:34:24 <jcorgan> how so?
 52 2015-09-18 02:35:09 <ludbb> the zmq changes were and are in the bitreserve's fork of bitcoin
 53 2015-09-18 02:35:11 <wumpus> obelisk uses zmq for many other things, like interprocess communication
 54 2015-09-18 02:35:23 <wumpus> we just use it for notification
 55 2015-09-18 02:36:17 <jcorgan> oic, i see the confusion
 56 2015-09-18 02:36:48 <jcorgan> since the original PR apparently several other companies have included the patch in their local configuration
 57 2015-09-18 02:37:14 <ludbb> (obelisk is apparently dead, libbitcoin-server is the one now)
 58 2015-09-18 02:37:42 <jcorgan> and yes, zmq can be used for many more things; the feature recently merged was intentionally limited to a very small set of "read-only" functions
 59 2015-09-18 02:38:10 <ludbb> sure
 60 2015-09-18 02:38:23 <wumpus> yes, I use libbitcoin-server and obelisk interchangably, has been a while that I looked at it
 61 2015-09-18 02:39:12 <jcorgan> perhaps someday bitcoind can use zmq for a bettery query interface than rpc, but, that's a huge ball of technical/political worms i have no interest in pursuing
 62 2015-09-18 02:39:54 <ludbb> maybe before that it could allow plugging in a different database, it's a considerable issue to have a database that is limited to a single process
 63 2015-09-18 02:40:24 <ludbb> much of what this zmq publisher is doing seem to not be necessary if such possibility were in place
 64 2015-09-18 02:41:14 <ludbb> in the end we will have duplicate databases and indexes around, although it's good to have something better for pushing this information with zmq
 65 2015-09-18 02:41:31 <wumpus> the database being limited to a single process wouldn't be as much of a problem if the query interface is efficient
 66 2015-09-18 02:41:57 <wumpus> most databases are maintained by a single process (unless you count sharding etc)
 67 2015-09-18 02:42:29 <ludbb> uhm, that's not what I meant
 68 2015-09-18 02:42:42 <ludbb> you can't open the leveldb from multiple processes
 69 2015-09-18 02:42:52 <ludbb> you can certainly connect from many different processes in most of the typical databases
 70 2015-09-18 02:42:53 <wumpus> e.g. postgres is also a single process, you just can access it from multiple clients efficiently
 71 2015-09-18 02:43:00 <wumpus> so can you in bitcoind
 72 2015-09-18 02:43:10 <ludbb> not really without going through rpc
 73 2015-09-18 02:43:20 <wumpus> postgres's intercace is also an RPC
 74 2015-09-18 02:43:50 <wumpus> there are very few databases that allow concurrent read/write access to the files by different processes without having a central management process
 75 2015-09-18 02:43:56 <ludbb> I'm not sure if you're saying it's actually an implementation of json over http in a different path or something different
 76 2015-09-18 02:44:01 <ludbb> because it's completely different
 77 2015-09-18 02:44:41 <wumpus> no, it's not json over http, but it is an interprocess communication interface using a protocol, thus an RPC
 78 2015-09-18 02:44:55 <wumpus> each database tends to have its own wire protocol, I don't see how that is relevant
 79 2015-09-18 02:46:03 <wumpus> bitcoind happens to use json over http, so do some other databases
 80 2015-09-18 02:46:04 <ludbb> you don't see how the different protocol is relevant or you don't see the difference I'm trying to mention?
 81 2015-09-18 02:47:51 <wumpus> as I see it it just doesn't matter. I think, no matter what, it is a bad idea to access bitcoind's files while it's running, just like you wouldn't access postgressql's
 82 2015-09-18 02:48:22 <ludbb> the difference I'm trying to tell you is this: bitcoind writes to some database, the access to that database is no longer handled by bitcoind
 83 2015-09-18 02:48:26 <wumpus> that the current RPC implementation does locking all over the place, resulting in lock contention and inefficiency when it gets a lot of queries is an issue
 84 2015-09-18 02:48:36 <ludbb> sure, so that's what I'm saying
 85 2015-09-18 02:48:43 <ludbb> if we go back one or two pages
 86 2015-09-18 02:48:49 <wumpus> but that has nothing to do with accessing bitcoind's files while it's running
 87 2015-09-18 02:49:01 <wumpus> that's just plainly a bad idea. the on-disk format is not an interface
 88 2015-09-18 02:49:15 <ludbb> that's not what I'm saying, at all
 89 2015-09-18 02:49:33 <wumpus> ok, then we're talking past each other, never mind
 90 2015-09-18 02:49:42 <ludbb> so right now bitcoind writes to a database using leveldb, yes?
 91 2015-09-18 02:50:39 <ludbb> now imagine some time from now it's possible to use a different database, one that is not leveldb
 92 2015-09-18 02:50:47 <ludbb> now bitcoind writes to that database
 93 2015-09-18 02:51:35 <ludbb> and that database happens to be something like postgresql, which provides a way to access it
 94 2015-09-18 02:51:48 <ludbb> that's what I'm saying, being able to use a different database which has different capabilities
 95 2015-09-18 02:52:23 <ludbb> I'm not trying to argue that any of these typical sql database are going to more space efficient or anything like that
 96 2015-09-18 02:52:49 <wumpus> yes, I understand. would be quite straightforward by changing txdb.cpp/h, but mind that it's optimized for bitcoind's usage and performance characteristics, and what is written to the database is not an external interface supposed to be used by other applications
 97 2015-09-18 02:53:27 <ludbb> well, that's today's reality which could change
 98 2015-09-18 02:53:47 <ludbb> what happens is that people duplicate that to achieve this
 99 2015-09-18 02:54:14 <wumpus> as said, it wouldn't be an issue of bitcoind's query interface was faster, then you could just query bitcoind any time you need data
100 2015-09-18 02:54:20 <jcorgan> yeah, i tried my hand at a sql database for bitcoin, it got ugly pretty quickly
101 2015-09-18 02:54:23 <wumpus> bitcoind manages the databases
102 2015-09-18 02:54:51 <wumpus> whether leveldb, or lmdb, or some custom file layer is used internally is just an implementation detail
103 2015-09-18 02:55:18 <wumpus> it's bound to change from version to version depending on what is fastest for bitcoind's purposes
104 2015-09-18 02:55:32 <jcorgan> well, having bitcoind manage the database would be fine, but there is strong antipathy against having the bitcoind database having tables/indexes of a more general nature than what bitcoind needs
105 2015-09-18 02:55:52 <wumpus> yes, there is, for good reason: we don't need those
106 2015-09-18 02:55:56 <jcorgan> right
107 2015-09-18 02:56:10 <jcorgan> so for people who do want those, an external database is necessary
108 2015-09-18 02:56:22 <wumpus> if you do need those, yes you need an external database, that makes sense
109 2015-09-18 02:56:29 <ludbb> hah :)
110 2015-09-18 02:56:34 <wumpus> not everything has to be built into bitcoind
111 2015-09-18 02:56:41 <jcorgan> i completely agree
112 2015-09-18 02:56:56 <jcorgan> was part of the original motiviation for the zmq notification system :-)
113 2015-09-18 02:57:00 <wumpus> bitcoind should be optimized to do what it does, providing validation and (partial) storage of the chain
114 2015-09-18 02:57:04 <wumpus> right :)
115 2015-09-18 02:57:53 <wumpus> I'd epect using e.g. pgsql would be awful for validation performance
116 2015-09-18 02:58:20 <morcos> jcorgan: i definitely did not have a clean build without libzmq.  i got compile errors.
117 2015-09-18 02:58:32 <jcorgan> morcos: would love to hear more detail
118 2015-09-18 02:58:38 <ludbb> I would expect the same wumpus, but for some cases it doesn't really matter
119 2015-09-18 02:58:40 <wumpus> morcos: even after cleaning the tree / re-running autogen.sh?
120 2015-09-18 02:58:45 <ludbb> the benefits are greater than that
121 2015-09-18 02:58:52 <wumpus> ludbb: maybe not, but that's the focus of the project
122 2015-09-18 02:59:10 <ludbb> and I'm not asking to move to postgresql
123 2015-09-18 02:59:23 <ludbb> I'm saying that it would be worth considering the possibility of plugging in a different database
124 2015-09-18 02:59:28 <morcos> wumpus: i never tried completely cleaning the tree and rerunning autogen.sh until after i installed libzmq
125 2015-09-18 02:59:41 <ludbb> it's going to solve a couple different problems, including the one that the zmq interface is trying to help with
126 2015-09-18 02:59:57 <jcorgan> i would expect that if you had an existing build and then just did a git pull, a rebuild would break badly
127 2015-09-18 03:00:01 <wumpus> ludbb: yes, and for stated reasons I don't agree
128 2015-09-18 03:00:02 <morcos> but rerunning configure before i installed libzmq still left ENABLE_ZMQ or whatever set to 1
129 2015-09-18 03:00:21 <wumpus> morcos: going to try. I don't have libzmq on this machine so I'll find out soon enough
130 2015-09-18 03:00:37 <morcos> jcorgan: oh.  well isn't that what people usually do
131 2015-09-18 03:00:53 <morcos> most changes don't require you to clean the tree or rerun autogen.sh
132 2015-09-18 03:00:57 <jcorgan> you can usually "get away" with it
133 2015-09-18 03:01:05 <wumpus> morcos: dependency changes usually do
134 2015-09-18 03:01:13 <jcorgan> but sometimes things change in the build system that make that impossible
135 2015-09-18 03:01:18 <morcos> ok fair enough
136 2015-09-18 03:01:36 <morcos> i was able to build without libzmq by specifically typing configure -disable-zmq
137 2015-09-18 03:01:54 <morcos> and then later i installed libzmq
138 2015-09-18 03:02:01 <jcorgan> that is the design, but again, even if you don't do that, if it doesn't see libzmq, it turns it off
139 2015-09-18 03:02:09 <wumpus> so it fails configure if you don't have libzmq?!
140 2015-09-18 03:02:26 <jcorgan> it shouldn't.  i just retested that.
141 2015-09-18 03:02:51 <jcorgan> a default git clone, without zmq installed, configures/builds perfectly
142 2015-09-18 03:02:51 <ludbb> -> use by default if it is present, don't otherwise
143 2015-09-18 03:02:55 <morcos> all i'm saying is that if you don't rerun autogen.sh...  then configure does not detect that you don't have libzmq
144 2015-09-18 03:03:30 <wumpus> morcos: ok, that could be, that's normal - in theory you always have to re-run autogen.sh when something in the build system changes
145 2015-09-18 03:03:42 <wumpus> morcos:  usually you can get away with not doing that, but it may fail
146 2015-09-18 03:04:12 <morcos> yeah it works OFTEN enough that i don't automatically remember to just rerun autogen.sh everytime something breaks
147 2015-09-18 03:04:22 <wumpus> I understand :)
148 2015-09-18 03:04:24 <morcos> i don't know why often is capitalized
149 2015-09-18 03:04:33 <jcorgan> i pretty much do a make uninstall; git clean -dxf; git pull; autogen.sh
150 2015-09-18 03:04:34 <wumpus> I usually re-run autogen.sh also only when it fails
151 2015-09-18 03:06:51 <wumpus> just checked, running autogens.sh / configure works fine on a clean tree w/out libzmq installed. On al old tree, ENABLE_ZMQ will not be defined in bitcoin-configure.h resulting in potentially undefined behavior
152 2015-09-18 03:07:14 <wumpus> anyhow, that's somewhat  to be expected and not really an issue
153 2015-09-18 03:07:37 <wumpus> all our build guides include running autogen.sh I hope?
154 2015-09-18 03:11:57 <morcos> wumpus: re locking in GetTransaction, getrawtransaction already called cs_main before even calling GetTransaction, so its only in the call from rest_tx that main is held slightly longer
155 2015-09-18 03:12:25 <morcos> in addition many other functions call ReadBlockFromDisk and were all already holding cs_main
156 2015-09-18 03:12:47 <morcos> so i agree we should improve it, sipa mentioned some ideas earlier today, but in the interim, this barely makes the locking worse
157 2015-09-18 03:12:57 <wumpus> morcos: unfortunate but true
158 2015-09-18 03:13:44 <wumpus> I've also utACKed the pull, the solution is ok for now
159 2015-09-18 03:14:34 <morcos> yeah, i snuck in another unnecessary scoping block removal.   is it preferred to not do that?
160 2015-09-18 03:14:55 <wumpus> I think that comment made sense, yes
161 2015-09-18 03:15:40 <morcos> yeah i did that and removed other extra braces at the top.  it's never clear to me when is a good time to do a little clean up in the area of code you're working on.
162 2015-09-18 03:15:41 <wumpus> such a trivial change is ok
163 2015-09-18 03:15:44 <morcos> ok
164 2015-09-18 03:16:43 <wumpus> the time for sensible cleanups is always when you're touching some code (although doing so in a seperate commit is good)
165 2015-09-18 03:17:16 <morcos> not sure if you saw my comments earlier about LoadExternalBlockFile, oh i guess i said I'll file an issue, but that's totally missing any locking whatsoever..
166 2015-09-18 03:18:34 <wumpus> that sounds like a bug
167 2015-09-18 03:19:06 <morcos> yeah it is..., but throwing a giant cs_main around LEBF sounded like a bad idea. :)
168 2015-09-18 03:19:23 <wumpus> yes, it should only lock while processing a block
169 2015-09-18 03:19:35 <wumpus> not e.g. while reading from disk
170 2015-09-18 03:20:13 <wumpus> I suppose we've always gotten away with how it is now because nothing else is writing blocks during LEBF active
171 2015-09-18 03:20:26 <wumpus> e.g. P2P sync is disabled
172 2015-09-18 03:20:38 <wumpus> but that's kind of fragile
173 2015-09-18 03:21:21 <morcos> yeah submitblock could get unlucky
174 2015-09-18 03:23:06 <morcos> yeah so then we're left with the lack of locking on ReadBlockFromDisk if we aren't protecting the reading from disk.  I haven't been able to think how anything could be causing a problem there.  I suppose submit block could cause pruning, but it seems it shouldn't cause pruning of a blockfile you'd still need to be reading by definition, but that sure seems fragile as well.
175 2015-09-18 03:23:29 <wumpus> yep. Although there too, submitting a block while reindexing is not a likely scenario
176 2015-09-18 03:24:54 <wumpus> pruning *during* LoadExternalBlockFile?
177 2015-09-18 03:25:10 <wumpus> whoa, hadn't thought of that one
178 2015-09-18 03:25:25 <wumpus> pruning does make all kinds of things that used to be obvious kind of tricky
179 2015-09-18 03:27:42 <morcos> yeah but if its LoadExternalBlockFile causing the pruning itself, it'll all be in the same thread
180 2015-09-18 03:27:49 <morcos> that does happen for sure
181 2015-09-18 03:28:48 <wumpus> and it wouldn't make sense to be pruning the block that it's reading, although I suppose it could decide so if a file contains only old blocks (after all, they don't have to be in order...)
182 2015-09-18 03:29:30 <morcos> oh
183 2015-09-18 03:29:37 <wumpus> you'd need a mechanism to signal 'leave this block file alone, I'm using it'
184 2015-09-18 03:29:41 <morcos> hmm
185 2015-09-18 03:32:09 <wumpus> absent that, we indeed need to make sure cs_main is held during all block reads
186 2015-09-18 03:33:07 <wumpus> though in the case of LEBF that may not be enough, if it could decide to prune based on information it has about a file that is still being read
187 2015-09-18 03:33:35 <wumpus> what about pruning only when LEBF is complete?
188 2015-09-18 03:34:14 <wumpus> this will also make sure that you can restart bitcoind without LEBF failing next time because some files have already been deleted
189 2015-09-18 03:34:22 <morcos> i was getting confused and thinking you couldn't do that b/c what if you didn't have the space, but i suppose you probably already do
190 2015-09-18 03:34:30 <morcos> going to look at code for a min
191 2015-09-18 03:34:35 <wumpus> (eg if you quit during reindex)
192 2015-09-18 03:34:51 <wumpus> (otherwise you're pretty screwed and need to redownload)
193 2015-09-18 03:56:18 <morcos> ok i need to go through this more carefully in the morning, but i think its ok right now
194 2015-09-18 03:57:04 <morcos> when pruning it goes through all the blockfiles, up to and not including the last one
195 2015-09-18 03:57:27 <morcos> by last one, i mean nLastBlockFile
196 2015-09-18 03:57:41 <morcos> which should be the one it is in the process of reading if reindexing
197 2015-09-18 03:57:51 <morcos> or writing to if its reading from an external file
198 2015-09-18 03:58:59 <morcos> also, i think
199 2015-09-18 03:59:08 <morcos> pruning won't happen until you're trying to allocate more space
200 2015-09-18 03:59:20 <jgarzik> github and travis appear to work on the same model - get you hooked on the free crack (public repos), and then charge you for the good stuff (private repos)
201 2015-09-18 03:59:25 <morcos> so in the case of a reindex, it shouldnt' prune until the end
202 2015-09-18 03:59:57 <morcos> but even if thats all correct it seems also fragile and at the very least needs better documentation
203 2015-09-18 04:00:54 <jgarzik> *PR tester
204 2015-09-18 04:02:31 <morcos> ok time for zzzzzzzzzzzz
205 2015-09-18 04:06:38 <wumpus> jgarzik: it's better than advertising like sourceforge :)
206 2015-09-18 04:06:54 <jgarzik> that's the truth..
207 2015-09-18 04:46:57 <maaku> jgarzik: i didn't know you could do that either. thank you
208 2015-09-18 06:31:50 <jgarzik> nice
209 2015-09-18 06:32:03 <jgarzik> +   - osx
210 2015-09-18 06:32:17 <jgarzik> one line to .travis.yml and univalue tests are built on both linux and osx
211 2015-09-18 06:36:19 <jgarzik> only a couple lines of "brew install" to get another project of mine, itd, building & passing tests on osx w/ travis
212 2015-09-18 06:40:27 <phantomcircuit> there isn't a reverse dependency list for the orphan map is there?
213 2015-09-18 06:40:50 <phantomcircuit> would anybody be opposed to adding one, so that we can efficiently determine when a transaction is no longer orphaned?
214 2015-09-18 06:43:12 <phantomcircuit> wait mapOrphanTransactionsByPrev maybe is
215 2015-09-18 06:46:03 <phantomcircuit> hmm
216 2015-09-18 07:19:37 <Luke-Jr> jonasschnelli: your doodle thing only lets me select 3 possible times per day?
217 2015-09-18 07:19:48 <jonasschnelli> Luke-Jr: let me check it.
218 2015-09-18 07:20:11 <jonasschnelli> Luke-Jr: yes. I have only proposed these three timeslots per day... Do you need another time slot?
219 2015-09-18 07:20:32 <Luke-Jr> jonasschnelli: IMO every of the 24 hours ought to be possible to vote for :p
220 2015-09-18 07:21:06 <jonasschnelli> Luke-Jr: Right... but i guess not possible with doodle. Let me write a comment.
221 2015-09-18 07:21:06 <Luke-Jr> (I will of course probably not vote for nearly all of them)
222 2015-09-18 07:21:20 <jonasschnelli> Luke-Jr: do you propose another slot? From when to when?
223 2015-09-18 07:21:25 <Luke-Jr> jonasschnelli: maybe there's a specific tool for such time coordination?
224 2015-09-18 07:22:07 <Luke-Jr> eg, one that lets people set a series of day(s)+time begin+time end
225 2015-09-18 07:22:48 <jonasschnelli> Maybe with fixed time slots (maybe 5 instead of 3) would be more easy to find consensus about a slot?
226 2015-09-18 07:23:42 <jonasschnelli> I think the three slots 15:00/19:00/21:00 UTC are the only slots that make sense for US/Euro?
227 2015-09-18 07:24:08 <jonasschnelli> (to keep it within business, evening hours)
228 2015-09-18 07:27:23 <Luke-Jr> IMO 16:00 is probably the most likely to work for the most people: 8 AM in California, noon in eastern US, later in the evening in China, etc
229 2015-09-18 07:30:35 <jonasschnelli> Luke-Jr: okay. let me add it
230 2015-09-18 07:31:49 <jonasschnelli> added
231 2015-09-18 08:33:56 <btcdrak> jonasschnelli: 17:00-18:00 CEST are popular times for other projects.
232 2015-09-18 08:34:56 <btcdrak> I dont think it's good planning in UTC, since a lot of people are on summer adjusted timezones, so it's confusing
233 2015-09-18 08:37:04 <btcdrak> Luke-Jr: 16:00 is actually 09:00 in California
234 2015-09-18 09:11:30 <jonasschnelli> btcdrak: CEST 17:00 would be a good timeslot. But we should also agree on a day. Thursday may not possible for everyone, maybe another day is.
235 2015-09-18 09:11:41 <jonasschnelli> (Thursday would not be possible for me)
236 2015-09-18 10:33:13 <Luke-Jr> btcdrak: DST?
237 2015-09-18 10:37:09 <btcdrak> Luke-Jr: yeah. because in the end, we need to convert what the actual time is for wherever we are. so 1600UTC is 0900 localtime in CA (and SF). http://everytimezone.com/#2015-9-18,-1200,cn3
238 2015-09-18 10:37:49 <Luke-Jr> btcdrak: 8 AM part of the year and 9 AM during DST sounds pretty good? :P
239 2015-09-18 10:38:02 <btcdrak> yeah it's crazy :))
240 2015-09-18 13:05:27 <jgarzik> did you know you can enable Travis on your own repo (e.g. jgarzik/bitcoin.git or jonasschnelli/bitcoin.git)?  I just discovered this.
241 2015-09-18 13:05:42 <jgarzik> No need to wait for bitcoin/bitcoin PR to trigger a build
242 2015-09-18 13:05:48 <jonasschnelli> jgarzik: yes. I'm using it since some month.
243 2015-09-18 13:06:37 <jonasschnelli> jgarzik: on my repository, it uses a slightly different environment though
244 2015-09-18 13:15:31 <jgarzik> w00t.  Travis Just Works(tm) when enabling the OSX build for picocoin (libccoin).
245 2015-09-18 13:19:41 <jonasschnelli> jgarzik: Yeah. Looks good. The guys at travis did a really good job: https://travis-ci.org/jgarzik/picocoin/jobs/81011387
246 2015-09-18 13:20:42 <jonasschnelli> jgarzik: if you wan't to get rid of the openssl deprecation warnings on osx: https://github.com/jonasschnelli/dbb-app/blob/master/src/libdbb/crypto.cpp#L17
247 2015-09-18 13:20:48 <jgarzik> jonasschnelli, now to fix all the warnings etc. :)
248 2015-09-18 13:20:48 <jonasschnelli> s/wan't/want
249 2015-09-18 13:21:22 <jgarzik> jonasschnelli, Well...  what is the core reason why BN_init() etc. is deprecated?  Ideally want to use modern solution
250 2015-09-18 13:21:43 <jonasschnelli> jgarzik: osx uses it's own crypto/ssl implementation...
251 2015-09-18 13:22:16 <jonasschnelli> since 10.7 openssl is deprecated. Most projects place a "#pragma clang diagnostic ignored "-Wdeprecated-declarations"" somewhere.
252 2015-09-18 13:23:24 <jonasschnelli> jgarzik: if you wan't to know why: http://adcdownload.apple.com/wwdc_2011/adc_on_itunes__wwdc11_sessions__pdf/212_nextgeneration_cryptographic_services.pdf
253 2015-09-18 13:23:25 <jgarzik> hm
254 2015-09-18 13:54:59 <jcorgan> was there a decision made on the weekly time?
255 2015-09-18 14:03:17 <jonasschnelli> jcorgan: i don't think so. wumpus did propose 15:00 till 16:00 UTC on Thursday, i did open a Doodle http://doodle.com/poll/cihug53sa8u4h2in#table to see if when people like to/can attend
256 2015-09-18 14:27:27 <wumpus> jonasschnelli: filled in the doodle
257 2015-09-18 14:27:49 <jonasschnelli> wumpus: Yes. I just got a mail. Lets wait for others.
258 2015-09-18 14:28:03 <wumpus> luckily we're deprecating openssl as well, so we don't need to worry about OSX's deprecation schedule unless they remove it completely before us
259 2015-09-18 14:32:42 <jgarzik> IMO Europe early evening is good
260 2015-09-18 14:33:17 <jgarzik> that should capture US-Eastern, US-West.  midnight for Asia alas.
261 2015-09-18 14:33:20 <wumpus> and even if they do it's not that bad: we compile it ourselves anyway
262 2015-09-18 14:33:25 <jgarzik> hackers stay up late :)
263 2015-09-18 14:33:48 <wumpus> right
264 2015-09-18 14:34:18 <wumpus> I don't really mind as long as it's not friday evening europe time
265 2015-09-18 14:34:37 <sipa> if we want to include asia, 11pm europe / 5pm eastcoast / 2pm westcoast works well (at blockstream)
266 2015-09-18 14:34:46 <sipa> otherwise, a few hours eaelier
267 2015-09-18 15:13:32 <morcos> 5pm eastern is a pushing it a bit for me, but i could probably make it work most of the time.  even just 30 mins earlier would be good, but i think anything 4pm or after is not great for sdaftuar
268 2015-09-18 15:14:04 <morcos> in any case we could pick the 2 best times and alternate between them at least until we see who actually shows up for these meetings
269 2015-09-18 15:15:50 <wumpus> alternating between two times could work too, but yes let not get ahead of ourselves and overdesign the schedule before we have an idea whether it will be an success or not
270 2015-09-18 15:15:53 <gmaxwell> I will make any time on thursday (in any timezone).
271 2015-09-18 15:17:57 <wumpus> if the schedule becomes more complex we can make an online calendar, something like the tor-dev one https://www.google.com/calendar/embed?src=dt92shou5q1ooe1kptubhclo4s%40group.calendar.google.com
272 2015-09-18 15:19:53 <jgarzik> wumpus, agreed.  just pick one time..
273 2015-09-18 15:20:08 <jgarzik> simple > complex
274 2015-09-18 15:29:38 <wumpus> yes, building the moxie toolchain takes quite long, makes sense to cache it
275 2015-09-18 15:30:48 <jgarzik> BTW, why does travis not build bitcoin on osx?  Nobody got around to making that work yet?
276 2015-09-18 15:31:18 <wumpus> O
277 2015-09-18 15:31:23 <wumpus> I thought it does?
278 2015-09-18 15:31:33 <wumpus> it just doesn't run the tests
279 2015-09-18 15:31:49 <jgarzik> wumpus, it builds cross-mac, when native mac + tests are available
280 2015-09-18 15:32:43 <wumpus> oh sure, but cross-mac is what we actually release, *testing* on native mac would be nice though
281 2015-09-18 15:32:53 <jgarzik> that's what I'm thinking
282 2015-09-18 15:33:04 <jgarzik> build & test on native mac
283 2015-09-18 15:34:42 <jgarzik> *bitcoin qa/rpc-tests
284 2015-09-18 15:34:44 <morcos> what do you mean cache the rpc tests?
285 2015-09-18 15:35:48 <morcos> hmm, do the tests that use the default 200 block chain not benefit from that on travis
286 2015-09-18 15:36:04 <jgarzik> pregenerate a snapshots of db + process core dump of nodes, following setup
287 2015-09-18 15:36:31 <jgarzik> then running a test involves restoring & running the core dump, much quicker
288 2015-09-18 15:36:52 <morcos> would be interesting to know how much time of each test is setup
289 2015-09-18 15:37:32 <morcos> for instance some of the longer tests that aren't run in travis, isn't bc setup is expensive but b/c there is a lot of mining and transaction creation as part of the tests themselves
290 2015-09-18 15:40:06 <jgarzik> There is always a range of which tests are useful to run.  I was just this second looking at which rpc-tests could be moved to "make check"
291 2015-09-18 15:41:08 <jgarzik> autotools provides useful "make distcheck" target which performs an out-of-srcdir build with testing and distribution release management...  however you don't want to put overly time consuming tests in 'make check' by default
292 2015-09-18 15:41:54 <jgarzik> that's more for unit tests than stress tests or timed ("do X, watch for timeout") tests.
293 2015-09-18 15:47:12 <jgarzik> As an aside, "rpc-tests" sucks as a name
294 2015-09-18 15:47:32 <jgarzik> it is really "we like this way of testing better than java, it is much more productive to write tests this way"
295 2015-09-18 15:47:41 <morcos> wumpus: 6695 was not meant to fix the rpcbind_test.py hanging problem right?
296 2015-09-18 15:47:52 <jgarzik> s/rpc-tests/tests/
297 2015-09-18 15:48:12 <wumpus> rpc-tests is really "functional tests" in constrast to "unit tests"
298 2015-09-18 15:48:27 <wumpus> the 'rpc' name makes less sense now that it includes P2P components too
299 2015-09-18 15:48:51 <wumpus> then again, there's more important things to do than thinking of a better directory name :)
300 2015-09-18 15:49:05 <wumpus> morcos: nope, haven't looked at the rpcbind test yet
301 2015-09-18 15:49:31 <morcos> ok, i'm testing other aspects of it now
302 2015-09-18 15:50:21 <wumpus> the RPC bind test has always been a kind of difficult animal, which is why it is disabled by default, it may not even have worked in your setup before the RPC server change
303 2015-09-18 15:51:11 <morcos> i fairly often run rpc-tests.sh -extended
304 2015-09-18 15:51:35 <morcos> so i think it broke recently, but not sure exactly when
305 2015-09-18 15:57:20 <wumpus> I do remember trying out the rpcbind test at least once when developing the evhttp code, but it's hanging here too
306 2015-09-18 15:58:17 <jgarzik> another todo item is a better language for describing rpc+wallet functional tests.  for a single test, you want to be able to say 'give me a standard wallet test environment with some local coins, and potentially a few 'remote actor' coins and pubkeys'
307 2015-09-18 15:58:29 <jgarzik> and then write a few lines for your specific test
308 2015-09-18 16:02:15 <wumpus> some library functions/framework to set that up would be useful, yes
309 2015-09-18 16:29:24 <wumpus> morcos: the problem in rpc bind test has to do with bitcoin-cli being unable to connect to IPv6 localhost (::1) - not sure why, yet
310 2015-09-18 16:29:59 <morcos> wumpus: i can't recreate that SIGPIPE issue in master, how did you do it?
311 2015-09-18 16:31:46 <wumpus> morcos: I haven't been able to reproduce it either. But I've seen other reports about SIGPIPE issues and libevent, so ignoring the signal is wise.
312 2015-09-18 16:32:16 <morcos> ok, i'll stop driving myself crazy then!
313 2015-09-18 16:33:38 <wumpus> you're trying on OSX?
314 2015-09-18 16:33:59 <wumpus> I don't think Linux has the issue
315 2015-09-18 16:38:23 <morcos> ahh. i didn't realize that
316 2015-09-18 16:38:26 <morcos> i was trying on linux
317 2015-09-18 16:38:30 <morcos> ok i just commented on PR
318 2015-09-18 16:38:41 <morcos> neither of the other 2 issues seem fixed for me
319 2015-09-18 16:45:18 <wumpus> morcos: what do you get when you run  grep _EVENT_NUMERIC_VERSION /usr/include/event2/event-config.h  ?
320 2015-09-18 16:46:05 <morcos> 0x02001500
321 2015-09-18 16:46:17 <wumpus> it's possible that the event_enable_debug_logging line doesn't get compiled at all, I had to delimit it with a version because older libevent don't have the functionality
322 2015-09-18 16:46:41 <wumpus> #if LIBEVENT_VERSION_NUMBER >= 0x02010100   .... right
323 2015-09-18 16:47:04 <morcos> ah, yes
324 2015-09-18 16:47:07 <morcos> hmm
325 2015-09-18 16:47:12 <wumpus> 0x02001500 doesn't have that call, so I don't think there's anything that can be done
326 2015-09-18 16:47:35 <morcos> ok
327 2015-09-18 16:47:58 <wumpus> your libevent is compiled with the debug flags on
328 2015-09-18 16:48:28 <wumpus> which is strange, does it come from the OS?
329 2015-09-18 16:49:56 <morcos> i checked on 3 different computers, it was all the same, but i guess they are all Ubuntu, i think i just did apt-get install libevent-dev
330 2015-09-18 16:50:20 <wumpus> I have ubuntu 14.04 and don't get the message, and haven't done anything specific to configure ipv6
331 2015-09-18 16:50:42 <morcos> well my normal ubuntu machines have ipv6
332 2015-09-18 16:50:51 <morcos> the only one that doesn't is the Google Compute Engine instance
333 2015-09-18 16:51:49 <morcos> this maybe seems like its not worth worrying about?
334 2015-09-18 16:54:37 <morcos> regarding the rpc timeouts..  before 6695, you could fix the problem by starting the bitcoind's in the python tests with -rpctimeout=1500 (or whatever)
335 2015-09-18 16:54:56 <morcos> doesn't that imply that somehow it was the timeout on the server end that was the problem?
336 2015-09-18 16:58:10 <wumpus> yes
337 2015-09-18 16:58:33 <wumpus> although the change in #6695 is necessary too, otherwise long queries will fail
338 2015-09-18 16:59:40 <wumpus> the python http client cannot cope with getting disconnected between requests, maybe?
339 2015-09-18 17:00:51 <morcos> that sounds like maybe what suhas was saying
340 2015-09-18 17:00:58 <wumpus> ah yes he said that first
341 2015-09-18 17:01:34 <wumpus> I'll have a look at that issue too
342 2015-09-18 17:02:17 <morcos> thanks! ok i'm afk for a couple hours
343 2015-09-18 21:51:12 <Luke-Jr> hm, another option if DST-based scheduling is desired: have a "break" during the divergent week
344 2015-09-18 21:51:27 <Luke-Jr> doesn't help those of us who use a sane non-changing time system, but..
345 2015-09-18 21:51:53 <sipa> just define the meeting times in the area with the most participants
346 2015-09-18 21:52:09 <sipa> the rest will have to adapt up to two weeks per year
347 2015-09-18 21:54:21 <gmaxwell> maybe the meeting should just be every 1008 blocks. :P
348 2015-09-18 21:54:31 <Luke-Jr> >_<
349 2015-09-18 21:57:13 <sipa> we should get google calendar to support blockchain-based times
350 2015-09-18 21:57:31 <Luke-Jr> sipa: no objections here. but get UTC at the same time?
351 2015-09-18 21:57:42 <sipa> haha
352 2015-09-18 21:58:00 <Luke-Jr> (♥ if you get tonal too!)
353 2015-09-18 22:29:32 <jcorgan> i'm concerned that having the meeting be only scheduled for an hour artificially limits what can be done.  i think we should have the meeting times automatically scheduled for 2, then, 4, then 8 hours based on block height.
354 2015-09-18 22:30:35 <sipa> every multiple of 1008 blocks, we hold a 1-hour meeting, every 2016 blocks a 2-hour meeting, every 4032 blocks a 4-hour meeting, ...?
355 2015-09-18 22:31:25 <ludbb_> should check with miners if they're ok with that
356 2015-09-18 22:32:11 <ludbb_> did you check with top companies what is their opinion on the matter?
357 2015-09-18 22:32:44 <jcorgan> i'm sure reddit will have something insightful to say about it
358 2015-09-18 22:32:52 <ludbb_> of course, and reddit
359 2015-09-18 22:49:46 <CodeShark> the length of the meeting should be HMAC(sha256(r/bitcoin html data), ripemd160(block height)) mod 12 [where the HMAC result is interpreted as a big endian integer]
360 2015-09-18 22:59:50 <jcorgan> i predict a the rise of asics that grind on r/bitcoin until the meeting length becomes 0