1 2015-03-09 00:01:10 <midnightmagic> are the block writes potentially out-of-order also, or are they only sequential writes (with the blocks themselves the only out-of-order things)? Are block writes single-threaded?
  2 2015-03-09 00:01:57 <gmaxwell> Yes. Yes.
  3 2015-03-09 00:02:04 <gmaxwell> Why?
  4 2015-03-09 00:08:25 <midnightmagic> I was thinking of failing write coalescing or underlying device write coalescing being additional pressure in the event that writes might have been happening simultaneously or via delayed-flush, or, if someone had been enterprising and committed them, iovec scatter writes being messed with at the device level/storage card level. I've seen overlapping db pages commited in a paged database, but in that case there were significantly
  5 2015-03-09 00:08:31 <midnightmagic> more complex arrangements of data than serialized block writes.
  6 2015-03-09 00:11:08 <gmaxwell> yea, no nothing that complex going on.
  7 2015-03-09 00:13:45 <midnightmagic> But.. that said, I have in fact witnessed heavy-concurrency highly special-purposes databases become corrupt in similar data-overlap fashion with no obvious explanation, with proximal OS-level events.
  8 2015-03-09 00:14:08 <midnightmagic> it made debugging after-the-fact basically impossible.
  9 2015-03-09 00:16:02 <midnightmagic> obviously a locking failure isn't an issue if the writer is single-threaded
 10 2015-03-09 03:17:08 <Meisterbrau> hello i am synching QT and im 2 days out....and i get a Cannot Read Block error box message
 11 2015-03-09 03:17:13 <Meisterbrau> first time....
 12 2015-03-09 03:17:26 <Meisterbrau> where would i figure out whats going on and how to get the fix/
 13 2015-03-09 03:17:54 <Meisterbrau> dont know if this is a ~reindex thing or what
 14 2015-03-09 03:18:14 <Meisterbrau> qt just shutting down, and didn't do that last night
 15 2015-03-09 03:31:19 <gmaxwell> coryfields: re #5864 say blocks are at disk offsets   10,1,2,3,4,5,6 won't the code there end up with you having an nsize of 31, which is far too much?
 16 2015-03-09 03:32:54 <gmaxwell> oh duh, no because it will traverse in order, nevermind me.
 17 2015-03-09 03:35:49 <phantomcircuit> Meisterbrau, reindex
 18 2015-03-09 03:36:31 <Meisterbrau> debug, console, then is the command ~reindex
 19 2015-03-09 03:36:35 <Meisterbrau> just like that?
 20 2015-03-09 03:40:38 <phantomcircuit> Meisterbrau, have you restarted?
 21 2015-03-09 03:41:05 <Meisterbrau> well i shut it down and am restarting the QT
 22 2015-03-09 03:41:18 <Meisterbrau> it keeps shutting down so dont think it will restart....
 23 2015-03-09 03:41:40 <Meisterbrau> may have to do it from the terminal here on linux and not exactly sure how
 24 2015-03-09 03:41:47 <Meisterbrau> though know it can be done
 25 2015-03-09 03:45:12 <phantomcircuit> Meisterbrau, the easiest way is to edit a shortcut to bitcoin-qt to include -reindex=1 after the exe path
 26 2015-03-09 03:45:43 <phantomcircuit> iirc it's supported to prompt for reindex if it's needed but maybe that's broken in this edge case
 27 2015-03-09 03:46:38 <Meisterbrau> id need instructions cause dont think id figure that out
 28 2015-03-09 03:46:45 <Meisterbrau> but following! kinda of
 29 2015-03-09 03:46:51 <Meisterbrau> thx
 30 2015-03-09 03:47:19 <Meisterbrau> im confused if it wont open cause debug and console is how id do it but dont know how to if i cant get in there cause it closes!
 31 2015-03-09 03:50:26 <phantomcircuit> Meisterbrau, unfortunately i dont think you can issue a reindex from the debug terminal
 32 2015-03-09 03:50:50 <Meisterbrau> says its loading addresses but i think its going to shut down i just saw the error cant read blocks
 33 2015-03-09 03:50:53 <Meisterbrau> or block
 34 2015-03-09 03:53:51 <Meisterbrau> ok it keeps shutting down have to do this  from the terminal in linux
 35 2015-03-09 03:53:56 <Meisterbrau> would you know the commands/
 36 2015-03-09 03:54:04 <Meisterbrau> im thining CD something....
 37 2015-03-09 03:54:40 <Meisterbrau> CD /.bitcoin
 38 2015-03-09 03:54:43 <Meisterbrau> something....
 39 2015-03-09 03:55:59 <Meisterbrau> or super cow powers
 40 2015-03-09 03:57:01 <Meisterbrau> i need help if anybody can come up with the command lines...i made it in sudo apt and did my password and thats about as far as illl get
 41 2015-03-09 04:22:28 <JustAnotherVogon> Meisterbrau, can you add the -reindex=1 command when you start QT from the terminal?
 42 2015-03-09 04:23:16 <Meisterbrau> i just luanch from the launch i dont go through terminal
 43 2015-03-09 04:23:28 <Meisterbrau> id need instructions
 44 2015-03-09 04:24:02 <Meisterbrau> i just dont know where to start
 45 2015-03-09 04:24:15 <Meisterbrau> dont know if ls -la
 46 2015-03-09 04:24:17 <Meisterbrau> or something
 47 2015-03-09 04:24:57 <Meisterbrau> my question is why is it not being read?
 48 2015-03-09 04:25:00 <Meisterbrau> what broke?
 49 2015-03-09 04:26:32 <JustAnotherVogon> Meisterbrau, It's my thought that your index for the blocks has been corrupted.
 50 2015-03-09 04:28:06 <Meisterbrau> how
 51 2015-03-09 04:28:49 <Meisterbrau> rather what do i do to fix it!
 52 2015-03-09 04:28:53 <Meisterbrau> got to be a workaround!
 53 2015-03-09 04:28:58 <Meisterbrau> get over that block!
 54 2015-03-09 04:32:32 <JustAnotherVogon> Meisterbrau, without knowing the entire situation try running "bitcoind --daemon -reindex" from the terminal. I will take some time ( it has to rebuild the blockchain from scratch but using the already downloaded blocks )
 55 2015-03-09 04:32:35 <gmaxwell> phantomcircuit: you need to ask him for his debug log.
 56 2015-03-09 04:32:50 <gmaxwell> because otherwise you cannot tell if he's sucessfully passing reindex to the process.
 57 2015-03-09 04:32:59 <Meisterbrau> 
 58 2015-03-09 04:33:08 <Meisterbrau> i just tried that permission denied
 59 2015-03-09 04:33:47 <Meisterbrau> yah im stuck
 60 2015-03-09 04:34:08 <gmaxwell> phantomcircuit: see?
 61 2015-03-09 04:34:39 <Meisterbrau> no hurry just figured this out today
 62 2015-03-09 04:34:45 <gmaxwell> Meisterbrau: can we move this conversation into #bitcoin? I'm over there
 63 2015-03-09 04:35:08 <Meisterbrau> ok
 64 2015-03-09 04:36:14 <Meisterbrau> ok im the only one in there now
 65 2015-03-09 04:36:40 <gmaxwell> uh  #bitcoin has >1000 people in it, check your spelling. :)
 66 2015-03-09 04:37:14 <Meisterbrau> ok
 67 2015-03-09 04:37:17 <Meisterbrau> yah i had ?
 68 2015-03-09 04:37:21 <gmaxwell> oh sorry.
 69 2015-03-09 08:48:02 <dansmith_btc> Hi, is my understanding correct that with a recent lift on isStandard it is now possible to have fancier p2sh multisig redeem scripts. Say, I have 3 sets (A,B,C) of 3 pubkeys (9 pubkeys total). Can I redeem if (2-of-A AND 2-of-B) OR (2-of-C) ?
 70 2015-03-09 08:57:58 <wumpus> dansmith_btc: yes, such expressions should be possible
 71 2015-03-09 09:00:43 <dansmith_btc> ah, ok, it's good to know
 72 2015-03-09 09:04:06 <wumpus> dansmith_btc: any script with <15 sigops will be accepted, see https://github.com/bitcoin/bitcoin/commit/7f3b4e95695d50a4970e6eb91faa956ab276f161
 73 2015-03-09 09:53:14 <fanquake> wumpus What’s the plan for a 0.10.1 release?
 74 2015-03-09 09:53:42 <fanquake> If there is any that is.
 75 2015-03-09 09:53:52 <sipa> soon.
 76 2015-03-09 09:55:29 <wumpus> fanquake: it makes sense to do one, but no immediate plans
 77 2015-03-09 09:57:40 <fanquake> Yep. A few annoying bugs fixes. As well as #5850 to be dealt with
 78 2015-03-09 10:02:34 <wumpus> some minor things, but nothing that necessitates a minor releae so quickly after the major release
 79 2015-03-09 13:12:09 <fanquake> ;;blocks
 80 2015-03-09 13:12:10 <gribble> 346861
 81 2015-03-09 13:54:35 <jtimon> mhmm, bitcoin-tx.cpp and script/sign.cpp use STANDARD_SCRIPT_VERIFY_FLAGS...
 82 2015-03-09 13:55:28 <jtimon> not good if we want to encapsulate STANDARD_SCRIPT_VERIFY_FLAGS and still leave policy.o in the server group of modules
 83 2015-03-09 13:56:03 <sipa> i think we will need to get rid of those "standard" flags anyway
 84 2015-03-09 13:56:47 <sipa> part of it is just policy, part depends on the current consensus rules
 85 2015-03-09 13:57:04 <sipa> but for signing the first is likely not required; it's just a sanity check
 86 2015-03-09 13:58:36 <jtimon> yep, getting rid of them (well just turning a couple of them into non-exposed attributes of CStandardPolicy) is what I was trying to do when I realized this
 87 2015-03-09 14:04:55 <wumpus> cfields: do we need to change all BOOST_AUTO_TEST_SUITE to BOOST_FIXTURE_TEST_SUITE? I just had to convert wallet and main to avoid them running with uninitialized parameters
 88 2015-03-09 14:08:23 <wumpus> seems the combination of #5852 and #5855 is causing problems with the tests
 89 2015-03-09 14:09:10 <sipa> wumpus: that's exactly the sort of problem 5855 was supposed to bring to light :)
 90 2015-03-09 14:09:31 <jtimon> sipa: what you think we should do to decouple bitcoin-tx from policy? defining a constant there or in script/interpreter like ALL_SCRIPT_VERIFY_FLAGS or SIGNING_SCRIPT_VERIFY_FLAGS (wouldn't be simpler to not move MANDATORY_SCRIPT_VERIFY_FLAGS to policy.o and put it in consensus.o?)
 91 2015-03-09 14:10:12 <wumpus> sipa: but does this mean all BOOST_AUTO_TEST_SUITE need to be replaced with BOOST_FIXTURE_TEST_SUITE, to make sure the initialization is run?
 92 2015-03-09 14:11:14 <wumpus> right now, there are some *_tests.cpp that use the former,and some that use the latter, all in all it is passing but there's no saying when it will break again w/ a different seed
 93 2015-03-09 14:14:31 <sipa> wumpus: well those that don't access main or wallet shouldn't need it
 94 2015-03-09 14:14:45 <wumpus> ... or chainparams
 95 2015-03-09 14:14:51 <sipa> Ah
 96 2015-03-09 14:15:32 <sipa> i guess we can have different fixtures in test_bitcoin.cpp, for different chainparams, and for different sets of initialization (just main, main and wallet, none, ...)
 97 2015-03-09 14:15:33 <wumpus> though if that is the only thing could also put SelectParams(CBaseChainParams::MAIN); at the top
 98 2015-03-09 14:15:43 <wumpus> (or SelectParams(CBaseChainParams::UNITTEST); )
 99 2015-03-09 14:15:55 <sipa> if there is no global state, just selectparams in the body should suffice, indeed
100 2015-03-09 14:16:14 <wumpus> indeed, the only global state there is which chain is selected
101 2015-03-09 14:19:16 <jtimon> I was about to say that, since I had similar problems that were solved with SelectPolicy("standard") at the top, or using Policy("standard") explicitly instead of just Policy() was sufficient in many cases...
102 2015-03-09 14:20:48 <jtimon> /test/test_bitcoin.cpp, /test/script_P2SH_tests.cpp and qt/test/test_main.cpp were the only 3 places where I need to call SelectPolicy()
103 2015-03-09 14:23:32 <jtimon> mhmm, we already have a SelectParams(CBaseChainParams::UNITTEST); in TestingSetup::TestingSetup()
104 2015-03-09 14:23:52 <sipa> we should get rid of that UnitTest chainparams
105 2015-03-09 14:27:56 <jtimon> and use main there?
106 2015-03-09 14:28:10 <sipa> i think the tests should use main as much as possible
107 2015-03-09 14:28:27 <sipa> only those where mining is involved, testnet or regtest (or weaker) may be needed
108 2015-03-09 14:28:51 <jtimon> unittest is basically main changing some of the params for specific tests
109 2015-03-09 14:29:07 <sipa> yes, but doing it in a mutable way
110 2015-03-09 14:29:17 <sipa> which is weird... the actual params should never change during their usage
111 2015-03-09 14:29:25 <sipa> if you need different params, you should switch to different params
112 2015-03-09 14:29:38 <sipa> not modify the existing ones, as that can have weird inter-test side effects
113 2015-03-09 14:29:47 <jtimon> yes, I agree, but maybe the solution to the non-mutability is precisely keeping that mode
114 2015-03-09 14:30:41 <sipa> ?
115 2015-03-09 14:31:57 <jtimon> I don't know the cases in depth, sergio lerner did it, whatever he was doing he could probably have done it without the modifiable params, but maybe he really needed a new mode
116 2015-03-09 14:32:16 <jtimon> s/mode/chainparam
117 2015-03-09 14:32:31 <sipa> no objection to a separate chainparams if it's needed for unit tests
118 2015-03-09 14:32:37 <sipa> the ugly part is just the mutability
119 2015-03-09 14:32:45 <jtimon> agreed
120 2015-03-09 14:32:57 <wumpus> sipa: yes, Sergio introduced that, I'm also not convinced it's necesary
121 2015-03-09 14:33:16 <wumpus> are there tests where we need to tweak a certain parameter?
122 2015-03-09 14:33:33 <sipa> afaik, the ones that needed that were never merged
123 2015-03-09 14:33:41 <wumpus> 
ACTION wishes we could get rid of global chainparams completely, and just pass around whatever is needed

124 2015-03-09 14:33:48 <jgarzik> wumpus, +100
125 2015-03-09 14:33:48 <sipa> wumpus: ack ack ack
126 2015-03-09 14:34:14 <sipa> that however also requires encapsulating most of main's state into objects
127 2015-03-09 14:34:20 <jtimon> CModifiableParams has 7 setters
128 2015-03-09 14:34:24 <wumpus> and there have been recent developments in that direction already, making code independent of chainparams
129 2015-03-09 14:34:50 <jtimon> chainparams could be loaded from a file
130 2015-03-09 14:34:51 <jgarzik> state encapsulation is useful in any case
131 2015-03-09 14:34:57 <wumpus> (for example the packet id bytes in net)
132 2015-03-09 14:35:09 <jgarzik> in some projects I went to extremes and created a global state class ;p
133 2015-03-09 14:35:20 <sipa> jgarzik: absolutely, we need to do that no matter what
134 2015-03-09 14:35:21 <jgarzik> as an interim step
135 2015-03-09 14:35:23 <wumpus> the god objeect
136 2015-03-09 14:35:42 <sipa> CAxiom
137 2015-03-09 14:35:46 <jgarzik> heh
138 2015-03-09 14:35:58 <wumpus> hehe
139 2015-03-09 14:36:17 <jgarzik> in practice you need to start out with N classes.  Otherwise the god class becomes more a problem than solution.
140 2015-03-09 14:38:04 <wumpus> in any case, I do like the chainparams idea, having the chain parameters configurable as a structure, but it shouldn't be a global but something that is passed to whatever needs it
141 2015-03-09 14:43:01 <jtimon> Yep, we're probably abusing the Params() factory function to the pselectedParams singleton, passing it down more often would be better, in the case of the tests, I would suggest to use a more generic factory like Params(type) or even Params(string), if they're passed around enough, no SelectParams() call should be required
142 2015-03-09 14:43:57 <jtimon> as you should be able to explicitly select Params("main") or Params("unittest") from the tests
143 2015-03-09 14:49:02 <wumpus> jtimon: python-bitcoinlib did a similar thing
144 2015-03-09 14:51:48 <wumpus> just checked: there is nothing using the UNITTEST functionality, so if we don't intend to add any tests that use modifyable parameters we can revert it easily
145 2015-03-09 14:52:28 <jtimon> then the tests can use main instead
146 2015-03-09 14:52:55 <wumpus> yes
147 2015-03-09 14:56:29 <sipa> wumpus: i guess it's easiest to have a simple fixtute that just sets chainparams, and call that from the texts that currently do not have a fixture
148 2015-03-09 14:57:07 <jtimon> mhmm cfields this may affect our previous conversation about Consensus::Params::fPowSkipProofOfWorkCheck (since it may disappear)
149 2015-03-09 14:57:24 <sipa> indeed, it can just go away for now
150 2015-03-09 14:57:39 <sipa> or leave it, and add a test that uses it (even better!)
151 2015-03-09 14:57:56 <wumpus> sipa: sounds good to me; some of them already handle setting the chain, though, such as the base58 tests (as they test multiple chains)
152 2015-03-09 14:58:22 <jtimon> probably better to just leave that parameter out of consensus if possible
153 2015-03-09 14:58:56 <wumpus> is there any reason for it
154 2015-03-09 14:59:22 <wumpus> REGTEST already hardly has a PoW
155 2015-03-09 14:59:35 <jtimon> I thought it was unittestparams
156 2015-03-09 14:59:53 <wumpus> so we just decided that unittestparams is going away
157 2015-03-09 15:00:15 <jtimon> no remaining reason for it then
158 2015-03-09 15:23:27 <sdaftuar> sipa: if you're around, there's one more thing i wanted to flag about the autoprune pull (5863), relating to duplicate block processing.
159 2015-03-09 15:24:22 <sipa> ok
160 2015-03-09 15:24:27 <sipa> i'm here
161 2015-03-09 15:24:43 <sdaftuar> so there was already a check in AcceptBlock to return early if we already had a block
162 2015-03-09 15:24:52 <sdaftuar> we modified that to return early if we already have it, or it's in the active chain
163 2015-03-09 15:25:08 <sdaftuar> but i don't see an easy way to avoid re-processing non-main-chain blocks, if someone were to serve them to us
164 2015-03-09 15:25:57 <sipa> can't you just check have_data, and bail out based on that?
165 2015-03-09 15:26:31 <sipa> or you mean avoid wasting time on blocks sent to us, outside of the chain we expect?
166 2015-03-09 15:26:32 <sdaftuar> well if we've already processed and pruned it, then we could process it again
167 2015-03-09 15:26:54 <sdaftuar> right, i'm worried about the case of someone wasting time having us re-process old blocks over and over
168 2015-03-09 15:27:30 <sipa> i think we could at least add a check that the block is an ancestor of the best header we have in common with the peer it is coming from
169 2015-03-09 15:27:43 <sipa> as those are the only ones we should ever request
170 2015-03-09 15:27:58 <sipa> but that doesn't solve your issue
171 2015-03-09 15:28:26 <sdaftuar> well i think we could more broadly not process blocks we didn't request since we track that already, right?
172 2015-03-09 15:28:35 <sdaftuar> but that seems like a more extreme measure
173 2015-03-09 15:28:36 <sipa> right, yes
174 2015-03-09 15:29:01 <sipa> i *think* that would be fine
175 2015-03-09 15:29:19 <sdaftuar> ignoring blocks we don't request?
176 2015-03-09 15:29:27 <sipa> yes
177 2015-03-09 15:30:09 <sipa> do we still push out newly mined blocks immediately without preceeding inv?
178 2015-03-09 15:30:22 <sipa> i forgot
179 2015-03-09 15:30:23 <sdaftuar> i don't think so
180 2015-03-09 15:31:49 <sdaftuar> well, bluematt's relay node will just dump blocks, so this would break that
181 2015-03-09 15:31:58 <sdaftuar> (i think)
182 2015-03-09 15:33:42 <sipa> i think it is already broken in some cases with headersfirst
183 2015-03-09 15:34:15 <sdaftuar> could we restrict that check to only processing new blocks if we hadn't requested them, but not reprocess an old block unless it was requested?
184 2015-03-09 15:34:15 <sipa> we should make it keep track of the last few headers and announce them together with the blocks
185 2015-03-09 15:34:56 <sdaftuar> old == in our block index
186 2015-03-09 15:35:44 <sipa> well you can always first just process the block's header, if you don't already have it, which is cheap
187 2015-03-09 15:36:57 <sipa> then, check whether it's (nTx == 0 && nChainWork > chainActive.nChainWork) || requested?
188 2015-03-09 15:37:57 <sdaftuar> nChainWork is only the work up to that block right?
189 2015-03-09 15:38:25 <sipa> yes
190 2015-03-09 15:38:36 <sipa> cummulative work of a block including all its ancestors to genesis
191 2015-03-09 15:39:09 <sipa> you're coming up with more edge cases than i had considered :)
192 2015-03-09 15:39:23 <sdaftuar> hah, great :)
193 2015-03-09 15:44:44 <sdaftuar> ok i think your suggestion makes sense to me -- i'll try to add that to the pull.  thanks!
194 2015-03-09 15:53:20 <sipa> thanks a lot!
195 2015-03-09 16:09:19 <jonasschnelli> damit! how do i calculate the age of a block? GetTimeMicros() - block.nTime seems not to work? int32/64 issue?
196 2015-03-09 16:10:24 <sipa> nTime is in seconds, nkt microseconds
197 2015-03-09 16:11:39 <jonasschnelli> sipa, thanks. Yes. I should have taken GetTime()... :)
198 2015-03-09 16:12:24 <sipa> also be aware that the accuracy of nTime is very low
199 2015-03-09 16:13:20 <jonasschnelli> sipa, okay. It's just for a live view of blocks and txs... accuracy doesn't matter that much.
200 2015-03-09 18:36:38 <cfields> wumpus: looks like you got your test issues worked out, but +1 on getting rid of the chainparams globals
201 2015-03-09 18:37:09 <cfields> and yes, some of the refactors in the works are working towards that
202 2015-03-09 18:37:55 <cfields> at least encapsulating the behavior/usage as a first step, then actually nuking the globals can be done much easier
203 2015-03-09 20:31:37 <bliljerk101> Anyone know if this is a standardized format and what the parts represent? f9608639-25ee-4388-8408-fc4a93bc50fe
204 2015-03-09 20:32:00 <sipa> looks like a uuid
205 2015-03-09 20:33:29 <bliljerk101> sipa thanks