1 2016-08-23 01:48:44 <rusty2> Why do the gitian instructions involve setting up an lxc Ubuntu Trusty x86_64 container inside a debian VM?  Any particular reason not to try just using a Trusty VM image?
 2 2016-08-23 05:16:53 <wumpus> gitian always uses an inner VM or container
 3 2016-08-23 05:17:32 <wumpus> this can be LXC or KVM (qemu) based, but as the guide described setting up a VM (so that it can work under any OS), LXC is used, as this is easier to use nested than KVM...
 4 2016-08-23 05:18:04 <wumpus> you can run gitian on a physical machine, or on a different OS VM, but using it without the inner container is not supported
 5 2016-08-23 05:19:21 <wumpus> you don't need to follow the guide it's just one way of doing things that works reasonably reliable if you follow the exact steps
 6 2016-08-23 05:20:20 <wumpus> building will at some point likely switch to using debian at all levels (as they work on deterministic building of packages and OS images), so it wasn't that of a bad choice
 7 2016-08-23 05:20:37 <rusty> wumpus: yes.  I've had to create my own script using the gbuild stuff and the gitian-linux.yml, and running on an actual Trusty box.  Just to see if I can reproduce.  Of course, a failure doesn't tell me much except that I probably missed something...
 8 2016-08-23 05:20:39 <wumpus> debian images are also much smaller tha ubuntu
 9 2016-08-23 05:21:12 <wumpus> sure, you can try to do that, but it's easy for it to go wrong due to wrong installed packages etc
10 2016-08-23 05:21:23 <wumpus> but obviously if you do a lot of work, you can reproduce that way
11 2016-08-23 05:21:49 <rusty> Yeah... hmm "Failed to parse FAKETIME timestamp: Success"  Not good.
12 2016-08-23 05:22:43 <wumpus> before running the yml as script you need to make sure you set all the environment settings exactly as gitian does
13 2016-08-23 05:22:53 <wumpus> this is important (e.g. even LOCALE can have some influence)
14 2016-08-23 05:23:07 <rusty> wumpus: indeed.  It's not that hard, though, reading gbuild  (ruby, but hey)
15 2016-08-23 05:24:06 <wumpus> adding another mode to gitian 'I hereby assert that this box is of the VM type required for building, and up to date, please build without container' has been talked about in the past
16 2016-08-23 05:25:02 <wumpus> but never implemented - it's not something you'd suggest to most users anyhow, as it guarantees hours of masochistic executables comparisons and re-builds etc
17 2016-08-23 05:26:10 <wumpus> of course, you're welcome to do it :)
18 2016-08-23 05:26:49 <rusty> wumpus: let's see if I ETIMEDOUT  on this particular windmill first :)
19 2016-08-23 05:28:20 <wumpus> it used to be more hairy because the gitian build used libraries from the OS, this is no longer the case with the depends system, so having e.g. gzip development headers installed should no longer influence the result
20 2016-08-23 05:29:34 <wumpus> so even having *more* packages on the VM than strictly necessary could influence the result like that
21 2016-08-23 05:30:02 <wumpus> this was compounded by that the exact packages expected on the VM differ per descriptor
22 2016-08-23 05:32:08 <wumpus> heh yes you're right - working on gitian is especially frustrating as it takes so long to see the result after a change
23 2016-08-23 05:32:27 <wumpus> which is why so many prefer to not touch it as long as the current incantation works :)
24 2016-08-23 05:35:23 <wumpus> ruby isn't my favourite programming language either - functionality-wise it's too similar to python to be worth learning (personal opinion, I happened to learn python first a long time ago), but the syntax is subtly different enough to confuse me
25 2016-08-23 05:39:34 <rusty> Yes, though gitian does surprisingly little.  Most of the heavy lifting is in bitcoin/depends (a directory I hadn't noticed previously!)
26 2016-08-23 05:40:37 <wumpus> it does surprisingly little, it sets up and creates a container (complicated by the fact it can handle KVM or LXC) and launch a script in it, and make sure to copy inputs/outputs
27 2016-08-23 05:43:53 <wumpus> the container could be a really small base system with the required toolchains, there's no reason why it would need to be a complete Ubuntu image, except that this is easy to do and this doesn't rely on bitcoin devs providing any of the inputs
28 2016-08-23 06:08:13 <rusty> wumpus: ok, it built... anyone got some genuine gitian binaries so I can see where I went wrong? bitcoin-0.13.0-x86_64-linux-gnu.tar.gz and src/bitcoin-0.13.0.tar.gz
29 2016-08-23 06:09:09 <rusty> 10cae03a14d23d2a9d4173a8c1e6a41bf2c983f86d99df3119ad310ece119a2d  /home/rusty/out/bitcoin-0.13.0-x86_64-linux-gnu.tar.gz
30 2016-08-23 06:09:09 <rusty> 9c13a3cd2271c7325fa8923ae65dace75f8fce2f65609e8e8a59f4086bade54c  /home/rusty/out/src/bitcoin-0.13.0.tar.gz
31 2016-08-23 06:23:03 <wumpus> interesting that even the source tarball is different
32 2016-08-23 06:23:08 <wumpus> I'd suggest looking at the metadata
33 2016-08-23 06:23:16 <wumpus> uploading those files now
34 2016-08-23 06:25:07 <rusty> wumpus: thanks!
35 2016-08-23 06:28:00 <jonasschnelli> should we add the git-verify script (./contrib/verify-commits/verify-commits.sh) to the build-*.md docs?
36 2016-08-23 06:29:32 <wumpus> tarball: https://dev.visucore.com/bitcoin/tmp/bitcoin-0.13.0.tar.gz
37 2016-08-23 06:29:47 <wumpus> jonasschnelli: I don't think it belongs in the build docs
38 2016-08-23 06:30:05 <wumpus> it verifies the executables on the bitcoin.org server, there's no use doing that if you build from source
39 2016-08-23 06:30:14 <wumpus> documenting it *somewhere* makes sense ofc
40 2016-08-23 06:30:33 <rusty> wumpus: Oh crap, I just clicked on a link on a bitcoin IRC channel.  Time to put a sledgehammer though this machine I guess :)
41 2016-08-23 06:31:20 <wumpus> yes, no hope for that machine, nuke it from orbit
42 2016-08-23 06:31:33 <jonasschnelli> wumpus: I just though giving self-compilers a hint from the build-*.md how they can verify the source code would be good
43 2016-08-23 06:31:51 <wumpus> OHH
44 2016-08-23 06:31:54 <wumpus> verify-commits, sorry
45 2016-08-23 06:31:59 <wumpus> I misread that as something else
46 2016-08-23 06:32:25 <wumpus> yes, makes sense to at least refer to that when people build from git
47 2016-08-23 06:32:51 <jonasschnelli> Yes. Either a short text passage or a link to the verify-commits readme
48 2016-08-23 06:33:14 <wumpus> a link to the readme makes most sense; the output is not trivial to interpret
49 2016-08-23 06:33:21 <wumpus> would be nice if it just printed GOOD or BAD
50 2016-08-23 06:33:36 <wumpus> but it prints a whole list of hashes and other things, the first time I ran it I was scared :)
51 2016-08-23 06:35:05 <rusty> wumpus: OK, the only difference is in our Makefile.in.  Line 92:
52 2016-08-23 06:35:06 <wumpus> linux amd64: https://dev.visucore.com/bitcoin/tmp/bitcoin-0.13.0-i686-pc-linux-gnu.tar.gz   debug symbols (in case it helps with figuring out discrepancies): https://dev.visucore.com/bitcoin/tmp/bitcoin-0.13.0-x86_64-linux-gnu-debug.tar.gz
53 2016-08-23 06:35:08 <rusty> -	build-aux/config.guess build-aux/config.sub \
54 2016-08-23 06:35:09 <rusty> +	build-aux/config.guess build-aux/config.sub build-aux/depcomp \
55 2016-08-23 06:35:19 <rusty> Second one is mine.  No idea why I have that file.
56 2016-08-23 06:35:49 <wumpus> probably you have a package installed that is not normally on a gitian builder
57 2016-08-23 06:36:50 <wumpus> or a different version of automake/autoconf, though that'd be strange if you have ubuntu 14.04 and updated it
58 2016-08-23 06:38:20 <rusty> wumpus: yes, it's an automake thing.  Hmm, wonder what gitian does; I have 1:1.14.1-2ub
59 2016-08-23 06:39:30 <wumpus> would be useful to do whatever is dumepd to the .assert (e.g. the package listing) and compare with other people's asserts
60 2016-08-23 06:53:42 <rusty> wumpus: that's the 32-bit binary, isn't it?  I only did the 64...
61 2016-08-23 06:54:31 <wumpus> oh did I uplod the wrong file?
62 2016-08-23 06:54:38 <wumpus> sorry, it's fairly early morning here
63 2016-08-23 06:54:46 <rusty> wumpus: I think so... dbug one is right, but that's a 10 minute downlaod :)
64 2016-08-23 06:55:50 <rusty> wumpus: thanks :)
65 2016-08-23 06:56:54 <wumpus> https://dev.visucore.com/bitcoin/tmp/bitcoin-0.13.0-x86_64-linux-gnu.tar.gz
66 2016-08-23 07:10:35 <rusty> Hmm, buildids are different of course, due to that autotools weirdness.  But binaries seem mildly different, too; looks like codegen randomness.
67 2016-08-23 07:13:29 <rusty> I'll keep pursuing this in my Copious Free Time...
68 2016-08-23 07:14:09 <wumpus> anyhow, now you maybe understand why a deterministically set up VM is used :)
69 2016-08-23 07:14:16 <wumpus> eh, container
70 2016-08-23 07:14:41 <rusty> wumpus: yeah, but I'm just paranoid enough to want to do this from scratch. :)