1 2017-09-29 10:56:51 <lxer> ZMQ zmqpubrawtx seems to send 3 messages for each transaction, the second having the actual data, so what are the other 2 for?
 2 2017-09-29 18:53:18 <esotericnonsense> lxer: what do they look like? :)
 3 2017-09-29 19:21:44 <lxer> esotericnonsense: ok, got it. first one translates to 'rawtx' . the 3rd one is a hex value like 9a510500  . what is it for?
 4 2017-09-29 19:26:09 <esotericnonsense> lxer: so the first one is the published zeromq topic (in multipart messages it's always that, that's how zmq pubsub works)
 5 2017-09-29 19:28:24 <esotericnonsense> the last one is a sequence number (little endian according to the code). looks quite high, not sure if it starts from 0, how long has bitcoind been running? :)
 6 2017-09-29 19:31:22 <lxer> are you sure that is how pubsub works? I already set a topic filter in socket.setsockopt(zmq.SUBSCRIBE, "rawtx") , shouldnt that filter it out already?  I've used ZMQ for other projects, but never the pubsub, so perhaps that is different from what I'm used to.. ?
 7 2017-09-29 19:31:43 <esotericnonsense> lxer: yes i've worked with pubsub rather a lot in the past :)
 8 2017-09-29 19:32:08 <esotericnonsense> lxer: it's not echoing your subscription back to you, rather it gives you the topic published
 9 2017-09-29 19:32:29 <lxer> I usually get just 1 message.
10 2017-09-29 19:32:43 <esotericnonsense> for example if the publisher sends out on "CHOCOLATEBAR" and you subscribe to "CHOC" the messages will be ["CHOCOLATEBAR", *parts]
11 2017-09-29 19:36:39 <cluelessperson> how large of a database do you think a bitcoin postgresql database would be?
12 2017-09-29 19:36:50 <esotericnonsense> cluelessperson: it really depends what you're storing
13 2017-09-29 19:37:45 <cluelessperson> esotericnonsense: I'm thinking tables for,   blocks, transactions, addresses
14 2017-09-29 19:38:24 <cluelessperson> esotericnonsense: I'm thinking about creating large flow visualizations
15 2017-09-29 19:39:12 <lxer> there have bee some projects about that. even making nice 3D images of transactions
16 2017-09-29 19:39:34 <esotericnonsense> cluelessperson: what i mean is the format of the things that you store. for example if you're indexing by txid anyway you can drop signature data.
17 2017-09-29 19:40:19 <esotericnonsense> if you store some big verbose transaction in a nonraw format then it will be much larger, etc.
18 2017-09-29 19:40:43 <cluelessperson> esotericnonsense: sorry, why can I drop signature data?
19 2017-09-29 19:40:52 <esotericnonsense> cluelessperson: well because you don't care for that specific purpose right.
20 2017-09-29 19:41:05 <cluelessperson> esotericnonsense: I want to be able to more easily/quickly parse through all the trnasactions
21 2017-09-29 19:41:30 <cluelessperson> esotericnonsense: render a data visualization
22 2017-09-29 19:41:39 <cluelessperson> but we're talking like 300million
23 2017-09-29 19:42:49 <lxer> esotericnonsense: but then why is it sending the zmq topic again as a message? I never needed that when using some request/reply pattern , and the 'topic' still works as usual.
24 2017-09-29 19:44:34 <esotericnonsense> lxer: it's not 'again'
25 2017-09-29 19:44:39 <esotericnonsense> it's just how pubsub works
26 2017-09-29 19:46:56 <esotericnonsense> often you'll encode data in it because the topic needs to do that anyway
27 2017-09-29 19:48:17 <esotericnonsense> for example if you have some stock ticker app or whatever and you have TICKER:AAPL, TICKER:GOOG, etc, if someone subscribes to TICKER, they get everything and can distinguish based on the topic rather than having to also include that in the payload
28 2017-09-29 19:49:14 <esotericnonsense> i'm not that up on the internal workings of ZeroMQ (it's been a while since I went digging there) but I suspect it can't even work through proxies if the topic weren't sent
29 2017-09-29 19:49:48 <esotericnonsense> (because the proxy wouldn't know where to route the messages)
30 2017-09-29 19:50:39 <lxer> I remember it differently. the topic filter is already set when creating the socket:  socket.setsockopt(zmq.SUBSCRIBE, topicfilter) , and the messages from then on are already filtered by topic
31 2017-09-29 19:51:01 <esotericnonsense> the filters can be set or unset at any time
32 2017-09-29 19:51:31 <lxer> that is not the point
33 2017-09-29 19:51:42 <esotericnonsense> say you have a bunch of publishers, with a hub in the middle, then some subscribers
34 2017-09-29 19:52:12 <esotericnonsense> the subscribers will sub to the hub, the hub will subscribe to the publishers, and when the hub receives a message, it uses the topic to route to the subscribers
35 2017-09-29 19:52:17 <esotericnonsense> without having that topic, it doesn't work
36 2017-09-29 19:52:48 <lxer> a 'hub'?  I always connect directly
37 2017-09-29 19:52:50 <esotericnonsense> (I can see how in the simple pub -> sub model with no middleman it doesn't seem necessary)
38 2017-09-29 19:52:59 <esotericnonsense> yeah, but zmq is designed to scale in that way
39 2017-09-29 19:53:38 <lxer> no sorry, I think this design is not right.
40 2017-09-29 19:53:52 <esotericnonsense> the point is that you could have many publishers on different boxes whilst a subscriber doesn't have to do service discovery, it just needs the hub
41 2017-09-29 19:54:07 <lxer> there is no need to create so hub to have a second layer of filtering
42 2017-09-29 19:54:17 <esotericnonsense> it's not for filtering purposes
43 2017-09-29 19:54:32 <esotericnonsense> imagine your publishers are doing computationally intensive tasks
44 2017-09-29 19:55:24 <lxer> no, if you want to scale through proxies, then simply make a new pub/sub on the proxies, with a topic filter
45 2017-09-29 19:55:29 <esotericnonsense> you can't put them all on one box, so you route everything through a hub which just serves as a zmq proxy (this is actually a builtin, http://api.zeromq.org/3-2:zmq-proxy)
46 2017-09-29 19:56:30 <esotericnonsense> imagine another case without proxy if you will
47 2017-09-29 19:56:45 <esotericnonsense> (in fact i'm being silly for not having mentioned this earlier, sorry heh)
48 2017-09-29 19:57:07 <esotericnonsense> if you set multiple subscription topics on the same socket, how would you distinguish them on receipt without having the topic?
49 2017-09-29 19:57:23 <esotericnonsense> you'd have to encode that in the payload somehow
50 2017-09-29 19:57:34 <esotericnonsense> but the topic is already there, so just use that :)
51 2017-09-29 19:59:22 <esotericnonsense> perhaps my defence of pubsub mechanics is not to your liking, but, it is the way it is :P
52 2017-09-29 20:00:29 <esotericnonsense> there are other weirdnesses in pubsub, like on the publisher end if you use verbose sockets you can catch the subscription messages and not bother to send if no-one is subscribed
53 2017-09-29 20:03:05 <lxer> I'm using a pub/sub for some other project, where I just receive messages about a topic, send by the publisher (who set the topic also). always just 1 message, and not 3.  and the topic-filter works as expected
54 2017-09-29 20:03:13 <cluelessperson> esotericnonsense: what size are we really talknb about here?
55 2017-09-29 20:04:43 <esotericnonsense> lxer: well you can send messages as topics if you want too
56 2017-09-29 20:05:10 <esotericnonsense> i guess i'm calling it topic because i've never used it in that way, then the topic is just the payload
57 2017-09-29 20:05:10 <lxer> no, that is not what I mean.
58 2017-09-29 20:06:21 <esotericnonsense> i think you can just publish without an envelope and make the first part of the message what you subscribe to (or subscribe to "")
59 2017-09-29 20:06:22 <lxer> I mean that it would just work the same if you left out message 1 and 3 , and only send the actual content (2nd message)
60 2017-09-29 20:07:33 <esotericnonsense> lxer: i think i'd understand this better with a code example
61 2017-09-29 20:08:22 <esotericnonsense> in python you could send non-multipart "something whatever" as a string, and subscribe to "something", it's the same just without the framing of multipart messages
62 2017-09-29 20:08:47 <lxer> ok, it is friday night over here and I want to do something else. lets talk about it again later
63 2017-09-29 20:08:57 <esotericnonsense> if you just send a blob of data with no topic then the subscriber can't filter messages
64 2017-09-29 20:09:30 <esotericnonsense> it would work in the specific case of bitcoind if you only published txids or only block hashes, but if both were enabled then you wouldn't know which was which
65 2017-09-29 20:09:41 <lxer> yes, the subscriber does filter, that is handled on another level, afaik.
66 2017-09-29 20:10:08 <esotericnonsense> argh.
67 2017-09-29 20:10:18 <esotericnonsense> the way you publish a message is to send a blob of data.
68 2017-09-29 20:10:45 <lxer> + a topic
69 2017-09-29 20:10:52 <esotericnonsense> you can send without a topic
70 2017-09-29 20:11:29 <esotericnonsense> (it's just that if you do that there is no topic! :P)
71 2017-09-29 20:12:00 <lxer> right, but the publisher (bitcoind) sends with topic 'rawtx'
72 2017-09-29 20:12:34 <esotericnonsense> yes, and it sends in the multipart manner
73 2017-09-29 20:12:59 <esotericnonsense> it could also send in one part, as "rawtx hash" instead of ["rawtx", "hash", "sequence"]
74 2017-09-29 20:13:07 <esotericnonsense> err, "rawtx hash sequence"
75 2017-09-29 20:13:23 <esotericnonsense> the sequence number is there so that you can detect dropped messages btw
76 2017-09-29 20:13:23 <lxer> then, if I create a subscriber  socket.setsockopt(zmq.SUBSCRIBE, "rawtx") ,  I only get 'rawtx' messages.  So, no need to send a message with 'rawtx' to announce it.
77 2017-09-29 20:13:56 <esotericnonsense> ????
78 2017-09-29 20:14:23 <esotericnonsense> the publisher has to announce the topic it's sending on, that is what sending on a topic is?
79 2017-09-29 20:14:42 <esotericnonsense> the first part of a multipart message, or the entire message if not multipart, is the topic
80 2017-09-29 20:16:10 <lxer> 'topic' and 'message' are different things
81 2017-09-29 20:16:28 <lxer> anyway.. i'm wanna get drunk and party :P
82 2017-09-29 20:17:35 <lxer> so, if I have time I'll make some example code.  is python ok?
83 2017-09-29 20:17:46 <esotericnonsense> i have an example here, just a sec
84 2017-09-29 20:17:53 <esotericnonsense> it is fine, yes
85 2017-09-29 20:19:46 <esotericnonsense> eh, can't get it to work in asyncio with one process.
86 2017-09-29 20:20:15 <esotericnonsense> basically, in python, make a subscriber to "TOPIC TOPIC". make a publisher that just does sock.send("TOPIC TOPIC PAYLOAD")
87 2017-09-29 20:20:23 <esotericnonsense> that's the non multipart way
88 2017-09-29 20:21:11 <esotericnonsense> alternatively you sock.send(["TOPIC TOPIC", "PAYLOAD"]), the client will get it by x = recv_multipart, x[0] will be the topic, x[1] the payload
89 2017-09-29 20:21:21 <esotericnonsense> err sock.send_multipart*
90 2017-09-29 20:21:34 <esotericnonsense> in both cases you're sending the topic, not sure what you're getting at :/
91 2017-09-29 20:32:31 <lxer> ok.. I always used  recv()  and not recv_multipart . I think I get what you were saying now.
92 2017-09-29 20:33:08 <lxer> anyway.. beertiem.
93 2017-09-29 20:33:31 <esotericnonsense> got it working. https://0bin.net/paste/eUGjYXpOBpK05qao#qnrNNT3fXZS6KG5eu4dt85xvi3FutMXHqQqbnI531AK
94 2017-09-29 20:33:41 <esotericnonsense> have fun :)
95 2017-09-29 20:35:47 <esotericnonsense> right. i see now. after playing with it.
96 2017-09-29 20:36:22 <esotericnonsense> if you send_multipart(['topic', 'payload']) and then recv, you will get two individual receive events of 'topic' and 'payload'.
97 2017-09-29 20:36:53 <esotericnonsense> now I will get food. :P
98 2017-09-29 21:53:48 <lime_> Hey all, trying to compile a node on armv7l hf, getting an error saying G++-arm-linux-gnueabihf isn't installed
99 2017-09-29 21:54:06 <lime_> is it looking for a cross compiler?