Documentation: Flow control, zmq_socket(3)

Mostly Flow control and additions to zmq_socket(3)
Removed/changed lots of text regarding message queues
More fixes for 2.0.7 changes

7 files changed, 205 insertions, 92 deletions

diff --git a/doc/zmq.txt b/doc/zmq.txt
index fd9f7ed..2d7f1c8 100644
--- a/doc/zmq.txt
+++ b/doc/zmq.txt
@@ -86,17 +86,9 @@ Message manipulation::
 
 Sockets
 ~~~~~~~
-Standard sockets present a _synchronous_ interface to either connection-mode
-reliable byte streams (SOCK_STREAM), or connection-less unreliable datagrams
-(SOCK_DGRAM). In comparison, 0MQ sockets present an abstraction of a
-asynchronous _message queue_, with the exact queueing semantics depending on
-the socket type in use. See linkzmq:zmq_socket[3] for the socket types
-provided.
-
-0MQ sockets being _asynchronous_ means that the timings of the physical
-connection setup and teardown, reconnect and effective delivery are organized
-by 0MQ itself, and that messages may be _queued_ in the event that a peer is
-unavailable to receive them.
+0MQ sockets present an abstraction of a asynchronous _message queue_, with the
+exact queueing semantics depending on the socket type in use. See
+linkzmq:zmq_socket[3] for the socket types provided.
 
 The following functions are provided to work with sockets:
 
@@ -118,9 +110,7 @@ Sending and receiving messages::
     linkzmq:zmq_send[3]
     linkzmq:zmq_recv[3]
 
-
-Input/output multiplexing
-^^^^^^^^^^^^^^^^^^^^^^^^^
+.Input/output multiplexing
 0MQ provides a mechanism for applications to multiplex input/output events over
 a set containing both 0MQ sockets and standard sockets. This mechanism mirrors
 the standard _poll()_ system call, and is described in detail in
diff --git a/doc/zmq_getsockopt.txt b/doc/zmq_getsockopt.txt
index 318d12e..f08748b 100644
--- a/doc/zmq_getsockopt.txt
+++ b/doc/zmq_getsockopt.txt
@@ -45,12 +45,16 @@ Applicable socket types:: all
 
 ZMQ_HWM: Retrieve high water mark
 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
-The 'ZMQ_HWM' option shall retrieve the high water mark for the _message queue_
-associated with the specified 'socket'.  The high water mark is a hard limit on
-the number of outstanding messages in the queue; if this limit has been reached
-the socket shall enter an "emergency" state and depending on the socket type,
-0MQ shall take appropriate action such as blocking or dropping new messages
-entering the queue.
+The 'ZMQ_HWM' option shall retrieve the high water mark for the specified
+'socket'.  The high water mark is a hard limit on the maximum number of
+outstanding messages 0MQ shall queue in memory for any single peer that the
+specified 'socket' is communicating with.
+
+If this limit has been reached the socket shall enter an exceptional state and
+depending on the socket type, 0MQ shall take appropriate action such as
+blocking or dropping sent messages. Refer to the individual socket descriptions
+in linkzmq:zmq_socket[3] for details on the exact action taken for each socket
+type.
 
 The default 'ZMQ_HWM' value of zero means "no limit".
 
@@ -63,10 +67,9 @@ Applicable socket types:: all
 ZMQ_SWAP: Retrieve disk offload size
 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
 The 'ZMQ_SWAP' option shall retrieve the disk offload (swap) size for the
-_message queue_ associated with the specified 'socket'. A socket which has
-'ZMQ_SWAP' set to a non-zero value may exceed it's high water mark; in this
-case outstanding messages shall be offloaded to storage on disk rather than
-held in memory.
+specified 'socket'. A socket which has 'ZMQ_SWAP' set to a non-zero value may
+exceed it's high water mark; in this case outstanding messages shall be
+offloaded to storage on disk rather than held in memory.
 
 The value of 'ZMQ_SWAP' defines the maximum size of the swap space in bytes.
 
diff --git a/doc/zmq_poll.txt b/doc/zmq_poll.txt
index 9106b5b..d96af3e 100644
--- a/doc/zmq_poll.txt
+++ b/doc/zmq_poll.txt
@@ -54,18 +54,16 @@ The 'events' and 'revents' members of *zmq_pollitem_t* are bitmasks constructed
 by OR'ing a combination of the following event flags:
 
 *ZMQ_POLLIN*::
-For 0MQ sockets, at least one message may be dequeued from the underlying
-_message queue_ associated with 'socket' without blocking. For standard sockets
-this is equivalent to the 'POLLIN' flag of the _poll()_ system call and
-generally means that at least one byte of data may be read from 'fd' without
-blocking.
+For 0MQ sockets, at least one message may be received from the 'socket' without
+blocking. For standard sockets this is equivalent to the 'POLLIN' flag of the
+_poll()_ system call and generally means that at least one byte of data may be
+read from 'fd' without blocking.
 
 *ZMQ_POLLOUT*::
-For 0MQ sockets, at least one message may be queued on the underlying
-_message queue_ associated with 'socket' without blocking. For standard sockets
-this is equivalent to the 'POLLOUT' flag of the _poll()_ system call and
-generally means that at least one byte of data may be written to 'fd'
-without blocking.
+For 0MQ sockets, at least one message may be sent to the 'socket' without
+blocking. For standard sockets this is equivalent to the 'POLLOUT' flag of the
+_poll()_ system call and generally means that at least one byte of data may be
+written to 'fd' without blocking.
 
 *ZMQ_POLLERR*::
 For standard sockets, this flag is passed through _zmq_poll()_ to the
@@ -82,10 +80,12 @@ of those interfaces in ways not defined in this documentation.
 RETURN VALUE
 ------------
 Upon successful completion, the _zmq_poll()_ function shall return the number
-of *zmq_pollitem_t* structures with events signaled in 'revents' or `0` if the
-'timeout' period has expired and no events have been signaled. Upon failure,
-_zmq_poll()_ shall return `-1` and set 'errno' to one of the values defined
-below.
+of *zmq_pollitem_t* structures with events signaled in 'revents' or `0` if no
+events have been signaled. Upon failure, _zmq_poll()_ shall return `-1` and set
+'errno' to one of the values defined below.
+
+IMPORTANT: The _zmq_poll()_ function may return *before* the 'timeout' period
+has expired even if no events have been signaled.
 
 
 ERRORS
diff --git a/doc/zmq_recv.txt b/doc/zmq_recv.txt
index 42df99a..79d3dc1 100644
--- a/doc/zmq_recv.txt
+++ b/doc/zmq_recv.txt
@@ -14,19 +14,17 @@ SYNOPSIS
 
 DESCRIPTION
 -----------
-The _zmq_recv()_ function shall dequeue a message from the underlying _message
-queue_ associated with the socket referenced by the 'socket' argument and store
-it in the message referenced by the 'msg' argument. Any content previously
-stored in 'msg' shall be properly deallocated. If there are no messages
-available to be dequeued from the underlying _message queue_ associated with
-'socket' the _zmq_recv()_ function shall block until the request can be
-satisfied. The 'flags' argument is a combination of the flags defined below:
+The _zmq_recv()_ function shall receive a message from the socket referenced by
+the 'socket' argument and store it in the message referenced by the 'msg'
+argument. Any content previously stored in 'msg' shall be properly deallocated.
+If there are no messages available on the specified 'socket' the _zmq_recv()_
+function shall block until the request can be satisfied. The 'flags' argument
+is a combination of the flags defined below:
 
 *ZMQ_NOBLOCK*::
 Specifies that the operation should be performed in non-blocking mode. If there
-are no messages available to be dequeued from the underlying _message queue_
-associated with 'socket', the _zmq_recv()_ function shall fail with 'errno' set
-to EAGAIN.
+are no messages available on the specified 'socket', the _zmq_recv()_ function
+shall fail with 'errno' set to EAGAIN.
 
 
 Multi-part messages
@@ -75,7 +73,7 @@ EXAMPLE
 zmq_msg_t msg;
 int rc = zmq_msg_init (&msg);
 assert (rc == 0);
-/* Block until a message is available to be dequeued from socket */
+/* Block until a message is available to be received from socket */
 rc = zmq_recv (socket, &msg, 0);
 assert (rc == 0);
 ----
@@ -89,7 +87,7 @@ do {
     zmq_msg_t part;
     int rc = zmq_msg_init (&part);
     assert (rc == 0);
-    /* Block until a message is available to be dequeued from socket */
+    /* Block until a message is available to be received from socket */
     rc = zmq_recv (socket, &part, 0);
     assert (rc == 0);
     /* Determine if more message parts are to follow */
diff --git a/doc/zmq_send.txt b/doc/zmq_send.txt
index 300f812..419e386 100644
--- a/doc/zmq_send.txt
+++ b/doc/zmq_send.txt
@@ -20,8 +20,8 @@ argument to be sent to the socket referenced by the 'socket' argument.  The
 
 *ZMQ_NOBLOCK*::
 Specifies that the operation should be performed in non-blocking mode. If the
-message cannot be queued on the underlying _message queue_ associated with
-'socket', the _zmq_send()_ function shall fail with 'errno' set to EAGAIN.
+message cannot be queued on the 'socket', the _zmq_send()_ function shall fail
+with 'errno' set to EAGAIN.
 
 *ZMQ_SNDMORE*::
 Specifies that the message being sent is a multi-part message, and that further
@@ -30,8 +30,7 @@ below for a detailed description.
 
 NOTE: A successful invocation of _zmq_send()_ does not indicate that the
 message has been transmitted to the network, only that it has been queued on
-the _message queue_ associated with the socket and 0MQ has assumed
-responsibility for the message.
+the 'socket' and 0MQ has assumed responsibility for the message.
 
 
 Multi-part messages
@@ -60,7 +59,7 @@ return `-1` and set 'errno' to one of the values defined below.
 ERRORS
 ------
 *EAGAIN*::
-Non-blocking mode was requested and the message cannot be queued at the moment.
+Non-blocking mode was requested and the message cannot be sent at the moment.
 *ENOTSUP*::
 The _zmq_send()_ operation is not supported by this socket type.
 *EFSM*::
diff --git a/doc/zmq_setsockopt.txt b/doc/zmq_setsockopt.txt
index 7721eff..93c5889 100644
--- a/doc/zmq_setsockopt.txt
+++ b/doc/zmq_setsockopt.txt
@@ -25,12 +25,16 @@ The following socket options can be set with the _zmq_setsockopt()_ function:
 
 ZMQ_HWM: Set high water mark
 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~
-The 'ZMQ_HWM' option shall set the high water mark for the _message queue_
-associated with the specified 'socket'.  The high water mark is a hard limit on
-the number of outstanding messages in the queue; if this limit has been reached
-the socket shall enter an "emergency" state and depending on the socket type,
-0MQ shall take appropriate action such as blocking or dropping new messages
-entering the queue.
+The 'ZMQ_HWM' option shall set the high water mark for the specified 'socket'.
+The high water mark is a hard limit on the maximum number of outstanding
+messages 0MQ shall queue in memory for any single peer that the specified
+'socket' is communicating with.
+
+If this limit has been reached the socket shall enter an exceptional state and
+depending on the socket type, 0MQ shall take appropriate action such as
+blocking or dropping sent messages. Refer to the individual socket descriptions
+in linkzmq:zmq_socket[3] for details on the exact action taken for each socket
+type.
 
 The default 'ZMQ_HWM' value of zero means "no limit".
 
@@ -42,11 +46,10 @@ Applicable socket types:: all
 
 ZMQ_SWAP: Set disk offload size
 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
-The 'ZMQ_SWAP' option shall set the disk offload (swap) size for the _message
-queue_ associated with the specified 'socket'. A socket which has 'ZMQ_SWAP'
-set to a non-zero value may exceed it's high water mark; in this case
-outstanding messages shall be offloaded to storage on disk rather than held in
-memory.
+The 'ZMQ_SWAP' option shall set the disk offload (swap) size for the specified
+'socket'. A socket which has 'ZMQ_SWAP' set to a non-zero value may exceed it's
+high water mark; in this case outstanding messages shall be offloaded to
+storage on disk rather than held in memory.
 
 The value of 'ZMQ_SWAP' defines the maximum size of the swap space in bytes.
 
diff --git a/doc/zmq_socket.txt b/doc/zmq_socket.txt
index 119ac31..23cc317 100644
--- a/doc/zmq_socket.txt
+++ b/doc/zmq_socket.txt
@@ -19,30 +19,89 @@ The 'zmq_socket()' function shall create a 0MQ socket within the specified
 argument specifies the socket type, which determines the semantics of
 communication over the socket.
 
+The newly created socket is initially unbound, and not associated with any
+endpoints. In order to establish a message flow a socket must first be
+connected to at least one endpoint with linkzmq:zmq_connect[3], or at least one
+endpoint must be created for accepting incoming connections with
+linkzmq:zmq_bind[3].
+
+.Key differences to conventional sockets
+Generally speaking, conventional sockets present a _synchronous_ interface to
+either connection-oriented reliable byte streams (SOCK_STREAM), or
+connection-less unreliable datagrams (SOCK_DGRAM). In comparison, 0MQ sockets
+present an abstraction of an asynchronous _message queue_, with the exact
+queueing semantics depending on the socket type in use. Where conventional
+sockets transfer streams of bytes or discrete datagrams, 0MQ sockets transfer
+discrete _messages_.
+
+0MQ sockets being _asynchronous_ means that the timings of the physical
+connection setup and teardown, reconnect and effective delivery are transparent
+to the user and organized by 0MQ itself. Further, messages may be _queued_ in
+the event that a peer is unavailable to receive them.
+
+Conventional sockets allow only strict one-to-one (two peers), many-to-one
+(many clients, one server), or in some cases one-to-many (multicast)
+relationships. With the exception of 'ZMQ_PAIR', 0MQ sockets may be connected
+*to multiple endpoints* using _zmq_connect()_, while simultaneously accepting
+incoming connections *from multiple endpoints* bound to the socket using
+_zmq_bind()_, thus allowing many-to-many relationships.
+
+.Socket types
 The following sections present the socket types defined by 0MQ, grouped by the
-general _messaging pattern_ built from related socket types.
+general _messaging pattern_ which is built from related socket types.
 
 
 Request-reply pattern
 ~~~~~~~~~~~~~~~~~~~~~
-The request-reply pattern is used for sending requests from a _client_ to a
-_service_, and receiving subsequent replies to each request sent.
+The request-reply pattern is used for sending requests from a _client_ to one
+or more instances of a _service_, and receiving subsequent replies to each
+request sent.
 
-Socket type:: 'ZMQ_REQ'
-Compatible peer sockets:: 'ZMQ_REP'
 
+ZMQ_REQ
+^^^^^^^
 A socket of type 'ZMQ_REQ' is used by a _client_ to send requests to and
 receive replies from a _service_. This socket type allows only an alternating
 sequence of _zmq_send(request)_ and subsequent _zmq_recv(reply)_ calls. Each
-request sent is load-balanced among all connected _services_.
+request sent is load-balanced among all _services_, and each reply received is
+matched with the last issued request.
 
-Socket type:: 'ZMQ_REP'
-Compatible peer sockets:: 'ZMQ_REQ'
+When a 'ZMQ_REQ' socket enters an exceptional state due to having reached the
+high water mark for all _services_, or if there are no _services_ at all, then
+any linkzmq:zmq_send[3] operations on the socket shall block until the
+exceptional state ends or at least one _service_ becomes available for sending;
+messages are not discarded.
+
+[horizontal]
+.Summary of ZMQ_REQ characteristics
+Compatible peer sockets:: 'ZMQ_REP'
+Direction:: Bidirectional
+Send/receive pattern:: Send, Receive, Send, Receive, ...
+Outgoing routing strategy:: Load-balanced
+Incoming routing strategy:: Last peer
+ZMQ_HWM option action:: Block
 
+
+ZMQ_REP
+^^^^^^^
 A socket of type 'ZMQ_REP' is used by a _service_ to receive requests from and
-send replies to a _client_.  This socket type allows only an alternating
+send replies to a _client_. This socket type allows only an alternating
 sequence of _zmq_recv(request)_ and subsequent _zmq_send(reply)_ calls. Each
-reply is routed to the _client_ that issued the last received request.
+request received is fair-queued from among all _clients_, and each reply sent
+is routed to the _client_ that issued the last request.
+
+When a 'ZMQ_REP' socket enters an exceptional state due to having reached the
+high water mark for a _client_, then any replies sent to the _client_ in
+question shall be dropped until the exceptional state ends.
+
+[horizontal]
+.Summary of ZMQ_REP characteristics
+Compatible peer sockets:: 'ZMQ_REQ'
+Direction:: Bidirectional
+Send/receive pattern:: Receive, Send, Receive, Send, ...
+Incoming routing strategy:: Fair-queued
+Outgoing routing stratagy:: Last peer
+ZMQ_HWM option action:: Drop
 
 
 Publish-subscribe pattern
@@ -50,21 +109,44 @@ Publish-subscribe pattern
 The publish-subscribe pattern is used for one-to-many distribution of data from
 a single _publisher_ to multiple _subscribers_ in a fanout fashion.
 
-Socket type:: 'ZMQ_PUB'
-Compatible peer sockets:: 'ZMQ_SUB'
 
+ZMQ_PUB
+^^^^^^^
 A socket of type 'ZMQ_PUB' is used by a _publisher_ to distribute data.
 Messages sent are distributed in a fanout fashion to all connected peers.
-The _zmq_recv()_ function is not implemented for this socket type.
+The linkzmq:zmq_recv[3] function is not implemented for this socket type.
 
-Socket type:: 'ZMQ_SUB'
-Compatible peer sockets:: 'ZMQ_PUB'
+When a 'ZMQ_PUB' socket enters an exceptional state due to having reached the
+high water mark for a _subscriber_, then any messages that would be sent to the
+_subscriber_ in question shall instead be dropped until the exceptional state
+ends.
 
+[horizontal]
+.Summary of ZMQ_PUB characteristics
+Compatible peer sockets:: 'ZMQ_SUB'
+Direction:: Unidirectional
+Send/receive pattern:: Send only
+Incoming routing strategy:: N/A
+Outgoing routing strategy:: Fanout
+ZMQ_HWM option action:: Drop
+
+
+ZMQ_SUB
+^^^^^^^
 A socket of type 'ZMQ_SUB' is used by a _subscriber_ to subscribe to data
 distributed by a _publisher_. Initially a 'ZMQ_SUB' socket is not subscribed to
-any messages, use the 'ZMQ_SUBSCRIBE' option of _zmq_setsockopt()_ to specify
-which messages to subscribe to. The _zmq_send()_ function is not implemented
-for this socket type.
+any messages, use the 'ZMQ_SUBSCRIBE' option of linkzmq:zmq_setsockopt[3] to
+specify which messages to subscribe to. The _zmq_send()_ function is not
+implemented for this socket type.
+
+[horizontal]
+.Summary of ZMQ_SUB characteristics
+Compatible peer sockets:: 'ZMQ_PUB'
+Direction:: Unidirectional
+Send/receive pattern:: Receive only
+Incoming routing strategy:: Fair-queued
+Outgoing routing strategy:: N/A
+ZMQ_HWM option action:: N/A
 
 
 Pipeline pattern
@@ -74,38 +156,76 @@ a pipeline. Data always flows down the pipeline, and each stage of the pipeline
 is connected to at least one _node_. When a pipeline stage is connected to
 multiple _nodes_ data is load-balanced among all connected _nodes_.
 
-Socket type:: 'ZMQ_DOWNSTREAM'
-Compatible peer sockets:: 'ZMQ_UPSTREAM'
 
+ZMQ_DOWNSTREAM
+^^^^^^^^^^^^^^
 A socket of type 'ZMQ_DOWNSTREAM' is used by a pipeline _node_ to send messages
 to downstream pipeline _nodes_. Messages are load-balanced to all connected
 downstream _nodes_. The _zmq_recv()_ function is not implemented for this
 socket type.
 
-Socket type:: 'ZMQ_UPSTREAM'
-Compatible peer sockets:: 'ZMQ_DOWNSTREAM'
+When a 'ZMQ_DOWNSTREAM' socket enters an exceptional state due to having
+reached the high water mark for all downstream _nodes_, or if there are no
+downstream _nodes_ at all, then any linkzmq:zmq_send[3] operations on the
+socket shall block until the exceptional state ends or at least one downstream
+_node_ becomes available for sending; messages are not discarded.
+
+[horizontal]
+.Summary of ZMQ_DOWNSTREAM characteristics
+Compatible peer sockets:: 'ZMQ_UPSTREAM'
+Direction:: Unidirectional
+Send/receive pattern:: Send only
+Incoming routing strategy:: N/A
+Outgoing routing strategy:: Load-balanced
+ZMQ_HWM option action:: Block
+
 
+ZMQ_UPSTREAM
+^^^^^^^^^^^^
 A socket of type 'ZMQ_UPSTREAM' is used by a pipeline _node_ to receive
 messages from upstream pipeline _nodes_. Messages are fair-queued from among
 all connected upstream _nodes_. The _zmq_send()_ function is not implemented
 for this socket type.
 
+[horizontal]
+.Summary of ZMQ_UPSTREAM characteristics
+Compatible peer sockets:: 'ZMQ_DOWNSTREAM'
+Direction:: Unidirectional
+Send/receive pattern:: Receive only
+Incoming routing strategy:: Fair-queued
+Outgoing routing strategy:: N/A
+ZMQ_HWM option action:: N/A
+
 
 Exclusive pair pattern
 ~~~~~~~~~~~~~~~~~~~~~~
 The exclusive pair is an advanced pattern used for communicating exclusively
 between two peers.
 
-Socket type:: 'ZMQ_PAIR'
-Compatible peer sockets:: 'ZMQ_PAIR'
 
+ZMQ_PAIR
+^^^^^^^^
 A socket of type 'ZMQ_PAIR' can only be connected to a single peer at any one
 time.  No message routing or filtering is performed on messages sent over a
 'ZMQ_PAIR' socket.
 
+When a 'ZMQ_PAIR' socket enters an exceptional state due to having reached the
+high water mark for the connected peer, or if no peer is connected, then
+any linkzmq:zmq_send[3] operations on the socket shall block until the peer
+becomes available for sending; messages are not discarded.
+
 NOTE: 'ZMQ_PAIR' sockets are experimental, and are currently missing several
 features such as auto-reconnection.
 
+[horizontal]
+.Summary of ZMQ_PAIR characteristics
+Compatible peer sockets:: 'ZMQ_PAIR'
+Direction:: Bidirectional
+Send/receive pattern:: Unrestricted
+Incoming routing strategy:: N/A
+Outgoing routing strategy:: N/A
+ZMQ_HWM option action:: Block
+
 
 RETURN VALUE
 ------------
@@ -120,8 +240,7 @@ ERRORS
 The requested socket 'type' is invalid.
 
 *EMTHREAD*::
-The number of application threads using sockets within this 'context' has been
-exceeded. See the 'app_threads' parameter of the _zmq_init()_ function.
+The maximum number of sockets within this 'context' has been exceeded.
 
 
 SEE ALSO
@@ -132,6 +251,7 @@ linkzmq:zmq_bind[3]
 linkzmq:zmq_connect[3]
 linkzmq:zmq_send[3]
 linkzmq:zmq_recv[3]
+linkzmq:zmq[7]
 
 
 AUTHORS