Transmission Control Protocols (TCP) and Internet Protocols (IP) form a protocolsuite universally deployed on the network. TCP/IP enables a reliablebidirectional communication channel between systems on the internet.
The ordered delivery of Cardano node communication protocols is guaranteed bythe TCP/IP protocol.
Operating systems limit the number of concurrent connections. By default, Linux,for example, can open 1,024 connections per process, whereas macOS limits thisnumber to 256. To avoid excessive use of resources and enable reliable means forconnection establishment, Cardano uses a multiplexer.
Connection management
The network layer handles a range of specific tasks besides the exchange ofblock and transaction information required by the Ouroboros protocol.
Generally, connection management implementation includes the performance of thefollowing tasks:
- opening a socket and/or acquiring resources from the OS
- negotiating the protocol version with the handshake mini-protocol
- spawning the thread that runs the multiplexer (which can be instructed tostart/stop various mini-protocols)
- discovering and classifying exceptions thrown by mini-protocols or themultiplexer itself
- shutting down the connection in case of an error
- handling a shutdown request from the peer
- shutting down the threads that run mini-protocols
- closing a socket.
Multiplexing
The multiplexing layer acts as a central crossing between mini-protocols and thenetwork channel. It runs severalmini-protocolsin parallel in a single channel ‒ TCP connection, for example.
Figure 1 reflects how data flows between two nodes, each running threemini-protocols using a multiplexer (MUX) and a de-multiplexer (DEMUX).
Figure 1. Data flow between the nodes through multiplexing
Data transmitted between nodes passes through the MUX/DEMUX of the nodes. Thereis a fixed pairing of mini-protocol instances, which means that each instanceonly communicates with its dual instance (an initiator and a responder side).
The implementation of the mini-protocol also handles serialization andde-serialization of its messages. Mini-protocols write chunks of bytes to theMUX and read chunks of bytes from the DEMUX. The MUX reads the data frommini-protocols, splits it into segments, adds a segment header, and transmitsthe segments to the DEMUX of its peer. The DEMUX uses the segment’s headers toreassemble byte streams for the mini-protocols on its side. The multiplexingprotocol (see the note below) itself is completely agnostic to the structure ofthe multiplexed data.
note
This is not a generic, but specialized, use of multiplexing. Individualmini-protocols have strict constraints on unacknowledged messages that can be inflight. The design avoids the conditions in which the use of general TCP overTCP multiplexing creates chaotic performance.
Data segments of the multiplexing protocol
Multiplexing data segments include the following details:
- Transmission time ‒ a timestamp based on the lower 32 bits of the sender’smonotonic clock with a resolution of one microsecond
- Mini-protocol ID ‒ the unique ID of the mini-protocol
- Payload length ‒ the size of the segment payload in bytes; the maximumpayload length supported by the multiplexing wire format is 216 − 1. Note thatan instance of the protocol can choose a smaller limit for the size ofsegments it transmits
- Mode ‒ the single bit M (the mode) is used to distinguish the dualinstances of a mini-protocol. The mode is set to 0 in segments from theinitiator (the side that initially has agency), and it is set to 1 in segmentsfrom the responder.
Cardano node communication protocols
Cardano uses inter-process communication (IPC) protocols to allow for theexchange of blocks and transactions between nodes, and to allow localapplications to interact with the blockchain via the node.
Node-to-node IPC overview
The Node-to-node (NtN) protocol transfers transactions between full nodes. NtNincludes three mini-protocols (chain-sync, block-fetch, and tx-submission),which are multiplexed over a single TCP channel using a network-mux package.
The following diagram represents the NtN operational flow:
NtN follows a pull-based strategy, where the initiator node queries for newtransactions and the responder node replies with the transactions if any exist.This protocol perfectly suits a trustless setting where both sides need to beprotected against resource consumption attacks from the other side.
NtN mini-protocols explained
A brief explanation of the NtN mini-protocols:
- chain-sync: a protocol that allows a node to reconstruct a chain of anupstream node
- block-fetch: a protocol that allows a node to download block bodies fromvarious peers
- tx-submission: a protocol that allows submission of transactions. Theimplementation of this protocol is based on a generic mini protocol framework,with one peculiarity: the roles of the initiator and the responder arereversed. The Server is the initiator that asks for new transactions, and theClient is the responder that replies with the transactions. This role reversalwas designed thus for technical reasons.
To ensure optimal networking service, the team has also implemented anadditional protocol:
- keep-alive: a protocol that ensures continuous connection between nodesand minimizes performance faults.
Node-to-client IPC overview
Node-to-client (NtC) is a connection between a full node and a client thatconsumes data but does not take part in the Ouroboros protocol (a wallet, forexample.)
The purpose of the NtC IPC protocol is to allow local applications to interactwith the blockchain via the node. This includes applications such as walletbackends or blockchain explorers. The NtC protocol enables these applications toaccess the raw chain data and to query the current ledger state, and it alsoprovides the ability to submit new transactions to the system.
The NtC protocol uses the same design as the node-to-node (NtN) protocol, butwith a different set of mini-protocols, and using local pipes rather than TCPconnections. As such, it is a relatively low-level and narrow interface thatexposes only what the node can provide natively. For example, the node providesaccess to all the raw chain data but does not provide a way to query data on thechain. The job of providing data services and more convenient higher-level APIsis delegated to dedicated clients, such as cardano-db-sync and the walletbackend.
NtC mini-protocols
The NtC protocol consists of three mini-protocols:
- chain-sync - used for following the chain and getting blocks
- local-tx-submission - used for submitting transactions
- local-state-query - used for querying the ledger state.
The NtC version of chain-sync uses full blocks, rather than just block headers.This is why no separate block-fetch protocol is needed. The local-tx-submissionprotocol is like the NtN tx-submission protocol but simpler, and it returns thedetails of transaction validation failures. The local-state-query protocolprovides query access to the current ledger state, which contains a lot ofinteresting data that is not directly reflected on the chain itself.
How NtC works
In NtC, the node runs the producer side of the chain-sync protocol only, and theclient runs the consumer side only.
This table shows which mini-protocols are enabled for NtC communication:
