Loopback#
Every transport in pamoja speaks the same publish-and-subscribe interface, so a
message flow can be exercised without the link that would normally carry it. The
loopback transport is that stand-in: a routing table living in the process,
shared by however many links are taken off one broker, matching topics with the
same + and # rules MQTT fixes. Nothing binds a port and nothing has to be
running, so a topic layout can be checked from a unit test instead of against a
deployment. It is also the link the Transport ladder and Engine
surface pages run on, where the point being made is the
composition, a ladder or a fault injector, rather than the network underneath
it.
What the example does#
It builds one broker, takes a publisher and a subscriber off it, and moves a
temperature reading from one to the other. The subscriber's filter is
single-level, and the reading a level deeper is published first, so what comes
back is 21.5 from the temperature topic and not the 2150 sent a moment
earlier under /raw. The subscriber then waits 50 milliseconds for anything
more, which is how a test shows that the raw reading never reached it.
A third link joins on the multi-level filter after both of those publishes have
gone out, and takes the next /raw reading. The publisher is disconnected
last, and one more send shows what an unusable link does with a reading.
It proves:
- A payload published on one link arrives on another carrying the topic it was sent to, with no port bound and no broker process running.
+matches exactly one level, so the filter takes the temperature topic and leaves the/rawreading a level below it, even though that one went out first.- A receive with a time limit runs out when nothing more matches, so a test proves a reading did not arrive without hanging on it.
#matches the levels that remain, so the second filter takes the deeper topic the single-level one passed over.- A link can join a broker that has already routed traffic, take a filter of its own, and receive a reading published after it connects.
- A disconnected link fails the send rather than accepting a reading it has no way to deliver.
Run it#
The example below is a program CI runs on every change, in each language, from a clone of the repository:
Rust
cargo run -p pamoja-examples --example loopbackTypeScript
npm --prefix bindings/node run guides -- loopbackPython
python bindings/python/guides/loopback.pyC#
dotnet run --project bindings/dotnet/samples/Pamoja.Guides -- loopbackRust#
In Rust, LoopbackBroker::new() makes the routing table and cloning it shares the table, so
every LoopbackTransport::new(broker.clone()) is a link on the same broker. A link implements
the Transport and Receive traits, which gives it the calls every link keeps, connect,
subscribe, send, send_text, and recv, and it adds is_connected() and disconnect(). To
stop waiting, wrap recv in tokio::time::timeout: a receive is cancel-safe, so the message it
would have taken stays queued for the next one. Faulty::new(link, n) makes any link fail its
next n sends, and fail_next(n) arms it again.
From examples/guides/loopback.rs:
use std::time::Duration;
use pamoja_core::{Receive, Transport};
use pamoja_loopback::{LoopbackBroker, LoopbackTransport};
// One broker and two links off it, all in this process. Nothing binds a port and
// nothing has to be running for the traffic below to flow, which is what makes this
// the link to develop a node against before it has a real one.
let broker = LoopbackBroker::new();
let mut publisher = LoopbackTransport::new(broker.clone());
let mut subscriber = LoopbackTransport::new(broker.clone());
publisher.connect().await?;
subscriber.connect().await?;
// A `+` stands for exactly one level, so this takes the mixer's temperature but not
// the raw reading a level below it.
subscriber.subscribe("line/+/temp").await?;
publisher.send_text("line/mixer/temp/raw", "2150").await?;
publisher.send_text("line/mixer/temp", "21.5").await?;
let message = subscriber.recv().await?.expect("a message");
let reading = message.text().expect("text");
println!("line/+/temp took {reading} from {}", message.topic);
// The raw reading went out first and never arrived, which a test proves by waiting a
// set time for anything more rather than forever. Giving up loses nothing: a message
// that came later would wait for the next receive.
let quiet = Duration::from_millis(50);
match tokio::time::timeout(quiet, subscriber.recv()).await {
Ok(_) => println!("line/+/temp took a second reading, which should never happen"),
Err(_) => println!(
"line/+/temp heard nothing more within {} ms",
quiet.as_millis()
),
}
// A `#` covers every level that remains, so a second link takes the whole subtree,
// including the reading the single-level filter passed over.
let mut watcher = LoopbackTransport::new(broker);
watcher.connect().await?;
watcher.subscribe("line/#").await?;
publisher.send_text("line/mixer/temp/raw", "2150").await?;
let deep = watcher.recv().await?.expect("a message");
let raw = deep.text().expect("text");
println!("line/# took {raw} from {}", deep.topic);
// A link that has been disconnected reports the failure instead of dropping the
// reading, which is the case a test wants to reach without unplugging anything.
publisher.disconnect();
match publisher.send_text("line/mixer/temp", "21.6").await {
Ok(_) => println!("a disconnected link took a reading, which should never happen"),
Err(error) => println!("disconnected refused the reading: {error}"),
}TypeScript#
In TypeScript, new LoopbackBroker() from @pamoja/loopback makes the broker, broker.link()
gives a LoopbackTransport to drive directly, and broker.rung() gives a Transport to hand to
a ladder or a wrapper. Every call returns a promise: connect, subscribe,
send(topic, bufferOrText), recv(timeoutMs?), isConnected, and disconnect. recv
resolves with a TransportMessage, { topic, payload, text?, number? }. Given a limit, it
rejects once the time runs out and the next message waits for the next receive, where racing a
plain recv() against a timer would leave it running to take that message.
From bindings/node/guides/loopback.ts:
import { LoopbackBroker, type TransportMessage } from '@pamoja/loopback'
const QUIET_MS = 50
async function main() {
// One broker and two links off it, all in this process. Nothing binds a port and nothing
// has to be running for the traffic below to flow, which is what makes this the link to
// develop a node against before it has a real one.
const broker = new LoopbackBroker()
const publisher = broker.link()
const subscriber = broker.link()
await publisher.connect()
await subscriber.connect()
// A `+` stands for exactly one level, so this takes the mixer's temperature but not the
// raw reading a level below it.
await subscriber.subscribe('line/+/temp')
await publisher.send('line/mixer/temp/raw', '2150')
await publisher.send('line/mixer/temp', '21.5')
const message = (await subscriber.recv())!
console.log(`line/+/temp took ${message.text!} from ${message.topic}`)
// The raw reading went out first and never arrived, which a test proves by waiting a
// set time for anything more rather than forever. Giving up loses nothing: a message
// that came later would wait for the next receive.
try {
await subscriber.recv(QUIET_MS)
console.log('line/+/temp took a second reading, which should never happen')
} catch {
console.log(`line/+/temp heard nothing more within ${QUIET_MS} ms`)
}
// A `#` covers every level that remains, so a second link takes the whole subtree,
// including the reading the single-level filter passed over.
const watcher = broker.link()
await watcher.connect()
await watcher.subscribe('line/#')
await publisher.send('line/mixer/temp/raw', '2150')
const deep = (await watcher.recv())!
console.log(`line/# took ${deep.text!} from ${deep.topic}`)
// A link that has been disconnected reports the failure instead of dropping the reading,
// which is the case a test wants to reach without unplugging anything.
await publisher.disconnect()
try {
await publisher.send('line/mixer/temp', '21.6')
console.log('a disconnected link took a reading, which should never happen')
} catch (error) {
console.log(`disconnected refused the reading: ${(error as Error).message}`)
}
return { message, deep }
}
main()Python#
In Python, LoopbackBroker() from pamoja.loopback makes the broker, broker.link() a link to
drive, and broker.rung() a Transport to compose. The calls are coroutines: connect,
subscribe, send(topic, bytes_or_text), recv, is_connected, and disconnect. recv
returns a Message whose payload is bytes and whose text and number read it.
asyncio.wait_for(link.recv(), seconds) stops waiting, and the receive it cancels leaves its
message queued for the next one. A failure raises PamojaError.
From bindings/python/guides/loopback.py:
import asyncio
from pamoja.core import PamojaError
from pamoja.loopback import LoopbackBroker
QUIET_MS = 50
async def main() -> None:
# One broker and two links off it, all in this process. Nothing binds a port and
# nothing has to be running for the traffic below to flow, which is what makes this
# the link to develop a node against before it has a real one.
broker = LoopbackBroker()
publisher = broker.link()
subscriber = broker.link()
await publisher.connect()
await subscriber.connect()
# A `+` stands for exactly one level, so this takes the mixer's temperature but not the
# raw reading a level below it.
await subscriber.subscribe("line/+/temp")
await publisher.send("line/mixer/temp/raw", "2150")
await publisher.send("line/mixer/temp", "21.5")
message = await subscriber.recv()
print(f"line/+/temp took {message.text} from {message.topic}")
# The raw reading went out first and never arrived, which a test proves by waiting a
# set time for anything more rather than forever. Giving up loses nothing: a message
# that came later would wait for the next receive.
try:
await asyncio.wait_for(subscriber.recv(), QUIET_MS / 1000)
print("line/+/temp took a second reading, which should never happen")
except asyncio.TimeoutError:
print(f"line/+/temp heard nothing more within {QUIET_MS} ms")
# A `#` covers every level that remains, so a second link takes the whole subtree,
# including the reading the single-level filter passed over.
watcher = broker.link()
await watcher.connect()
await watcher.subscribe("line/#")
await publisher.send("line/mixer/temp/raw", "2150")
deep = await watcher.recv()
print(f"line/# took {deep.text} from {deep.topic}")
# A link that has been disconnected reports the failure instead of dropping the
# reading, which is the case a test wants to reach without unplugging anything.
await publisher.disconnect()
try:
await publisher.send("line/mixer/temp", "21.6")
print("a disconnected link took a reading, which should never happen")
except PamojaError as error:
print(f"disconnected refused the reading: {error}")
return message, deep
message, deep = asyncio.run(main())C##
In C#, new LoopbackBroker() in Pamoja.Loopback makes the broker, Link() a
LoopbackTransport to drive, and Rung() a Transport to compose, all disposable. The calls
are ConnectAsync, SubscribeAsync, SendAsync(topic, bytesOrText), ReceiveAsync(),
ReceiveAsync(limit), IsConnectedAsync, and DisconnectAsync, and calls on one link run one at
a time. ReceiveAsync(limit) throws TimeoutException when the time runs out and leaves the next
message for the next receive, where a ReceiveAsync() abandoned through Task.WhenAny would take
it. A failure throws PamojaException.
From bindings/dotnet/samples/Pamoja.Guides/LoopbackGuide.cs:
// One broker and two links off it, all in this process. Nothing binds a port and
// nothing has to be running for the traffic below to flow, which is what makes
// this the link to develop a node against before it has a real one.
using var broker = new LoopbackBroker();
using LoopbackTransport publisher = broker.Link();
using LoopbackTransport subscriber = broker.Link();
await publisher.ConnectAsync();
await subscriber.ConnectAsync();
// A `+` stands for exactly one level, so this takes the mixer's temperature but
// not the raw reading a level below it.
await subscriber.SubscribeAsync("line/+/temp");
await publisher.SendAsync("line/mixer/temp/raw", "2150");
await publisher.SendAsync("line/mixer/temp", "21.5");
TransportMessage message = (await subscriber.ReceiveAsync())!;
Console.WriteLine(
$"line/+/temp took {message.Text}"
+ $" from {message.Topic}");
// The raw reading went out first and never arrived, which a test proves by waiting
// a set time for anything more rather than forever. Giving up loses nothing: a
// message that came later would wait for the next receive.
TimeSpan quiet = TimeSpan.FromMilliseconds(50);
try
{
await subscriber.ReceiveAsync(quiet);
Console.WriteLine("line/+/temp took a second reading, which should never happen");
}
catch (TimeoutException)
{
Console.WriteLine($"line/+/temp heard nothing more within {quiet.TotalMilliseconds} ms");
}
// A `#` covers every level that remains, so a second link takes the whole subtree,
// including the reading the single-level filter passed over.
using LoopbackTransport watcher = broker.Link();
await watcher.ConnectAsync();
await watcher.SubscribeAsync("line/#");
await publisher.SendAsync("line/mixer/temp/raw", "2150");
TransportMessage deep = (await watcher.ReceiveAsync())!;
Console.WriteLine(
$"line/# took {deep.Text}"
+ $" from {deep.Topic}");
// A link that has been disconnected reports the failure instead of dropping the
// reading, which is the case a test wants to reach without unplugging anything.
await publisher.DisconnectAsync();
try
{
await publisher.SendAsync("line/mixer/temp", "21.6");
Console.WriteLine("a disconnected link took a reading, which should never happen");
}
catch (PamojaException error)
{
Console.WriteLine($"disconnected refused the reading: {error.Message}");
}Values at a glance#
The broker and its links:
| What | Rust | TypeScript | Python | C# |
|---|---|---|---|---|
| a broker | LoopbackBroker::, cloned to share | new LoopbackBroker() | LoopbackBroker() | new LoopbackBroker() |
| a link to drive | LoopbackTransport:: | broker. | broker. | broker. |
| a link to compose | the same link, moved into the ladder | broker. | broker. | broker. |
| connect | link. | await link. | await link. | await link. |
| subscribe | link. | await link. | await link. | await link. |
| send | link., link. | await link. | await link. | await link. |
| receive | link. | await link. | await link. | await link. |
| receive with a limit | timeout( | await link. | await asyncio. | await link. |
| is it connected | link. | await link. | await link. | await link. |
| disconnect | link. | await link. | await link. | await link. |
| fail the next sends | Faulty::, then fail_next( | Transport. | Transport. | Transport. |
| take every link out of reach | broker. | broker. | broker. | broker. |
Out of reach, every link on the broker fails to connect, send, or subscribe with transport error: the broker is out of reach, including links a ladder or a wrapper now owns, and keeps its
connection and filters until the broker is back. The Transport ladder guide takes
three networks out of reach this way.
How a filter matches a topic. The rules are MQTT's, from sections 4.7.1 and 4.7.2 of the OASIS MQTT 3.1.1 standard, and they are the same on the loopback as on a broker:
| Filter | Topic | Delivered | Why |
|---|---|---|---|
line/ | line/ | yes | + takes exactly one level |
line/ | line/ | no | the topic has a level the filter does not |
line/ | line | no | + needs a level to take, and there is none |
line/ | line/ | yes | # takes every level that remains |
line/ | line | yes | # takes the parent level too |
+/ | /finance | yes | a leading / makes an empty first level, and + takes it |
+ | /finance | no | the topic has two levels |
# | $SYS/ | no | a filter that starts with a wildcard never takes a $ topic |
$SYS/ | $SYS/ | yes | naming $SYS first lets the rest match |
What the loopback does, and what a broker adds. Code that passes here meets these differences when it moves to MQTT; the section numbers are the MQTT 3.1.1 standard's:
| Behavior | Loopback | An MQTT broker |
|---|---|---|
| who hears a publish | every link whose filters match, the publisher included | every client with a matching subscription, the publisher included (3.3.5) |
| a publish that matches two of a link's filters | one copy | at least one, and it may send one per filter (3.3.5) |
| a message published before the subscribe | never delivered | never delivered, unless it was published to be retained (3.3.1.3) |
| a link slow to read | its queue grows without a limit | the broker's own limits apply |
| a disconnect | drops what was queued; the filters stay for the next connect | with pamoja's clean session, the subscriptions end too (3.1.2.4) |
| a topic the publisher may not use | there is no access control | the broker acknowledges and drops it, or closes the connection (3.3.5) |
When it goes wrong#
What the loopback refuses, and what it says:
| What happened | The message | What to check |
|---|---|---|
| a call on a link that is not connected | resource is closed | connect first, or again after disconnect |
| a fault injector refused a send | transport error: | nothing: that is its job |
| no message within the limit | no message arrived within 50 ms in TypeScript and C#, asyncio. in Python, Elapsed in Rust | nothing matched, which may be the point of the test |
| a composed link used after it was handed on | this transport was already added to a ladder or a wrapper | build another with rung() |
| a composed link handed on while a call runs | this transport is busy with a call | await the call first |
The mistakes that cost an afternoon:
- A test hangs instead of failing. It waits on a receive for a message that never comes, and a connected loopback link never ends on its own. Give the receive a limit, as the example does.
- A test raced a receive against a timer and lost the next message. In TypeScript and C# a
receive nobody awaits any more keeps running, and it takes the next message. Pass the limit to
recvorReceiveAsyncinstead. Rust and Python cancel the receive when the time runs out, so the message stays queued. - Two links that should talk never do. They were taken off two brokers:
LoopbackBroker::new()called twice rather than one broker cloned, or a secondnew LoopbackBroker(). Take every link off the same broker. - A link receives its own messages. A link subscribed to a topic it publishes on hears itself, as an MQTT client would. Keep commands and readings on topics of their own.
- Memory climbs over a long run. A link that subscribes and never reads keeps every matching message, because its queue has no limit. Read what a link subscribes to, or leave it unsubscribed.
- A reconnected link missed what was sent while it was away. Disconnecting drops what was queued for the link. Its filters stay, and apply again from the reconnect.
- Everything passes here and fails against a real broker. The loopback has no retained messages, no access control, and one copy per publish, as the table above sets out. Run the same code against a broker before it ships, which the MQTT guide does.
Where next#
- Simulators: Noisy and replay sensors, a recording actuator, a simulated robot that dead-reckons its pose, and a link that loses sends on a pattern.
- MQTT: An MQTT client with the topic and wildcard rules, acknowledged delivery, retained messages, a last will, and TLS, as the core transport.
- Your own device: A sensor and an actuator pamoja has never heard of, written against the core traits, run against a rule, and published with nothing plugged in.
- Also in Transports and testing: CoAP, Store and forward, Transport ladder, Event bus, Engine surface, Your own link.
Reference#
- Rust:
pamoja-loopback, install - TypeScript:
@pamoja/loopback, install - Python:
pamoja.loopback, install - C#:
Pamoja.Loopback, install