First things first: What is a Topic?
Publish-Subscribe (PubSub) architectures are designed to create a separation between data publishers and consumers. Using this approach rather than publishers sending data to specific consumers, as you would when sending an email, data is sent with tags that allow for consumers to find the information based on interest. The data tagging utilized by many middleware providers is called a Topic. For consumers to receive messages, they need to register their interest in one or more topics – like choosing to follow someone on Twitter. When dealing with Solace Messaging Appliances, the appliance acts as the broker that keeps track of everyone’s interests, and does the high performance matching to make sure that everyone gets each message that they are interested in.
Topics & Message Routing
A topic is a string or a sequence of strings separated by a delimiter. Different Messaging systems use different delimiters, the most common being dots (.) or slashes (/). In their simplest form topics can be thought of as something akin to the name of a TV channel where both broadcasters and subscribers can find the same topic simply by using the same name.
Some examples for topic names are:
- foods/waterloo/plantfloor/<machine id>/<sub system id>
In more complex situations wildcards allow receiving applications to subscribe to large swaths of a topic namespace with a single subscription. Much like delimiters, the wildcards you’ll have access to depend on which messaging middleware you’re using. Solace Messaging Middleware currently supports the following wildcards:
- ‘*” replaces any part or whole level identifier
- “>” replaces any number of level identifiers
As an example, imagine a food processing plant that has 100’s of machines, each identified by a <machine id> and each machine has several sub systems each identified by a <sub system id>. As you can imagine, the overall dashboard for KPIs of the plant would want to subscribe to something like:
Which would allow it to subscribe to every update from every machine on the plant floor. Whereas a dashboard that was responsible for just a subset of the plant floor like packaging, might want to use (assuming packaging machine id’s all started with 115:
This would allow the packaging dashboard to receive every published message from every machine who’s ID started with 115 – which should map to our packaging machine ids.
As you can see, being able to apply wildcards efficiently can depend on having a logical topic hierarchy that has taken into account possible uses of the information that is being published. Had my mythical food company just sequentially numbered each machine id, there would be no way to use wildcards to subscribe to just a portion of the data flow. This can lead to long lists of topics that have to be subscribed to individually – which slows down product development, opens the door for human error, and increases time required for testing and maintenance.
So What’s a Topic Namespace?
Topic namespaces are the naming conventions and policies that define how topic names are assigned and used. Topic namespaces are important because they dictate how users interact with the messaging layer, and how the organization manages and governs topics. Namespace governmance is a complex and important concept that I’ll address in a future article. (update: that post is now available)
In this blog post, I will be discussing a Solace feature called “Topic Subscription on Queues” and explore how this feature can be used to meet some complex messaging requirements elegantly. I will begin by covering few basic concepts, discuss the feature in detail and conclude with a list of benefits this feature offers.
The Messaging Model divide
JMS defines two distinct Messaging models : Point to Point and Publish/Subscribe. Point to Point (P2P or PTP) is built on the concept of message queues with publisher addressing messages to specific Queue and the receiving clients extracting messages from the queue. The Queue acts as a holding area and messages are retained in the Queue until they are consumed or expire. Queues remove the timing dependency between senders and receivers. As the name Point-to-Point implies, each message can be consumed by only one consumer, though multiple consumers can bind to a queue.
With Publish/Subscribe (PubSub), messages are addressed to Topics and multiple consumers can subscribe to and receive the messages. The messaging system takes care of distributing the messages to subscribed consumers. Except for durable subscribers, messages are not retained for inactive or slow subscribers. (More on durable subscribers later).
As a messaging application developer, you have to pick your side: PubSub for best effort fan-out subscriber use case or P2P for consumers requiring guaranteed delivery. But what if you wanted the best of both these messaging paradigms in your application? What if some of your subscribers require delivery guarantee while others don’t and you want publishers to be unaware of this dependency?
Now a note of caution before we proceed: To keep the discussion simple and the blog post focused, I am leaving out some related, but important JMS and Solace features deliberately. For e.g., in addition to P2P and PubSub, there are other message models such as Request/Reply; there are topic endpoints in addition to Queues; there are subtle but important differences in the way JMS/Solace delivery types (Persistent/Non-Persistent/Direct) are mapped to Solace transports and so on. We will set them aside for now, but you are encouraged to explore them in Solace Documentation page.
A Simple Application
Before we answer that question of mixing messaging models, let’s see how these messaging models differ with a very simple hypothetical application.
Simple P2P Application
Let’s consider a P2P application with three publishers, queues and consumers. Let’s assume that not all consumers are live: C2 is slow to consume and can’t keep up with the publish rate; C3 is offline can’t receive messages now.
With P2P, messages published to a queue are persisted by Solace on the corresponding Queue endpoints. Irrespective of the consumer status (Live, slow and offline), there is no message loss to any of the consumers. Messages will be delivered to each consumer as and when they are ready to consume. As long as the Queue quota is not reached, messages keep accumulating on the endpoint until they are consumed or expire. To keep the discussion simple, queue properties that influence this behavior such as queue access-types (exclusive or non-exclusive), expiry, max redelivery are not discussed here.
Simple PubSub Application
Let’s now consider a simple PubSub application with two publishers (P1 and P2) that publish on two topics (Topic1 and Topic2). To illustrate the fan-out nature of PubSub, there are more than one subscribers per topic. Subscriber S3 also subscribes to both topics. To keep the discussion simple, none of the subscribers are assumed to be durable.
As messages are published to each topic, they are fanned-out to all subscribers. However, messages are delivered only to subscribers that are live. Since S2 is slow to consume, it’s likely to lose some messages; S3 and S5 will lose all messages during the time they are offline.
Mixing PubSub and P2P
Let’s assume in “Simple PubSub Application”, we want to make S2, S3 and S4 guaranteed consumers. What could be done to make them immune to message loss?
One option is to define Queues for each of these subscribers requiring guaranteed delivery (S2, S3 and S4) and have the publisher double publish to a topic and respective queue as below:
Though this may be a workable solution, its less desirable for the following reasons:
- This adds more administrative overhead since a new queue needs to be created for each consumer/topic pair. For eg, in the case of S3, we will need to define two queues one each for Topic1 and Topic2.
- Publisher now has the additional workload of publishing to Topic as well as potentially multiple queues. As we add more consumers with guaranteed delivery requirement, this complexity increases.
- This also makes the publisher and subscriber tightly coupled that goes against one of the core messaging principles (publishers and subscribers should be loosely coupled)
Topic Subscription on Queues
Solace offers an interesting feature called “Topic subscription on Queues” which allows a set of topics to be added as “subscriptions” on Queues. This feature expands the utility of Queues to beyond being a mere P2P destination. Now Queues can receive and persist messages published to all matching Topics it is subscribing to, in addition to receiving and persisting messages published to the Queue destination. This feature allows any number of topics to be added as subscriptions on queues and supports topic wild carding as well.
With this feature in hand, each subscriber requiring guaranteed delivery (S2, S3 and S4) will define its own queues (Queue2, Queue3 and Queue4) and add corresponding topics as subscriptions on queues. Messages for these subscribers (S2, S3 and S4) are now safe stored even though they are published on topic destination.
And what’s interesting is that the publisher need not be aware of this consumer end differences and there is no change to the publisher end. This feature automatically “promotes” messages to guaranteed delivery for these (and only these) subscribers requiring delivery guaranty. Regular topic subscribers (S1 and S5) will continue to subscribe to the topic with no delivery guarantee. A related concept is message “message demotion” where publisher publishes messages with persistence, but the messages are demoted as “direct” if there are no subscribers requiring guaranteed delivery. Please see Message Promotion and Demotion for more info on this topic.
Adding topic subscriptions on Queues can be done either administratively (SolAdmin / CLI / SEMP) or programmatically from the application. This is covered in Topic to Queue Mapping Getting Started Guide.
What about Durable Subscribers?
Since JMS Durable subscribers do offer Queue like message reliability to topic subscribers, using that would be another option. If the consumer is interested in only a single topic (e.g S2 and S4 in the example above), they can create a Durable Subscriptions to topics. This is offered in Solace as well with Durable topic endpoint (DTE). A detailed discussion of the differences is beyond the scope of this blog. But it’s important to recognize that DTEs differ from Queues in features and behavior. For E.g:
- DTEs support a single consumer while multiple consumers can bind to Queues
- Durable endpoints are limited to a single topic subscription. Queues allow multiple topic subscriptions as well as topic wildcards.
- Any subscription change (eg: message filters) with Durable endpoints deletes the messages.
- Message filters are applied inbound with DTE while they are applied outbound with Queues. This means any messages not matching the filters will be discarded with DTEs while they are retained on the Queue.
This elegant solution makes the overall application design less complex and cleaner. The burden of managing the subscriptions and persistence requirements shifts to the consumer and the middleware domain. Publishers remain unaware of how the topics are subscribed to and consumed. For e.g., if Subscriber S3 decides not to receive Topic2, that subscription can be removed from Queue3 with no impact to publishers. Similarly, if a new consumer is to be added in future or additional topic needs to be added to existing queues, this can be done with no impact to the publisher.
Similarly, when topic subscriptions change on the Queue, there is no impact to the consumers either. All messages published on subscribed topics will be delivered to the consumer in the order in which they are received. This is true even when there are multiple or wildcard topic subscriptions on the Queue
Further, if the consumer requires to know the original topic where the messages were published to, this is possible too. The original destination (topic) where messages are published is available in the message header and the consumer can make use of that info.
This feature, combined with the topic hierarchy and wildcarding supported in Solace gives rise to very clean application design for some of the complex message exchange patterns. As long as the topic structure is rich enough, the complete de-coupling of publishers and consumers on destination and delivery type can “future-proof” the publisher from downstream changes.
Until next time!
Thanks for reading and I do hope this blog provided sufficient info to help you understand and appreciate the “Topic Subscription on Queue” feature. I will discuss few use-cases with samples in a follow-up blog. In the meanwhile, you are welcome to explore Solace documentation page for additional info. If you have any comments or Questions, please feel free to use “Leave a Reply” box below, or use our Developer Community.