What is Message Queue ?
A message queue is a form of asynchronous service-to-service communication used in serverless and microservices architectures. Messages are stored on the queue until they are processed and deleted. So what is queue and what is message can be ?
Queue
Queue is a line of things waiting to be handled, starting at the beginning of the line and processing it in sequential order. A Message Queue is a queue of messages sent between applications. It includes a sequence of work objects that are waiting to be processed. For example, imagine that you go to a coffee shop with a lot of customers waiting to order, and the staff tells everyone to line up to order. That line is a queue, when you go first you will order first, and the staff will processed your order like the service process the message, after the staff finished processing your order, you will move out the line, that like the service delete the message out the queue.
Message
Next message is a data transported between between two service, we can call two service is a sender and receiver, message can be a text, JSON,...
Message queue architecture
A Message Queue provides an asynchronous communications protocol, which is a system that puts a message onto a Message Queue and does not require an immediate response to continuing processing. Email is probably the best example of asynchronous communication. When an email is sent, the sender continues to process other things without needing an immediate response from the receiver and the sender can be receiver, the receiver can be the sender. But not some things not like a Email system, with email you can choose what email you want to read first, but with message queue you must choose the first email has come and process that, like example about the queue in the coffee shop.
Some type of Message queue
Point to Point: Message queues can be point-to-point, meaning we only have one queue , one sender and receiver to process messages in the queue.
Publisher-Subscriber: A message needs to be processed by more than one receiver, in publish/subscribe an receiver is known as a subscriber and the messages are queued on a queue identified by a subscription.
Next we will find more about Publisher - Subscriber
Publish/subscribe messaging
Publish/subscribe messaging is a asynchronous communication method in which messages are exchanged between applications without knowing the identity of the sender or receiver. In this arrangement, the publisher and subscriber rely on a message broker to send messages from the publisher to the subscribers. Messages (events) are sent out by the host (publisher) to a channel, which subscribers can join.
A typical publish/subscribe system has more than one publisher and more than one subscriber on many different topics, and often has more than one queue manager. An application can be both a publisher and a subscriber. A message in a publish/subscribe, where more than one subscriber has registered an interest with message, is processed by every interested subscriber.
Because there are many publishers and subscribers, the system does not know which messages sent from which publishers will go to which message queue, and which subscribers will receive messages from which message queue. To solve this problem, we need to provide addresses for both publishers and subscribers, and notify the message queue that it will receive and send messages to those addresses.
With a big system, we can have a million message in one day, so we need to add more queue to be saved the message. An important thing is the message can't be full, that very important, so if we have many message need to process, we need more queue and subscriber to process the message.
The pub/sub model makes the software more flexible. Publisher and subscribers are decoupled and work independently, allowing you to grow and scale independently. Adding or changing functionality won't have a ripple effect across the system because the pub/sub model gives you the flexibility to adjust how things work together.
References:
https://www.ibm.com/docs/en/ibm-mq/9.1?topic=overview-introduction-message-queuing
https://aws.amazon.com/vi/message-queue/
https://aws.amazon.com/what-is/pub-sub-messaging/?nc1=h_ls
https://www.ibm.com/docs/en/ibm-mq/9.1?topic=overview-publishsubscribe-messaging
https://www.enjoyalgorithms.com/blog/publisher-subscriber-pattern
https://ably.com/topic/pub-sub
https://www.youtube.com/watch?v=xErwDaOc-Gs&ab_channel=IBMTechnology
Tags:
mlops