I was faced with a problem recently. Develop a solution that can accept a customer transaction from almost any continent, at any time, from multiple servicing applications (existing and envisioned) and keep it as simple and inexpensive as possible. Oh, and since this is a feed from the ordering services applications (read revenue generating) to back-office billing and General Ledger (GL), guarantee the transactions are processed and not lost should outages occur. This can be (and was) a daunting task considering that it needs to support many hundreds of thousands of transactions per day.
At first I thought of creating a network of local servers, networked and coordinated through a complicated series of scheduled processes to extract, merge, sort, process, translate and transfer this order information and somehow get it delivered to the back office on time and guaranteed. Once I started to map out the servers, networks, and schedules, I soon discovered given the number of servers needed and the complications in the scheduling that A) inexpensive was not going to be achieved; and B) simple this was not. In addition, having all these moving parts means that there were a lot of potential points of failure and when possible I prefer to reduce as many explicit file transfers as possible since more than once a failed password/login has sunk an entire schedule.
After wracking my brain and doing some research, I decided to use Message Queues as the method of moving data from point A to point B.
For those not familiar with the technology, Wikipedia describes Message Queues as:
“… an asynchronous communications protocol, meaning that the sender and receiver of the message do not need to interact with the message queue at the same time. Messages placed onto the queue are stored until the recipient retrieves them.”
To use the email analogy, Message Queuing is akin to sending an email. When you send an email, you don’t worry about if the recipient is online and ready to receive your message. You send the message and are pretty sure that when they login to their email, the message will be there and available.
This was perfect. It meant that I could set up a single central collection point of data, make that point accessible to only those services involved in customer ordering and guarantee that the back office billing/GL processes would receive it. On the back-end of this Message Queue a process was developed to listen for when mew messages came in and to process them into Billing/GL so that revenue could be realized. Sounds simple? Well it was.
The only other decisions that needed to be made were:
- Which Message Queue software to use? There are many enterprise supporting products out there both for purchase and open source.
- Determine how the data is to be configured. This was handled by defining a number of XML message formats recognized by the receiving process.
Implementation was as easy as:
- Setting up a couple of servers (load balancing/fall back) where the message queues were defined and storage was available.
- Insuring that all firewall rules were in place as needed, this is a multi-continental endeavor.
- Installing a queuing client on the application servers.
- Changing the supporting data source systems to build the XML message and post it to the queue. Changes in most cases were trivial.
- Building the listener application that pulls the message from the queue and transforms it into data streams fed to Billing and GL.
This simple yet powerful solution service has been running without issue and more that 70 million customer orders have successfully made it from the customer servicing applications to back office and GL. In addition, expanding this process to new services that have been developed since its inception has been painless.
FYI Solutions has been a leader in staffing and solutions for over 30 years. For more information about message queues, contact FYI Solutions.