Best way to handle multiple Messages (EDI or Standards-based) in Microservices

Typical Integration Scenario

Let's paint a picture of a typical Enterprise scenario where Applications are talking to each other (i.e exchanging data) over standardized messages.

This article intentionally stays at a high level, attempting to explain the microservices design without delving into serious technical/technological details. If you need consultation specific to your requirements, contact us at — info@bridgeapps.co.uk

There are many types of standard message formats (not all of them prescriptive), for e.g. EDI (in manufacturing + a few other domains), ACCORD (in Insurance), SWIFT messages (in the financial world), etc.

Each of these formats defines a series of message format types — for e.g. MT101 (Swift — request for transfer), MT102 (Swift — Multiple Customer Credit Transfer), etc.; the intention being to define/ prescribe / exchange data for a specific business scenario/use case, making it easier for the community to transfer data and perform business.

Conventionally, these message transfers have always been between Enterprise monoliths, many of whom are based on COTS — aka. commercial the shelf — software, which is adept at handling these messages — leading to customized workflows within enterprises.

Change in Scenario — Introducing Microservices

As more and more enterprise monoliths are replaced by a Collection of Microservices in enterprises, let’s discuss the ideal Design to support this case.

Microservices Design

The question of Microservices design is one of the key questions to answer once the decision to replace a Monolith with Microservices has been decided (partially or completely).

The right design for Microservices mostly depends on the following factors

1. Granularity at which change is expected
2. Backend Design, Data Consistency
3. Deployment Strategy — the amount of automation planned/implemented
4. Client access
5. Orchestration ability among Microservices, potentially also to create/expose Coarse-grained Microservices (from fine-grained MS)
6. Quality of Services (QoS) requirements include performance, resilience, and security.

Going with the above checklist, for the current scenario, making typical assumptions of — a single DB as a backend, fully automated Deployment strategy, file-based (or) API-based access, and no or very less orchestration requirements among different message types (assuming each message type is a part of its unique industry-wide workflow) — the ideal design would be — to create a Microservice per Message type.

As for interaction layers

• for messages inbound, each of these Microservices could be exposed via a shared Managed File Transfer (MFT) capability (or) API Gateway (or) a message listener — exposing the services via File/ API or Messaging modes; regarding inbound messages processing, the first step after message receipt is to perform message validation, which could be a separate module (within a multi-module service), with validation rules embedded. The options for validation could be to use a rules engine (which may be overkill ) based on the complexity of requirements — or — a self-created validation rules component. Once the inbound message is considered “valid”, then this message should ideally be “published” via an Async flow to be picked up further for executing further workflow/processing.
• for messages outbound, i.e. where our Microservice is expected to generate standard message types, once the sufficient inputs are collected, and messages validated and constructed as per data in the pipeline, the generated message could be “published” via messaging / file transfers or “sent” via API calls. it is a good practice for outbound messages to be validated before sending out, which ensures high success rates for messages (and hence business interactions).

This design provides enough flexibility for stable B2B interactions, offering agility for business change without impacting the entire end-to-end process, along with “growth support” for adding more (scaling) vendors & customers, with minimal changes.

Publishing APIs or clients — with some documentation — offers a really effective way to enable easier onboarding (client/vendor) with minimal effort on the Business side.

Design Pros

The following are the key benefits — technical and business — of this design -

1. Improved Agility (aka. much easier to Change) — In an Enterprise monolith scenario — a change to a message involves extensive testing and potential delays due to the risk involved with 3rd Parties, e.g. vendors and customers. With the use of a Microservices architecture, the updates are limited to just one message at a time, enabling faster deployments, rollback, and further fixes — ensuring a much faster Client Onboarding, and quicker changes rollout — leading to better business turnaround.
2. Improved Scalability — In a Monolith scenario, the entire application needs to be scaled up for increased volume — for e.g. during Christmas or peak business period. This is a quite significant increase in cost. Similarly, as more customers are onboarded, the application needs to consider additional inbound data volume, affecting other existing customers. With a Microservice architecture Implementation on a per-message basis — infrastructure scaling to each message microservice can be achieved independently based on individual message volumes, enabling — faster customer or supplier onboarding, and increasing message implementation breadth to explore newer markets.

Others include

1. Technology Agnostic Implementation — allowing reuse of existing capabilities and newer microservices, maximizing ROI
2. Future-Proofing — allowing specific implementations to be replaced by COTS or other implementations.

Design Cons

One of the key concerns with this implementation could be the orchestration of services for multiple messages for the same use cases and its efficient rollout.

Others -

1. A large number of Microservices
2. There is a higher chance of failure during communication between different services.
3. The complexity introduced in distributed environments — e.g. Testing, transactions management. etc.

These cons are not specific to just this implementation, but to Microservice architecture in general, and there is enough literature around to counter this.

Summary

This article discusses a good design for handling multiple messages scenarios with a Microservices implementation. The focus of this article goes beyond the technical implementation into the Business benefits.

Originally published at https://bridgeapps.substack.com on June 19, 2022.