The objective of this post is to analyze the Integration Services to connect the IT components like Front to Back or Back to Back Systems offered by the four main providers of public cloud; AWS, GCP, Azure and Alibaba. In addition, the post will identify the use cases for each kind of Integration Service taking into account factors like latency, throughput, guaranty of delivery and more.
Integration Services
Integration Services are the services that allow the communication and orchestration among the IT components of one application. In a Cloud environment, those services are grouped under integration Platform as a Service (iPaaS) solutions.
In general, AWS, GCP, Azure and Alibaba have structured his Integration Services as follows:
- API Gateway or Web API
- API Management
- Message Queuing
- Event Engine
- Orchestration Engine
- Data Streaming
- Real Time Data Replication
API Gateway or Web API
API Gateway routing requests from clients to services. API Gateway allows support for stateful (WebSocket) and stateless (REST) APIs to call any Web Services or Microservices.
This is the basic communication that all the cloud providers offer.
API Gateway is a server that is the single entry point into the system and it might have other capabilities such as authentication, monitoring, load balancing and caching.
API Gateway can be used to call from one service to another; however under this case you need a low latency call especially when both services are in the same network (internal API-based microservice-to-microservice communication). In this situation the use of direct calls as gRPC (like Google offers) can improve the latency. A good Gateway service should incorporate this possibility for internal calls between services of the same system.
API Management
API Management is a full stack API Communication Platform more sophisticated and complete than an API Gateway.
With an API Management you will have out of the box the following additional capabilities:
- Internal and External APIs management
- SLA and Quotas depending of the client
- Security
- Authentication
- Authorization
- Role‑based access control
- Rate Limit
- Load Balance and High Scalability
- Traffic Management
- Monitoring (technical and functional)
- Analytics and Reports
- Monetize APIs
- Provide full life cycle of API development:
- Publish Portal for definition and publication
- Developer Portal for onboarding
API Management is especially useful when it is required to expose your services to the outside world or market. Some API Gateways implementations are very close to API Management so there is an overlap that generates confusion between API Gateway and API Management.
Message Queuing
Message Queuing allows the communication between IT Components in a loosely coupled way via asynchronous messaging. This service provides queues that hold messages until they can be picked up by the receiver. This lets applications and integration software communicate asynchronously, even across diverse technology platforms and protocols.
The concept of Message Queuing was born out of the need to move away from point-to-point synchronous integration, which becomes couple applications and does not scale because it creates rigid dependencies between applications.
Types of message queuing
- Point-to-point
One message is placed on the queue and one application receives that message. Messages accumulate on queues until they are retrieved by programs that service those queues.
In point-to-point messaging, a sending application must know information about the receiving application (the name of the queue to which to send the information) before it can send a message to that application.
- Publish/Subscribe
A copy of each message published by a publishing application is delivered to every interested application. There might be many, one, or no interested applications. In publish/subscribe an interested application is known as a subscriber and the messages are queued on a queue identified by a subscription. The subject of the information is identified by its topic.
Publish/subscribe messaging allows you to decouple the provider of information, from the consumers of that information. The sending application and receiving application do not need to know anything about each other for the information to be sent and received.
Message Queuing will provide the following capabilities
- Route messages between Queues
- Data Transformation
- Event queuing and sequencing
- FIFO ordering
- Basic Orchestration of messages
- Monitor and control routing of message exchange between Queues
- Control deployment and versioning of messages formats
- Implement multiple physical protocols with protocol conversion
- Scaling
- Message Consistency (transaction management)
- Security
Event Engine
In a message queuing approach the receiver of the message require to pull for new messages.
Under the Even Engine approach the receiver registers an event handler for the event source it’s interested in. The Event Engine then invokes that event handler when the specified event occurs (Push vs Pull in message queuing)
So the Event Engine is an implementation of the publish-subscribe model which subscriber services automatically perform work (push) in response to events triggered by publisher service.
More sophisticate Event Engines support Rules and Complex Event processing (CEP) that allows correlate events to identify patterns and takes specific actions all in real time.
In general, the event Engine don’t guaranty FIFO and occasionally duplicate messages at the subscriber end could happen.
In addition, all the Cloud Providers have a set of source events associate to his cloud ecosystem of services. This event can start another process or cloud service like a function.
Orchestration
The Orchestration allows automating a set of activities to perform complex process. There are four kind of Orchestration Software:
- Configuration Management (**to be covered in the Management post)
For maintaining computer Systems and Software in a known, consistent state. It is based on a scripting language like chef, puppet, and ansible. - Batch Processing
Batch process jobs that can run without any end-user interaction and can be scheduled to start up at specific time or whenever a condition occur. The jobs are processed in parallel over a pool of compute nodes. - Workflow Management (or BPM)
That define a workflow process model and execute instances of the model to generate tasks. A process model consists of a series of tasks and events from the start of the process to its termination points. The tasks could be assigned to a user or a group of users or automatize by a service and also flagged with a due date or start date. In addition, you also have a Report in real time of the status and KPI’s of all the instances of Workflow - Case Management
Case Management is an evolution of Workflow Management where interactions between people, process, data, and content can be dynamic, ad hoc, and unpredictable against the Workflow Management model that is perfect for linear processes.
Data Streaming
Data streaming is the process of sending data records continuously to a Data Lake, Storage o Database System. This may include a wide variety of data sources such as telemetry from connected devices, log files, e-commerce transactions, or information from social networks.
It is usually an event engine specialized in IoT or data transformation ready for a high flow of information.
Real Time Data Replication
Real Time Data Replication is the process of replicate the data as soon as it changes from a source to another Site or Database System.
Real Time Data Replication is useful in improving the availability data and performance. It can be use also to move legacy Data to a new Cloud Data Model during the coexistence of both platforms.
The Cloud providers offer real time replication for their databases in his cloud. For Databases out of the Cloud each Cloud provider offers a partial solution with only a subset of Databases.
On the other side, companies like Attunity (https://www.attunity.com/solutions/database/enterprise-data-replication/ ) offers a better portfolio of Database real time Replication but it is not managed by the Cloud Provider (at the moment).
However, for Cloud Storage , almost all the cloud providers offers a solution for real time synchronization with on premise or other sites.
In the post I will only analyze External Real Time Data Replication (with third party).
Integration Services Use Cases & Recommendations
Pros | Cons | Use Cases |
|
|---|---|---|---|
API Gateway | |||
API Management | |||
Message Queuing | |||
Event Engine | |||
Orchestration | |||
Data Streaming | |||
Real Time Data Replication |
AWS Integration Services
API Gateway and API Management
Amazon offers API Amazon API Gateway for both API Gateway and API Management.
The reality is that Amazon API Gateway was born as API Gateway and then Amazon has added API Management features, but it still does not have the API Management category. Capabilities like Onboarding Portal, API Monetization and Real Time Analytics are missing.
Furthermore, it is missing the possibility of invoking internal services within the same network with low latency (such as gRPC)
Message Queueing
Amazon offers two Message Queueing Implementations:
- Amazon SQS; a cloud native fully managed message queuing implementation
- Amazon MQ: a managed message broker service for Apache ActiveMQ that simplify the connection of your current applications to Amazon MQ.
Amazon SQS manages two types of queues:
- Standard queues that provide at-least-once delivery.
- FIFO queues provide exactly-once processing and the order in which messages are sent and received is strictly preserved.
Amazon MQ provides compatibility with many popular message brokers perfect for migrating applications from existing message brokers that rely on compatibility with APIs such as JMS or protocols such as AMQP, MQTT, OpenWire, and STOMP.
Event Engine
Amazon Simple Notification Service (SNS) is the serverles service for Events.
Amazon SNS can filter (a subscription filter policy) and fanout (replicated and pushed to multiple endpoints) events to the following destinations to support event-driven computing use cases:
- Amazon Simple Queue Service
- AWS Lambda
- Mobile Push
- Webhook (HTTP/S)
- Email or SMS
And almost all the internal AWS services can work as a publisher.
In addition, Amazon offers specialized event service for telemetry called AWS IoT Events that is a fully managed IoT service that makes it easy to detect and respond to events from IoT sensors and applications.
Orchestration
Workflow
Amazon offers two solutions for Workflow Orchestration:
- AWS Step Functions allows coordinate multiple AWS services into serverless workflows. You define state machines (in JSON) that describe your workflow as a series of steps, their relationships, and their inputs and outputs.
- Amazon SWF helps developers build, run, and scale background jobs that have parallel or sequential steps. In this case you need to code the logic.
As Amazon said; “You should consider using AWS Step Functions for all your new applications, since it provides a more productive and agile approach to coordinating application components using visual workflows. If you require external signals to intervene in your processes, or you would like to launch child processes that return a result to a parent, then you should consider Amazon Simple Workflow Service (Amazon SWF). With Amazon SWF, instead of writing state machines in declarative JSON, you write a decider program to separate activity steps from decision steps. This provides you complete control over your orchestration logic, but increases the complexity of developing applications. You may write decider programs in the programming language of your choice, or you may use the Flow framework to use programming constructs that structure asynchronous interactions for you.”
In addition Amazon offers a specialized workflow service; AWS Data Pipeline that helps you reliably process and move data between different AWS compute and storage services, as well as on-premises data sources, at specified intervals.
Batch
Amazon offers AWS Batch for batch processing at any scale.
Data Streaming
Amazon offers two solutions for Data Streaming to cover IoT and Data:
- AWS IoT Core is a managed cloud service that lets connected devices easily and securely interacts with cloud applications and other devices. AWS IoT Core can support billions of devices and trillions of messages, and can process and route those messages to AWS endpoints and to other devices reliably and securely.
The Message Broker is a high throughput pub/sub message broker that securely transmits messages to and from all of your IoT devices and applications with low latency. - Amazon Kinesis
- Amazon Kinesis Data Streams (KDS) is a massively scalable and durable real-time data streaming service. KDS can continuously capture gigabytes of data per second from hundreds of thousands of sources such as website clickstreams, database event streams, financial transactions, social media feeds, IT logs, and location-tracking events. The data collected is available in milliseconds to enable real-time analytics use cases such as real-time dashboards, real-time anomaly detection, dynamic pricing, and more.
- Amazon Kinesis Data Firehose is the easiest way to reliably load streaming data into data lakes, data stores and analytics tools. It can capture, transform, and load streaming data into Amazon S3, Amazon Redshift, Amazon Elasticsearch Service, and Splunk.
Data Replication
Amazon offers two solutions for real time replication to cover Storage and Databases:
- AWS DataSync is a data transfer service that makes it easy for you to automate moving data between on-premises storage and Amazon S3 or Amazon Elastic File System (Amazon EFS).
- AWS Database Migration Service helps you migrate databases to AWS, but also you can use AWS Database Migration Service for continuous data replication. Continuous replication can be done from your data center to the databases in AWS or in the reverse, replicating to a database in your datacenter from a database in AWS. Ongoing continuous replication can also be done between homogeneous or heterogeneous databases. Supported databases for CDC are SQL Server, Oracle and MySQL.
Others; Data Access
Amazon in addition offers as an integration Service; AWS AppSync that simplifies application development by letting you create a flexible API to securely access, manipulate, and combine data from one or more data sources.
Azure Integration Services
API Gateway and API Management
Azure unlike Amazon offers an API solution Management; Azure API Management for both API Gateway and API Management.
Azure is focused on developers and adds capabilities such as policy management, basic onboarding and SOAP support, but miss other required capabilities like API Monetization and Real Time Analytics.
Message Queueing
Azure offers Azure Service Bus as a Message Queueing Implementation.
In addition Azure has Azure Service Bus Relay for hybrid deployments (Connect your existing on-premises systems to cloud solutions)
Azure, unlike Amazon, offers a single product for both point to point and publish/subscribe scenarios.
Finally Azure Service Bus has more advanced features that Amazon that enable you to solve more complex messaging problems.
However, the Azure implementation seems to be less serverless than Amazon. You need to Partitioning your traffic across multiple queues to improve throughput, but you can use Partitioned queues or Azure Auto Scale.
MQTT is not supported in Azure Service Bus but it is covered in Azure IoT Hub
Event Engine
Azure Event Grid is the serverles service for Events.
Azure Event Grid can filter (a subscription filter policy) and fanout (replicated and pushed to multiple endpoints) events to the following destinations to support event-driven computing use cases:
- Azure Automation
- Azure Functions
- Event Hubs
- Hybrid Connections
- Logic Apps
- Microsoft Flow
- Queue Storage
- WebHooks (HTTP/S)
And almost all the internal Azure services (but less than Amazon) can work as a publisher.
As Amazon, Azure offers specialized event service for telemetry under the Azure IoT Hub Service.
Orchestration
Workflow
Azure Logic Apps is a cloud service that helps you automate and orchestrate tasks, business processes, and workflows when you need to integrate apps, data, systems, and services across enterprises or organizations.
In addition there is another product; Microsoft Flow that is built on top of Logic Apps focus on SaaS and not require an Azure subscription to build flows.
Azure Logic Apps focuses on Enterprise Integration.
Finally there are a third alternative; WebJobs feature of App Service to run a script or code in the context of an App Service web app.
Batch
Azure Batch to run large-scale parallel and high-performance computing (HPC) batch jobs.
Data Streaming
Azure offers a single solution for Data Streaming to cover both IoT and Data: Event Hub that is a fully managed, real-time data ingestion service that’s simple, trusted, and scalable. Stream millions of events per second from any source to build dynamic data pipelines and immediately respond to business challenges.
Data Replication
Azure offers multiple solutions for real time replication to cover Storage and Databases:
- SQL Data Sync that is a service built on Azure SQL Database that lets you synchronize the data you select bi-directionally across multiple SQL databases and SQL Server instances.
- Storage and files ongoing data ingestion
- Azure Data Factory – Data Factory should be used to scale out a transfer operation, and if there is a need for orchestration and enterprise grade monitoring capabilities. Use Azure Data Factory to set up a cloud pipeline that regularly transfers files between several Azure services, on-premises, or a combination of the two. Azure Data Factory lets you orchestrate data-driven workflows that ingest data from disparate data stores and automate data movement and data transformation.
- Azure Data Box family for online transfers – Data Box Edge and Data Box Gateway are online network devices that can move data into and out of Azure. Data Box Edge uses artificial intelligence (AI)-enabled Edge compute to pre-process data before upload. Data Box Gateway is a virtual version of the device with the same data transfer capabilities.
- Azure File Sync replicates files from your on-premises Windows Server to an Azure file share.
Unlike amazon, azure does not support continuous replication in his Azure Database Migration Service.
Google Integration Services
API Gateway and API Management
Google offers the best API management solution based on Apigee with all the capabilities required by a good API Manager; however the main drawback is the price of Apigee platform that up to date follows legacy market models (no pay per use model).
The good news is that unlike Azure, Google offers a cheaper alternative for API Gateway solutions; Cloud Endpoint that also supports gRPC for internal calls (mainly inter containers and microservices calls) with low latency.
Message Queueing and Event Engine
Google offers a single solution for both Message Queueing and Events; Cloud Pub/Sub supporting Point to Point and Publish/Subscribe under pull and push approach.
Cloud Pub/Sub provides a very scalable environment (Serverless) with low latency but it has a great disadvantage; does not support FIFO and occasional duplicates are to be expected.
As google says; “Cloud Pub/Sub serves as a foundation for modern stream analytics pipelines”. In this case the aforementioned drawback is not an issue, but in other use cases you need to receive the messages in the right order (for instance any financial transaction). Google offers advices to support the ordering of messages by following some design patterns, but in my opinion it should be an optional capability of the service (like Amazon or Azure) that will have less throughput.
As Amazon and Azure, Google offers specialized event service for telemetry under the Cloud IoT Core Service.
It is surprising that Google has made such a strong investment with Apigee to cover the Api Management solution and the message queuing option does not support FIFO as an option.
Orchestration
Workflow
Google offers Cloud Composer as a fully managed workflow orchestration service built on Apache Airflow open source project and operated using the Python programming language.
Batch
Up to date Google does not have a Batch solution. However for data transformation google offers Cloud Dataflow for transforming and enriching data in stream (real time) and batch modes.
Data Streaming
Google offers Cloud Dataflow as a fully-managed service for transforming and enriching data in stream (real time) and batch.
Cloud Dataflow supports fast, simplified pipeline development via expressive SQL, Java, and Python APIs in the Apache Beam SDK.
Data Replication
Google offers Cloud Data Transfer Service for data transfer needs and get your data on the cloud quickly and securely. However, Cloud Data Transfer does not support continuous data replication from/to external database.
If you want real time replication you should use an external solution like Attunity Replicate that supports data replication, ingest and streaming across a wide range of heterogeneous databases, data warehouses and Big Data platforms. In fact, Attunity are doing a relevant campaign on Google platform.
Alibaba Integration Services
API Gateway and API Management
Alibaba as Amazon offers only an API Gateway; Alibaba API Gateway.
Alibaba unlike Amazon don’t publish his API Gateway as an API Management solution but it is more than basic API Gateway covering API lifecycle management services, including API publishing, management and maintenance.
Message Queueing
Alibaba offers two Message Queueing Implementations:
- Message Queue is a distributed message queue service based on RocketMQ that supports reliable message-based asynchronous communication among microservices, distributed systems, and serverless applications.
- Message Service is a message queuing and notification service that facilitates smooth transfer of messages between applications
Both are very similar. Message Queue is more mature (used at the largest e-commerce site in Alibaba implementations) and have better performance but Message Service seems to be a more modern service including event/notification supporting both Point to Point and Pub/Sub with push capabilities.
Therefore Message Service is the solution that up to day is promoting Alibaba
Event Engine
Alibaba Message Service works also as an Event Engine supporting Pub/Sub with push capabilities.
Orchestration
Workflow
Up to date Alibaba does not have a Workflow Service
Batch
Alibaba offers Batch Compute to supports massive concurrent jobs.
Data Streaming
Alibaba Offers Realtime Compute a one-stop, high-performance platform that enables real-time big data processing based on Apache Flink. It is widely used in diverse scenarios, such as streaming data processing, offline data processing, and data lake computing.
Data Replication
Alibaba offers a solutions for real time replication to cover Storage: Cloud Storage Gateway (beta version) that is a gateway service that can be deployed on-premises data center or in the cloud. It uses Alibaba Cloud OSS as the backend storage and supports industry-standard storage protocols (NFS/SMB/iSCSI) and provides low latency performance by caching frequently accessed data locally.
Cloud Storage Gateway supports File and Block Gateway.
However, Data Transmission Service (DTS) to migrate data between data storage types, such as relational database, NoSQL, and OLAP supports continuous data replication but only among Alibaba RDS instances.
API Gateway and API Management comparison
Message queuing comparison
Event Engine comparison
Workflow comparison
Batch comparison
Conclusion
All cloud providers of the analysis offers Integration services in their portfolio, but with some differences and scope:
- AWS offers the most consistent integration services covering all the blueprint of integration architecture and even adding a service to access aggregate data.
However, AWS should improve his API Manager service so that it really covers all the expected capabilities as Onboarding Portal, API Monetization and Real Time Analytics.
In addition AWS should incorporate low latency calls (such as gRPC) for internal services calls. - Azure, thanks to Microsoft’s experience in the implementation of corporate architectures in large companies, offers a more integrated approach focused on the developer and hybrid deployments. Azure also offers the most complete external real time replication solution.
However, the API Manager solution is also not complete (missing API Monetization and Real Time Analytics) and some services are not serverless because they were not originally born in the cloud. - Google offers a combination of very good products and ideas with incomprehensible shortcomings. The good part is:
- The purchase of Apigge, the best market solution of API Management
- The possibility of low latency calls based on gRPC
- Serverless approach in all their services
- Commitment to Open Source implementations
However, other decisions are not understood, such as:
- Offer a legacy contract model for the use of Apigee
- Don’t support FIFO in the Message Queuing service
- Don’t offer a general purpose Batch service
- Don’t offer external replication services in real time (although it has a very good relationship with Attunity)
- Alibaba tries to follow the trail of AWS, but very far from the maturity of the AWS integration services. It has shortcomings in the API Manager, Workflow and external replication services in real time.
In all the cloud services analyzed, advanced integration functions are missing, such as:
- Complex event processing (CEP) support in the Message Queue service
- Case Management Service that are currently replacing traditional Workflows
- Business activity monitoring (BAM) services
- Advanced real-time data replication service as provided by Attunity
- Full API Manager support (like Google’s Apigee but with the option of payment for use)
- Offer low latency alternatives for the call of microservices within the same network
- Offer a general purpose Rule Engine service