Integration in the Modern World

Whenever Integration is mentioned, our minds automatically recall integration formulas in mathematics but that’s not the case here. In the modern world where everything is connected to the internet and as more and more applications are created every day designed to stay connected, integration plays a crucial role to let them communicate smoothly and share functionalities. When the integration is properly implemented, it guarantees improved product quality and provides value to the organization.

What is integration?

Imagine two people with different languages trying to communicate but each is using their own language along with some signs and facial expressions with no mean of success, this is where the translator comes in handy, in our case: Integration.

Integration in engineering is defined as the process of linking sub-systems to act as one system and make sure it works coherently, bringing together different computing systems and software applications to be linked physically or functionally, it may be existing systems or under development. It also involves creating a coordinated system with joined databases and data sources. Integration not only connects, but it also adds value through the new functionalities provided by creating communication channels between different systems’ functions.

There’s no conventional or one way for integration instead, you’ll find multiple methods, each differ according to the organization or software needs. Let us take a glimpse on the popular methods of integration today:

Vertical Integration:

Each sub-system is represented as an information silo that works independently and the integration works on connecting these silos according to their functionality. This method has several benefits, such as it doesn’t require extensive coding knowledge, it’s cheap in the short term, and it provides isolation to each sub-system by sending individual requests rather than staying connected. On the other hand, when scaling existing systems or adding new functionalities is needed the only possible way to do so is to add new silos and when adding more and more silos along the way, the system gets cluttered and slow.

Enterprise Service Bus (ESB):

Also known as horizontal integration as opposed to vertical, the service bus is a sub-system on its own that acts as a translator and communicator between sub-systems. Each sub-system only needs one interface in order to connect to the ESB which is capable of translating the interface into another interface thus achieving the communication needed.

This type of integration allows more flexibility in replacing a whole sub-system without having to modify anything in the other integrated sub-systems only implementing a new interface communicating with the ESB. The service bus may have a slower communication speed due to all the data it’s processing, also if something goes wrong with one of the sub-systems, it may affect the others.

Spaghetti Integration:

Also called star integration or Point to Point, this method depends on connecting each system independently to the other sub-systems allowing systems to communicate without the need of ESB, but in case one of the sub-systems needs to be changed all the other systems are modified accordingly.

Planning this method is complex to ensure that all is connected properly to avoid the high risk of problems when too many are connected.

Star integration is a great choice for small number of sub-systems.

Road to Agile Integration

The idea of integrating systems has been around for decades now, as long as IT systems developed and grew over time. A solution was founded to end the duplicate efforts and let the systems exchange valuable documents seamlessly: Electronic Data Interchange. EDI is one of the first in system integration technologies when there was no protocol to manage the exchange. It provided a set of software tools that manage the exchange of documents.

The journey of integration development started with the SOA concept, Service-oriented architecture. It was typically implemented in the form of ESB aiming to provide connectivity to backend systems over web services. But soon it got caught in some drawbacks: Centralized ESB patterns often failed to cut down expenses for the enterprise as few interfaces could be re-used and the creation and maintenance of interfaces were expensive for projects to take on and SOA was more complex than just the implementation of an ESB. Centralized ESB did serve its purpose to simplify consistency and governance of implementation. However, many organizations have more fluid and dynamic requirements to manage which lead to microservices architecture and agile integration.

Microservices architecture is a way of writing an application as a set of smaller (microservice) components in a way that makes that application more agile, scalable, and resilient. Instead of writing the application as a large bulk of code running on the same server, the application is divided into smaller independent running components.

Agile integration is a software development practice that depends on short, continuous code releases over longer cycles to minimize instability and complexity when releasing software. There are three main related yet separate aspects in agile integration: Fine-grained integration deployment, Decentralized integration ownership and Cloud-native integration infrastructure. Following the concept of microservices, breaking up the centralized ESB into smaller more manageable small pieces solved the bottleneck issue for projects all integrating on one server, providing specialized right size containers for each server which supports the continuous enhancement and deployment of integrations without affecting the others. This decentralization allowed the application team to own the creation, maintain their application and manage the exposure of their own APIs rather than having a central integration team knowing their integration technologies well but not understanding applications they were integrating, so meeting requirements may be slow. Integration runtimes have changed through the years to allow the lightweight runtimes to be used in cloud-native ways and taking the burden of cluster management, scaling, availability off our shoulders and to be managed by the cloud platform in which they are running.

The agile integration shed some light on the major benefits of Microservices architecture: Greater agility, Elastic scalability and Discrete resilience. Individual integrations can be changed and redeployed without affecting the other flows, as well as each flow can be scaled independently without stealing resources from other flows since all are deployed in separate containers, making use of elastic scaling in cloud infrastructure.

So where does the integration Engineer Stand from all this

The role of integration engineers is important: more and more apps are designed to connect, whether they’re under construction or already deployed and working. Integration engineers develop and implement solutions to coordinate applications across the enterprise and departments. Many organizations tend to use existing backends and systems, giving engineers the role of evaluating the existing components and the integration requirements. The solution can be vague, it may vary from creating custom software to developing custom hardware and machinery.

IBM Cloud Pak for Integration

IBM Cloud Integration brings all the major integration capabilities into a coherent simple platform. It allows easily building powerful integrations and APIs, provides leading performance and scalability, and offers end-to-end capabilities with enterprise-grade security. IBM Cloud Pak for Integration is built on the open-source Kubernetes platform for container orchestration. This platform checks all the points on the list of what integration engineers need:

• Application and Data sources connections in the cloud so that data is available whenever needed.
• API Life Cycle management as reusable APIs for select developer communities both internal/external to your organization
• Ensures real-time information that is always available by providing reliable message delivery without message loss or complex recovery in the event of a system or network issue.
• High-Speed Data Transfer Moving huge amounts of data between on-premises and cloud or cloud-to-cloud at high speed with enhanced levels of security.
• Secure Gateway to extend Connectivity and Integration beyond the enterprise.

Conclusion

Integration isn’t a new concept anymore, rather it has been around since the 70’s, but it plays a vital role now in the software development industry and the few paragraphs above is all what you need to know about it. Whether it’s only data integration, application data integrations or enterprise systems integration, all you need is a reliable fast and secure integration platform.