Using Events in Highly Distributed Architectures（待翻译）

https://msdn.microsoft.com/en-us/library/dd129913.aspx

Using Events in Highly Distributed Architectures

David Chou

Summary: SOA succeeded in getting loosecoupling at a technical level; now, let us go for the same at abusiness-process level.

Contents

Introduction
Event-Driven Architecture
Fundamental Principles
Potential Benefits
Aspects of Implementation
State of the Art
Conclusion
Resources

Introduction

Many service-oriented architecture (SOA) efforts today arefocusing on implementing synchronous request-response interaction patterns(sometimes using asynchronous message delivery) to connect remote processes indistributed systems. While this approach works for highly centralizedenvironments, and can create loose coupling for distributed software componentsat a technical level, it tends to create tight coupling and added dependenciesfor business processes at a functional level. Furthermore, in the migrationtowards real-time enterprises, which are also constantly connected and alwaysavailable on the Web, organizations are encountering more diverse businessscenarios and discovering needs for alternative design patterns in addition tosynchronous request-driven SOA.

Event-driven architecture (EDA, or event-driven SOA) is oftendescribed as the next iteration of SOA. It is based on an asynchronousmessage-driven communication model to propagate information throughout anenterprise. It supports a more “natural” alignment with an organization’soperational model by describing business activities as series of events, anddoes not bind functionally disparate systems and teams into the samecentralized management model. Consequently, event-driven architecture can be amore suitable form of SOA for organizations that have a more distributedenvironment and where a higher degree of local autonomy is needed to achieveorganizational agility. This article discusses how EDA, together with SOA, canbe leveraged to achieve a real-time connected system in highly distributedenvironments, and be more effective at understanding organizational state. Wewill then discuss a pragmatic approach in designing and implementing EDA, whileconsidering the trade-offs involved in many aspects of the architecture design.

Event-Driven Architecture

So, what do we mean by events? Well, events are nothing new. Wehave seen various forms of events used in different areas such as event-drivenprogramming in modern graphical user interface (GUI) frameworks, SNMP andsystem notifications in infrastructure and operations, User Datagram Protocol(UDP) in networking, message-oriented middleware (MOM), pub/sub message queues,incremental database synchronization mechanisms, RFID readings, e-mail messages,Short Message Service (SMS) and instant messaging, and so on.

In the context of SOA however, an event is essentially asignificant or meaningful change in state. And an event in this perspectivetakes a higher level semantic form where it is more coarse-grained, andabstracted into business concepts and activities. For example, a new customerregistration has occurred on the external Website, an order has completed thecheckout process, a loan application was approved in underwriting, a market tradetransaction was completed, a fulfillment request was submitted to a supplier,and so on. Fine-grained technical events, such as infrastructure faults,application exceptions, system capacity changes, and change deployments arestill important but are considered more localized in terms of scope, and not asrelevant from an enterprise-level business alignment perspective.

As event sources publish these notifications, event receivers canchoose to listen to or filter out specific events, and make proactive decisionsin near real-time about how to react to the notifications. For example, updatecustomer records in internal systems as a result of a new customer registrationon the Website, update the billing and order fulfillment systems as a result ofan order checkout, update customer account as a result of a back office processcompletion, transforming and generating additional downstream events, and soon.

Event-driven architecture is an architectural style that buildson the fundamental aspects of event notifications to facilitate immediateinformation dissemination and reactive business process execution.

In an event-driven architecture, information can be propagated innear-real-time throughout a highly distributed environment, and enable theorganization to proactively respond to business activities. Event-drivenarchitecture promotes a low latency and highly reactive enterprise, improvingon traditional data integration techniques such as batch-oriented datareplication and posterior business intelligence reporting. Modeling businessprocesses into discrete state transitions (compared to sequential processworkflows) offers higher flexibility in responding to changing conditions, andan appropriate approach to manage the asynchronous parts of an enterprise.

 

Figure 1. Request-drivenarchitecture (logical view)

 

Figure 2. Event-drivenarchitecture (logical view)

Fundamental Principles

The way systems and business processes are connected in anevent-driven architecture represents another paradigm shift from request-drivenSOA. Here, we will discuss some of the fundamental aspects.

Asynchronous “Push”-Based Messaging Pattern

Instead of the traditional “pull”-based synchronousrequest/response or client/server RPC-style interaction models, event-drivenarchitecture builds on the pub/sub model to “push” notifications out tointerested listeners, in a “fire-and-forget” (does not block and wait for asynchronous response), uni-directional, asynchronous pattern.

Autonomous Messages

Events are transmitted and communicated in the form of autonomousmessages — each message contains just enough information to represent a unit ofwork, and to provide decision support for notification receivers. Each eventmessage also should not require any additional context, or any dependencies onthe in-memory session state of the applications; it is intended to communicatethe business state transitions of each application, domain, or workgroup in anenterprise.

For example, an externally-facing Web application publishes anevent that a new customer registration was completed with these additionalattributes: timestamp, ID=123, type=business, location=California,email=[e-mail]; information that is relevant to the rest of theenterprise, and supports any process-driven synchronous lookups for additionaldetails to published data services from distributed systems.

Higher decoupling of distributed systems

This asynchronous message-based interaction model furtherencapsulates internal details of distributed systems in an environment. Thisform of communication does not need to be as concerned with some aspects ofrequest-response models such as input parameters, service interface definitionssuch as WSDL in SOAP-based communication and hierarchical URI definitions inREST-based communication, and fine-grained security. The only requirement is awell-defined message semantic format, which can be implemented as plain oldXML-based (POX) payload.

The event-driven pattern also logically decouples connectedsystems, compared to the technical loose-coupling achieved in therequest-driven pattern. That is, functional processes in the sender system,where the business event took place, do not depend on the availability andcompletion of remote processes in downstream distributed systems. Whereas in arequest-driven architecture, the sender system needs to know exactly whichdistributed services to invoke, exactly what the provider systems need in orderto execute those services, and depends on the availability and completion ofthose remote functions in order to successfully complete its own processing.

Furthermore, the reduced logical dependencies on distributedcomponents also have similar implications on the physical side. In synchronousrequest-driven architectures, connected and dependent systems are oftenrequired to meet the service-level requirements (i.e., availability andthroughput) and scalability/elasticity of the system that has the highesttransaction volume. But in asynchronous event-driven architectures, thetransaction load of one system does not need to influence or depend on theservice levels of downstream systems, and allows application teams to be moreautonomous in designing their respective physical architectures and capacityplanning.

This also means that changes to each connected system can bedeployed more independently (and thus more frequently) with minimal impact toother systems, due to the overall reduced dependencies.

In the case of an externally facing Web application that has justcompleted a new customer registration process, that information needs to becommunicated to an internal customer management system. The Web application’sonly concern is to publish the event notification according to previouslydefined format and semantics; it does not need to remotely execute a service inthe customer management system to complete its own processing, or bind thecustomer management system into a distributed transaction to make sure that theassociated customer data is properly updated. The Web application leaves thedownstream customer management system to react to the event notification andperform whatever it needs to do with that information.

Receiver-Driven Flow Control

The event-driven pattern shifts much of the responsibility of controllingflow away from the event source (or sender system), and distributes/delegatesit to event receivers. Event source systems do not need to explicitly definethe scope of distributed transactions and manage them, or connected systems areless dependent on centralized infrastructure components such as transactionprocess systems (TPS), business process management (BPM), and enterpriseservice bus (ESB) products. The connected systems participating in anevent-driven architecture have more autonomy in deciding whether to furtherpropagate the events, or not. The knowledge used to support these decisions isdistributed into discrete steps or stages throughout the architecture, andencapsulated where the ownerships reside, instead of burdening the transaction-initiatingsender systems with that knowledge.

For example, in the new customer registration event scenario, thecustomer management system that receives the event notification owns thedecision on executing a series of its own workflows, populating its owndatabase of this information, and publishing a potentially different event thatthe marketing department may be interested in. All these downstream activitiesrely on decisions owned and made at each node, in a more independent andcollective model.

 

Figure 3. Hybrid requestand event-driven architecture (logical view)

Near Real-Time Propagation

Event-driven architecture is only relevant when events arepublished, consumed, and propagated in near real-time. If this approach isapplied in passive event-processing scenarios such as batch-oriented fixedschedule, then its effectiveness is not much different from the traditionaldata integration techniques used today.

This is primarily due to a simple fact: Events that are “in-motion”are much more valuable than events that are “at rest.” Gaining accurate andnear real-time insight has been a complex and difficult challenge forenterprises; even with the advent of SOA, it is still relatively complicatedgiven the existing focus on integrating distributed processes in synchronousinteraction models. Event-driven architecture provides a comparatively simplerapproach to achieve near–real-time information consistency, as we just need toapply a different form of system interaction model using existing technologiesand infrastructure investments.

Potential Benefits

By now, some potential benefits of using an event-drivenarchitecture pattern may have become more apparent:

Effective data integration—Insynchronous request-driven architectures, focus is typically placed on reusingremote functions and implementing process-oriented integration. Consequently,data integration is not well supported in a process-oriented model, and isoften an overlooked aspect in many SOA implementations. Event-drivenarchitecture is inherently based on data integration, where the eventsdisseminate incremental changes in business state throughout the enterprise;thus it presents an effective alternative of achieving data consistency, asopposed to trying to facilitate data integration in the process layer.

Reduced informationlatency—As distributed systems participate in an event-driven architecture,event occurrences are enabled to ripple throughout the connected enterprise innear real-time. This is analogous to an enterprise nervous system where theimmediate propagation of signals and notifications form the basis of nearreal-time business insight and analytics.

Enablement of accurateresponse—As business-critical data propagates throughout the enterprise ina timely manner, operational systems can have the most accurate, current viewof the state of business. This enables workgroups and distributed systems toprovide prompt and accurate business-level responses to changing conditions.This also moves us closer to being able to perform predictive analysis in an adhoc manner.

Improved scalability—Asynchronoussystems tend to be more scalable than synchronous systems. Individual processesblock less and have reduced dependencies on remote/distributed processes. Also,intermediaries can be more stateless, reducing overall complexity. Similarpoints can be made for reliability, manageability, and so forth.

Improved flexibility—Discreteevent-driven processes tend to be more flexible in terms of the ability toreact to conditions not explicitly defined in statically structured sequentialbusiness processes, as state changes can be driven in an ad hoc manner. And asmentioned earlier, the higher level of decoupling between connected systemsmeans changes can be deployed more independently and thus more frequently.

Improved business agility—Finally,the ultimate goal of SOA implementations. Event-driven architecture promisesenhanced business agility, as it provides a closer alignment with operationalmodels, especially in complex enterprise environments consisting of diversefunctional domains and requiring high degrees of local autonomy. Also, businessconcepts and activities are modeled in simpler terms with responsibilitiesappropriately delegated to corresponding owners. This creates an environmentwhere decisions and business processes in one area are less dependent on theoverall enterprise environment, compared to synchronous request-drivenarchitecture’s centralized view of logically coupled, sequentially structuredbusiness processes, where one component is often dependent on a number ofdistributed components for its processing. The reduced dependencies allowtechnical and functional changes to be planned and released more independentlyand frequently, achieving a higher level of IT agility and, hence, businessagility.

Aspects of Implementation

At a high level, event-driven architecture models businessoperations and processes using a discrete state machine pattern, whereasrequest-driven architecture tend to use more structurally static sequential flows.This results in some fundamental differences between the two architecturalstyles, and how to design their implementation approaches. However, the twoarchitectural styles are not intended to be mutually exclusive; they are verycomplementary to each other as they address different aspects of operationalmodels in an organization. Furthermore, it is the combination of request-drivenand event-driven architectures that makes the advanced enterprise capabilitiesachievable. Here, we will discuss some of the major consideration areas, fromthe perspective of building on top of existing service-oriented architectureprinciples.

Event Normalization and Taxonomy

First and foremost, event-driven architecture needs a consistentview of events, with clearly defined and standardized taxonomy so that theevents are meaningful and can be easily understood—just as when creatingenterprise-level standards of data and business process models, anenterprise-level view of events needs to be defined and maintained.

Process Design and Modeling

Modeling business processes into discrete event flows is similarto defining human collaboration workflows using state machines. A key is todistinguish between appropriate uses of synchronized request-driven processesor asynchronous event-driven processes.

For example, one organization used the synchronous request-drivenpattern to implement a Web-based user registration process, in terms ofpopulating data between multiple systems as a result of the registrationrequest. The implementation leveraged an enterprise service bus solution, andincluded three systems in a synchronized distributed transaction. The resultingregistration process waited until it received “commit successful” responsesfrom the two downstream systems, and often faulted due to remote exceptions. Itwas a nonscalable and error-prone solution that often negatively affected endusers for a seemingly simple account-registration process. On the other hand,as discussed previously, this type of scenario would obtain much betterscalability and reliability if an asynchronous event-driven pattern wasimplemented.

However, if in this case the user registration process requiresan identity proofing functionality implemented in a different system, in orderto verify the user and approve the registration request, then a synchronousrequest-response model should be used.

As a rule of thumb, the synchronous request-driven pattern isappropriate when the event source (or client) depends on the receiver (orserver) to perform and complete a function as a component of its own processexecution; typically focused on reusing business logic. On the other hand, theasynchronous event-driven pattern is appropriate when connected systems arelargely autonomous, not having any physical or logical dependencies; typicallyfocused on data integration.

Communication

Event-driven architecture can leverage the same communicationinfrastructure used for service-oriented architecture implementations. And justthe same as in basic SOA implementations, direct and point-to-point Webservices communication (both SOAP and REST-based) can also be used tofacilitate event-driven communication. And standards such as WS-Eventing andWS-Notification, even Real Simple Syndication (RSS) or Atom feeds can be usedto support delivery of event notifications.

However, intermediaries play a key role in implementing anevent-driven architecture, as they can provide the proper encapsulation,abstraction, and decoupling of event sources and receivers, and the much-neededreliable delivery, dynamic and durable subscriptions, centralized management,pub/sub messaging infrastructure, and so on. Without intermediaries to managethe flow of events, the communication graphs in a point-to-point eventing modelcould become magnitudes more difficult to maintain than a point-to-pointservice-oriented architecture.

Event Harvesting

The use of intermediaries, such as the modern class of enterpriseservice bus solutions, message-oriented middleware infrastructure, andintelligent network appliances, further enhance the capability to capturein-flight events and provide business-level visibility and insight. This is thebasis of the current class of business activity monitoring (BAM) and business eventmanagement (BEM) solutions.

Security

Security in event-driven architecture can build on the sametechnologies and infrastructures used in service-oriented architectureimplementations. However, because of the highly decoupled nature ofevent-driven patterns, event-based security can be simpler as connected systemscan leverage a point-to-point, trust-based security model and context, comparedto the end-to-end security contexts used in multistep request-drivendistributed transactions.

Scope of Data

So, how much data should be communicated via events? How is thisdifferent from traditional data replication? Data integration is intrinsic toevent-driven architecture, but the intention is to inform other system“occurrences” of significant changes, instead of sending important data asevents over the network. This means that only the meaningful data that uniquelyidentifies an occurrence of a change should be part of the event notificationpayload, or in other words, just an adequate amount of data to help eventreceivers decide how to react to the event. In the example discussed earlierwith the customer registration event, the event message does not need tocontain all user data elements captured/used in the customer registrationprocess, but the key referential or identifying elements should be included.This helps to maintain clear and concise meaning in events, appropriate dataownerships, and overall architectural scalability. If downstream systemsrequire more data elements or functional interaction to complete theirprocesses, alternative means, such as synchronous Web services communication,can be used to facilitate that part of the communication.

State of the Art

Here, we will discuss several emerging trends that influenceadvanced applications of event-driven processing.

Complex Event Processing

So far, we have only explored using individual events as a meansof information dissemination and achieving enterprise-wide business stateconsistency across highly distributed architectures. Multiple events can beinterpreted, correlated, and aggregated based on temporal, historical, and/orcomputational algorithms to discern meaningful patterns. Basically, in additionto the information individual events carry, combinations (or the lack) ofmultiple events can provide significant meaning.

Similar capabilities can be found in today’s SOA implementations,but are typically implemented locally to a system examining a request againsthistorical and environmental data at rest, where visibility is limited and interpretationrules are encapsulated. But the use of service intermediaries providescentralized event management, higher visibility, and more relevantinterpretation of event streams. Also, as complex event processing often makesuse of rules engines to drive the pattern recognition behaviors, changes to howevents are processed can be deployed quickly, allowing business decisions to beimplemented quickly.

This information can also be derived into composite or syntheticevents to prompt downstream actions. For example, in practice today, complexevent processing is often used in fraud detection to identify patternconsistency (for example, banking transactions conducted in the samegeography), pattern anomaly (for example, a transaction requested from a differentgeography when other requests are consistently within the same geography), andeven lack of events. As soon as the fraud-detection rule identifies a potentialattack, the monitoring system can send that information as another event toalert other systems in the enterprise, where specific actions can be taken inresponse to this event.

Events in the Cloud

As we move towards cloud computing, event processing can also beleveraged to capture and correlate event streams running in the open cloud.However, unlike event-driven architectures within an enterprise, where thefocus is organizing business state changes in distributed systems into aconsistent view, integrating event streams from the public cloud follows a muchless structured form and approach, as the data and context in publiclyavailable events are highly heterogeneous. Information latencies are alsoexpected to be higher in the open cloud. From an implementation perspective,tapping into public event streams could be as simple as subscribing to RSS orAtom feeds directly from other Websites and organizations (including feedaggregators), direct point-to-point Web services integration (either viaSOAP-based SOA or REST-based WOA approaches), or facilitated through theemerging class of cloud-based “Internet service bus” intermediaries (such asMicrosoft’s Cloud Services platform) where message delivery, service levels,identity and access control, transactional visibility, and contextualizedtaxonomies can be more formally managed between multiple enterprises.Consequently, the interpreted information from cloud-based event streams can becorrelated against event streams internal to an enterprise. The result is ahighly reactive enterprise that is cognitive of market and environmentalchanges as they occur, in which business decisions (whether manual orautomated) can be made with significantly higher relevancy.

Context-Aware Computing

The next perceived step in business application architecture istowards context-aware computing. Context from this perspective refers toinformation that is not explicitly bound to the data elements of a particularprocess or transaction, but is relevant via implicit relationships. Forexample, geo-location, time (of the day, week, month, seasons, holidays,special occasions), real-life events such as presidential elections, Olympicgames (for example, subsequent potential impact on retailers and their supplychains), stock market conditions, interest rates, raw material over supply orshortage, new media blockbuster hit, new invention, and even weather, can allbe information (or event occurrences) that influence human behaviors anddecisions, which also convert into interactions with online systems. Thiscontextual information is useful as it provides additional decision supportfactors, but most importantly, aids towards understanding user intent, suchthat different responses can be generated for similar requests, influenced bydifferent contextual factors. As a result, context-aware systems are much moreconnected with the environment and thus can be more relevant, compared totoday’s systems that are mostly siloed within the limited views of their owndatabases.

Event-driven architecture can play a key role in facilitatingcontext-aware computing, as it is the most effective and scalable form of dataintegration for highly distributed architectures. As discussed previously inthis article, higher level decoupling of distributed systems and reduceddependencies in event-driven architectures can significantly simplify effortsin harvesting and interpreting business events within an enterprise, plustapping into massive amounts of cloud-based events. As a result, event-drivenarchitecture can help systems keeping an accurate pulse of its relevantenvironments, and becoming more context-aware.

Conclusion

Event-driven architecture brings about the convergence ofdisparate forms of internal and external events, such as system and ITmanagement notifications, business activities and transactions, public market,technical, and social events. The organization of these events into a unifiedand consistent view promises enhanced alignment between business and IT, andmoves towards a connected, dynamic enterprise that has accurate and consistentbusiness insight, and can proactively respond to changing conditions in nearreal-time.

However, asynchronous event-driven architecture is not intendedto be a replacement of existing forms of synchronous request-drivenservice-oriented architectures. In fact, they are very complementary to eachother when appropriately modeling the different aspects of an organization’soperational model. From a technical perspective, event-driven architecture canbe implemented on top of an existing service-oriented infrastructure, withoutadding more dependencies and complexities. An effective mix of botharchitectural patterns will create a highly consistent and responsiveenterprise, and establish a solid foundation for moving towards always-on andalways-connected environments.

Resources

“Complex Event Processing in Distributed Systems,” David C.Luckham and Brian Frasca, Stanford University Technical Report CSL-TR-98-754. (http://pavg.stanford.edu/cep/fabline.ps)

“Event-Based Execution Architectures for Dynamic SoftwareSystems,” James Vera, Louis Perrochon, David C. Luckham, Proceedings of theFirst Working IFIP Conf. on Software Architecture. (http://pavg.stanford.edu/cep/99wicsa1.ps.gz)

“Complex Event Processing: An Essential Technology for InstantInsight into the Operation of Enterprise Information Systems,” David C.Luckham, lecture at the Stanford University Computer Systems Laboratory EE380Colloquium series. (http://pavg.stanford.edu/cep/ee380abstract.html)

“Service-Oriented Architecture: Developing the EnterpriseRoadmap,” Anne Thomas Manes, Burton Group. (http://www.burtongroup.com/Client/Research/Document.aspx?cid=136)

“The Unification of SOA and EDA and More,” Randy Heffner,Forrester. (http://www.forrester.com/Research/Document/Excerpt/0,7211,35720,00.html)

“Clarifying the Terms ‘Event-Driven’ and ‘Service-Oriented’Architecture,” Roy W. Shulte, Gartner. (http://www.gartner.com/DisplayDocument?doc_cd=126127&ref=g_fromdoc)

“Event-Driven Architecture Overview,” Brenda M. Michelson,Patricia Seybold Group. (http://www.psgroup.com/detail.aspx?id=681)

About the author

David Chou is anarchitect in the Developer & Platform Evangelism group at Microsoft, wherehe collaborates with local organizations in architecture, design, andproof-of-concept projects. David enjoys helping customers create value by usingobjective and pragmatic approaches to define IT strategies and solutionarchitectures. David maintains a blog at http://blogs.msdn.com/dachou,and can be reached at [e-mail].

 

This article was published in the Architecture Journal, a printand online publication produced by Microsoft. For more articles from thispublication, please visit the Architecture Journal Web site.