CQRS - Command/Query responsability segregation (overview)

[Merrion]

What is CQRS?
CQRS is an architecture that recognises that the model used to display or query data differs to the model used to insert or update data. By splitting your application into separate legs - one for Command and one for Query - you can tune and scale each independently to match the usage characteristics of your application.

Other advantages:
1) Reduces the model contention that typically happens if you have a large team of developers working on the same application.
2) Insulates areas from each other - it is much less likely that a change in one area will impact another area.
3) Allows for more/easier testing as each module only needs the tiny part of the model it directly impacts to be mocked rather than the whole model.

CQRS is ideally suited to web applications because, down at the deepest level all web applications are sitting on top of a CQRS system :- HTTP. Matching your application architecture to this can significantly reduce the headaches of trying to bolt things like data concurrency onto an essentially stateless environment.

CQRS is nearly always paired with domain driven design (DDD) and event sourcing (ES) but this is not mandatory - you can build CQRS into an application that sits on a single SQL server database and uses fairly standard CRUD technologies and has a MVC front end if you so desire. However the true "enterprise" advantage comes from using these three technologies together.

Related concepts
* How cloud computing requires us to rethink application development

 

[Merrion]

Event Sourcing

Event sourcing is a way of storing data in your system, not as a point in time view but rather as a historical chain of events that have occurred to that object. using this history it is possible to get a view of the state of any given object at any given point in time by "playing" the history of events.

Getting your head around event sourcing

For developers schooled in the relational database model, event sourcing can seem to be a very confusing way of doing things.
Hopefully the following hints can help:
1.Events only get added to the end of the event list (you can conceptualise this as a stack that doesn't have a pop option)
2.Events are stored by the thing they occur to rather than the type of event that the are. For example we don't have a separate table for "payments" and "standing orders" in a bank account type of system - these are just different events that occur in the life of the bank account.
3.Events cannot be deleted - if you need to "undo" something that has happened a reversal or undo event needs to be added to the event store
4.Events are past-tense.

However, if you get your head out of the CRUD you will see some benefits


1.You automatically get a complete audit trail for everything that occurred (If you have fields like "Last Modified", "Updated By" in your database tables and a set of triggers that write to an audit table whenever a
field is updated in the database you are already doing this and doing it backwards)
2.You can query the event history in ways that weren't anticipated when the system was first created
3.Your event payload schema can be very flexible - if a particular attribute doesn't apply to a particular type of event you don't need to store a "null" there

 

[Merrion]

View attachment 135611 (Full CQRS on Azure framework)

View attachment 135021 (just event sourcing)

The attached code is the "framework" part of a CQRS / Event Sourcing system that is hosted on Azure Storage.
The backing technology for the event streams can be File, Blob or Table based - personally I recommend "AppendBlob" as it gives extraordinarily good performance.

To use this you need to create your classes that represent the aggregates (the thing that events can happen to), the events (the things that can happen) and projections (the views of what has happened to an aggregate).

 

[Merrion]

Choosing the backing store type to use

There are a number of different implementations I have included in the above code - and in fact there is no reason why you couldn't mix and match so different aggregates are stored using different technology. However as a rule of thumb I would recommend the following:-

Azure Tables

Azure Tables are a NoSQL type solution whereby a table can have different dynamic fields for each row, with only the partition key and row key (which uniquely identify the row between them) being common to all rows.

The easiest way to create an event stream on top of an Azure Tables solution is to use the partition key to hold the unique aggregate identifier and the row key to hold the event sequence.

Pros

There is reasonable tooling to allow the inspection of data within the event stream for diagnostic purposes.

If the partition key is chosen well the Azure Table can be made use of in a highly parallel system.

Cons

There is a restriction of 252 attributes available for any given event as there are only 255 fields in a table.

The data are stored as a name:value pair which means that you end up storing (and transferring) a large amount of data that describes the event. This adds to cost and reduces the data transport speed.

Azure Blob Storage

A binary large object is a way of storing data that is either unstructured or has implied structure. In implementing an event stream on blob storage I would recommend using a data structure that is common for the context of each event and then having the dynamic payload of the event follow that.

Microsoft have recently added an "append only" type of Blob which is perfectly suited to the needs of an event stream.

Pros

Each event stream can grow to 50,000 records and having a separate append blob for each unique event stream means a potential data size of 500TB is possible.

Cons

Because the data in the Blob are unstructured you need to create your own tools to investigate that data. (Typically this would be some form of binary serialisation/deserialisation)

Conclusions

I would use SQL Azure if the event streaming aspect were only a small part of a much larger data system and if there was no need for very highly parallel processing.

I would use Azure Table storage for hybrid cases, where there is a need easily to inspect the event stream data and also for prototyping systems.

I would use append blob systems where I wanted to maximise the system performance and scale to the very top end of the cloud based system size.