Architecture (shift to PostgreSQL)
This is the third of a four-part series of posts related to ChronosES, its implementation, and its usage.
- Basic usage (Hello, world!)
- In-depth usage (including a case-study of one of our own internal ChronosES applications!)
In this post, we’re going to detail how and why we recently changed the default ChronosES data store and describe the new ChronosES architecture.
Out with the old, in with the new
We previously listed some of the reasons behind our decision to use Redis as the default ChronosES data store; however, after using ChronosES in production for the majority of last year, we’ve realized that there are even better reasons to use PostgreSQL in its place:
- More reliable (all data is stored to disk, rather than memory)
- Cheaper to store large amounts of data
- Supports constraints
- Table partitioning keeps queries performant while data grows unbounded
- JSON support allows for flexible schemas
- Indices can be stored along with all of the other data
Changes to ChronosES architecture
Before diving into the specific changes that we made, it will be helpful to give a brief overview of the ChronosES architecture. We have mentioned that there are two main components, a server and a client. Additionally, the server itself can be broken down into two separate components, the Gateway and the ChronosProcess. The Gateway is the heart of ChronosES. It is responsible for communicating with clients and managing the individual processing threads associated with each Aggregate. Each Aggregate that is registered with ChronosES gets its own ChronosProcess, which is where all event processing takes place. This structure ensures that Aggregates are kept entirely separate, which increases performance and prevents issues with one Aggregate from affecting any others.
Now that we have a basic understanding of the ChronosES design, we can introduce some of the more advanced features of the architecture. From the beginning, ChronosES has been designed to be flexible to meet the user’s needs. While this quality has always been true in the form of ChronosES’s configurable plugins, the switch to PostgreSQL has allowed us to extend this functionality to include indexing. Let’s dive into both of these individually to see how ChronosES can adapt to the environment in which it is used.
To make ChronosES as flexible as possible, it relies on a suite of configurable plugins. These plugins allow ChronosES to use different technologies for storage, notification, and client/server interaction. Because we were previously using Redis for both storage and notification, one plugin was responsible for notification, storage reading, and storage writing. We now support a separate plugin for each of these responsibilities, making it easier to leverage different technologies for each component. This has the added bonus of aligning our architecture more closely with the tenants of CQRS. All told, there are configurable plugins for the LogicStore, TransportLayer, ServiceProxyManager, EventPersister, EventReader, Notifier, CrossAggregateAccess, CoreProvider, and the GatewayStore. More information about each of these plugins can be found in the source code documentation, found here.
Defining aggregate-specific indices and constraints is now supported internally by ChronosES, so Aggregate logic no longer needs to import SQLAlchemy. Every ChronosES Aggregate still requires an index definition in the python logic file and Chronos.Core.ChronosIndex is still the abstract base for all Aggregate indices. However, instead of being in the form of a SQLAlchemy declarative base class, index definitions must now comply with ChronosES’s own domain specific language. At the time of this post, ChronosES supports four Chronos.Core.ChronosConstraint types that may be used to specify Chronos indices: Unique, Index, NonNull, and NoCase. The ChronosConstraint base class requires that every constraint is uniquely named (for future identification) with exclusively alpha-numeric characters (to prevent SQL-injection). The only exception to this rule is that NoCase constraints need not be named. This deviation, in fact, underscores the inherent flexibility of having ChronosES’s own DSL for index definition that we will expand upon further after we take a closer look at the available types.
- Unique: Creates database unique constraints on the attributes provided. If more than one attribute is passed, the unique constraint will be on the composition of the keys. If it is desired to have multiple fields be unique individually, a Unique must be created for each attribute specifically.
- Index: Creates a database index to speed up queries on the specified attribute.
- NonNull: Creates database non-null constraints.
- NoCase: Specifies the values to be case insensitive. Typically NoCase constraints would be coupled with a Unique constraint to restrict attribute names without considering case. For example, if we had a bank account Aggregate with an ‘owner’ attribute designated Unique and NoCase and attempted to create two Aggregate instances with owners ‘JohnDoe’ and ‘johndoe’, we would get an error for violating our unique constraint. (Note: NoCase ChronosConstraints need not be named).
As alluded to above, there are significant benefits to having defined ChronosES’s own DSL. First, ChronosES no longer depends on SQLAlchemy and SQLite. This makes installation and deployment simpler, as one no longer needs to worry about third-party requirements nor deploy index requirements to /var/lib/chronos as originally instructed in the first ChronosES blog post. Second, ChronosES has become more extensible and flexible given that the user can now define their own ChronosConstraints in a way that was not possible with the previous installation. By simply implementing a concrete class in Chronos.Core that inherits from Chronos.Core.ChronosConstraint the user has the power to restrict their Aggregate’s attributes in any way they’d like. Over time, we anticipate adding more default constraint types that will make ChronosES even more configurable and easy to use.
With the new DSL, index definition looks cleaner than it did before. Attributes no longer need to be defined individually. Instead, they are now designated with the attributes field, which makes defining several attributes at once much cleaner. It is worth noting that both the attributes and constraints variables are expected to be tuples, so if your logic definition requires only one of either field, it must end with a trailing comma as seen below in the constraints field.
class BankAccountIndex(ChronosIndex): attributes = (‘owner’, ‘accountNumber’) constraints = (Unique('owner', name='ux'),)
In sum, ChronosES experienced some large scale changes under the hood that made the current version more configurable, extensible, and reliable. Thanks to the wonders of abstraction, clients only need to change the way they specify their constraints and indices in the aggregate logic. All other interfacing parts were kept exactly the same. As always, thank you for following ChronosES’s progress. We’re excited to share with you next time an in-depth usage of one of our own internal ChronosES applications.