AWS — When to use Amazon Aurora instead of DynamoDB

Amazon DynamoDB as managed database will work for you if you prefer code-first methodology. You will be able to easily scale it if your application inserts data and reads data by your hash key or primary key (hash+sort key). It is also good if your application is doing some queries on the data as long as the resultset of these queries returns less than 1Mb of data. Basically if you stick to functionality that is typically required by websites in real-time, then DynamoDB will perform for you. Obviously you will need to provision the reads and writes properly and you will need to implement some auto-scaling on DynamoDB WCUs and RCUs, but after you do all of the homework, it will be smooth for you without needing to manage much.

However, there are cases when you will need to go back to relational databases in order to accomplish your business requirements and technical requirements.

For example, let’s assume that your website calls one of your microservices which in turn inserts data into its table. Then let’s assume that you need to search the data in this table and perform big extracts which then have to be sent to a 3rd party that deals with your data in a batch-oriented way. If you need to for example query and extract 1 million records from your DynamoDB table, it will take you up to 4.7 hours based on my prototypes using standard AWS DynamoDB library from Python or C# application. The way you read this amount of data is by using LastEvaluatedKey within DynamoDB where you query/scan and get 1Mb (due to the cutoff) and then if the LastEvaluatedKey is not the end of resultset, you need to loop through and continue fetching more results until you exhaust the list. This is feasible but not fast and not scalable.

My test client was outside VPC and obviously if you run it within the VPC, you will almost double your performance, but it comes to bigger extracts, it still takes long. If you are dealing with less than 100,000 records, it is manageable within DynamoDB, but when you exceed 1 million records, it gets unreasonable.

So what do you do in this case? I am sure that you can improve the performance of the extract by using Data Pipeline and similar approaches that are more optimized, but you are still limited.

Basically, your solution would be to switch to a relational database where you can manage your querying much faster and you have a concept of transaction that helps with any concurrency issues you might have been challenged with. If you want to stay within the Amazon managed world, then Amazon Aurora looks very attractive. It has limitations on the amount of data, but most likely those limits are not low enough for your business. As for the big extract performance challenge, your extracts will go from hours (within DynamoDB) to minutes with Aurora.

Please consider this in your designs. Performing big extracts is opposite of the event driven architecture, but these type of requirements still exist due to a need to support legacy systems that you need to interact with or systems that have not adjusted their architecture to your methodologies.

Thank you for reading.

Almir Mustafic.

Knowing when to put code into your API/microservice vs. outside API (wrapper microservice)

As you are defining the purpose of each one of your microservices, and as you are developing these microservices, it is very important to know if you need to make changes to an existing microservice, or you need to build a wrapper microservice.

What I am talking about here are not rules; I am just giving you some examples so it can trigger you to think about this while you are going through the similar exercise. I just want to make developers aware of the differences.

So the main question is:

• Put code into an existing microservice/API
• OR develop a wrapper microservice API?

Let’s assume you have a “Package” microservice/API and this API gives you the list of packages and their details:

• /api/packages (this gives you all the packages)
• /api/packages/{id} (this gives you a specific package based on that ID)
• /api/packages?type=type101 (this gives you all the packages of type=type101)

Let’s assume that behind this microservice you are using a NoSQL table “Package” and that table has the packageId as the hash key of the table and then you have a bunch of other attributes that you are saving as part of the JSON object.

Let’s say that this API has been used in your production system for many months. Then you get a requirement from your product team that they want to introduce the concept of “Package Bundles”. A package bundle would be just a collection of different packages. For example, you can create the Bundle 1 that contains:

• package 1
• package 2
• package 3

Whereas the bundle 2 could contain:

• package 1
• package 7
• package 8
• package 9

Ultimately the product team instead of presenting individual packages to customers and having them pick one by one, they want to also present some pre-configured bundles of packages to customers to simplify their experience.

Now the big question is: How do you as a software engineer and solutions architect design this, or in other words how do you incorporate this requirement into your existing design of your microservices?

Before I get into the design, one unwritten rule in software development is that you need to keep things backwards compatible regardless of what type of design you come up with.

Let’s get into the design a bit. If you have been working in monolithic applications for many years (as many of us have), you would be naturally inclined to open up the existing “Package” microservice and you could be possibly introducing the list of bundle IDs in the existing data of the package. Or you could be adding another table within the Package microservice to introduce the concept of bundles and that database structure would be leaning towards a relation DB design. I talked about NoSQL and avoidance of relation DB design in my YouTube video: https://www.youtube.com/watch?v=49YvbcqjEiA

As you get deeper into this approach, you can realize that you are polluting the purpose of the Package microservice and you are slowly turning it into a macro-service. I have a full post about “micro” in micro-services: https://medium.com/@almirx101/micro-in-microservices-677e183baa5a

You also can realize that the API routes cannot stay RESTful. How do you actually search to get the bundles using /api/packages API route?

That’s when you conclude that you need to change your design.

The solution that I recommend for this specific requirement or use case is to introduce a new “PackageBundle” microservice. I really classify this as a wrapper microservice. This PackageBundle microservice would have the following type of API routes:

• /api/packagebundles (this gives you all the package bundles)
• /api/packagebundles/{id} (this gives a specific package bundle by id)
• /api/packagebundles?package=packageA (this gives you the list of bundles that contain packageA)

This microservice would have a NoSQL table with the following data structure:

- packageBundleId   (hash key for the table)

- packages [ ]   (list of packages)
   - packageId
   - packageName
   - other package attributes

- Other attributes needed to represent the bundle

In JSON, this would look like this:

{
  "packageBundleId": "",
  "packages": [
    {
      "packageId": "",
      "packageName": "",
      "packageDescription": ""
    },
    {
      "packageId": "",
      "packageName": "",
      "packageDescription": ""
    }
  ],
  "otherPackageBundleInfo": ""
}

Now let’s visualize how package bundles would be created. Imagine a business-enablement tool that allows you to create a bundle and then let’s assume that you drag packages into a bundle. Behind the scenes, that tool would be calling the Package API to retrieve the details on the package that is dragged into a bundle and then as the final step the tool would be calling the PacakgeBundle API to save all those details into the record for that given bundle. That’s what this business-enablement tool would do. On the other hand, the main website that customers are using would be calling the /api/packagebundles API to get a list of bundles or a specific bundle to display on the screen for customers.

With this approach you kept your existing microservice as micro and you introduced a new microservice that has a sole purpose to manage collection of packages. If I wanted to create some new packages, I am not dependent on the PackageBundle API; I can directly call the Package API to perform the actions. If the product team tomorrow decides that the concept of bundles is not needed any more, then I can just stop using the PackageBundle API and get rid of it in the long term without any impact on the Package API.

I hope this example can help you save some time if you have similar type of requirements and design.

Thank you for reading.

Almir Mustafic.