Top 25 things to achieve Enterprise-Level Microservices — What your team should accomplish

Developing applications / microservices is a definitely a skill that your team needs to gain, but what does it mean when you say “I am done”. What is the definition of “DONE”?

A lot of times you hear from us software engineers the following statements:

• “I am done coding that part but I still need to wrap up my unit-testing”.
• “My user story is done, but we just need to finish writing the automation tests.”
• “I am done, but I just need to add better logging in the code and add some better exception handling.”
• “I am done, but I have not deployed to QA environment yet; it works on my machine.”

We all have been in these shoes. We need to step back and look at this from the software engineering 10,000-foot level and truly understand what it means to deliver something and call it done and be enterprise worthy at the same time.

Here is my recommended list of things that your TEAM needs to accomplish in order to truly have production and enterprise worthy applications/microservices:

(1) Architecture diagram representing interactions of your family of microservices:

After going through the design and contract definitions of your APIs/microservices, your team needs to put together a single diagram that explains the interaction among microservices (owned by your team) and how these microservices interact with other teams’ microservices. This is basically the blueprint for your team’s work and this picture needs to be constantly updated as you are evolving your services.

(2) Service names, boundaries (domain lines) and API definitions/standards:

Names mean things. So you first need to properly name your services and that’s the names that you would use when talking to your teammates and clients of your services/APIs.

I have a separate article on how you go about defining what a microservice is: https://medium.com/@almirx101/micro-in-microservices-677e183baa5a

Essentially, you need define the purpose and boundaries of your service.

Then you get into API routes and properly defining them for each service. The goal is to keep the routes RESTful and if you run into the situation when they are not, then it should trigger you to revisit the purpose and boundaries of that given microservice.

(3) Application CONFIG file defining your service:

The assumption would be that your platform is set up in such a way that you as the owner of a microservice would properly set up your application CONFIG file which defines your microservice and all the resources that it needs. This config file defines anything from what database you need, database details, what cache you need, what queues you need, how much memory and CPU you want your application to use, what physical server you want your application to run on, and etc.

Basically this config file is the contract between you (as the owner of a microservice) and DevOps scripts/tools that take this config file and use as the definition to deploy your service into a given environment whether it is in the cloud or on-prem.

You own this config file but with ownership comes responsibility. It is important that this config file and code-reviewed properly and managed properly to keep the stability of your application.

(4) API Dependencies (Direct or via Pub/Sub approach):

As you are defining your microservices and API contracts, it is important that you explicitly think about API dependencies. Think about the type of dependency you want to achieve. You can directly call another microservice via http call (sync or async), or you can use the publish/subscribe methodology. It depends on your use case and there is no right or wrong here. The important thing is that you give it an explicit thought as you are designing your family of microservices and as you are designing the interaction among them and interaction with outside services. Ideally you could have this API dependency definition within the config file mentioned above.

(5) API Dependency Analysis Tool:

As you have all these API dependencies, they need to be managed somehow. You need a holistic view of all your services. Therefore, you need to develop a tool that performs static analysis through your config files and maps a graph of all microservices and how they interact with each other. To supplement this static analysis tool, you would also need a tool that in real-time maps all the microservices interactions that are actually taking place in production.

(6) Logging / correlationId / Log-Levels:

First agree on the format of your log messages. Then implement a logging framework in a library that all your microservices can utilize.

Make sure that you introduce the concept of correlationId that can be used to map the flow especially between microservices. For example, your microservice could be logging a requestId or some form of a GUID unique to that service, but in order to connect things among multiple components/services, you need the concept or correlationId.

Now that you have the foundation and the principles down, your team needs to have the right mindset; it looks simple on paper, but there is much more to it. How many times have you been in a situation when you were troubleshooting a production issue and you said to yourself: “I wish had a log line for this condition”. When it comes to logging, here are some tips:

• Log positive as well as negative cases
• Any time you add an IF condition in your code, you need to think about what you will log in the IF, ELSE IF and ELSE.
• Log your exceptions if you caught them and you need to enrich your log with the context you have at that level and not at the level above in the stack-trace.
• Introduce the concept of log-levels and make sure that the production environment is not logging data that should not be logged (personal info) and that the excessive logging does not negatively impact performance.

Unwritten rule with logging is that you do NOT put a log statement in a loop unless you truly understand the number of iterations will not impact your performance with multiple requests going into your service.

(7) Log Aggregator Tool:

Your team needs or your company overall needs to pick a log aggregator tool that allows you to view all your flows through a website and it should allow you to perform fancy queries on your logs. For example, you can pick Splunk. Your service will use a library to stream all the logs into Splunk and Splunk would be your eyes into the system.

Now that you have a tool like Splunk, you need to take it to the next level. Each team should be responsible to build dashboards in Splunk that track the health of your microservices. These dashboards could be used by any technical person troubleshooting things in production.

Then the next thing to do is for your Network Operation Center to set up alerts in Splunk so that they get informed when certain limits are reached and when to open a severity 1 or 2 ticket with the right microservices team.

(8) Exception Handling in your code:

In your choice of general purpose language, you have the concept of exception handling (try-catch or try-except blocks). Even though your services would be micro-services, it does not mean that the whole microservice internally contains only one layer. Most likely you will have your API level or controller layer or resource layer. That’s the code where your API routes are defined and where you have the entry points for each route. That code typically should do just some validation and then proxy your call down into your business layer and your business layer makes calls into the database/repository layer and/or into another another API via your adapter layer. As you are writing code in these layers, you need to think about the try/catch blocks if you need it or not. For example, there is no reason to catch an exception at the lower level unless you need to log something that you have ONLY available at that layer and not at the layer of the invoker.

I don’t want to get into too much details on this topic as this deserves a totally separate article. I just wanted to point out that the exception handling needs to be taken seriously if you want to achieve that enterprise grade.

(9) Error Handling:

Don’t mistake the Error handling with Exception handling. Error handling is different. An error could be a result of a specific IF condition in your code or it could be a result of an exception that got caught. Basically your team needs to decide what how to translate all the errors within the microservice into HTTP codes and you can also enrich the JSON response with some extra details. Don’t forget that you can for example use specific range of 5xx custom http codes to provide a more meaningful response to your client. Your team needs to analyze all the paths in the code and decide what HTTP codes will be sent to the client. Don’t just have all errors be HTTP 500; that does not help your clients nor you if you are troubleshooting your service.

(10) Timeouts:

Timeouts in your API calls are something that is typically very lightly taken. We engineers would just code it and then when things stop working, you are wondering what your defaults are and if those defaults make sense. Understand what the defaults are in the library that you are using to perform HTTP/API calls. Then assess what is reasonable from client’s point of view.

Dynamically variable timeouts are something even better if your team has time to implement. Let’s assume you have a soft timeout of 15 seconds for outbound API calls from your microservice. Now your service is calling another microservice and it starts timing out at 15 seconds. You could build smarts into your system where timeouts dynamically change from the 15-sec soft limit to 20 seconds. Then your application knows what percentage improvement you made by increasing the timeout to 20 seconds. Then your application based on data can decide if to stay at 20 seconds or to increase the timeout to 25 seconds or go back down to 15 seconds. This mechanism allows your application to still function and service customers and obviously you would need a hard limit for your timeout because you need to draw a line and determine what timeout is truly something that is not acceptable for end customers. This mechanism allows you to be a bit more aggressive by starting with lower soft-limits which improves the overall performance of your microservice because you don’t have that many connections opened due to longer timeouts.

(11) API Contracts & Stubbed or Mock services:

This item is more about the development methodology that I would strongly recommend. I have a short article that talks about developing the layers of cake instead of slice of cake: https://medium.com/@almirx101/developing-layers-of-cake-instead-of-slices-of-cake-in-your-software-engineering-projects-71483aff2df1

What I strongly recommend that you force yourself to fully define the API contracts for all your microservices. As you go through this exercise, you will realize that you need to reach out to other teams to agree on how you will invoke their APIs and at the same time, you should reach out to your clients and agree on what the best contract for your API is.

After defining the contracts, my strong recommendation would be to develop the stubbed or mock versions of your APIs honoring these contracts. So if I called any of your microservices, I would get proper JSON responses. That means you internally would have that JSON harded or sitting in some temporary JSON file until you eventually implement the data/repository layer and your choice of database itself.

If everybody did this for each one of their microservices, then you are in theory building a thin layer of the cake or a thin layer of your microservice platform. This would enable your Front End team developing a website or a mobile application to implement the full journey A to Z by calling all these mock services. The sooner this full UI journey is given to product management team, the sooner they will get a chance to feel how good their requirements are and the sooner they will be able to adjust things and steer the ship in the right direction so that your end customers ultimately win.

(12) API Documentation and Swagger docs:

Don’t underestimate the power of documentation. I am not a big fan of LONG documents, but I am a big proponent of single-page documents. If it is longer than a page, the chance that people will read it is low. However a singe-page document will be read and if supplemented by good examples, it is even more powerful. Please follow this recommendation for your API documentation. In theory, the Swagger documentation could replace the majority of your API documentation.

(13) Postman app in Chrome:

Postman app for Chrome is what I have been using to try out different APIs. If you use this app or similar type of app, I strongly recommend that you keep the collection of API calls saved and committed into your GitHub repo so that other members of the team don’t have to figure it out from scratch. Examples are the best way to share the documentation as long as you maintain these examples.

(14) Service/API Versioning and Regular clean-up:

In the API world, you can introduce API versions and you can require from clients to pass you the version of the API that they want to call. Typically you can follow the Major.Minor.Patch versioning. For example, you can start with version 1.0.0 of your API. As you are making small changes and fix defects, you can keep incrementing the patch version. If you introduce some new features, then you can increment the minor version. However, if you introduce a big set of new features and also refactor your API to break some contracts, then you would increment the major version.

My recommendation is that you as a team don’t maintain/support too many versions at any given time. As you depreciate the old API versions, please make sure that you properly clean up all the resources that have been used by that version in order to save the costs. For example, that could be anything from dedicated servers for that API version, dedicated cache, dedicated database, configuration files, and etc.

(15) Cache:

Cache can be used as a mechanism to save some configuration values and improve the performance for accessing values in this cache. Cache can also be used to cache transactional information. For example, you could have a Customers microservice that could be saving the transactional information into cache for 15 minutes because the Front End (HTML/Javascript) code of your website accesses your Customers API on many pages. That’s where you can improve the performance and cache this information and obviously any time customer’s information is changed, you need to update the cache. My recommendation is that you analyze your use cases and see how and if you need to use cache. If you do decided to use cache, then make sure that your application/microservice cache size can be determined on service by service basis. Some services need more and some less cache and this approach will keep your costs down.

(16) Backwards Compatibility:

One way to retrieve backwards compatibility in APIs is to introduce a new API version, but if you did that for every little thing, you would end up in API version management hell very soon. Introduce guidelines in your teams for incrementing new API versions. But let’s focus a bit on keeping backwards compatibility within the existing API version. When it comes to model classes in your Java/C#/Python/NodeJS code and the data structure/scheme in your choice of database, it is important that these changes are always backwards compatible. For example, you could introduce a new attributes in these model classes, but if you make these new attributes mandatory, then you need to think how your code would read the pre-existing records in database. You need to make sure that your code gracefully handles that. As for changing the existing attributes in your model classes, that should never happen; that’s an interface change and you should never do interface changes.

Therefore, my general advice is to always think about backwards compatibility and make it a part of your programming culture.

(17) Security behind your API routes and header information in POST/PUT calls:

Assuming that you have a mechanism to configure your API routes as internal to the platform or external to outside client, you need to have a process around tracking the security of your API routes. This is important especially if your external routes could have different roles depending on what token you get authenticated for.

Then when we get a bit deeper into the HTTP call itself, it is important that you have standards around what is acceptable in the header of the POST/PUT HTTP calls. This will save you a lot of headaches. Now that you know what is acceptable in the header, you need to have mechanisms for deciding what if the given header attribute can be injected by your external client or not. Or even if it is injected, it would not matter if you further down this HTTP call, you strip out that information and replace it with correct values; if you have this type of capability, you are that much better.

(18) Security — Encryption of data at the attribute/column level:

To be at the enterprise level and pass PCI regulations, you need to have a library for your microservices that fully abstracts the encryption of data and the management of keys needs to be done on dedicated appliances.

(19) API Automation approach:

First, you need to implement an automation framework that your teams can use.

Then your team needs to focus on automating the most likely real-life scenarios with APIs instead of trying to exhaust the permutations on each individual API.

Initial automation tests should exercise how your family of APIs work together. Then you can expand your horizon and focus on more holistic view joining forces with automation engineers from other teams.

(20) Batch process within your platform:

The microservice architecture has the event driven methodology built it in, but there are still 3rd parties that interact with your platform and require a batch oriented design. Therefore, your team needs to make sure that you have a reference architecture to handle batch oriented implementations. Depending on whether you use NoSQL or Relational type of database, this batch oriented solution will be different, but you need to have it.

(21) Load and Performance testing:

Load and performance testing needs to be done for each microservice before that microservice can be accepted into production environment and opened to real customers. Your team needs to be able with confidence to determine for every microservice:

• What is the comfortable memory & CPU usage while the service is functioning fine but close to crashing?
• What is the memory & CPU usage when the service starts crashing in the load test.
• How many requests your service can handle per minute or per second?

Your team needs to document all of this in order to be accepted into production.

(22) Circuit-breaker pattern especially when calling external (3rd party) APIs:

In order to prevent the back-pressure effect on your microservices, you need to introduce the circuit-break pattern for situations when your microservice calls into another service/API. This is especially crucial when your microservice calls into an API external to your company; you don’t want the stability of 3rd parties to have direct impact on your application and cause so much back-pressure that the other pieces of your service are affected.

(23) Database design:

First you need to decide if you will embrace NoSQL world or you will stick to more traditional SQL/Relational database world.

Let’s assume you decide to pick NoSQL, then you need to keep in mind that there is NO SUCH THING as no schema. You technically always have some data structure but your database system may not be enforcing it, and with code-first methodology, your application code would be enforcing all of this. Even if you pick NoSQL, your data still needs to be eventually streamed or replicated into some form of relational database if you want your reporting/analytics team to perform analysis on it.

Therefore, regardless of NoSQL or SQL choice, your team needs to implement the streaming/replication of data into a reporting database.

Don’t forget about the database backups. I am talking about the cold backups or snapshots. For example, if you realized that your data has been unreliable for last few hours, or the system fully went down, then you need to have a choice to pull the backups and restore from earlier that day.

(24) Database capacity provisioning:

Your team needs to provision your database for your scale of traffic and the amount of data. For example in AWS DynamoDB NoSQL world, you need to provision the reads and writes per second and you pay for what you provisioned. At the same time you need to be able to handle quick bursts of traffic or longer or consistent increases in traffic. Similar concepts need to be applied in the relational database world as well. For example, in Amazon Aurora you don’t necessarily provision tables ahead of time, but you do need to worry about the overall amount of data you can store in Aurora instances.

(25) Data / Data structure Governance & Reporting/Analytics:

What do I mean by “data governance”? Your team needs to have procedures with the goal of preventing breaking changes to data structures and negatively impacting the integrity of data. For example, in the NoSQL world, code comes first, but to manage data structure changes, there are no stored procedures; you really need to have procedures for software engineers to inform the data team about any model classes changes in your Java/C#/Python/Go/NodeJS code. You need to be always backwards compatible and since this NoSQL data eventually ends up in some form of relation database, you need to keep informing the data team about the attributes that have been added in the data structure. Ideally you would need an automated process that does static code analysis and always detects changes in these model classes even if programmers forget to inform each other and the database teammates.

On the other hand, if you are in the relational database world, you are controlling things through your stored procedures that are abstracting the data structure from software engineers and the database team is in more control.

Reporting and analytics has always been of those after-thoughts. In theory, you should start with this. Before a line of code is written, you could specify what defines “success” for you and you should be able to break it down into multiple measures. I am sure that the requirements and designs will change throughout the implementation phase, but at the principle level, it stays the same. Therefore, introduce into your work environment the concepts of reporting and analytics from early phases of design and development.

In conclusion, writing code is relatively simple and when your software engineers say they are done, you should clearly define what “done” means from enterprise point of view.

Thank you for reading this long article and I hope you found it useful as you go down this path of implementing and releasing enterprise level applications/microservices.

Almir Mustafic

Amazon WorkSpaces — Desktop in the cloud for non-technical users and even software engineers

I am not sure how many of you have heard of Amazon WorkSpaces. Let me give you some information and my analysis after using it for a week.

What is it?

It is the Amazon’s solution to desktops in the cloud.

What’s the point?

Well, if you want to use a basic less powerful laptop or even use a Chromebook as your daily computer, then you will not be missing the full functionality of a powerful Windows 10 machine. For example, you can use your Chromebook to connect to your Amazon WorkSpace machine and then you get the full Windows 10 experience. The point is that it gives you the experience as if your remote/workspace Windows 10 machine is really part of your Chromebook. Where it makes the most sense is using Chromebooks to connect to your Amazon WorkSpace machine as Chromebooks with Chrome OS are fast for anything you do in the browser and they are very secure; Chromebooks are also generally cheaper than other laptops. Then when you want to some more advanced work (even software development), you can connect to your remote workspace with the client app that Amazon provides you. It basically turns your Chromebook laptop into a Windows 10 laptop; it gives you that experience.

Instead of using a Chromebook as your laptop, you can also use your an old Windows or an old Mac OS device and still remote into your Amazon WorkSpace machine and use the more powerful Windows 10 machine for all the work that you need to perform.

Who would consider this?

If your computer (Windows / MacOS) is getting old or you already have a Chromebook, then you may consider setting up an Amazon WorkSpace Windows 10 machine and paying monthly for it instead of buying a new laptop in your local tech store or online. You really need to work out for yourself what makes sense by doing some cost analysis. You need to ask yourself how often you buy new laptops. Based on my rough calculations, if you buy new laptops every 2-3 years because you need to keep up with demands, then the Amazon WorkSpace could be a good option for you. However, if you are change your laptops every 4 or 5+ years, then from the cost point of view, it makes sense that you continue buying new laptops instead of getting the Amazon WorkSpace.

On the other hand, in the enterprise world, companies get solutions like Amazon WorkSpaces because they want to reduce their maintenance/support costs. If you as a helpdesk specialist set up your employees with Chromebooks and then you give them access to an Amazon WorkSpace Windows 10 machine that already has MS Office and other apps installed, then you will spend less time supporting your users and you will have more control managing these machines in the cloud. Pushing updates and rolling out bigger OS changes will be much simpler for your helpdesk department.

Here is the pricing information:

You can see that based on these machine specifications a business person and most employees can easily utilize the “Standard” bundle. I am actually experimenting with the “Standard” bundle for the purposes of software engineering and it seems to be fast enough for some lightweight open-source type of development. The “Standard” bundle performs as good as my Macbook Air with 4Gb of memory. If I wanted to do some more intensive software development work, I am sure that the “Performance” bundle would do just fine even though the 7.5 Gb memory seems a bit low for what you would typically get if you were buying a laptop for software development; you would typically go for 16 Gb or 32 Gb memory. The “Graphics” bundle seems to be a good option if you want to occasionally use it for some heavy processing, but it is a costly option for us consumers.

The important thing is that you are the local admin on the machine in the workspace. Therefore, you can install different softwares on that Windows 10 machine. For example, here is the list of applications that I installed so far: Chrome, Dropbox, WinSCP, Python, a list of Python modules, PyCharm IDE for Python development, AWS CLI, Git for Windows. I also plan to install Java and an IDE for Java.

To set up Amazon WorkSpaces, just follow the instructions here: https://aws.amazon.com/workspaces/

Once you set it up on the AWS website and install the client app on your laptop, you can open the client app, set up the credentials and then it will connect you into the workspace Windows 10 machine. Here are some screenshots:

Login windows in the client app

You are connected and in

Start menu

A few windows opened in the workspace machine

PyCharm IDE for Python coding

Switching between virtual desktops

You can also expand your client app across two monitors and the screen resolution adjusts perfectly.

Amazon WorkSpaces client app expanded across multiple monitors

I hope you found article useful. I will continue using Amazon WorkSpaces for a couple of months and I will keep you updated.

Almir Mustafic

AWS — When to use Amazon Aurora instead of DynamoDB

Amazon DynamoDB as managed database will work for you if you prefer code-first methodology. You will be able to easily scale it if your application inserts data and reads data by your hash key or primary key (hash+sort key). It is also good if your application is doing some queries on the data as long as the resultset of these queries returns less than 1Mb of data. Basically if you stick to functionality that is typically required by websites in real-time, then DynamoDB will perform for you. Obviously you will need to provision the reads and writes properly and you will need to implement some auto-scaling on DynamoDB WCUs and RCUs, but after you do all of the homework, it will be smooth for you without needing to manage much.

However, there are cases when you will need to go back to relational databases in order to accomplish your business requirements and technical requirements.

For example, let’s assume that your website calls one of your microservices which in turn inserts data into its table. Then let’s assume that you need to search the data in this table and perform big extracts which then have to be sent to a 3rd party that deals with your data in a batch-oriented way. If you need to for example query and extract 1 million records from your DynamoDB table, it will take you up to 4.7 hours based on my prototypes using standard AWS DynamoDB library from Python or C# application. The way you read this amount of data is by using LastEvaluatedKey within DynamoDB where you query/scan and get 1Mb (due to the cutoff) and then if the LastEvaluatedKey is not the end of resultset, you need to loop through and continue fetching more results until you exhaust the list. This is feasible but not fast and not scalable.

My test client was outside VPC and obviously if you run it within the VPC, you will almost double your performance, but it comes to bigger extracts, it still takes long. If you are dealing with less than 100,000 records, it is manageable within DynamoDB, but when you exceed 1 million records, it gets unreasonable.

So what do you do in this case? I am sure that you can improve the performance of the extract by using Data Pipeline and similar approaches that are more optimized, but you are still limited.

Basically, your solution would be to switch to a relational database where you can manage your querying much faster and you have a concept of transaction that helps with any concurrency issues you might have been challenged with. If you want to stay within the Amazon managed world, then Amazon Aurora looks very attractive. It has limitations on the amount of data, but most likely those limits are not low enough for your business. As for the big extract performance challenge, your extracts will go from hours (within DynamoDB) to minutes with Aurora.

Please consider this in your designs. Performing big extracts is opposite of the event driven architecture, but these type of requirements still exist due to a need to support legacy systems that you need to interact with or systems that have not adjusted their architecture to your methodologies.

Thank you for reading.

Almir Mustafic.

Success of good tech leads is really attributed to the SUM of many LITTLE good moves

How many times have you been asked as a tech lead what you do to make things successful?

If the answer to that question were simple, then the experience of tech leads would be totally undervalued.

We all know that there is no silver bullet for the success of projects that certain tech leads have been achieving. The success of good tech leads is really attributed to the sum of many little good moves.

It starts from day 1 when requirements are being delivered to the team. For example, analyzing the business requirements, detecting what are the true requirements and removing all requirements that contain implementation details is very crucial in the overall equation. Sometimes you, as the tech lead, will need to put a project management hat and a product management hat on in order to steer the ship in the right direction, and at the same time you need to be very engaged in the low-level technical details with your fellow software engineers. This is just the beginning and the moves made at this phase are the foundation to the rest of challenges that follow. In the agile methodology world, this requirements phase happens very often and that even puts more pressure on you as the tech lead to keep the foundation solid.

Then you get to the point where you are taking the ironed out business requirements (epics) and turning them into a collection of user stories that different squads will be receiving into their squad’s backlogs. Understanding how to do this breakdown and estimation is crucial in laying the foundation for all the squads and the backlogs that squads would take on. You, as the tech lead, plays an instrumental role in this process because you would be working very closely with solutions architects in determining the high-level design and architecture. The decisions that you and solutions architects make at this step are impacting multiple squads and the squads’ timelines (pre-planned sprints) and the overall calculated production launch date for the given milestone/release.

Then you get the point where you take the stories from your squad’s backlog and you work with fellow engineers on your squad/team. Every interaction and every little move or piece of advice that you give to developers is important; every recommendation that is given to you needs to be properly assessed. It starts before even a line of code is written. It starts with the coding methodology and mindset. It is anything from introducing guidelines on how to define boundaries of microservices/APIs to unit-testing culture to database design in NoSQL vs. SQL to concepts of backwards compatibility to introduction of feature toggles for product team and launch risk mitigations and etc. There are so many little things that create the sum that defines the success.

If you are a developer on this team, learn to listen and hear what the good tech leads are preaching and also provide your feedback as great tech leads also know how to accept your input and build on it. If you are a tech lead, show the values you care about through examples and let the team experience it, and then sense when you need to step back a bit and let the team move forward and ride on the momentum and the vibe you have established on the floor.

Keep in mind that while you are going through this exercise, you do NOT think that you are smartest person in the room because if you think you are, then it really means that you are in the wrong room. If nobody applies any humbleness, then it means everybody is in the wrong room and the concept the team disappears. Understand your skills and understand the skills your teammates bring on the table and try to create a culture where complimenting each other happens naturally. This is the culture where you respect each other so much at the professional level that you are happily sacrificing your time to save each other’s work.

Geek out and enjoy the process because it requires a lot of patience and perseverance.

Thank you for reading.

Almir Mustafic

Knowing when to put code into your API/microservice vs. outside API (wrapper microservice)

As you are defining the purpose of each one of your microservices, and as you are developing these microservices, it is very important to know if you need to make changes to an existing microservice, or you need to build a wrapper microservice.

What I am talking about here are not rules; I am just giving you some examples so it can trigger you to think about this while you are going through the similar exercise. I just want to make developers aware of the differences.

So the main question is:

• Put code into an existing microservice/API
• OR develop a wrapper microservice API?

Let’s assume you have a “Package” microservice/API and this API gives you the list of packages and their details:

• /api/packages (this gives you all the packages)
• /api/packages/{id} (this gives you a specific package based on that ID)
• /api/packages?type=type101 (this gives you all the packages of type=type101)

Let’s assume that behind this microservice you are using a NoSQL table “Package” and that table has the packageId as the hash key of the table and then you have a bunch of other attributes that you are saving as part of the JSON object.

Let’s say that this API has been used in your production system for many months. Then you get a requirement from your product team that they want to introduce the concept of “Package Bundles”. A package bundle would be just a collection of different packages. For example, you can create the Bundle 1 that contains:

• package 1
• package 2
• package 3

Whereas the bundle 2 could contain:

• package 1
• package 7
• package 8
• package 9

Ultimately the product team instead of presenting individual packages to customers and having them pick one by one, they want to also present some pre-configured bundles of packages to customers to simplify their experience.

Now the big question is: How do you as a software engineer and solutions architect design this, or in other words how do you incorporate this requirement into your existing design of your microservices?

Before I get into the design, one unwritten rule in software development is that you need to keep things backwards compatible regardless of what type of design you come up with.

Let’s get into the design a bit. If you have been working in monolithic applications for many years (as many of us have), you would be naturally inclined to open up the existing “Package” microservice and you could be possibly introducing the list of bundle IDs in the existing data of the package. Or you could be adding another table within the Package microservice to introduce the concept of bundles and that database structure would be leaning towards a relation DB design. I talked about NoSQL and avoidance of relation DB design in my YouTube video: https://www.youtube.com/watch?v=49YvbcqjEiA

As you get deeper into this approach, you can realize that you are polluting the purpose of the Package microservice and you are slowly turning it into a macro-service. I have a full post about “micro” in micro-services: https://medium.com/@almirx101/micro-in-microservices-677e183baa5a

You also can realize that the API routes cannot stay RESTful. How do you actually search to get the bundles using /api/packages API route?

That’s when you conclude that you need to change your design.

The solution that I recommend for this specific requirement or use case is to introduce a new “PackageBundle” microservice. I really classify this as a wrapper microservice. This PackageBundle microservice would have the following type of API routes:

• /api/packagebundles (this gives you all the package bundles)
• /api/packagebundles/{id} (this gives a specific package bundle by id)
• /api/packagebundles?package=packageA (this gives you the list of bundles that contain packageA)

This microservice would have a NoSQL table with the following data structure:

- packageBundleId   (hash key for the table)

- packages [ ]   (list of packages)
   - packageId
   - packageName
   - other package attributes

- Other attributes needed to represent the bundle

In JSON, this would look like this:

{
  "packageBundleId": "",
  "packages": [
    {
      "packageId": "",
      "packageName": "",
      "packageDescription": ""
    },
    {
      "packageId": "",
      "packageName": "",
      "packageDescription": ""
    }
  ],
  "otherPackageBundleInfo": ""
}

Now let’s visualize how package bundles would be created. Imagine a business-enablement tool that allows you to create a bundle and then let’s assume that you drag packages into a bundle. Behind the scenes, that tool would be calling the Package API to retrieve the details on the package that is dragged into a bundle and then as the final step the tool would be calling the PacakgeBundle API to save all those details into the record for that given bundle. That’s what this business-enablement tool would do. On the other hand, the main website that customers are using would be calling the /api/packagebundles API to get a list of bundles or a specific bundle to display on the screen for customers.

With this approach you kept your existing microservice as micro and you introduced a new microservice that has a sole purpose to manage collection of packages. If I wanted to create some new packages, I am not dependent on the PackageBundle API; I can directly call the Package API to perform the actions. If the product team tomorrow decides that the concept of bundles is not needed any more, then I can just stop using the PackageBundle API and get rid of it in the long term without any impact on the Package API.

I hope this example can help you save some time if you have similar type of requirements and design.

Thank you for reading.

Almir Mustafic.

Defense is the BEST Offense — how do you apply this concept in Software Engineering?

Defense is the BEST offense. If you play sports, this is very familiar to you. I played high-school basketball and I learned that this is very true.

As for applying this concept in software engineering field, let me start by saying that I am NOT talking about developing software in such a way to protect your job; I am totally against that. I am talking about something else here.

I am talking about a mindset in software engineering that allows you to be offensive and the buzz word that describes this is “being disruptive”. You can develop software very fast, but can you consistently repeat this without creating chaos?

One real life example is the following: You developed a piece of software or a list of features and your QA and Stage testing passed perfectly. Then you are about to go to production and on the day of the production launch, the decision gets made not to launch those features for technical or business reasons. Now the question becomes: Can you easily disable these features and still launch everything to production with these features turned off? If the answer to this question is a CONFIDENT YES, then you are implementing things the right way and you have development methodologies in place that present you with a lot of capabilities and options. It is easy to say this, but this methodology starts before you even write a line of code. On the other hand, one might say “why do I need to develop the feature toggle if I can just be fast and go for it?”. I would say that going fast and taking chances is fine if you have the foundation, but without the foundation and methodology to support you, it is just reckless.

The confident YES in above scenario gives your product management and your senior management enough confidence to try different things without worrying about failing because the failures or sudden decisions will not cause chaos. I provided just one example that accomplishes this goal.

In conclusion, you are building a defensive capability called “feature toggle” but in turn you are actually providing offensive capabilities for you and your leadership team to innovate and disrupt. It sounds counter-active, but it is not.

Thank you for reading.

Almir Mustafic.

Make components reusable and NOT necessarily the orchestration layers in your software platform

Making things reusable in software engineering is very important. That’s why all the general purpose programming languages have methods/functions and they also have object oriented capabilities. Using these capabilities you can implement different patterns. Yes, different languages implement these patterns differently, but my point is that they are all achievable.

Generally speaking junior programmers write code that is simple, but it may not be as reusable as we would like it to be. On the other hand, when programmers gain some experience and learn these different programming patterns, they may get over-excited and start applying these patterns without figuring out first what problem they are trying to solve. They may kind of look at the list of patterns and say: “Oh, this one would be cool”. It is totally fine to use a known programming pattern, but how you arrive at using one decides if you are doing it for the right reasons. Here is what I mean.

First, understand what problem you are trying to solve.

Then figure out the solution for this problem without being disrupted by available patterns. Focus strictly on the solution and work it out on paper or a piece of napkin.

Now that you have a solution to your problem, you can look at the list of known programming patterns at your disposal and most likely one those patterns would be the one that exactly fits the solution. However, there could be a chance that you don’t have a known pattern or it may mean that you need to use a combination of two patterns with slight modifications.

You may be wondering where I am going with this. I am basically trying to get to my next point which is the main subject of this article. I want to talk about making your individual components in your platform reusable and NOT necessarily the actual orchestration layers that you may have.

Let’s assume that you have the following components:

• Component A
• Component B
• Component C
• Component D
• Component E

You need to develop each one of these components in such a way that they are reusable so that the orchestration layer can invoke those components without needing to learn about the internals of your components. For the sake of this example, I will assume that those components are just methods/functions. Then let’s assume that the orchestration layer looks as following:

Orchestration Layer:

MethodA();
Async call to MethodB();
MethodC();
Wait for MethodB() response
MethodD();
MethodE();

Then what could happen is that you try to make this orchestration layer reusable by creating a template for it. When you do that, you could create a workflow or some type of state-map that ties all these calls and you have some fancy configuration that allows you to connect state to state and you have a mechanism to incorporate a new step in the state-map without writing any code.

I understand this conceptually, but my question to you is: Why? What do you gain by doing this? What problem are trying to solve? Are you solving a problem at the root?

A lot of times the response is: We need to do it because we want to introduce new steps in this step-factory (state-map) without needing to push any code.

Yes, there are valid reasons to introduce the step-factory pattern (state-map) in your design, but most of the time you don’t have a good reason and I will explain why.

You maybe introducing this step-factory design for wrong reasons. It could be that in your organization it is very hard to deploy a code/binary change to production due to a lot of processes. So you ended up changing your design (even for a simple case as above) and introducing the step-factory to get around the process so that you can deploy a config change quickly to production. First, you are complicating your design and implementation by introducing this without good reasons. Second, you need skilled developers to make code changes within it and you are actually adding more risk.

Let me use the following analogy with the car engine and other cars parts under the hood of your car. Under the hood in the engine bay, you have the following:

• engine
• battery
• alternator
• water pump
• brake cylinders
• air intake and air filter
• radiator
• fan
• …etc

You can picture those car parts as the individual components that I described above in your software platform. The engine bay itself would be the orchestration layer. Everything in this engine bay (orchestration layer) is connected a specific way and it works great. If something goes wrong, you can go inside the individual parts (components) and you can replace the full part or the insides of that part. For example, the air filter could be dirty and you just open the air filter box (component) and put a new air filter. So this setup with this engine bay (orchestration layer) and these parts (components) is great if all you are doing is changing the insides of a part/component or replacing the whole part/component with another one that fits in (using the same interface). So by changing these parts, you are pretty much changing just the configuration within your step-factory. If that is all you are doing, that is perfectly fine and this engine bay / orchestration layer / workflow is perfectly good for you.

However, the world of software engineering (especially microservices world) is not as static as the engine bay of your car. You typically have a need to put things together using different permutations or you have a need to orchestrate in a different way. That’s where you need to step back and ask yourself if you really need to turn your orchestration layer into some template/step-factory/state-map when you are actually constantly changing what that orchestration looks like? This is where you realize that the complexity of your state-map/workflow via step-factory pattern is actually complicating things and it is slowing you down, and it is adding the unnecessary risk. There is nothing wrong with making a code change in your orchestration layer in order to meet a requirement. Tell yourself, it is ok. It is not 100% configurable, but I know that making it 100% configurable would slow me down because of added complexity.

As for my example above about using the configurability to get around your process, you really need to think about solving that problem at the root. That company probably introduced the process to slow developers down because they were manually deploying things to production and breaking things. So the typical reaction from your Change Control team is to tighten the SDLC process. What you are missing is the continuous integration and automated deployments that increase the confidence, success and reduce the risk. That is your solution for that problem and your Change Control team will welcome that because it is repeatable and predictable.

In conclusion, please think about the complexity of your code. Make your individual components modular/reusable and use them in your code. After you use them enough in your orchestration layer, you can determine if you want to keep your orchestration layer simple, or you need to build a template (i.e. step factory or workflow) for it. Think about the maintenance here and your use cases. If you are constantly changing the flow of your orchestration layer, then the complexity you are introducing with the workflows/step-factory is very unnecessary. Then stick with the basics, and it is ok to change code in order to adjust the orchestration layer. It is ok, but streamline your deployments throughout the pre-prod environments and into production to make this experience very smooth.

Thank you for reading.

Almir Mustafic