HOME | Blog | YouTube | LinkedIn | About Me         || Calculators    | SoftEng/Tech Posts    | Code/Scripts

Friday, May 26, 2017

Factory pattern in microservice architecture — Are you doing it right?

In the software engineering field we all have been using the factory pattern for many years. However, something changed with the introduction of the microservice architecture. How did this affect your factory pattern implementations and have you realized that you need to change your implementation to align it with the microservice architecture?
In the factory pattern, you basically have the following:
  • Interface that contains your method signatures
  • Base class
  • Class A that inherits from Base and implements the interface
  • Class B that inherits from Base and implements the interface
  • Then you have the Factory class
  • Then you have a client or some form of Manager class that creates an instance of the factory and invokes the methods defined in the interface
Let’s now use an example. Let’s say you have a website that accepts the customer’s address and then you need to verify if that address is valid. Let’s assume that behind the scenes you have two different external (3rd party) providers that you can use to do the address verification and you want to abstract that from your clients (Front End code). Then you would end up coding a service as following:
  • IVerifyAddress interface with verifyAddress(…) method signature
  • BaseAddressVerification class
  • ProviderA class that inherits from BaseAddressVerification and implements IVerifyAddress interface
  • ProviderB class that inherits from BaseAddressVerification and implements IVerifyAddress interface
  • AddressVerificationProviderFactory class
  • AddressVerificationManager that creates and instance of the factory and invokes the verifyAddress(…) method
In the n-tier architecture or a monolithic application, the verifyAddress(…) method would be fully implemented within the ProviderA and ProviderB classes within that existing application/service.
However, if you do this type of approach in your microservice architecture, your service really becomes a macro-service instead of a micro-service. You just need to go back to basics of the microservice architecture and go through that exercise of cleanly defining what micro means in the microservice (https://medium.com/@almirx101/micro-in-microservices-677e183baa5a). Once you go through this exercise, then you would realize that you need to do some refactoring in the above code.
Basically the full implementation of the verifyAddress(…) method is NOT going to be within the ProviderA and ProviderB classes of the existing service. The verifyAddress(…) method is those two classes would just be a shell or wrapper method and within that shell you would be making a call into TWO different microservices that you need to implement. One microservice would be AddressProviderA microservice and the other one would be AddressProviderB microservice. Those two microservices would contain the code that integrates behind the scenes to the EXTERNAL (3rd party) ProviderA and ProviderB.
With this approach, we truly decoupled things and each microservice has its single purpose:
  • My Address Factory microservice has the sole purpose to orchestrate things using factory pattern.
  • AddressProviderA microservice has my code on how to integrate with the external ProviderA and verifyAddress() method is fully implemented here.
  • AddressProviderB microservice has my code on how to integrate with the external ProviderB microservice and verifyAddress() method is fully implemented here.
  • and then you have the EXTERNAL (3rd party) APIs that you talk to and they are the actual external ProviderA APIs and ProviderB APIs.
The following exercise is what I like to go through to explain if you implemented the factory pattern properly within the microservice architecture. Let’s say you implemented your service as the macro-service described in the beginning. Then I tell you that I want to implement a batch process and verify a bunch of addresses through provider A leveraging what you have built so far. Would you be able to give me an API route that I can call to specifically verify addresses via provider A? I am sure you could expose an API route within your existing macro-service, but that API route would mostly likely NOT be RESTful. Now going back to the cleaned-up version of your microservices, I could just read the API documentation of your AddressProviderA microservice and I could invoke it directly (without talking to your Factory microservice) because in my batch process I want to run certain address only through provider A.
In conclusion, the factory pattern as the shell is your orchestration microservice and the actual full implementations of the interface are in separate microservices that you need to implement.
Thank you for reading this article.
Almir Mustafic

Thursday, May 25, 2017

Metrics in Software Development Life Cycle — Do they make teams and projects successful?

I like math and science and that’s why I studied Computer Engineering in school and ended up getting into the software engineering field. Numbers were the major part of my education. However, when it comes to making teams more successful, metrics and numbers are not everything.
They are ok as long as they are NOT in the way of team’s work. The numbers and metrics themselves are good, but the problem is in methods how those numbers/metrics are attained. As soon as you get into that territory of putting more emphasis on tools and metrics over working software and people, you are putting a dent into the team’s culture and their productivity. We are leaders in our companies and we don’t get paid for applying metrics that might have worked for somebody else. We are better than that. If it worked for somebody else, it does not mean that it will work for me. You have to do a temperature read in your company and you need to understand the culture on the floor, and most importantly you have be on the floor and live it for yourself as a leader. That’s when you will feel what works and what does not.
I talk a lot about the “feel” and “passion” in software engineering. If I could give you the formula for the “feel”, than I could also be able to give you a scientific explanation for what made successful leaders successful. If that were the case, we would not have leaders hosting seminars and we would not have speakers giving motivational speeches; everything could’ve been calculated through mathematical formulas. We know that’s not the case.
All you can do is try to get closer to understanding what that “feel” is and have others attain it by show-casing it through examples. The “feel” is something that comes with experience and we know it is tough to convey your experience.
I have worked on many high-profile projects that were carefully monitored by the senior leadership teams. Besides having a strong technical team on the floor, what makes these type of projects successful is the good relationship that the project manager and the tech lead of the project build. They compliment each other and the tech lead is the one that provides you that overall “feel” about the project status, and what needs to be done to keep correcting the direction of the ship in the open waters of the oceans. So the metrics can be used by managers to some degree, but ultimately you hire great tech leads to steer the ship at the technical level and these tech leads are closest to engineers on the floor, and that’s why they have the power to build a great culture in the company. With power comes responsibility.
In conclusion, please step back and think about what makes projects successful. Yes, you can use metrics, but be very careful on how you attain these metrics. Ultimately we know that the success of those projects is directly dependent on the authenticity of certain individuals who make things happen. It is about all the intangibles that make it happen and you can’t put a number to intangibles.
Thank you for reading this article. 
Almir Mustafic.

Sunday, May 21, 2017

Micro in Microservices

What does “micro” in the microservices mean?
How do you define what a microservice is? How do you define the boundaries?
This is not an exact science. However, when you get close to the boundaries of violating what the word “micro” means, you will know, and you will feel bothered making changes in that service because it will not feel right. It is still good to have some guidelines on defining the boundaries of your microservice. Here is the approach that I like to use.
Force yourself to write down on paper what your service does. Write it in first person to give it a little more emphasis.
I am the microservice A and I exist so that I can perform X.
This should be the first description in your API documentation so it is clear for your clients and it is also a good reminder for your team.
Let’s see an example. Let’s say you want to develop a “Subscription” microservice, and after some initial discussions you write the definition for that service:
I am the Subscription microservice and I exist so that I can create, read, update, delete subscriptions and I also exist so that I can activate, deactivate benefits and terms for the subscription. I also exist so that I can correct the failed subscriptions into a good state.
This definition is overwhelming. If the Subscription service ends up doing all of that work, then it is really not a micro-service. It is a macro-service. If the definition of your service has a lot of “ands” and “ifs” in the sentence(s), then you should worry about the size of this service.
Here is how this Subscription service could be turned into a micro-service.
I am the Subscription microservice and I exist so that I can perform the CRUD operations on subscriptions.
Now that your Subscription service is a micro-service, you need to introduce other services that will perform the remaining actions. For example, you can add the following services to perform the remaining actions:
I am the Fulfillment microservice and I exist so that I can orchestrate the creation of a subscription and the fulfillment of that subscription which is really the activation and deactivation of benefits with a given subscription.
I am the Fulfillment-Recovery microservice and I exist so that I can recover the failed subscription into a good state.
This is just one example of how you can define the boundaries for the “micro” in the microservices. One may argue that you can break the above services into smaller services, and in my opinion that’s where you are on the other spectrum where your microservices would be turning into nano-services. That is ok as long as you don’t get over-excited and negatively impact performance. If you go in this direction too much, that’s where you are starting to feel the overhead of too many RESTful calls chained together and each http call over the wire adds up. That’s where you need to be very realistic and decide how much you are gaining by doing this given your use cases.
Another way to more closely define the “micro” in your microservice is to define the API routes for all the actions that you mentioned in the definition of your microservice. If you go through this exercise, you quickly discover that the API routes for the original long definition of the Subscription microservice are not going to stay RESTful. That is your signal to start breaking up that service into smaller services.
In conclusion, introduce some guidelines within your company on how to define what a microservice is and keep each other in check when adding new services and when adding functionality to you existing microservices.
Thank you for reading.
Almir Mustafic.

When you know your tech team is so skilled to …

Your company could be categorized to be in financial industry sector (non-tech company), and you may have a small tech team that is focused to deliver what business needs and building huge in-house solutions might be the last thing on your mind.
Then you end up expanding your technical team from year to year and creating a cool tech culture to work in. After a while your technical team becomes very strong and they end up implementing a lot of solutions and the business grows.
In the process of maturing as a technical team, you might have gone through building different type of architectures.
Then you get to the point where you built a scalable microservice platform in the cloud and this platform is generic enough to support any type of business model.
That’s when it clicks to you that you have put together an amazing technical team capable of implementing any idea. That’s when you realize that you have a platform capable of being the heart of all solutions within your corporation across multiple divisions. That’s when you realize that you are not any more just a company in a financial sector with a tech support team, and you are really a tech company with both technical and business mindset.
Yes, there is a lot more maturity that can be attained, but you feel proud of what you have accomplished so far and you know this is just the beginning of something great. You know you have built the heart of something big and you should take a moment and let it sink a bit, recharge and dive into the next adventure now that you have a good foundation. This is when you reach out to your senior leadership team and thank them for their vision, and this is when your senior leadership team thanks the tech team on the floor for their perseverance, dedication and ability to be open-minded to new tech ideas and cutting-edge solutions.
Thank you for reading.
Almir Mustafic.

Should you achieve Architectural Elegance?

A lot of us solutions architects want to make things perfect. The need to achieve that architectural elegance is in our technical nature, but is that the right thing to do for your organization?
Business requirements are given to the technical team and the tech team needs to provide solutions. The job of solutions architects is to solve problems, not to achieve architectural elegance. Don’t misunderstand this statement.
  • This does not mean that you don’t have any architecture.
  • It does not mean that your code should be written the quickest way without any re-usability.
  • This just means that your ultimate goal is to solve problems meeting the business requirements and in the process of doing that work, you can have architectural designs, coding standards, code reviews, and you can strive towards that ultimate goal of architectural elegance.
However, you cannot let the obsession for architectural elegance take over and negatively impact what business needs. You need to keep each other in check.
One may argue that if you met that architectural elegance means that you are not challenged enough by business and your company is really in cruise control without any innovation happening.
The analogy I like to use is: “Controlling a big ship in the ocean is not precise but that control is definitely steering the ship in the right direction”. And by the way, this ship is never stopping and it is constantly circling our planet. Therefore, you are hired to solve problems and in the process of solving problems you can keep the architectural elegance in your sight. As long as it is in your sight, you are heading in the right direction.
It is the “strive to get to the destination” and not the destination itself that keeps you going.
In conclusion, it is very acceptable for solutions architects to stand on their principles as long as those principles support the business and not just architectural elegance for the sake of architecture.
Thank you for reading.
Almir Mustafic.

Opening Chrome using Windows command-line

Here is a quick post about opening Chrome browser in Windows command-line. You may be wondering why you need this. This is very useful if you want to open a bunch of tabs in Chrome and pre-load them with your favorite URLs.
This is what you need to type in Windows command-line or a Windows batch script:
@echo off

echo Opening chrome
start "c:\program files (x86)\google\chrome\application\chrome.exe" "https://www.youtube.com"
I am not sure if you need the same thing but I am often have a need to pre-load a YouTube video to a specific time within that video. Here is how you can do this:
Let’s assume that the link for the YouTube video is: https://www.youtube.com/watch?v=49YvbcqjEiA
In order to position the video to a specific time position, you just need to append the following to that URL: &t=1m44s
Then you get the following links: https://www.youtube.com/watch?v=49YvbcqjEiA&t=1m44s
This will position the video to 1 minute and 44 seconds.
Here is how you would do it in command-line or a batch script:
@echo off

echo Opening chrome
start "c:\program files (x86)\google\chrome\application\chrome.exe" "https://www.youtube.com/watch?v=49YvbcqjEiA&t=1m44s"
Thank you for reading.
Almir Mustafic

XML to Dictionary and JSON Conversion in Python

If you calling somebody else’s API and that API is returning XML, then you can convert that into a dictionary and/or JSON to use it in your Python code.
Here is what you need to do:
$ pip install xmltodict
Here is the code:
import xmltodict
import json

def main():
    print("main program")
    dict01 = convert_xml_to_dictionary("sample.xml")
    print("To: " + str(dict01["note"]["to"]))
    full_json_string = json.dumps(dict01, indent=4)
    print("Full json string: " + full_json_string)

def convert_xml_to_dictionary(xml_file, xml_attribs=True):
    with open(xml_file, "rb") as f:
        print("Doing the conversion...")
        my_dictionary = xmltodict.parse(f, xml_attribs=xml_attribs)
        return my_dictionary

if __name__ == "__main__": main()
Here is the XML file referenced in the code above:
    <body>Don't forget about the requirements</body>
Thank you for reading.
Almir Mustafic.

7 things about working in Scrum Teams and Microservice Architecture

  1. 1. SAFe is not scaled if you get requirements in big bulks every 6 months. This is fully explained in this article: https://medium.com/@almirx101/scaled-agile-framework-safe-isnt-really-scaled-if-you-f933fc78a738. This can be avoided.
  2. 2. Microservices are micro if you follow the rule of “single purpose, singe microservice”. You need to establish the guidelines for your teams and you need to keep each other in check. Otherwise, they become macro-services or even worse, monoliths in the cloud. Old habits are hard get rid of, but it is possible.
  3. 3. Story estimation by complexity value may not work for you. This may not work when you are doing sprint planning because you would not know how much you can commit into the sprint. It also does not help you predict how much work you have in your product backlog. Complexity does not directly proportional to the level of effort. That’s why story point estimation generally works better when you talk about the level of effort.
  4. 4. Watch for this anti-pattern: Building an enterprise-bus type of solution centralized for everybody with microservices under the hood does not really make your architecture a microservice type of architecture. If you find yourself centralizing things or coding a hub that everybody shares, it means that your design is becoming an anti-pattern in your microservice architecture.
  5. 5. From DevOps point of view, automate the heck out of everything or at least have that attitude with the team on the floor because good attitude radiates.
  6. 6. Manually test to get the insight of how things work or are supposed to work, and then write API automation code. The common mistake is that you tend to roll up the sleeves right away wanting to write automation code, but you end up automating without understanding. The QA automation engineer should know the API as good as any other client of these APIs and even better. In a nutshell, manual testing is about getting the insight into your application and automated testing is about making sure what worked before still works
  7. 7. Understand microservice dependencies, paint the big picture and have low-level engineering monitoring in place for each group of microservices. Having this type of view into your system will save you so much time with troubleshooting issues in production; it also gives you that extra confidence to launch more often because you have ability to see the impact (positive or negative) very easily. The value of understanding microservice dependencies is explained in this article: https://medium.com/@almirx101/microservice-dependency-analysis-tool-trust-me-you-need-this-cdd25b934c6c.
Thank you for reading.
Almir Mustafic

Learning somebody else’s APIs — What is the most important thing?

We all have been on projects when you have to learn somebody else’s APIs in order to accomplish the requirements that you were given.
Yes, you need to first see if they are SOAP or RESTful APIs. Then you are most likely curious if the response from these APIs is XML or JSON. Then you analyze the elegance of the input parameters and how the GET/POST/PUT/DELETE https calls work for each API. Then you learn about the security for these APIs and what you as the client have to do.
However, the most important thing that you need to watch out in consuming other APIs is how they work together in a flow. What do I mean by this? APIs when investigated in a standalone way may all look good, but how do they work in a flow and can you connect them together without complicating your design?
For example, you may provide me a “Customers” API that allows me to create a customer. You also provide me a “Subscriptions” API that allows me to create a subscription for the given customer. Let’s assume that you neglected to return the subscriptionId in the successful response for the creation of the subscription. Now I don’t have the subscriptionId and I need to connect some of my data with the subscriptionId that you should’ve returned to me before I can give my customer the access to the subscription features. You may provide a mechanism for me to get this subscriptionId in near time by listening to some events from your system, but this means that I have to implement this whole near-time capability in my system and plus the near-time solution may not be good enough for me because the customer is waiting in the browser and I cannot technically proceed without connecting my records with your subscriptionId as I will increase the chance of ending up with orphan records.
Do you see how a simple lack of subscriptionId in the response of the Subscriptions (POST) API introduces complexities in the flow of your application? This is just a simple example, but there are more complicated cases that you can detect as you get deeper into analyzing somebody else’s APIs. You may have some flexibility if the APIs you are trying to consume are developed within your company, but if these APIs are from an external vendor, then your decisions may be affected in the negotiation with this vendor.
Please pay attention to this as the consumer of somebody else’s APIs and if you are developing APIs for others to consume, then try to keep it simple for your clients so that their flows are simple.
Thank you for reading.
Almir Mustafic.

Opening Chrome via Terminal window using command-line interface on Mac OS

Here is a quick post about opening Chrome browser in terminal window using command-line. You may be wondering why you need this. This is very useful if you want to open a bunch of tabs in Chrome and pre-load them with your favorite URLs. Yes, this is for the Mac OS techies out there.
This is what you need to type in the terminal window:
/usr/bin/open -n "/Applications/Google Chrome.app" --args 'http://www.google.com'
I am not sure if you need the same thing but I am often have a need to pre-load a YouTube video to a specific time within that video. Here is how you can do this:
Let’s assume that the link for the YouTube video is: https://www.youtube.com/watch?v=49YvbcqjEiA
In order to position the video to a specific time position, you just need to append the following to that URL: &t=1m44s
Then you get the following links: https://www.youtube.com/watch?v=49YvbcqjEiA&t=1m44s
This will position the video to 1 minute and 44 seconds.
Here is how you would do it in command-line in the Terminal window:
/usr/bin/open -n "/Applications/Google Chrome.app" --args 'https://www.youtube.com/watch?v=49YvbcqjEiA&t=1m44s'
Thank you for reading.
Almir Mustafic

One thing you need understood and specified in the SOW with your next vendor

This article gets into the most important part of preparing the Statement of Work (SOW) with a vendor, and how you need to protect your business from the factors that could be lost in translation as you are getting excited about using this vendor’s APIs and UI interface.
Yes, you need to analyze this vendor’s APIs and see if their business model fits into your business model.
Yes, you need to analyze the vendor’s front-end tool that business could be using and you need to make sure that it will meet your requirements.
Yes, you need to make sure that the vendor’s overall feature set will meet your requirements or they have a roadmap where in short-term they will have features that you will need.
Yes, you need to identify what custom items your vendor will need to implement and fully understand their timelines and those timelines align with your timelines.
Yes, you need to identify the level and type of engagement. There are companies that are still based on the waterfall approach and that might fit well into your agile methodology and how engineers in your company approach software development.
However, the most important thing that needs to be specified in the SOW is the performance and load metrics that this vendor can handle. None of other things matter if this is not satisfied. You can spend weeks and months developing the system on your side integrating with this vendor’s APIs and it will all work fine in pre-production environment. Then when you deploy your code to production, you may realize that the performance is not on par and then you end up polluting your design by introducing crow-bars in your code in order get around these deficiencies. All of this can be prevented by spending time with the vendor’s engineers and first understanding the performance and loads they can handle. Then ideally you would want to perform the actual load test with them. Then all of this needs to be specified in the SOW.
Thank you for reading.
Almir Mustafic.

Production Launches — Getting/Having the confidence that your code works

There are many approaches that you can take in verifying that your code is robust. Let me list a few of them:
  • Writing robust code that is easily maintainable. Maintainable code becomes robust in the long term. Keep it simple.
  • Having good code-reviews that detect and prevent severity #1 issues in your production environment. Typical issues that can be prevented are the ones that are abusing the frequency of I/O, memory utilization and string manipulation that is the subset of memory manipulation and CPU utilization.
  • Automated unit-testing and continuous deployments to your environments. Automated integration testing is even more important/valuable.
  • Having two types of QA resources: One QA resource would be executing the test-cases that cannot be automated. Another QA resource would be doing what I call free-form testing. Free-form testing requires you to have full understanding of all your products and you have to be the super-user of that application/site/system.
  • Having deployments automated to the point where you just clicking button but at the same time having post-deployment scans to verify if deployments were done on correct servers, correct folders and in completion.
  • Performance and Load testing needs to be performed on your services so you can with confidence say that you service X can handle Y amount of load and peform at Z speeds.
  • Dashboard monitoring at the high-level with graphs showing the high-level state of your production environment.
  • Monitoring the production environment (low-level events, logs…) before and after the launch so that you can tell the percentage change in the errors logged. Don’t mix this with the dashboard monitoring that tracks the high-level state of your production environment.
You can do all the great work on the first 4 items, but what matters at the end of the day is if the last two items above are coming back with the positive results. If you don’t have the last two items implemented, you pretty much don’t have any visibility on the health of your systems in production and what customers are experiencing.
In what order do you implement the items listed above?
It all depends on your organization; however, you cannot be operating a business if you don’t have the last two items implemented from the checklist above. You can end up writing bad code and the last two items will save your day in the case when you deploy bad code to your production environment and in that case you can quickly roll back.
Thank you for reading.
Almir Mustafic