Track Request Between Micro services : Importance of Response time taken by each micro service and how to implement

As every one know in micro service world , Orchestration service calls multiple micro services to fulfill the request . We have to track the request between micro services for multiple purpose , one of the important aspect is know the response taken by each micro service which helps us to work on micro service taking more time . We can get the response time taken by each micro service by using tools like spring sleuth if your using the spring libraries but what if your not using the spring libraries or using different http client where you cant use the interceptor provided by the spring.

What are the important aspects we should know at each micro service ?

1. Time taken by micro service which is sum of the time taken for auth check + Application logic +DB calls + External api if any + time taken by other micro service with in your product or project
2. For every call micro service we do some auth checks so we should calculate the time taken for checks
3. Time taken by any external call we make i.e calls to external service which are not part of the your micro services
4. Time taken by the db calls
5. Time taken by the other micro services
6. Actual time taken by logic in side micro service

Event log names :(Suggestion )

1. All external Calls name the event name as ExternalAPI and url path as URI
2. All Security calls log with event name as : AuthCheck
3. DB calls event name as : DB
4. Calls to other micro services event name as : OtherMicroService

How to implement if your using different HTTP client and using spring eco systems (sleuth for log correlation ) :

Write simple interceptor which intercept each request and add time stamps for all out bound . I have written simple interceptor which interceptor all our bound calls from micro service and log the response time but give appropriate log event name depends on the call your going to make .

How it helped us :

We have taken each request and looked at the time taken by each micro services and in micro services how much taken by each of above log events . After looking at all events we have improved performance by doing one or some or all of the following

1. Cached the external call api responses
2. Added indexes
3. Refactored code to perform better
4. In some cases actual time is taken by other micro services which is called from this micro service so we applied one of the above in other micro service