Problem Statement :
Introduce Alerts mechanism in Kibana and integrate with Teams channel
a. For HTTP Errors
b. For Performance
For HTTP Errors
Query Preparation for Error Extraction
Where
Translated JSON looks like this
WildCard Query:
Match Phrase Query:
Create the Action
Give mandatory details and Select Destination
Add teams destination
Select the teams channel configuration
Teams specific task
1. You have to create the teams channel
2. Add the App name in there Incoming Webhook this will give you Webhook URL which you can use for the Teams alert integration.
Create the message which you want to share on teams once alerts occur and Copy in Action Message steps
b. For Performance
All the steps are same except the Query extraction are same . Query extraction which has to be done very carefully
Create your Query For example I have created the Query which
a.Fetch API from ELECTORNICS which are taking more than 3 seconds
Introduce Alerts mechanism in Kibana and integrate with Teams channel
a. For HTTP Errors
b. For Performance
For HTTP Errors
Query Preparation for Error Extraction
- Prepare the Query in "Query workbench and do the explain ,Copy this Json somewhere
Select count(*) from logs-json-* where logs.status >= 400 and logs.localServerName is "api.cuxuzunfzu-public.model-t.cc.commerce.ondemand.com" and logs.requestFirstLine LIKE "%electronics%"Where
electronics is Site ID for which you want to monitor errors on api Node . You can customize the Query based on your needsTranslated JSON looks like this
{
"from": 0,
"size": 0,
"query": {
"bool": {
"filter": [
{
"bool": {
"must": [
{
"bool": {
"must": [
{
"range": {
"logs.status": {
"from": 400,
"to": null,
"include_lower": true,
"include_upper": true,
"boost": 1
}
}
},
{
"term": {
"logs.localServerName.keyword": {
"value": "api.cuxuzunfzu-public.model-t.cc.commerce.ondemand.com",
"boost": 1
}
}
},
{
"wildcard": {
"logs.requestFirstLine": {
"wildcard": "*electronics*",
"boost": 1
}
}
},
{
"range": {
"time": {
"from": "now-10m",
"to": null,
"include_lower": true,
"include_upper": true,
"boost": 1
}
}
}
],
"adjust_pure_negative": true,
"boost": 1
}
}
],
"adjust_pure_negative": true,
"boost": 1
}
}
],
"adjust_pure_negative": true,
"boost": 1
}
},
"_source": {
"includes": [
"COUNT"
],
"excludes": [
]
},
"aggregations": {
"COUNT_0": {
"value_count": {
"field": "_index"
}
}
}
}
You can also change the wildcard from exact match Translated json like this- Please update the wildcard query with match-phrase query in above Json.
WildCard Query:
"wildcard": {
"logs.requestFirstLine": {
"wildcard": "*electronics*",
"boost": 1
}
}
Match Phrase Query:
"match_phrase": {Note: If we have to use hyphen "-" is not recognized as a character in the LIKE/WILDCARD query, so you can use MATCH_PHRASE instead. If you want to monitor error occured in last 10m or nay frequency this is the code Snippet
"logs.requestFirstLine": {
"query": "electronics",
"boost": 1
}
}
{} The above Jso has to copied in Extraction Query after craeting the montor Create the Monitor like below
"range": {
"time": {
"from": "now-10m",
"to": null,
"include_lower": true,
"include_upper": true,
"boost": 1
}
}
}Create the Trigger condition You can define priorty based on your business needs like in 10 mins more that 5 erros are high alerts Here in below example any error is High
Create the Action
Give mandatory details and Select Destination
Add teams destination
Select the teams channel configuration
Teams specific task
1. You have to create the teams channel
2. Add the App name in there Incoming Webhook this will give you Webhook URL which you can use for the Teams alert integration.
Create the message which you want to share on teams once alerts occur and Copy in Action Message steps
{
"@type":"MessageCard",
"@context":"https://schema.org/extensions",
"summary":"Error for ELECTRONICS",
"themeColor":"FF0000",
"sections":[
{
"activityImage":"URL",
"activityTitle": "Errors Produced for ELECTRONICS SITE in Last 10 minutes ",
"facts":[
{
"name":"Reason: ",
"value":"ERROR-CODE - More than 400. <b>Number of Occurrences: {{ctx.results.0.hits.total.value}}</b>""
}
],
"text":"Api Calls failed.",
"potentialAction":[
{
"@type":"OpenUri",
"name" : "Check Kibana",
"targets":[
{
"os":"default",
"uri":"URL FOR KIBANA "
}
]
}
]
}
]
}
b. For Performance
All the steps are same except the Query extraction are same . Query extraction which has to be done very carefully
Create your Query For example I have created the Query which
a.Fetch API from ELECTORNICS which are taking more than 3 seconds
SELECT count(*) FROM logs-json-* where logs.responseTime is not null and logs.localServerName is "api.cuxuzunfzu-public.model-t.cc.commerce.ondemand.com" and logs.requestFirstLine like "%ELECTRONCIS%"
and CAST(logs.responseTime AS INTEGER) > 3000
{
"from": 0,
"size": 0,
"query": {
"bool": {
"filter": [
{
"bool": {
"filter": [
{
"bool": {
"filter": [
{
"script": {
"script": {
"source": "XXXXXXXXXXXXXcWwuZXhwcmVzc2lvbi5mdW5jdGlvbi5GdW5jdGlvbkRTTCQy7+eYHhdeXG4CAARMAA12YWwkYXJndW1lbnRzdAAQTGphdmEvdXRpbC9MaXN0O0wADHZhbCRmdW5jdGlvbnQAPUxvcmcvb3BlbnNlYXJjaC9zcWwvZXhwcmVzc2lvbi9mdW5jdGlvbi9TZXJpYWxpemFibGVGdW5jdGlvbjtMABB2YWwkZnVuY3Rpb25OYW1ldAA1TG9yZy9vcGVuc2VhcmNoL3NxbC9leHByZXNzaW9uL2Z1bmN0aW9uL0Z1bmN0aW9uTmFtZTtMAA52YWwkcmV0dXJuVHlwZXQAJ0xvcmcvb3BlbnNlYXJjaC9zcWwvZGF0YS90eXBlL0V4cHJUeXBlO3hyADBvcmcub3BlbnNlYXJjaC5zcWwuZXhwcmVzc2lvbi5GdW5jdGlvbkV4cHJlc3Npb26yKjDT3HVqewIAAkwACWFyZ3VtZW50c3EAfgABTAAMZnVuY3Rpb25OYW1lcQB+AAN4cHNyABNqYXZhLnV0aWwuQXJyYXlMaXN0eIHSHZnHYZ0DAAFJAARzaXpleHAAAAABdwQAAAABc3IAMW9yZy5vcGVuc2VhcmNoLnNxbC5leHByZXNzaW9uLlJlZmVyZW5jZUV4cHJlc3Npb274AO0rxWvMkAIAA0wABGF0dHJ0ABJMamF2YS9sYW5nL1N0cmluZztMAAVwYXRoc3EAfgABTAAEdHlwZXEAfgAEeHB0ABFsb2dzLnJlc3BvbnNlVGltZXNyABpqYXZhLnV0aWwuQXJyYXlzJEFycmF5TGlzdNmkPL7NiAbSAgABWwABYXQAE1tMamF2YS9sYW5nL09iamVjdDt4cHVyABNbTGphdmEubGFuZy5TdHJpbmc7rdJW5+kde0cCAAB4cAAAAAJ0AARsb2dzdAAMcmVzcG9uc2VUaW1lfnIAOm9yZy5vcGVuc2VhcmNoLnNxbC5vcGVuc2VhcmNoLmRhdGEudHlwZS5PcGVuU2VhcmNoRGF0YVR5cGUAAAAAAAAAABIAAHhyAA5qYXZhLmxhbmcuRW51bQAAAAAAAAAAEgAAeHB0ABdPUEVOU0VBUkNIX1RFWFRfS0VZV09SRHhzcgAzb3JnLm9wZW5zZWFyY2guc3FsLmV4cHJlc3Npb24uZnVuY3Rpb24uRnVuY3Rpb25OYW1lC6g4Tc72Z5cCAAFMAAxmdW5jdGlvbk5hbWVxAH4ACnhwdAALaXMgbm90IG51bGxxAH4ACHNyACFqYXZhLmxhbmcuaW52b2tlLlNlcmlhbGl6ZWRMYW1iZGFvYdCULCk2hQIACkkADmltcGxNZXRob2RLaW5kWwAMY2FwdHVyZWRBcmdzcQB+AA5MAA5jYXB0dXJpbmdDbGFzc3QAEUxqYXZhL2xhbmcvQ2xhc3M7TAAYZnVuY3Rpb25hbEludGVyZmFjZUNsYXNzcQB+AApMAB1mdW5jdGlvbmFsSW50ZXJmYWNlTWV0aG9kTmFtZXEAfgAKTAAiZnVuY3Rpb25hbEludGVyZmFjZU1ldGhvZFNpZ25hdHVyZXEAfgAKTAAJaW1wbENsYXNzcQB+AApMAA5pbXBsTWV0aG9kTmFtZXEAfgAKTAATaW1wbE1ldGhvZFNpZ25hdHVyZXEAfgAKTAAWaW5zdGFudGlhdGVkTWV0aG9kVHlwZXEAfgAKeHAAAAAGdXIAE1tMamF2YS5sYW5nLk9iamVjdDuQzlifEHMpbAIAAHhwAAAAAHZyAEdvcmcub3BlbnNlYXJjaC5zcWwuZXhwcmVzc2lvbi5vcGVyYXRvci5wcmVkaWNhdGUuVW5hcnlQcmVkaWNhdGVPcGVyYXRvcgAAAAAAAAAAAAAAeHB0ADtvcmcvb3BlbnNlYXJjaC9zcWwvZXhwcmVzc2lvbi9mdW5jdGlvbi9TZXJpYWxpemFibGVGdW5jdGlvbnQABWFwcGx5dAAmKExqYXZhL2xhbmcvT2JqZWN0OylMamF2YS9sYW5nL09iamVjdDt0AEdvcmcvb3BlbnNlYXJjaC9zcWwvZXhwcmVzc2lvbi9vcGVyYXRvci9wcmVkaWNhdGUvVW5hcnlQcmVkaWNhdGVPcGVyYXRvcnQAG2xhbWJkYSRpc05vdE51bGwkODc3ODgwNzckMXQAVChMb3JnL29wZW5zZWFyY2gvc3FsL2RhdGEvbW9kZWwvRXhwclZhbHVlOylMb3JnL29wZW5zZWFyY2gvc3FsL2RhdGEvbW9kZWwvRXhwclZhbHVlO3EAfgAncQB+ABl+cgApb3JnLm9wZW5zZWFyY2guc3FsLmRhdGEudHlwZS5FeHByQ29yZVR5cGUAAAAAAAAAABIAAHhxAH4AFXQAB0JPT0xFQU4=",
"lang": "opensearch_query_expression"
},
"boost": 1.0
}
},
{
"wildcard": {
"logs.requestFirstLine.keyword": {
"wildcard": "*ELECTRONICS*",
"boost": "1.0"
}
}
}
],
"adjust_pure_negative:true,boost": "1.0"
}
},
{
"wildcard": {
"logs.requestFirstLine.keyword": {
"wildcard": "*locationId*",
"boost": "1.0"
}
}
},
{
"range": {
"time": {
"from": "now-10m",
"to": null,
"include_lower": true,
"include_upper": true,
"boost": 1
}
}
}
],
"adjust_pure_negative": "true",
"boost": "1.0"
}
},
{
"script": {
"script": {
"source": "XXXXXXXX/macCAARMAA12YWwkYXJndW1lbnRzdAAQTGphdmEvdXRpbC9MaXN0O0wADHZhbCRmdW5jdGlvbnQAP0xvcmcvb3BlbnNlYXJjaC9zcWwvZXhwcmVzc2lvbi9mdW5jdGlvbi9TZXJpYWxpemFibGVCaUZ1bmN0aW9uO0wAEHZhbCRmdW5jdGlvbk5hbWV0ADVMb3JnL29wZW5zZWFyY2gvc3FsL2V4cHJlc3Npb24vZnVuY3Rpb24vRnVuY3Rpb25OYW1lO0wADnZhbCRyZXR1cm5UeXBldAAnTG9yZy9vcGVuc2VhcmNoL3NxbC9kYXRhL3R5cGUvRXhwclR5cGU7eHIAMG9yZy5vcGVuc2VhcmNoLnNxbC5leHByZXNzaW9uLkZ1bmN0aW9uRXhwcmVzc2lvbrIqMNPcdWp7AgACTAAJYXJndW1lbnRzcQB+AAFMAAxmdW5jdGlvbk5hbWVxAH4AA3hwc3IAE2phdmEudXRpbC5BcnJheUxpc3R4gdIdmcdhnQMAAUkABHNpemV4cAAAAAJ3BAAAAAJzcgA0b3JnLm9wZW5zZWFyY2guc3FsLmV4cHJlc3Npb24uZnVuY3Rpb24uRnVuY3Rpb25EU0wkMu/nmB4XXlxuAgAETAANdmFsJGFyZ3VtZW50c3EAfgABTAAMdmFsJGZ1bmN0aW9udAA9TG9yZy9vcGVuc2VhcmNoL3NxbC9leHByZXNzaW9uL2Z1bmN0aW9uL1NlcmlhbGl6YWJsZUZ1bmN0aW9uO0wAEHZhbCRmdW5jdGlvbk5hbWVxAH4AA0wADnZhbCRyZXR1cm5UeXBlcQB+AAR4cQB+AAVzcQB+AAcAAAABdwQAAAABc3IAMW9yZy5vcGVuc2VhcmNoLnNxbC5leHByZXNzaW9uLlJlZmVyZW5jZUV4cHJlc3Npb274AO0rxWvMkAIAA0wABGF0dHJ0ABJMamF2YS9sYW5nL1N0cmluZztMAAVwYXRoc3EAfgABTAAEdHlwZXEAfgAEeHB0ABFsb2dzLnJlc3BvbnNlVGltZXNyABpqYXZhLnV0aWwuQXJyYXlzJEFycmF5TGlzdNmkPL7NiAbSAgABWwABYXQAE1tMamF2YS9sYW5nL09iamVjdDt4cHVyABNbTGphdmEubGFuZy5TdHJpbmc7rdJW5+kde0cCAAB4cAAAAAJ0AARsb2dzdAAMcmVzcG9uc2VUaW1lfnIAOm9yZy5vcGVuc2VhcmNoLnNxbC5vcGVuc2VhcmNoLmRhdGEudHlwZS5PcGVuU2VhcmNoRGF0YVR5cGUAAAAAAAAAABIAAHhyAA5qYXZhLmxhbmcuRW51bQAAAAAAAAAAEgAAeHB0ABdPUEVOU0VBUkNIX1RFWFRfS0VZV09SRHhzcgAzb3JnLm9wZW5zZWFyY2guc3FsLmV4cHJlc3Npb24uZnVuY3Rpb24uRnVuY3Rpb25OYW1lC6g4Tc72Z5cCAAFMAAxmdW5jdGlvbk5hbWVxAH4ADnhwdAALY2FzdF90b19pbnRxAH4ADHNyACFqYXZhLmxhbmcuaW52b2tlLlNlcmlhbGl6ZWRMYW1iZGFvYdCULCk2hQIACkkADmltcGxNZXRob2RLaW5kWwAMY2FwdHVyZWRBcmdzcQB+ABJMAA5jYXB0dXJpbmdDbGFzc3QAEUxqYXZhL2xhbmcvQ2xhc3M7TAAYZnVuY3Rpb25hbEludGVyZmFjZUNsYXNzcQB+AA5MAB1mdW5jdGlvbmFsSW50ZXJmYWNlTWV0aG9kTmFtZXEAfgAOTAAiZnVuY3Rpb25hbEludGVyZmFjZU1ldGhvZFNpZ25hdHVyZXEAfgAOTAAJaW1wbENsYXNzcQB+AA5MAA5pbXBsTWV0aG9kTmFtZXEAfgAOTAATaW1wbE1ldGhvZFNpZ25hdHVyZXEAfgAOTAAWaW5zdGFudGlhdGVkTWV0aG9kVHlwZXEAfgAOeHAAAAAGdXIAE1tMamF2YS5sYW5nLk9iamVjdDuQzlifEHMpbAIAAHhwAAAAAXNxAH4AHwAAAAZ1cQB+ACIAAAAAdnIAP29yZy5vcGVuc2VhcmNoLnNxbC5leHByZXNzaW9uLm9wZXJhdG9yLmNvbnZlcnQuVHlwZUNhc3RPcGVyYXRvcgAAAAAAAAAAAAAAeHB0ADtvcmcvb3BlbnNlYXJjaC9zcWwvZXhwcmVzc2lvbi9mdW5jdGlvbi9TZXJpYWxpemFibGVGdW5jdGlvbnQABWFwcGx5dAAmKExqYXZhL2xhbmcvT2JqZWN0OylMamF2YS9sYW5nL09iamVjdDt0AD9vcmcvb3BlbnNlYXJjaC9zcWwvZXhwcmVzc2lvbi9vcGVyYXRvci9jb252ZXJ0L1R5cGVDYXN0T3BlcmF0b3J0ABtsYW1iZGEkY2FzdFRvSW50JDg3Nzg4MDc3JDF0AFQoTG9yZy9vcGVuc2VhcmNoL3NxbC9kYXRhL21vZGVsL0V4cHJWYWx1ZTspTG9yZy9vcGVuc2VhcmNoL3NxbC9kYXRhL21vZGVsL0V4cHJWYWx1ZTtxAH4ALXZyADJvcmcub3BlbnNlYXJjaC5zcWwuZXhwcmVzc2lvbi5mdW5jdGlvbi5GdW5jdGlvbkRTTAAAAAAAAAAAAAAAeHBxAH4AKHEAfgApcQB+ACp0ADJvcmcvb3BlbnNlYXJjaC9zcWwvZXhwcmVzc2lvbi9mdW5jdGlvbi9GdW5jdGlvbkRTTHQAJWxhbWJkYSRudWxsTWlzc2luZ0hhbmRsaW5nJDg3ODA2OWM4JDF0AJEoTG9yZy9vcGVuc2VhcmNoL3NxbC9leHByZXNzaW9uL2Z1bmN0aW9uL1NlcmlhbGl6YWJsZUZ1bmN0aW9uO0xvcmcvb3BlbnNlYXJjaC9zcWwvZGF0YS9tb2RlbC9FeHByVmFsdWU7KUxvcmcvb3BlbnNlYXJjaC9zcWwvZGF0YS9tb2RlbC9FeHByVmFsdWU7cQB+AC1xAH4AHX5yAClvcmcub3BlbnNlYXJjaC5zcWwuZGF0YS50eXBlLkV4cHJDb3JlVHlwZQAAAAAAAAAAEgAAeHEAfgAZdAAHSU5URUdFUnNyAC9vcmcub3BlbnNlYXJjaC5zcWwuZXhwcmVzc2lvbi5MaXRlcmFsRXhwcmVzc2lvbkVCLfCMx4IkAgABTAAJZXhwclZhbHVldAApTG9yZy9vcGVuc2VhcmNoL3NxbC9kYXRhL21vZGVsL0V4cHJWYWx1ZTt4cHNyAC5vcmcub3BlbnNlYXJjaC5zcWwuZGF0YS5tb2RlbC5FeHBySW50ZWdlclZhbHVlpmwHmYm3bkMCAAB4cgA1b3JnLm9wZW5zZWFyY2guc3FsLmRhdGEubW9kZWwuQWJzdHJhY3RFeHByTnVtYmVyVmFsdWUb79eD5r/h5AIAAUwABXZhbHVldAASTGphdmEvbGFuZy9OdW1iZXI7eHIAL29yZy5vcGVuc2VhcmNoLnNxbC5kYXRhLm1vZGVsLkFic3RyYWN0RXhwclZhbHVl5d/SeNHJxJQCAAB4cHNyABFqYXZhLmxhbmcuSW50ZWdlchLioKT3gYc4AgABSQAFdmFsdWV4cgAQamF2YS5sYW5nLk51bWJlcoaslR0LlOCLAgAAeHAAAAu4eHNxAH4AHHQAAT5xAH4ACHNxAH4AHwAAAAZ1cQB+ACIAAAABc3EAfgAfAAAABnVxAH4AIgAAAAB2cgBIb3JnLm9wZW5zZWFyY2guc3FsLmV4cHJlc3Npb24ub3BlcmF0b3IucHJlZGljYXRlLkJpbmFyeVByZWRpY2F0ZU9wZXJhdG9yAAAAAAAAAAAAAAB4cHQAPW9yZy9vcGVuc2VhcmNoL3NxbC9leHByZXNzaW9uL2Z1bmN0aW9uL1NlcmlhbGl6YWJsZUJpRnVuY3Rpb25xAH4AKXQAOChMamF2YS9sYW5nL09iamVjdDtMamF2YS9sYW5nL09iamVjdDspTGphdmEvbGFuZy9PYmplY3Q7dABIb3JnL29wZW5zZWFyY2gvc3FsL2V4cHJlc3Npb24vb3BlcmF0b3IvcHJlZGljYXRlL0JpbmFyeVByZWRpY2F0ZU9wZXJhdG9ydAAZbGFtYmRhJGdyZWF0ZXIkYTAxY2U0MzAkMXQAfShMb3JnL29wZW5zZWFyY2gvc3FsL2RhdGEvbW9kZWwvRXhwclZhbHVlO0xvcmcvb3BlbnNlYXJjaC9zcWwvZGF0YS9tb2RlbC9FeHByVmFsdWU7KUxvcmcvb3BlbnNlYXJjaC9zcWwvZGF0YS9tb2RlbC9FeHByVmFsdWU7cQB+AE1xAH4AL3EAfgBJcQB+AClxAH4ASnEAfgAwdAAlbGFtYmRhJG51bGxNaXNzaW5nSGFuZGxpbmckYTUwMDUyODEkMXQAvChMb3JnL29wZW5zZWFyY2gvc3FsL2V4cHJlc3Npb24vZnVuY3Rpb24vU2VyaWFsaXphYmxlQmlGdW5jdGlvbjtMb3JnL29wZW5zZWFyY2gvc3FsL2RhdGEvbW9kZWwvRXhwclZhbHVlO0xvcmcvb3BlbnNlYXJjaC9zcWwvZGF0YS9tb2RlbC9FeHByVmFsdWU7KUxvcmcvb3BlbnNlYXJjaC9zcWwvZGF0YS9tb2RlbC9FeHByVmFsdWU7cQB+AE1xAH4AQX5xAH4AM3QAB0JPT0xFQU4=",
"lang": "opensearch_query_expression"
},
"boost": "1.0"
}
}
],
"adjust_pure_negative:true,boost": "1.0"
}
},
"sort": [
{
"_doc": {
"order": "asc"
}
}
],
"aggregations": {
"count(*)": {
"value_count": {
"field": "_index"
}
}
}
}
DO the Explain and copy the inner translated json in New Performance Error monitor
- SAP Managed Tags:
