High CPU usage per core(s)-juniper-junos

error
health-checks
junos
juniper
High CPU usage per core(s)-juniper-junos
0

#1

High CPU usage per core(s)-juniper-junos

Vendor: juniper

OS: junos

Description:
High CPU usage is a symptom of a system which is unable to handle " +
"the required load or a symptom of a specific issue with the system " +
"and the applications and services running on it. Indeni will monitor the CPU usage " +
"of each core separately and alert if any of the cores’ CPU usage crosses the threshold.

Remediation Steps:
Determine the cause for the high CPU usage of the listed cores.
|||The Juniper SRX device may start dropping packets if CPU utilization reaches 100%. In order to determine the root cause of high CPU usage:
|1. Check the CPU status in the routing engine by running “show chassis routing-engine” in the command-line interface.
|2. Identify the top running processes which hold most of CPU cycle by running the command “show system processes extensive”. Consider restarting or ending processes if too many events are being handled (e.g. sampling, traceoptions, syslog, snmp).
|3. Check CPU utilization in the forwarding engine by running “show chassis forwarding”. If CPU is high it may be indicative of the device reaching capacity. |
|See Juniper Tech Library on CPU load thresholds or contact Juniper technical support for further troubleshooting.

How does this work?
This script and others use the CLI over SSH to retrieve the current status of multiple different CPU elements.

Why is this important?
The control and data plane CPU utilization of a Juniper JUNOS device are important to track to ensure a smooth operation. A high CPU utilization of the control plane may impact the management interface, while a high CPU utilization in the data plane may impact traffic handling.

Without Indeni how would you find this?
CPU utilization information at both the control and data plane levels is available via SNMP and can be monitored using an SNMP-based tool. An administrator can then define thresholds against this.

junos-show-chassis-routing-engine

#! META
name: junos-show-chassis-routing-engine
description: JUNOS get routing engine stats (CPU/mem)
type: monitoring
includes_resource_data: true
monitoring_interval: 1 minute
requires:
    vendor: juniper
    os.name: junos
    product: firewall
    high-availability: 
        neq: true

#! COMMENTS
cpu-usage:
    why: |
        The control and data plane CPU utilization of a Juniper JUNOS device are important to track to ensure a smooth operation. A high CPU utilization of the control plane may impact the management interface, while a high CPU utilization in the data plane may impact traffic handling.
    how: |
        This script and others use the CLI over SSH to retrieve the current status of multiple different CPU elements.
    without-indeni: |
        CPU utilization information at both the control and data plane levels is available via SNMP and can be monitored using an SNMP-based tool. An administrator can then define thresholds against this.
    can-with-snmp: true
    can-with-syslog: false
memory-usage:
    why: |
        The various memory components of a Juniper JUNOS device are important to track to ensure a smooth operation. This includes the routing engine's memory element (RE) as well as the variety of data plane elements.
    how: |
        This script and others use the CLI over SSH to retrieve the current status of multiple different memory elements.
    without-indeni: |
        Some of the memory elements' status is accessible over SNMP, but many of the memory elements in the data plane are solely accessible over SSH. An administrator would need to write their own scripts to collect this information.
    can-with-snmp: false
    can-with-syslog: false

#! REMOTE::SSH
show chassis routing-engine | display xml

#! PARSER::XML
_vars:
    root: /rpc-reply//route-engine-information[1]
_metrics:
    -
        _tags:
            "im.name":
                _constant: "cpu-usage"
            "live-config":
                _constant: "true"
            "display-name":
                _constant: "CPU Usage"
            "im.dstype.displayType":
                _constant: "percentage"
            "cpu-id":
                _constant: "RE"
            "cpu-is-avg":
                _constant: "false"
            "resource-metric":
                _constant: "true"
            "im.identity-tags":
                _constant: "cpu-id"
        _temp:
            "cpu_idle":
                _text: ${root}/route-engine/cpu-idle
        _transform:
            _value.double: |
                {
                    idle_cpu = 100 - temp("cpu_idle")
                    print idle_cpu 
                }

# CONTROL PLANE MEMORY
    -
        _tags:
            "im.name":
                _constant: "memory-total-kbytes"
            "live-config":
                _constant: "true"
            "display-name":
                _constant: "Memory Used"
            "im.dstype.displayType":
                _constant: "kilobytes"
            "name":
                _constant: "Control Plane"
            "im.identity-tags":
                _constant: "name"

        _temp:
            "cp_total_mem":
                _text: ${root}/route-engine/memory-control-plane
        _transform:
            _value.double: |
                {
                    cp_total_memory = temp("cp_total_mem") * 1024
                    print cp_total_memory
                }
    -
        _tags:
            "im.name":
                _constant: "memory-free-kbytes"
            "live-config":
                _constant: "true"
            "display-name":
                _constant: "Memory Free"
            "im.dstype.displayType":
                _constant: "kilobytes"
            "name":
                _constant: "Control Plane"
            "im.identity-tags":
                _constant: "name"

        _temp:
            "cp_total_mem":
                _text: ${root}/route-engine/memory-control-plane
            "cp_used_mem":
                _text: ${root}/route-engine/memory-control-plane-used
        _transform:
            _value.double: |
                {
                    cp_free_memory = (temp("cp_total_mem") - temp("cp_used_mem")) * 1024
                    print cp_free_memory 
                }
    -
        _tags:
            "im.name":
                _constant: "memory-usage"
            "live-config":
                _constant: "true"
            "display-name":
                _constant: "Memory Usage"
            "im.dstype.displayType":
                _constant: "percentage"
            "name":
                _constant: "Control Plane"
            "resource-metric":
                _constant: "true"
            "im.identity-tags":
                _constant: "name"
        _temp:
            "cp_mem_usage":
                _text: ${root}/route-engine/memory-control-plane-util
        _transform:
            _value.double: |
                {
                    print temp("cp_mem_usage")
                }

# DATA PLANE MEMORY
    -
        _tags:
            "im.name":
                _constant: "memory-total-kbytes"
            "live-config":
                _constant: "true"
            "display-name":
                _constant: "Memory Used"
            "im.dstype.displayType":
                _constant: "kilobytes"
            "name":
                _constant: "Data Plane"
            "im.identity-tags":
                _constant: "name"

        _temp:
            "dp_total_mem":
                _text: ${root}/route-engine/memory-data-plane
        _transform:
            _value.double: |
                {
                    dp_total_memory = temp("dp_total_mem") * 1024
                    print dp_total_memory
                }
    -
        _tags:
            "im.name":
                _constant: "memory-free-kbytes"
            "live-config":
                _constant: "true"
            "display-name":
                _constant: "Memory Free"
            "im.dstype.displayType":
                _constant: "kilobytes"
            "name":
                _constant: "Data Plane"
            "im.identity-tags":
                _constant: "name"

        _temp:
            "dp_total_mem":
                _text: ${root}/route-engine/memory-data-plane
            "dp_used_mem":
                _text: ${root}/route-engine/memory-data-plane-used
        _transform:
            _value.double: |
                {
                    dataplane_memory = (temp("dp_total_mem") - temp("dp_used_mem")) * 1024
                    print dataplane_memory
                }
    -
        _tags:
            "im.name":
                _constant: "memory-usage"
            "live-config":
                _constant: "true"
            "display-name":
                _constant: "Memory Usage"
            "im.dstype.displayType":
                _constant: "percentage"
            "name":
                _constant: "Data Plane"
            "im.identity-tags":
                _constant: "name"
        _temp:
            "dp_mem_usage":
                _text: ${root}/route-engine/memory-data-plane-util
        _transform:
            _value.double: |
                {
                    print temp("dp_mem_usage")
                }

high_per_core_cpu_use_by_device

package com.indeni.server.rules.library

import com.indeni.apidata.time.TimeSpan
import com.indeni.ruleengine.expressions.conditions.{And, ConditionHelper, GreaterThanOrEqual}
import com.indeni.ruleengine.expressions.core.{StatusTreeExpression, _}
import com.indeni.ruleengine.expressions.data.{SelectTagsExpression, SelectTimeSeriesExpression, TimeSeriesExpression}
import com.indeni.ruleengine.expressions.math.MinExpression
import com.indeni.ruleengine.expressions.scope.ScopeValueExpression
import com.indeni.server.common.ParameterValue
import com.indeni.server.common.data.conditions.True
import com.indeni.server.params.ParameterDefinition
import com.indeni.server.params.ParameterDefinition.UIType
import com.indeni.server.rules._
import com.indeni.server.rules.config.expressions.DynamicParameterExpression
import com.indeni.server.rules.library.core.PerDeviceRule
import com.indeni.server.sensor.models.managementprocess.alerts.dto.AlertSeverity

case class HighPerCoreCpuUsageRule(context: RuleContext) extends PerDeviceRule with RuleHelper {

  private val highThresholdParameterName = "high_threshold_of_cpu_usage"
  private val highThresholdParameter = new ParameterDefinition(highThresholdParameterName,
    "",
    "High Threshold of CPU Usage",
    "What is the threshold for the CPU usage for which once it is crossed an alert will be issued.",
    UIType.DOUBLE,
    new ParameterValue((70.0).asInstanceOf[Object])
  )

  private val minimumNumberOfCoresParameterName = "higher_than_threshold_cores"
  private val minimumNumberOfCoresParameter = new ParameterDefinition(minimumNumberOfCoresParameterName,
    "",
    "Number of Cores",
    "The number of CPU cores with usage above the value set in " +
      "\"" + highThresholdParameterName + "\"" +
      " before an alert is issued.",
    UIType.INTEGER,
    new ParameterValue((1).asInstanceOf[Object])
  )

  private val timeThresholdParameterName = "time_threshold"
  private val timeThresholdParameter = new ParameterDefinition(timeThresholdParameterName,
    "",
    "Time Threshold",
    "The CPU cores need to remain above the value set in " +
      "\"" + highThresholdParameterName + "\"" +
      " for this amount of time before an alert is issued.",
    UIType.TIMESPAN,
    TimeSpan.fromMinutes(10))


  override def metadata: RuleMetadata = RuleMetadata.builder("high_per_core_cpu_use_by_device", "All Devices: High CPU usage per core(s)",
    "High CPU usage is a symptom of a system which is unable to handle " +
      "the required load or a symptom of a specific issue with the system " +
      "and the applications and services running on it. Indeni will monitor the CPU usage " +
      "of each core separately and alert if any of the cores' CPU usage crosses the threshold.",
    AlertSeverity.ERROR,
    Set(RuleCategory.HighAvailability)).
    configParameters(highThresholdParameter, minimumNumberOfCoresParameter, timeThresholdParameter).
    build()

  override def expressionTree: StatusTreeExpression = {
    val inUseValue = MinExpression(TimeSeriesExpression[Double]("cpu-usage"))

    StatusTreeExpression(
      // Which objects to pull (normally, devices)
      SelectTagsExpression(context.metaDao, Set(DeviceKey), True),

      // What constitutes an issue
      StatusTreeExpression(

        // The additional tags we care about (we'll be including this in alert data)
        SelectTagsExpression(context.tsDao, Set("cpu-id", "cpu-is-avg"), True),

        StatusTreeExpression(
          // The time-series we check the test condition against:
          SelectTimeSeriesExpression[Double](context.tsDao, Set("cpu-usage"), historyLength = getParameterTimeSpanForRule(timeThresholdParameter)),

          // The condition which, if true, we have an issue. Checked against the time-series we've collected
          And(
            ScopeValueExpression("cpu-is-avg").visible().isIn(Set("true")).not,
            GreaterThanOrEqual(
              inUseValue,
              getParameterDouble(highThresholdParameter)))

          // The Alert Item to add for this specific item
        ).withSecondaryInfo(
          scopableStringFormatExpression("${scope(\"cpu-id\")}"),
          scopableStringFormatExpression("Current CPU utilization is: %.0f%%", inUseValue),
          title = "Cores with High CPU Usage"
        ).asCondition(minimumIssueCount = DynamicParameterExpression.withConstantDefault(minimumNumberOfCoresParameter.getName, minimumNumberOfCoresParameter.getDefaultValue.asInteger().intValue()))
      ).withoutInfo().asCondition()
    ).withRootInfo(
      getHeadline(),
      ConstantExpression("Some CPU cores are under high usage."),
      ConditionalRemediationSteps("Determine the cause for the high CPU usage of the listed cores.",
        ConditionalRemediationSteps.VENDOR_CP -> "An extremely helpful article on high CPU utilization on Check Point firewalls is available here: <a target=\"_blank\" href=\"https://supportcenter.checkpoint.com/supportcenter/portal?eventSubmit_doGoviewsolutiondetails=&solutionid=sk98348\">Best Practices - Security Gateway Performance</a>.",
        ConditionalRemediationSteps.VENDOR_PANOS ->
          """|For MP (management plane) CPU, look at <a target="_blank" href="https://live.paloaltonetworks.com/t5/Featured-Articles/Tips-amp-Tricks-Reducing-Management-Plane-Load/ta-p/64681">Tips & Tricks: Reducing Management Plane Load</a>.
             |For DP (data plane), read <a target="_blank" href="https://live.paloaltonetworks.com/t5/Featured-Articles/How-to-Troubleshoot-High-Dataplane-CPU/ta-p/73000">How to Troubleshoot High Dataplane CPU</a>.""".stripMargin,
        ConditionalRemediationSteps.VENDOR_JUNIPER ->
          """|The Juniper SRX device may start dropping packets if CPU utilization reaches 100%. In order to determine the root cause of high CPU usage:
             |1. Check the CPU status in the routing engine by running "show chassis routing-engine" in the command-line interface.
             |2. Identify the top running processes which hold most of CPU cycle by running the command "show system processes extensive". Consider restarting or ending processes if too many events are being handled (e.g. sampling, traceoptions, syslog, snmp).
             |3. Check CPU utilization in the forwarding engine by running "show chassis forwarding". If CPU is high it may be indicative of the device reaching capacity.             |
             |See <a target="_blank" href="https://kb.juniper.net/InfoCenter/index?page=content&id=KB20989">Juniper Tech Library</a> on CPU load thresholds or contact Juniper technical support for further troubleshooting.""".stripMargin,
        ConditionalRemediationSteps.VENDOR_FORTINET ->
          """
             |1. Login via https to the Fortinet firewall and go to menu "System > Dashboard > Status" and look at the system resources widget to review the current CPU utilization graph.
             |2. Login via ssh to the Fortinet firewall and run the FortiOS command “get system performance status”. The first line of output shows the CPU usage by category. The other lines of the output, such as average network usage, average session setup rate, viruses caught, and IPS attacks blocked can also help to determine why system resource usage is high. For example, if network usage is high it will result in high traffic processing on the FortiGate; or if the session setup rate is very low or zero the proxy may be overloaded and not able to do its job.
             |3. Login via ssh to the Fortinet firewall and run the FortiOS command “get system performance top”. This command shows all the top processes running on the FortiGate unit and their CPU usage. If a process is using most of the CPU cycles, investigate it to determine if it’s normal activity. If the top few entries are using most of the CPU, note which processes they are and investigate those features to try and reduce their CPU load. Some common examples of processes you will see include:  ipsengine, scanunitd (antivirus), iked and sshd.
             |4. For more information review: https://docs.fortinet.com/uploaded/files/2924/troubleshooting-54.pdf
             |5. If the problem persists, contact Fortinet Technical support at https://support.fortinet.com/ for further assistance.""".stripMargin,
        ConditionalRemediationSteps.VENDOR_BLUECOAT ->
        """
            |1. Login via https to the ProxySG and go to Statistics > System > Resources > CPU. Review the current CPU utilization graph.
            |2. Login via ssh to the ProxySG and run the command "show cpu" which provides information about current memory usage.
            |3. Try to troubleshoot the cause using the CPU monitor feature: https://x.x.x.x:8082/Diagnostics/CPU_Monitor/statistics
            |4. Check the ICAP service maximum number of connections. For more information review the following Bluecoat guide:
            |- https://support.symantec.com/en_US/article.TECH242911.html
            |5. If the problem persists, contact Symantec Technical support at https://support.symantec.com for further assistance.
        """.stripMargin
      )
    )
  }
}