High disk space utilization-paloaltonetworks-panos
Vendor: paloaltonetworks
OS: panos
Description:
Many systems store vital configuration and operational data on their " +
"storage devices at different mount points and on different " +
"disk devices. It is important to monitor the usage of the " +
"different storage devices to ensure they do not fill up and " +
"cause issues in the on-going operation of the system.
Remediation Steps:
Determine the cause for the high disk usage of the listed file systems.
||Review https://live.paloaltonetworks.com/t5/Management-Articles/How-and-When-to-Clear-Disk-Space-on-the-Palo-Alto-Networks/ta-p/55736 and https://live.paloaltonetworks.com/t5/tkb/articleprintpage/tkb-id/FeaturedArticles/article-id/89",
How does this work?
This alert uses the Palo Alto Networks API to retrieve the current utilization of the different disks and mount points (the equivelant of using the “show system disk-space” command).
Why is this important?
Each device has multiple disk sections, or even separate disks. If any of those fill up, device stability may be impacted. Most mount points available in Palo Alto Networks firewalls have an auto-cleanup feature. Some do not, and require constant monitoring.
Without Indeni how would you find this?
An administrator will use the CLI to retrieve the storage utilization, normally once an issue occurs.
panos-show-system_disk-space
name: panos-show-system_disk-space
description: fetch disk space utilization
type: monitoring
monitoring_interval: 10 minutes
requires:
vendor: paloaltonetworks
os.name: panos
comments:
disk-usage-percentage:
why: |
Each device has multiple disk sections, or even separate disks. If any of those fill up, device stability may be impacted. Most mount points available in Palo Alto Networks firewalls have an auto-cleanup feature. Some do not, and require constant monitoring.
how: |
This alert uses the Palo Alto Networks API to retrieve the current utilization of the different disks and mount points (the equivelant of using the "show system disk-space" command).
can-with-snmp: true
can-with-syslog: true
disk-used-kbytes:
why: |
Tracking used disk on the system is critical. Used to display how much, in kilobytes, of the Disk is being used.
how: |
This alert uses the Palo Alto Networks API to retrieve the current utilization of the different disks and mount points (the equivelant of using the "show system disk-space" command).
can-with-snmp: true
can-with-syslog: false
disk-total-kbytes:
why: |
Determining total disk on the system is important to be able to check there is still space left. Used to display how much, in kilobytes, total disk size.
how: |
This alert uses the Palo Alto Networks API to retrieve the current utilization of the different disks and mount points (the equivelant of using the "show system disk-space" command).
can-with-snmp: true
can-with-syslog: false
steps:
- run:
type: HTTP
command: /api?type=op&cmd=<show><system><disk-space></disk-space></system></show>&key=${api-key}
parse:
type: AWK
file: show-system-disk-space.parser.1.awk
high_disk_space_usage
package com.indeni.server.rules.library.core
import com.indeni.ruleengine.expressions.conditions.{And, ConditionHelper, GreaterThanOrEqual}
import com.indeni.ruleengine.expressions.core.{ConstantExpression, StatusTreeExpression}
import com.indeni.ruleengine.expressions.data._
import com.indeni.ruleengine.expressions.math.AverageExpression
import com.indeni.ruleengine.expressions.scope.ScopeValueExpression
import com.indeni.server.common.ParameterValue
import com.indeni.server.common.data.conditions.True
import com.indeni.server.params.ParameterDefinition
import com.indeni.server.rules._
import com.indeni.server.rules.library.{ConditionalRemediationSteps, PerDeviceRule, RuleHelper}
import com.indeni.server.sensor.models.managementprocess.alerts.dto.AlertSeverity
case class HighDiskSpaceUsageRule() extends PerDeviceRule with RuleHelper {
private val excludeDisks = Set("/dev", "/mnt/cdrom", "/proc", "/dev/shm", "/dev/shm", "/junos", "/junos/dev")
val highThresholdParameter: ParameterDefinition = new ParameterDefinition(
"High_Threshold_of_Space_Usage",
"",
"High Threshold of Space Usage",
"What is the threshold for the mount point's disk usage for which once it is crossed " + "an issue will be triggered.",
ParameterDefinition.UIType.DOUBLE,
new ParameterValue((80.0).asInstanceOf[Object])
)
override def metadata: RuleMetadata =
RuleMetadata.builder("high_disk_space_usage",
"High disk space utilization",
"Many systems store vital configuration and operational data on their " +
"storage devices at different mount points and on different " +
"disk devices. It is important to monitor the usage of the " +
"different storage devices to ensure they do not fill up and " +
"cause issues in the on-going operation of the system.",
AlertSeverity.ERROR, categories = Set(RuleCategory.HealthChecks), deviceCategory = DeviceCategory.AllDevices).configParameter(highThresholdParameter).build()
override def expressionTree(context: RuleContext): StatusTreeExpression = {
val diskUsage = AverageExpression(TimeSeriesExpression[Double]("disk-usage-percentage"))
val diskNotExcluded = ScopeValueExpression("file-system").visible().isIn(excludeDisks).not
val diskUsageGreaterThanThreshold =
GreaterThanOrEqual(diskUsage, getParameterDouble(highThresholdParameter))
StatusTreeExpression(
SelectTagsExpression(context.metaDao, Set(DeviceKey), True),
StatusTreeExpression(
SelectTagsExpression(context.tsDao, Set("file-system"), True),
And(
diskNotExcluded,
StatusTreeExpression(
SelectTimeSeriesExpression[Double](context.tsDao, Set("disk-usage-percentage"), denseOnly = false),
diskUsageGreaterThanThreshold
).withSecondaryInfo(
scopableStringFormatExpression("${scope(\"file-system\")}"),
scopableStringFormatExpression("Current disk space utilization is: %.0f%%", diskUsage),
title = "Affected Disks / File Systems"
).asCondition()
),
).withoutInfo().asCondition()
).withRootInfo(
getHeadline(),
ConstantExpression("Some disks or file systems are under high usage."),
ConditionalRemediationSteps("Determine the cause for the high disk usage of the listed file systems.",
RemediationStepCondition.VENDOR_PANOS -> "Review https://live.paloaltonetworks.com/t5/Management-Articles/How-and-When-to-Clear-Disk-Space-on-the-Palo-Alto-Networks/ta-p/55736 and https://live.paloaltonetworks.com/t5/tkb/articleprintpage/tkb-id/FeaturedArticles/article-id/89",
RemediationStepCondition.VENDOR_JUNIPER ->
"""|1. On the devices command line interface execute "show system storage detail" command to review storage utilization.
|2. Run "request system storage disconnect" command to free storage space on the SRX by rotating log files and proposing a list of files for deletion.
|3. Remove the debug files after debug is done.
|4. Consider configuring logs to be sent to remote log servers.
|5. Review the following article on Juniper tech support site: <a target="_blank" href="https://www.juniper.net/documentation/en_US/junos/topics/reference/command-summary/request-system-storage-disconnect.html">Operational Commands: request system storage disconnect</a>.""".stripMargin,
RemediationStepCondition.OS_CISCO_ASA ->
"""Run the “show file system” command in privileged EXEC mode to display the size, bytes available, type of media, flags, and prefix information about the disk file system
|Execute the “show file information <filename>” command to view information about a specific file, including partner application package files
|The contents of this filesystem can be verified with any of these commands:
|dir: Displays the contents of the current directory
|show flash: Displays the contents of the internal flash memory
|show disk0: Displays the contents of the internal flash memory
|show disk1: Displays the contents of the external flash memory
|Run the “delete <filename>” to remove unused files. The “/noconfirm” switch of ‘delete’ command with eliminate the need for interactive confirmation. It is common to have unused files from older ASA or ASDM images which can be safely removed.
|If the flash filesystem is properly accessed by the device and works properly, the device indicates this with a Solid Green on the Flash LED on the front panel of the device. Refer to the next cisco troubleshooting guide in case that you have get flash errors messages:
|https://www.cisco.com/c/en/us/support/docs/security/asa-5500-x-series-next-generation-firewalls/113266-asa-flash-error-ts.html
|Use the “File Transfer…” dialog under “Tools” menu of the ASDM to perform file system manipulations""".stripMargin
)
)
}
}