Interesting logs found-paloaltonetworks-panos

error
panos
paloaltonetworks
Interesting logs found-paloaltonetworks-panos
0

#1

Interesting logs found-paloaltonetworks-panos

Vendor: paloaltonetworks

OS: panos

Description:
For each supported device, indeni will look for logs that are deemed “interesting” and trigger an issue when these are found.

Remediation Steps:
For each line, extended information is included with specific remediation steps.

How does this work?
This alert logs into the Palo Alto Networks device through SSH and runs “grep” on multiple different mp-log files, looking for known issues. When found, these log lines are passed on to analysis and alerting.

Why is this important?
Most logs can be retrieved through syslog and sent for analysis in a variety of products and components (indeni’s Dendron, Splunk and other syslog databases). Some logs cannot be sent via syslog. Palo Alto Network’s mp-logs are an example of such logs. These are saved solely on the device itself as textual files and must be retrieved over SSH. Coincidentally, these logs contain important information regarding the failure of certain components and features and should be analyzed.

Without Indeni how would you find this?
An administrator is required to log into the Palo Alto Networks device manually and use the “grep mp-log” and “less mp-log” commands to review the mp-log files manually.

panos-grep-mp-log

#! META
name: panos-grep-mp-log
description: Fetch mp-logs and analyze them
type: monitoring
monitoring_interval: 60 minute
requires:
    vendor: paloaltonetworks
    os.name: panos

#! COMMENTS
interesting-logs:
    why: |
        Most logs can be retrieved through syslog and sent for analysis in a variety of products and components (indeni's Dendron, Splunk and other syslog databases). Some logs cannot be sent via syslog. Palo Alto Network's mp-logs are an example of such logs. These are saved solely on the device itself as textual files and must be retrieved over SSH. Coincidentally, these logs contain important information regarding the failure of certain components and features and should be analyzed.
    how: |
        This alert logs into the Palo Alto Networks device through SSH and runs "grep" on multiple different mp-log files, looking for known issues. When found, these log lines are passed on to analysis and alerting.
    without-indeni: |
        An administrator is required to log into the Palo Alto Networks device manually and use the "grep mp-log" and "less mp-log" commands to review the mp-log files manually.
    can-with-snmp: false
    can-with-syslog: false

#! REMOTE::SSH
grep mp-log useridd.log pattern rror.*bind
grep mp-log useridd.log pattern "Message from socket exceeds internal buffer"
grep mp-log useridd.log pattern "enabling rate limiting for"
grep mp-log ehmon.log pattern Alarm.*True
grep mp-log ms.log pattern "log.*purged"
grep mp-log ms.log pattern "Failed to purge"
grep mp-log ms.log pattern "error while loading serial number"
grep mp-log ms.log pattern "Content time below threshold"
grep mp-log ms.log pattern "Template lab mismatch"
grep mp-log pan_dhcpd.log pattern "Unable to enable cfgagent"
grep mp-log ikemgr.log pattern "Due to negotiation timeout"
grep mp-log controlplane-console-output.log pattern pca954x_select
grep mp-log devsrvr.log pattern "Unable to save MP Cache"
grep mp-log devsrvr.log pattern "Firewall unable.*for.*service"
grep mp-log ms.log pattern "Error: _pan_email"
grep mp-log ha_agent.log pattern "Error:Domain Name Invalid"
grep mp-log ha_agent.log pattern "mgmtsrvr insync: NO"
grep mp-log ms.log pattern "pan_comm_get_tcp_conn_gen.*COMM: cannot connect"
grep mp-log supervisor.log pattern "pan_ssl_load_tcas.*failed"
grep mp-log pan_bc_download.log pattern "Error downloading latest URL database"
grep mp-log sslmgr.log pattern "pan_ocsp_parse_response.*status query is unavailable"
grep mp-log dnsproxyd.log pattern "Error reading dnsproxy persistent cache xml file"
grep mp-log authd.log pattern "unsuccessful authentication attempts threshold reached"
grep mp-log authd.log pattern "Unexpected error from radius server"
grep mp-log masterd.log pattern "Virtual memory limit exceeded"

#! PARSER::AWK
BEGIN {
    linesmatched = 0
}

function addLine(filename, line, info) {
    # Grab the log's timestamp
    logtimestamp = 0

    if (match($0, /([0-9]{4})-([0-9]){2}-([0-9]){2} ([0-9]){2}:([0-9]){2}:([0-9]){2}/)) {
        _year = substr($0, RSTART, 4)
        _month = substr($0, RSTART+5, 2)
        _day = substr($0, RSTART+8, 2)
        _hh = substr($0, RSTART+11, 2)
        _mm = substr($0, RSTART+14, 2)
        _ss = substr($0, RSTART+17, 2)
        logtimestamp = datetime(_year, _month, _day, _hh, _mm, _ss)
    } else {
        logtimestamp = now()
    }

    # Need to be careful we don't flood the server with a massive metric
    if ((now() - logtimestamp) <= 3600*24*7) {
        linesmatched += 1
        logs[linesmatched, "line"] = filename ": " line
        logs[linesmatched, "info"] = info
        logstimestamps[linesmatched] = logtimestamp

        # We limit the logs array's size, so once we've hit a certain number of
        # matched lines we work to ensure we always remove the oldest
        if (linesmatched > 50) {
            # Remove oldest log
            oldest = now()
            logtoremove = linesmatched
            for (logid in logstimestamps) {
                timestamp = logstimestamps[logid]
                if (timestamp < oldest) {
                    oldest = timestamp
                    logtoremove = logid
                }
            }

            delete logs[logtoremove, "line"]
            delete logs[logtoremove, "info"]
            delete logstimestamps[logtoremove]
        }
    }
}

/rror.*ldap.*bind/ {
    addLine("useridd.log", $0, "Review https://live.paloaltonetworks.com/t5/Management-Articles/Useful-CLI-Commands-to-Troubleshoot-LDAP-Connection/ta-p/57217")
}

/Alarm.*True/ {
    addLine("ehmon.log", $0, "Review https://live.paloaltonetworks.com/t5/Learning-Articles/CLI-Commands-to-View-Hardware-Status/ta-p/61027")
}

/log.*purged/ {
    addLine("ms.log", $0, "Review https://live.paloaltonetworks.com/t5/Management-Articles/Difference-Between-Data-in-Logs-and-Predefined-Reports/ta-p/56869")
}

/Unable to enable cfgagent/{
    addLine("pan_dhcpd.log", $0, "Review https://live.paloaltonetworks.com/t5/Management-Articles/CLI-Commands-to-Troubleshoot-DHCP/ta-p/59416")
}

/Due to negotiation timeout/ {
    addLine("ikemgr.log", $0, "Review https://live.paloaltonetworks.com/t5/Management-Articles/IPSec-VPN-Error-IKE-Phase-2-Negotiation-is-Failed-as-Initiator/ta-p/60725")
}

/pca954x_select/ {
    addLine("controlplane-console-output.log", $0, "Review https://live.paloaltonetworks.com/t5/Management-Articles/Palo-Alto-Networks-PA-5000-Series-Power-Supply-Unit-Error/ta-p/54124")
}

/Unable to save MP Cache/ {
    addLine("devsrvr.log", $0, "Review https://live.paloaltonetworks.com/t5/Management-Articles/BrightCloud-Error-in-devsrvr-log-Unable-to-save-MP-Cache/ta-p/54166")
}

/Failed to purge old uploaded files/ {
    addLine("ms.log", $0, "Review https://live.paloaltonetworks.com/t5/Management-Articles/Software-Download-Error-Failed-to-download-due-to-server-error/ta-p/57458")
}

/Error: _pan_email/ {
    addLine("ms.log", $0, "Review https://live.paloaltonetworks.com/t5/Learning-Articles/How-to-Send-a-Test-Email-to-Verify-Email-Profile-Settings/ta-p/60956")
}

/Error:Domain Name Invalid/ {
    addLine("ha_agent.log", $0, "Review https://live.paloaltonetworks.com/t5/Management-Articles/HA-Sync-Failure-Due-to-Inconsistent-Management-Settings/ta-p/59260")
}

/pan_comm_get_tcp_conn_gen/ {
    addLine("ms.log", $0, "Review https://live.paloaltonetworks.com/t5/tkb/articleprintpage/tkb-id/Management-TKB/article-id/514")
}

/pan_ssl_load_tcas/ {
    addLine("supervisor.log", $0, "Review https://live.paloaltonetworks.com/t5/Management-Articles/Wrong-Certificate-used-when-SSL-Decryption-is-enabled/ta-p/58710")
}

/Error downloading latest URL database/ {
    addLine("pan_bc_download.log", $0, "Review https://live.paloaltonetworks.com/t5/Management-Articles/PAN-DB-URL-Activation-Changes-to-Not-Active-after-Refresh/ta-p/53783")
}

/mgmtsrvr insync: NO/ {
    addLine("ha_agent.log", $0, "Review https://live.paloaltonetworks.com/t5/Management-Articles/HA-Active-Passive-Unable-to-Push-Config-to-Peer/ta-p/60850")
}

/Message from socket exceeds internal buffer/ {
    addLine("useridd.log", $0, "Review https://live.paloaltonetworks.com/t5/tkb/articleprintpage/tkb-id/Management-TKB/article-id/1584")
}

/enabling rate limiting for/ {
    addLine("useridd.log", $0, "Review https://live.paloaltonetworks.com/t5/Management-Articles/Unknown-IP-Rate-Limit-Mitigation-for-User-ID-Mappings/ta-p/58581")
}

/Firewall unable.*for.*service/ {
    addLine("devsrvr.log", $0, "Review https://live.paloaltonetworks.com/t5/tkb/articleprintpage/tkb-id/Management-TKB/article-id/3627")
}

/error while loading serial number/ {
    addLine("ms.log", $0, "Review https://live.paloaltonetworks.com/t5/tkb/articleprintpage/tkb-id/Management-TKB/article-id/896")
}

/pan_ocsp_parse_response/ {
    addLine("sslmgr.log", $0, "Review https://live.paloaltonetworks.com/t5/tkb/articleprintpage/tkb-id/ConfigurationArticles/article-id/973")
}

/Error reading dnsproxy persistent cache xml file/ {
    addLine("dnsproxyd.log", $0, "Contact Palo Alto Networks TAC and refer to this log line.")
}

/Content time below threshold/ {
    addLine("ms.log", $0, "Review https://live.paloaltonetworks.com/t5/Management-Articles/Dynamic-updates-scheduled-with-a-threshold-set-but-are-never-or/ta-p/65952")
}

/unsuccessful authentication attempts threshold reached/ {
    addLine("authd.log", $0, "Review https://live.paloaltonetworks.com/t5/Management-Articles/How-to-Unlock-Users-on-the-Lock-List-for-Failed-Maximum/ta-p/66195")
}

/Unexpected error from radius server/ {
    addLine("authd.log", $0, "Review https://live.paloaltonetworks.com/t5/Management-Articles/How-to-Identify-Secret-Key-Mismatch-Between-Palo-Alto-Networks/ta-p/54612")
}

/Template lab mismatch/ {
    addLine("ms.log", $0, "Review https://live.paloaltonetworks.com/t5/Management-Articles/Panorama-Error-template-incompatible-Due-to-Multi-VSYS/ta-p/60881")
}

#Dec 31 10:20:19 INFO: mgmtsrvr: User restart reason - Virtual memory limit exceeded (2575744 > 2560000)
/Virtual memory limit exceeded/ {
    addLine("masterd.log", $0, "Review https://live.paloaltonetworks.com/t5/Management-Articles/Error-mgmtsrvr-virtual-memory-limit-exceeded-restarting/ta-p/65992")
}

END {
    writeComplexMetricObjectArray("interesting-logs", t, logs)
}

cross_vendor_interesting_logs

package com.indeni.server.rules.library

import com.indeni.ruleengine._
import com.indeni.ruleengine.expressions.core.{StatusTreeExpression, _}
import com.indeni.ruleengine.expressions.data._
import com.indeni.server.common.data.conditions.True
import com.indeni.server.rules._
import com.indeni.server.rules.library.core.PerDeviceRule
import com.indeni.server.sensor.models.managementprocess.alerts.dto.AlertSeverity

/**
  * This rule allows ind scripts to pull in their own interesting logs, separate from Dendron.
  * This is usually done for log infromation that cannot be sent via syslog, or is best analyzed
  * over SSH by using grep/awk or something similar.
  */
case class InterestingLogsRule() extends PerDeviceRule with RuleHelper {

  override val metadata: RuleMetadata = RuleMetadata.builder("cross_vendor_interesting_logs", "All Devices: Interesting logs found",
    "For each supported device, indeni will look for logs that are deemed \"interesting\" and trigger an issue when these are found.", AlertSeverity.ERROR).build()

  override def expressionTree(context: RuleContext): StatusTreeExpression = {
    StatusTreeExpression(
      // Which objects to pull (normally, devices)
      SelectTagsExpression(context.metaDao, Set(DeviceKey), True),

      // What constitutes an issue
      StatusTreeExpression(
        SelectSnapshotsExpression(context.snapshotsDao, Set("interesting-logs")).multi(),
        StatusTreeExpression(
          IterateSnapshotDimensionExpression("interesting-logs"),
          com.indeni.ruleengine.expressions.conditions.True
        ).withSecondaryInfo(
            scopableStringFormatExpression("${scope(\"interesting-logs:line\")}"),
            scopableStringFormatExpression("${scope(\"interesting-logs:info\")}"),
            "Interesting Logs",
          Set[InvisibleScopeKey](InvisibleScopeKey("line",Some("interesting-logs")), InvisibleScopeKey("info",Some("interesting-logs")))
        ).asCondition()
      ).withoutInfo().asCondition()
      // Details of the alert itself
    ).withRootInfo(
        getHeadline(),
        ConstantExpression("Interesting logs have been found on this device. Review the list below for which logs were found and what action should be taken."),
        ConstantExpression("For each line, extended information is included with specific remediation steps.")
    )

  }
}




#2