Degraded Mode

UI Column

Degraded Mode

Logged As

NODE_ALARM_DEGRADED_MODE

Meaning

The node has entered degraded mode due to database or MFS performance issues.

Every 15 minutes, MFS checks the node. The alarm is raised if any of the following conditions are met in that 15-minute interval:
  • Any operation in the DB memory work area is stuck for more than 6 minutes.
  • More than 100 DHLs are seen in in the 15-minute interval for any DB or MFS operations.
  • More than 100 large DB scans with low selectivity in the 15-minute interval.
    NOTE
    The following settings are not tunable:
    • The 15-minute evaluation interval
    • The threshold of 100 DHLs in a 15‑minute period
    • The threshold of 100 large low‑selectivity DB scans in a 15‑minute period
    • The 6‑minute DB operation timer
    • The alarm duration; after the alarm is raised, it will remain raised for one hour.
Resolution
Investigate the workload and health of the node to identify and reduce the conditions that cause degraded mode:
  • Check for DB operations that are stuck or running unusually long in the DB memory work area, and terminate or tune those operations if possible.
  • Review recent DB and MFS activity for a high number of DHLs, and address any underlying hardware, network, or configuration issues that might be causing frequent failures or retries.
  • Identify workloads that trigger large, low‑selectivity DB scans (for example, full‑table scans with broad filters), and optimize those queries or jobs to reduce scan volume and improve selectivity.
  • Ensure the node has sufficient CPU, memory, and disk resources, and that there are no underlying hardware or I/O problems contributing to stalls or excessive scans.

The alarm remains raised for 1 hour, so if the alarm persists for longer than 1 hour after you have addressed these issues, collect relevant logs (MFS, DB, and node metrics) and contact your support representative for further analysis.