Skip to main content

Optimizing System Configuration

There are a virtually unlimited number of unique individual setups that cannot be covered in this document. As well, even similar builds and configurations can behave differently due to external factors, so your results may vary. Here are some general guidelines to use as a starting point. Be cautious, make incremental changes, testing and observing before moving forward. Always focus on only one specific area at a time - avoid making changes to memory, storage, and CPU configs all at once. Diagnosing potential problems becomes nearly impossible otherwise.

Memory Management

The following settings in /etc/sysctl.conf can optimize memory usage and disk I/O patterns:
Apply changes: sudo sysctl -p

Network Stack

The following settings in /etc/sysctl.conf may improve network performance:

Storage Configuration

For NVMe drives, optimize I/O scheduling: Storage Optimization Commands

Infrastructure Monitoring

Monitoring is one of the most critical components of network infrastructure. performance tuning, and alerting configuration for Cosmos-SDK/Tendermint nodes.

Prometheus Setup

First, install Prometheus:
Example Prometheus configuration:

Grafana Integration

Install and configure Grafana:

Alert Management

Install Alertmanager:

Log Management

Loki Setup

Using Loki for log aggregation:

Log Rotation

Configure logrotate to manage log files:

Security Configuration

Network Security

UFW firewall configuration:

Rate Limiting

Validator-Specific Monitoring

Status Query

Query validator status through SDK:
Query through REST API:

Critical Metrics

Monitor these validator-specific metrics:

Backup Management

Host System Monitoring

Resource Usage Tracking

Install and configure node_exporter:
Add to Prometheus configuration:

EVM RPC OpenTelemetry Metrics

The EVM RPC layer emits OpenTelemetry metrics through the process-wide MeterProvider (for example, a Prometheus exporter). These are emitted in parallel with the legacy sei_* metrics so you can migrate dashboards incrementally.

Available EVM RPC Metrics

The evmrpc_request_latency_seconds histogram carries the following labels:

Migrating from Legacy EVM RPC Metrics

The following legacy sei_* metrics remain available today but are deprecated and scheduled for removal once dashboards migrate to the evmrpc_* OpenTelemetry metrics: Update your Prometheus and Grafana dashboards to consume the evmrpc_* metrics before the legacy metrics are removed. Note that latency changed units from milliseconds (sei_rpc_request_latency_ms) to seconds (evmrpc_request_latency_seconds), so adjust any thresholds and panel formatting accordingly.

FlatKV OpenTelemetry Metrics

The FlatKV state store emits OpenTelemetry metrics through the process-wide MeterProvider (for example, a Prometheus exporter). These metrics let node operators observe commit throughput, catchup progress, snapshotting, rollbacks, and snapshot imports.

Available FlatKV Metrics

Labels

FlatKV metrics carry the following labels where applicable:

Enabling Pebble Internal Metrics

Pebble’s internal (per-DB) metrics are governed by a single FlatKV-level knob: enable-pebble-metrics under [state-commit.flatkv] in app.toml (default true). The key is honored when present but is not part of the app.toml that seid init generates, so add it manually to change the default. Its value is propagated to every data DB (account, code, storage, legacy, and metadata) during initialization and overrides any per-DB EnableMetrics settings, so configure Pebble metrics through this knob rather than the individual per-DB settings.

LittDB OpenTelemetry Metrics

LittDB now emits its metrics through the process-wide OpenTelemetry MeterProvider instead of a private Prometheus client. When MetricsEnabled is set, LittDB configures a Prometheus exporter on the global provider and serves /metrics on MetricsPort (default 9101). The previous MetricsNamespace and MetricsRegistry config fields have been removed; all metric names now use a fixed litt_ prefix.

Available LittDB Metrics

Attributes

Migrating from Legacy LittDB Metrics

Metric names, units, and shape changed with the OpenTelemetry migration, so existing Prometheus and Grafana dashboards must be updated:
  • Latency metrics moved from millisecond summaries (for example {namespace}_read_latency_ms) to second histograms (litt_read_latency_seconds). Adjust thresholds and panel formatting from milliseconds to seconds accordingly.
  • Counters and gauges gained a fixed litt_ prefix and explicit units, for example bytes_read became litt_bytes_read and the cache weight gauge became litt_chunk_cache_weight_bytes.
  • The per-cache series that were previously separate metric names (for example chunk_read_cache_* and chunk_write_cache_*) are now the shared litt_chunk_cache_* metrics distinguished by the cache attribute.
  • The MetricsNamespace and MetricsRegistry config fields no longer exist. Metric names are fixed, and metrics are always backed by the global OTel provider; supply the scrape port via MetricsPort.

Performance Testing

For specific customizations or additional metrics, consult the Sei technical communities in Telegram or Discord.