Tip: Logging Capabilities in Solace Agent Mesh

giri · May 13, 2026, 10:06am

Tip: Logging Capabilities in Solace Agent Mesh

Effective logging is critical for troubleshooting issues, monitoring system health, and managing production deployments. Solace Agent Mesh provides a flexible, Python-native logging system that gives you centralized control over log output across all components—agents, gateways, and services.

This guide covers what SAM surfaces, how to access logs, and how to configure logging for development and production scenarios.

Overview

SAM uses Python’s built-in logging module, providing:

Centralized Control: A single configuration file manages logging for all components
Python Native: Built on Python’s standard logging module for maximum compatibility
Flexible Formats: Supports YAML and JSON configuration files using Python’s dictConfig
Production-Ready: Industry-standard approach with support for rotating file handlers, structured JSON logging, and log aggregation systems

Quick Start

When you run sam init, SAM automatically generates a configs/logging_config.yaml file with sensible defaults. To use it, set the environment variable:

export LOGGING_CONFIG_PATH=configs/logging_config.yaml

Or add it to your .env file:

LOGGING_CONFIG_PATH=configs/logging_config.yaml

Then run your SAM application:

sam run

Logs will output to the console (with color coding) and to a rotating log file (sam.log by default).

Configuration Methods

Simple Configuration (Per-Agent/Gateway)

Individual agent or gateway YAML files can include a basic log: section:

log:
  stdout_log_level: INFO
  log_file_level: DEBUG
  log_file: my-agent.log

Field	Type	Required	Default	Description
`stdout_log_level`	string	Yes	`INFO`	Logging level for console output. Valid values: `DEBUG`, `INFO`, `WARNING`, `ERROR`, `CRITICAL`
`log_file_level`	string	No	`INFO`	Logging level for file output
`log_file`	string	No	None	Path to the log file. If not specified, file logging is disabled

Note: The simple log: section has lower precedence and is overridden when a dedicated logging configuration file is provided via LOGGING_CONFIG_PATH.

Advanced Configuration (Recommended)

For production deployments and advanced scenarios, use a dedicated logging configuration file. Here’s the default configuration generated by sam init:

# Python logging configuration version (always 1)
version: 1

# Don't disable existing loggers when this config is loaded
disable_existing_loggers: false

# Formatters control the structure and appearance of log messages
formatters:
  # Simple human-readable format
  simpleFormatter:
    format: "%(asctime)s | %(levelname)-5s | %(threadName)s | %(name)s | %(message)s"

  # Colored simple human-readable format 
  coloredFormatter:
    class: solace_ai_connector.logging.ColoredFormatter
    format: "%(asctime)s | %(levelname)-5s | %(threadName)s | %(name)s | %(message)s"
    
  # JSON format for structured logging
  jsonFormatter:
    "()": pythonjsonlogger.json.JsonFormatter
    format: "%(timestamp)s %(levelname)s %(threadName)s %(name)s %(message)s"
    timestamp: "timestamp"

# Handlers determine where log messages go
handlers:
  # Stream handler - outputs logs to console (stdout)
  streamHandler:
    class: logging.StreamHandler
    formatter: coloredFormatter
    stream: "ext://sys.stdout"
  
  # Rotating file handler - writes to log files with automatic rotation
  rotatingFileHandler:
    class: logging.handlers.RotatingFileHandler
    formatter: simpleFormatter
    filename: ${LOGGING_FILE_NAME, sam.log}
    mode: a             # Append mode
    maxBytes: 52428800  # 50 MB - rotate when file reaches this size
    backupCount: 10     # Keep up to 10 historical log files

# Loggers 
loggers:
  solace_ai_connector:
    level: ${LOGGING_SAC_LEVEL, INFO}
    handlers: []
  
  solace_agent_mesh:
    level: ${LOGGING_SAM_LEVEL, INFO}
    handlers: []
  
  # Special trace logger for detailed troubleshooting
  sam_trace:
    level: ${LOGGING_SAM_TRACE_LEVEL, INFO}
    handlers: []

# Root logger
root:
  level: ${LOGGING_ROOT_LEVEL, WARNING}
  handlers:
    - streamHandler
    - rotatingFileHandler

Logger Hierarchy

SAM defines three primary loggers organized in a hierarchical namespace:

Logger	Purpose	Default Level
`solace_ai_connector`	Core connector framework logs	`INFO`
`solace_agent_mesh`	SAM-specific component logs (agents, gateways, tools)	`INFO`
`sam_trace`	Detailed troubleshooting traces (A2A messages, payloads)	`INFO`

Loggers use dot-separated names forming a tree structure. Child loggers inherit configuration from parents. For example:

solace_agent_mesh.gateway.http_sse inherits from solace_agent_mesh
solace_agent_mesh.agent.sac inherits from solace_agent_mesh

The `sam_trace` Logger

The sam_trace logger is a special trace logger designed for detailed troubleshooting. When set to DEBUG, it outputs verbose information including:

Full A2A message payloads
SSE event details
Complete context objects for task execution

This logger is used throughout SAM for conditional verbose logging:

if trace_logger.isEnabledFor(logging.DEBUG):
    trace_logger.debug(
        "%s A2A Context (shared service model): %s",
        component.log_identifier,
        a2a_context,
    )

Enable it for deep debugging:

export LOGGING_SAM_TRACE_LEVEL=DEBUG

Handlers and Formatters

Built-in Handlers

Handler	Description
`streamHandler`	Outputs logs to console (stdout) with color coding
`rotatingFileHandler`	Writes to log files with automatic rotation (default: 50MB, 10 backups)

For additional handlers, see Python’s supported handlers documentation.

Built-in Formatters

Formatter	Description
`simpleFormatter`	Human-readable format: timestamp, level, thread, logger name, message
`coloredFormatter`	Same as simple, but with color coding for console readability
`jsonFormatter`	JSON format for log aggregation systems (Datadog, Splunk, Elasticsearch)

Environment Variable Substitution

All configuration formats support environment variable substitution:

${VARIABLE_NAME, default_value}

Examples from the default configuration:

filename: ${LOGGING_FILE_NAME, sam.log}
level: ${LOGGING_SAM_LEVEL, INFO}
level: ${LOGGING_ROOT_LEVEL, WARNING}

Set these in your .env file or export them:

export LOGGING_SAM_LEVEL=DEBUG
export LOGGING_FILE_NAME=/var/log/sam/application.log

Common Configuration Scenarios

Structured Logging for Log Aggregation

Enable JSON formatting for integration with Datadog, Splunk, or Elasticsearch:

handlers:
  rotatingFileHandler:
    class: logging.handlers.RotatingFileHandler
    formatter: jsonFormatter  # Changed from simpleFormatter
    filename: ${LOGGING_FILE_NAME, sam.log}
    mode: a
    maxBytes: 52428800
    backupCount: 10

Add contextual fields for better filtering:

formatters:
  jsonFormatter:
    "()": pythonjsonlogger.json.JsonFormatter
    format: "%(asctime)s %(levelname)s %(threadName)s %(name)s %(message)s"
    static_fields:
      service: ${SERVICE_NAME, my-agent-mesh}
      env: ${ENV, production}

Output example:

{
   "asctime": "2025-10-30 22:25:56,960",
   "levelname": "INFO",
   "threadName": "MainThread",
   "name": "solace_agent_mesh.agent",
   "message": "Processing request",
   "service": "my-agent-mesh",
   "env": "production"
}

Customizing Log Levels for Specific Components

Increase verbosity for specific modules while keeping others quiet:

loggers:
  solace_ai_connector:
    level: INFO
    handlers: []
  
  solace_agent_mesh:
    level: INFO
    handlers: []
  
  sam_trace:
    level: INFO
    handlers: []
  
  # Increase verbosity for HTTP SSE gateway troubleshooting
  solace_agent_mesh.gateway.http_sse:
    level: DEBUG
    handlers: []
  
  # Increase verbosity for external library
  google_adk:
    level: INFO
    handlers: []

root:
  level: WARNING
  handlers:
    - streamHandler
    - rotatingFileHandler

Discovering Logger Names

To find available logger names, temporarily set the root logger level to DEBUG:

root:
  level: DEBUG
  handlers:
    - streamHandler

Run your application and observe the logger names in the output. Then restore the root level to WARNING and add specific logger configurations.

Agent-Specific Log Files

Isolate an agent’s logs by running it in a separate process with its own log file:

# Process 1: Run main components with default logging
sam run configs/gateways/webui.yaml configs/agents/orchestrator.yaml

# Process 2: Run isolated agent with dedicated log file
export LOGGING_FILE_NAME=my-isolated-agent.log && sam run configs/agents/my-agent.yaml

Using Logging in Custom Agents

When building custom agents, use Python’s standard logging module:

import logging
from solace_agent_mesh.agent.tools import ToolResult

log = logging.getLogger(__name__)

async def my_custom_tool(
    param1: str,
    tool_context=None,
    tool_config=None
) -> ToolResult:
    """
    A custom tool with proper logging.
    """
    log_identifier = "[MyTool]"
    
    # Info level for normal operation
    log.info("%s Processing request with param1=%s", log_identifier, param1)
    
    try:
        # Your logic here
        result = process_data(param1)
        
        # Debug level for detailed troubleshooting
        log.debug("%s Processing complete, result: %s", log_identifier, result)
        
        return ToolResult.ok("Success", data={"result": result})
        
    except ValueError as e:
        # Warning for recoverable issues
        log.warning("%s Invalid input: %s", log_identifier, e)
        return ToolResult.error(f"Invalid input: {e}")
        
    except Exception as e:
        # Error for unexpected failures
        log.error("%s Unexpected error: %s", log_identifier, e, exc_info=True)
        return ToolResult.error(f"Processing failed: {e}")

Log Identifier Pattern

SAM components use a consistent log identifier pattern for traceability:

log.info("%s Initialized successfully", self.log_identifier)
# Output: 2025-04-27 10:15:30 | INFO  | MainThread | my_agent.tools | [MyTool] Initialized successfully

This makes it easy to filter logs by component in production.

Tips

Start with defaults. The generated logging_config.yaml works well for development. Customize only when needed.
Use environment variables. Configure log levels via environment variables for easy adjustment across environments without changing config files.
Enable sam_trace for debugging. Set LOGGING_SAM_TRACE_LEVEL=DEBUG to see full A2A message payloads and execution context.
Use structured logging in production. JSON formatting integrates seamlessly with log aggregation systems and enables powerful filtering and alerting.
Rotate log files. The default rotating file handler prevents disk space issues. Adjust maxBytes and backupCount based on your retention needs.
Isolate noisy components. Use specific logger configurations to increase verbosity for components you’re debugging while keeping others quiet.
Include log identifiers. Follow SAM’s pattern of using log_identifier strings (e.g., [MyTool]) for easy filtering and correlation.

Learn More

Full Documentation: Logging | Solace Agent Mesh
Python Logging Reference: Python Logging Module
Python Handlers Reference: Logging Handlers
Repository: github.com/SolaceLabs/solace-agent-mesh

Topic		Replies	Views
Tip: How to Build a Custom Agent in Solace Agent Mesh Solace Agent Mesh	0	37	May 28, 2026
Creating agents in Solace Agent Mesh Solace Agent Mesh agent	0	112	March 16, 2026
Solace Agent Mesh Resources Solace Agent Mesh	0	216	April 28, 2025
Announcement: Solace Agent Mesh Enterprise Now Generally Available Solace Agent Mesh	2	160	March 2, 2026
Need help to trigger events from local storage Solace Agent Mesh	1	89	October 20, 2025

Tip: Logging Capabilities in Solace Agent Mesh

Tip: Logging Capabilities in Solace Agent Mesh

Overview

Quick Start

Configuration Methods

Simple Configuration (Per-Agent/Gateway)

Advanced Configuration (Recommended)

Logger Hierarchy

The sam_trace Logger

Handlers and Formatters

Built-in Handlers

Built-in Formatters

Environment Variable Substitution

Common Configuration Scenarios

Structured Logging for Log Aggregation

Customizing Log Levels for Specific Components

Discovering Logger Names

Agent-Specific Log Files

Using Logging in Custom Agents

Log Identifier Pattern

Tips

Learn More

Related topics

The `sam_trace` Logger