Building agentic workflows with SageMaker AI and Bedrock AgentCore

A common challenge in building agentic workflows is mixing managed foundation models (FMs) with your own cost-optimized or domain-specific models, without rewriting your agent framework to do it. In this post, we show you how to combine OpenAI-compatible endpoints on Amazon SageMaker AI with Amazon Bedrock AgentCore runtime, a capability of Amazon Bedrock AgentCore, and its managed deployment. Specialized agents can collaborate on complex tasks while each uses the model best suited to its job. This combination gives you cost optimization, data residency, and model flexibility in a single production-ready architecture.

We walk through deploying Qwen 3.5 9B on Amazon SageMaker AI, integrating it into a Strands Agents multi-agent system alongside models on Amazon Bedrock, and shipping the entire workflow to Amazon Bedrock AgentCore runtime. The focus is on the integration mechanics including how to get token-level observability from SageMaker endpoints, which Strands doesn’t provide by default.

Solution overview

The architecture connects three model-hosting paths through a single Amazon Bedrock AgentCore container:

• Orchestrator agent (Claude Haiku 4.5 on Bedrock) – Classifies user intent and routes tasks through Global cross-Region inference.
• Budget agent (Claude Sonnet 4.6 on Bedrock) – Handles 50/30/20 budget breakdowns with structured Pydantic output.
• Financial analysis agent (Qwen 3.5 9B on Amazon SageMaker AI) – Stock analysis and portfolio construction using tool-calling.

Amazon Bedrock model availability varies by AWS Region. See Supported models by AWS Region in Amazon Bedrock.

A user request enters the orchestrator agent running inside the Amazon Bedrock AgentCore runtime. The orchestrator uses the agents as tools pattern from Strands Agents to route the request to either the budget agent or the financial analysis agent. Both specialized agents call their respective models. The budget agent invokes Claude Sonnet 4.6 through Amazon Bedrock, and the financial analysis agent invokes Qwen 3.5 9B through a SageMaker AI real-time endpoint using the OpenAI-compatible API. Results flow back through the orchestrator to the user. For the complete source code, see the accompanying GitHub repository. The following diagram illustrates this architecture.

Figure 1: Architecture of the multi-agent workflow across Amazon Bedrock and Amazon SageMaker AI

Prerequisites

You must have the following prerequisites to follow along with this post.

• An AWS account with permissions for Amazon SageMaker AI, Amazon Bedrock, and AgentCore.

pip install sagemaker-core openai httpx strands-agents[otel] yfinance pydantic bedrock-agentcore.

• An AWS Identity and Access Management (IAM) role with sagemaker:InvokeEndpoint and sagemaker:CallWithBearerToken.
• Bedrock model access for Claude Haiku 4.5 and Claude Sonnet 4.6.
• Python 3.12+.

Step 1: Deploy Qwen 3.5 9B on SageMaker AI

Deploy Qwen 3.5 9B using the vLLM Deep Learning Container (DLC), image vllm:0.22.1-gpu-py312-cu130, on ml.g6e.2xlarge.

region = "us-west-2"
model_id = "Qwen/Qwen3.5-9B"
instance_type = "ml.g6e.2xlarge"  # 1x L40S (48GB VRAM)
num_gpu = 1

# vLLM 0.22.1, Python 3.12, CUDA 13.0, Ubuntu 22.04
inference_image = f"763104351884.dkr.ecr.{region}.amazonaws.com/vllm:0.22.1-gpu-py312-cu130-ubuntu22.04-sagemaker"

env = {
    "SM_VLLM_MODEL": model_id,
    "SM_VLLM_TENSOR_PARALLEL_SIZE": "1",
    "SM_VLLM_MAX_MODEL_LEN": "32768",
}

# Create Model
sm.create_model(
    ModelName=model_name,
    ExecutionRoleArn=role,
    PrimaryContainer={"Image": inference_image, "Environment": env},
)

# Create Endpoint Config + Endpoint
sm.create_endpoint_config(
    EndpointConfigName=endpoint_config_name,
    ProductionVariants=[{
        "VariantName": "v1",
        "ModelName": model_name,
        "InstanceType": instance_type,
        "InitialInstanceCount": 1,
        "ContainerStartupHealthCheckTimeoutInSeconds": 1200,
        "InferenceAmiVersion": inference_ami_version,
    }],
)

sm.create_endpoint(EndpointName=endpoint_name, EndpointConfigName=endpoint_config_name)

Step 2: Build the multi-agent system

The OpenAI-compatible API of SageMaker AI expects a bearer token. Tokens expire, so for any long-running agent session you need a way to refresh them on every request. Set up auto-refreshing bearer tokens with an httpx.Auth subclass:

import httpx
from openai import AsyncOpenAI
from sagemaker.core.token_generator import generate_token

class SageMakerAuth(httpx.Auth):
    def __init__(self, region): self.region = region
    def auth_flow(self, request):
        request.headers["Authorization"] = f"Bearer {generate_token(region=self.region)}"
        yield request

strands_client = AsyncOpenAI(
    base_url=f"https://runtime.sagemaker.{REGION}.amazonaws.com/endpoints/{ENDPOINT_NAME}/openai/v1",
    api_key="sagemaker",
    http_client=httpx.AsyncClient(auth=SageMakerAuth(region=REGION)),
)

Build using Strands Agents’ agents as tools pattern with fresh agent instances per invocation.

from strands import Agent, tool
from strands.models.openai import OpenAIModel

qwen_model = OpenAIModel(
    client=strands_client, model_id="",
    params={"temperature": 0.7, "max_tokens": 4096, "stream_options": {"include_usage": True}},
)

@tool
def financial_analysis_agent_tool(query: str) -> str:
    fresh = Agent(model=qwen_model, tools=[...], callback_handler=None)
    return str(fresh(query))

orchestrator = Agent(
    model=BedrockModel(model_id="global.anthropic.claude-haiku-4-5-20251001-v1:0"),
    tools=[budget_agent_tool, financial_analysis_agent_tool],
)

Step 3: Deploy to Amazon Bedrock AgentCore runtime

Deploy using the bedrock-agentcore-starter-toolkit. See deploy_agentcore.ipynb for the full deployment notebook.

from bedrock_agentcore_starter_toolkit import Runtime

agentcore_runtime = Runtime()
agentcore_runtime.configure(
    entrypoint="main.py", auto_create_execution_role=True,
    auto_create_ecr=True, requirements_file="requirements.txt",
    region="ap-south-1", agent_name="personal_finance_agent",
)

launch_result = agentcore_runtime.launch(
    env_vars={
        "SAGEMAKER_ENDPOINT_NAME": "qwen35-9b-260612-082732",
        "SAGEMAKER_REGION": "ap-south-1",
        "AGENT_OBSERVABILITY_ENABLED": "true",
    }
)

Configure observability for SageMaker endpoints

Amazon Bedrock AgentCore runtime instruments your agents with OpenTelemetry automatically, but that instrumentation doesn’t extend equally to every model provider. Before you can monitor cost and latency for the Qwen model on Amazon SageMaker AI, you must understand where the default instrumentation falls short and how to close that gap.

The challenge: Invisible token usage

Amazon Bedrock AgentCore runtime automatically instruments agents using OpenTelemetry. However, there is a critical gap:

• Amazon Bedrock model calls get full generative AI spans with token counts automatically. No extra work is needed.
• Amazon SageMaker OpenAI-compatible endpoints (through Strands OpenAIModel) don’t get automatic token telemetry. The instrumentation doesn’t recognize them as generative AI calls.

This means tokens consumed by the financial analysis agent calling Qwen 3.5 9B on Amazon SageMaker are completely invisible in traces. You cannot monitor cost, detect regressions, or debug latency.

Root cause: Strands’ OTEL integration emits spans for tool calls and agent lifecycle events, but it doesn’t emit gen_ai.chat spans with token attributes for the OpenAIModel provider. The auto-instrumentation of AgentCore only recognizes Amazon Bedrock model inference calls (made through boto3) as generative AI operations.

The solution: Custom OpenTelemetry spans

Manually emit a gen_ai.chat span that wraps the Amazon SageMaker agent invocation and extracts token usage from Strands’ internal AgentResult.metrics.accumulated_usage:

from opentelemetry import trace

tracer = trace.get_tracer("financial_analysis_agent")

@tool
def financial_analysis_agent_tool(query: str) -> str:
    """Route investment queries to Qwen on SageMaker with observability."""
    with tracer.start_as_current_span("gen_ai.chat", attributes={
        "gen_ai.system": "openai",
        "gen_ai.request.model": f"qwen3.5-9b ({SAGEMAKER_ENDPOINT_NAME})",
        "gen_ai.operation.name": "chat",
    }) as span:
        fa_agent = Agent(
            model=OpenAIModel(
                client=strands_client, model_id="",
                params={"temperature": 0.7, "max_tokens": 4096,
                        "stream_options": {"include_usage": True}},
            ),
            system_prompt=FINANCIAL_ANALYSIS_PROMPT,
            tools=[get_stock_analysis, create_diversified_portfolio, compare_stock_performance],
            callback_handler=None,
        )
        result = fa_agent(query)
        # Extract token usage from Strands agent metrics
        usage = result.metrics.accumulated_usage
        span.set_attribute("gen_ai.usage.input_tokens", usage.get("inputTokens", 0))
        span.set_attribute("gen_ai.usage.output_tokens", usage.get("outputTokens", 0))
        span.set_attribute("gen_ai.usage.total_tokens", usage.get("totalTokens", 0))
        return str(result)

Key detail: Strands tracks token usage internally with keys inputTokens, outputTokens, and totalTokens. This dict is populated only if the model provider returns usage data.

Why stream_options is mandatory for vLLM

By default, vLLM doesn’t include a usage chunk in streaming responses. Strands receives text chunks but never a final usage object. As a result, accumulated_usage stays at zero. Adding stream_options: {"include_usage": True} tells vLLM to send an extra final chunk with token counts:

qwen_model = OpenAIModel(
    client=strands_client,
    model_id="",
    params={
        "temperature": 0.7,
        "max_tokens": 4096,
        "stream_options": {"include_usage": True},  # Critical for token tracking
    },
)

Without this parameter, your gen_ai.chat spans report 0 tokens. This defeats the purpose of the custom span.

Step-by-step configuration

1. Turn on Amazon CloudWatch Transaction Search (one-time per account or Region):

   aws xray update-trace-segment-destination --region ap-south-1 --destination CloudWatchLogs
   
   aws xray update-indexing-rule --region ap-south-1 --name "Default" \
     --rule '{"Probabilistic": {"DesiredSamplingPercentage": 100}}'

2. Install Strands with OTEL extras: strands-agents[otel]>=1.0.0.
3. Set AGENT_OBSERVABILITY_ENABLED=true in your code or env vars.
4. Use opentelemetry-instrument as the container CMD.
5. Add stream_options: {"include_usage": True} to OpenAIModel params.
6. Create custom gen_ai.chat span wrapping the SageMaker agent call.

Example trace output

{
    "name": "gen_ai.chat",
    "attributes": {
        "gen_ai.system": "openai",
        "gen_ai.request.model": "qwen3.5-9b (qwen35-9b-260612-082732)",
        "gen_ai.operation.name": "chat",
        "gen_ai.usage.input_tokens": 1391,
        "gen_ai.usage.output_tokens": 1432,
        "gen_ai.usage.total_tokens": 2823
    },
    "durationNano": 37237386894
}

Agent trajectory on Bedrock AgentCore Observability dashboard

This trace view shows the gen_ai.chat span for the Amazon SageMaker AI hosted Qwen model alongside the automatically instrumented Amazon Bedrock AgentCore spans, with token counts now visible for both. Building this end-to-end observability surfaced several implementation details worth calling out.

Figure 2: AgentCore observability trace with token counts for the SageMaker-hosted model

Key learnings

1. Amazon Bedrock AgentCore auto-instruments Bedrock calls – No extra work for Claude or Amazon Nova.
2. SageMaker OpenAI endpoints need manual spans – Strands doesn’t emit gen_ai.chat spans for OpenAIModel.
3. Token usage requires stream_options – vLLM doesn’t send usage in streaming by default.
4. Use result.metrics.accumulated_usage – Keys: inputTokens, outputTokens, totalTokens.
5. AWS X-Ray sampling rate matters – Default 1 percent drops most traces. Use 100 percent during development.
6. Fresh agent instances per request – Singletons cause concurrent invocation errors.

Extending the pattern

This architecture is composable. A few directions to explore:

1. Swap in fine-tuned models: Point SM_VLLM_MODEL to your fine-tuned checkpoint on Amazon Simple Storage Service (Amazon S3). The auth layer, OTEL spans, and AgentCore deployment stay unchanged.
2. A/B test with inference components: Deploy base and fine-tuned variants on the same Amazon SageMaker endpoint. Add a variant attribute to your OTEL span to compare quality in traces.
3. Cost-aware routing: Check query complexity before dispatch. Route simple lookups to Haiku on Amazon Bedrock. Reserve the Amazon SageMaker GPU endpoint for multi-step reasoning tasks.

Cleaning up

To avoid incurring future charges, delete the resources:

agentcore_control = boto3.client("bedrock-agentcore-control", region_name=region)
agentcore_control.delete_agent_runtime(agentRuntimeId=launch_result.agent_id)
sagemaker_client.delete_endpoint(EndpointName=ENDPOINT_NAME)
sagemaker_client.delete_endpoint_config(EndpointConfigName=f"qwen35-9b-epc-{TIMESTAMP}")
sagemaker_client.delete_model(ModelName=f"qwen35-9b-{TIMESTAMP}")

Conclusion

In this post, we showed how to connect a self-hosted model on Amazon SageMaker AI to Amazon Bedrock AgentCore runtime, and critically, how to get full token-level observability from Amazon SageMaker endpoints that Strands Agents doesn’t instrument by default.

• httpx.Auth + generate_token() + AsyncOpenAI – Production-ready SageMaker authentication inside AgentCore.
• Custom gen_ai.chat OTEL span + stream_options: {"include_usage": True} – Full token visibility for Amazon SageMaker endpoints.
• result.metrics.accumulated_usage – The Strands API for extracting token counts.

To get started, clone the accompanying repository and see OBSERVABILITY.md for the complete reference.

Related resources

• OpenAI-compatible API for SageMaker AI
• Strands Agents — agents as tools
• Amazon Bedrock AgentCore Observability
• OpenTelemetry generative AI semantic conventions

---

About the authors

Ayush Sharma

Ayush is a Senior AI Specialist Solutions Architect at AWS. He helps ISV and startup customers build production-ready generative AI solutions on AWS, specializing in multi-agent architectures and model deployment on Amazon SageMaker AI.

Shabna MT

Shabna is a Senior AI/ML Specialist Solutions Architect at AWS with more than 20 years of experience designing enterprise-scale, distributed software systems in the cloud. She specializes in generative AI and machine learning, and works with enterprise customers to take generative AI and ML applications from prototype to production at scale.

Vivek Gangasani

Vivek is a Worldwide Leader for Solutions Architecture, SageMaker Inference. He leads Solution Architecture, Technical Go-to-Market (GTM), and Outbound Product strategy for SageMaker Inference. He also helps enterprises and startups deploy and optimize generative AI models and build AI workflows with SageMaker and GPUs.

from Artificial Intelligence https://ift.tt/Fq6kEbw