Vantage MCP tool-selection evals

Generated 2026-09-25T21:06:37.022Z. JSON under evals/results/<model>/<resource>/<tool>.json is the source of truth.

100.0%pass rate
71passed
0failed
0errors
27tools
Tool Phrasing Prompt gpt-5.6-luna · mixedgpt-5.6-luna · isolatedgpt-6-luna · mixed
add-team-memberdirectUse add-team-member to add pat@example.com to Vantage Team team_abc123 as an integration owner.pass
add-team-member({"team_token":"team_abc123","user_email":"pat@example.com","role":"integration_owner"})
pass
add-team-member({"team_token":"team_abc123","user_email":"pat@example.com","role":"integration_owner"})
add-team-memberinferredMake lee@example.com a viewer on the Vantage Team team_abc123.pass
add-team-member({"team_token":"team_abc123","user_email":"lee@example.com","role":"viewer"})
pass
add-team-member({"team_token":"team_abc123","user_email":"lee@example.com","role":"viewer"})
create-financial-commitment-reportdirectUse create-financial-commitment-report to create a report titled AWS Commitment Coverage in workspace wrkspc_abc123 for the last 30 days, grouped by resource_account_id and service.pass
create-financial-commitment-report({"workspace_token":"wrkspc_abc123","title":"AWS Commitment Coverage","date_interval":"last_30_days","groupings":["resource_account_id","service"]})
create-financial-commitment-reportinferredSet up a new Vantage financial commitment report named Linked Account Savings Plan Coverage in workspace wrkspc_abc123. Use the last month and break it down by the account whose resources consumed the commitment and by commitment type.pass
create-financial-commitment-report({"workspace_token":"wrkspc_abc123","title":"Linked Account Savings Plan Coverage","date_interval":"last_month","groupings":["resource_account_id","commitment_type"]})
create-kubernetes-efficiency-reportdirectUse create-kubernetes-efficiency-report to create Production K8s Efficiency in workspace wrkspc_abc123 for last month, grouped by cluster_id, namespace, and label:app, with costs aggregated by cost efficiency.pass
create-kubernetes-efficiency-report({"workspace_token":"wrkspc_abc123","title":"Production K8s Efficiency","date_interval":"last_month","aggregated_by":"cost_efficiency","groupings":["cluster_id","namespace","label:app"]})
create-kubernetes-efficiency-reportinferredSet up a saved Vantage Kubernetes efficiency report named Namespace Idle Spend in workspace wrkspc_abc123. Use the last 30 days, show daily data, group by namespace, and rank it by idle cost.pass
create-kubernetes-efficiency-report({"workspace_token":"wrkspc_abc123","title":"Namespace Idle Spend","date_interval":"last_30_days","aggregated_by":"idle_cost","date_bucket":"day","groupings":["namespace"]})
create-saved-filterdirectUse create-saved-filter to create AWS Spend in workspace wrkspc_abc123 with VQL filter (costs.provider = 'aws').pass
create-saved-filter({"title":"AWS Spend","workspace_token":"wrkspc_abc123","filter":"(costs.provider = 'aws')"})
create-saved-filterinferredSave a reusable Vantage Cost Report filter named Azure Spend in workspace wrkspc_abc123 using VQL (costs.provider = 'azure').pass
create-saved-filter({"title":"Azure Spend","workspace_token":"wrkspc_abc123","filter":"(costs.provider = 'azure')"})
create-teamdirectUse create-team to create a Vantage Team named Platform Ops in workspace wrkspc_abc123 and give user usr_def456 the editor role.pass
create-team({"name":"Platform Ops","workspace_tokens":["wrkspc_abc123"],"user_tokens":["usr_def456"],"role":"editor"})
pass
create-team({"name":"Platform Ops","workspace_tokens":["wrkspc_abc123"],"user_tokens":["usr_def456"],"role":"editor"})
create-teaminferredSet up a new Vantage Team called FinOps Enablement.pass
create-team({"name":"FinOps Enablement"})
pass
create-team({"name":"FinOps Enablement"})
delete-financial-commitment-reportdirectUse delete-financial-commitment-report to permanently delete Vantage financial commitment report fncl_cmnt_rprt_abc123.pass
delete-financial-commitment-report({"financial_commitment_report_token":"fncl_cmnt_rprt_abc123"})
delete-financial-commitment-reportinferredRemove the obsolete Vantage financial commitment report fncl_cmnt_rprt_legacy456 for good.pass
delete-financial-commitment-report({"financial_commitment_report_token":"fncl_cmnt_rprt_legacy456"})
delete-kubernetes-efficiency-reportdirectUse delete-kubernetes-efficiency-report to permanently delete report kbnts_eff_rprt_abc123.pass
delete-kubernetes-efficiency-report({"kubernetes_efficiency_report_token":"kbnts_eff_rprt_abc123"})
delete-kubernetes-efficiency-reportinferredRemove the obsolete Vantage Kubernetes efficiency report kbnts_eff_rprt_legacy456 for good.pass
delete-kubernetes-efficiency-report({"kubernetes_efficiency_report_token":"kbnts_eff_rprt_legacy456"})
delete-saved-filterdirectUse delete-saved-filter to delete svd_fltr_abc123.pass
delete-saved-filter({"saved_filter_token":"svd_fltr_abc123"})
delete-saved-filterinferredRemove obsolete Vantage Cost Report filter svd_fltr_old456 permanently.pass
delete-saved-filter({"saved_filter_token":"svd_fltr_old456"})
delete-teamdirectUse delete-team to delete Vantage Team team_abc123.pass
delete-team({"team_token":"team_abc123"})
pass
delete-team({"team_token":"team_abc123"})
delete-teaminferredPermanently remove the obsolete Vantage Team team_legacy456.pass
delete-team({"team_token":"team_legacy456"})
pass
delete-team({"team_token":"team_legacy456"})
get-cost-provider-accountsdirectUse get-cost-provider-accounts to list the AWS accounts in workspace wrkspc_abc123.pass
get-cost-provider-accounts({"workspace_token":"wrkspc_abc123","provider":"aws"})
get-cost-provider-accountsinferredI need human-readable titles for all connected billing accounts in workspace wrkspc_abc123 so I can use their account IDs in a VQL filter.pass
get-cost-provider-accounts({"workspace_token":"wrkspc_abc123"})
get-cost-provider-accountsinferredIn workspace wrkspc_abc123, search connected billing accounts for titles matching McpTest.pass
get-cost-provider-accounts({"workspace_token":"wrkspc_abc123","q":"McpTest"})
get-financial-commitment-reportdirectUse get-financial-commitment-report to retrieve the saved configuration for Vantage financial commitment report fncl_cmnt_rprt_abc123.pass
get-financial-commitment-report({"financial_commitment_report_token":"fncl_cmnt_rprt_abc123"})
get-financial-commitment-reportinferredShow me the title, date range, groupings, and filter configured on Vantage financial commitment report fncl_cmnt_rprt_coverage789.pass
get-financial-commitment-report({"financial_commitment_report_token":"fncl_cmnt_rprt_coverage789"})
get-kubernetes-efficiency-reportdirectUse get-kubernetes-efficiency-report to retrieve the saved configuration for kbnts_eff_rprt_abc123.pass
get-kubernetes-efficiency-report({"kubernetes_efficiency_report_token":"kbnts_eff_rprt_abc123"})
get-kubernetes-efficiency-reportinferredShow me the title, date range, groupings, aggregation metric, and VQL filter configured on Vantage Kubernetes efficiency report kbnts_eff_rprt_platform789.pass
get-kubernetes-efficiency-report({"kubernetes_efficiency_report_token":"kbnts_eff_rprt_platform789"})
get-myselfdirectUse get-myself to inspect the current Vantage credentials.pass
get-myself({})
get-myselfinferredWhat is my default workspace?pass
get-myself({})
get-saved-filterdirectUse get-saved-filter to inspect svd_fltr_abc123.pass
get-saved-filter({"saved_filter_token":"svd_fltr_abc123"})
get-saved-filterinferredWhat VQL query is stored in Vantage Cost Report filter svd_fltr_abc123?pass
get-saved-filter({"saved_filter_token":"svd_fltr_abc123"})
get-teamdirectUse get-team to retrieve Vantage Team team_abc123.pass
get-team({"token":"team_abc123"})
pass
get-team({"token":"team_abc123"})
get-teaminferredShow me the details for the Vantage Team with token team_platform789.pass
get-team({"token":"team_platform789"})
pass
get-team({"token":"team_platform789"})
get-team-membersdirectUse get-team-members to show page 3 of the members of Vantage Team team_abc123.pass
get-team-members({"team_token":"team_abc123","page":3})
pass
get-team-members({"team_token":"team_abc123","page":3})
get-team-membersinferredWho belongs to Vantage Team team_abc123? Show me the second page.pass
get-team-members({"team_token":"team_abc123","page":2})
pass
get-team-members({"team_token":"team_abc123","page":2})
get-teamsdirectUse get-teams to search page 2 for Vantage Teams named Platform that can access workspace wrkspc_abc123.pass
get-teams({"page":2,"q":"Platform","workspace_token":"wrkspc_abc123"})
pass
get-teams({"page":2,"q":"Platform","workspace_token":"wrkspc_abc123"})
get-teamsinferredWhich Vantage Teams can access workspace wrkspc_abc123? Show page 3.pass
get-teams({"page":3,"workspace_token":"wrkspc_abc123"})
pass
get-teams({"page":3,"workspace_token":"wrkspc_abc123"})
list-financial-commitment-reportsdirectUse list-financial-commitment-reports to search page 2 for Vantage financial commitment reports titled Savings Plans in workspace wrkspc_abc123.pass
list-financial-commitment-reports({"page":2,"q":"Savings Plans","workspace_token":"wrkspc_abc123"})
list-financial-commitment-reportsinferredWhich saved Vantage financial commitment reports are available in workspace wrkspc_finops789? Show the first page.pass
list-financial-commitment-reports({"page":1,"workspace_token":"wrkspc_finops789"})
list-kubernetes-efficiency-reportsdirectUse list-kubernetes-efficiency-reports to search page 2 for reports titled Production in workspace wrkspc_abc123.pass
list-kubernetes-efficiency-reports({"q":"Production","workspace_token":"wrkspc_abc123","page":2,"limit":128})
list-kubernetes-efficiency-reportsinferredWhich saved Vantage Kubernetes efficiency reports are available in workspace wrkspc_platform789? Show the first page.pass
list-kubernetes-efficiency-reports({"workspace_token":"wrkspc_platform789","page":1,"limit":128})
list-saved-filtersdirectUse list-saved-filters to search page 2 for saved filters titled AWS in workspace wrkspc_abc123.pass
list-saved-filters({"page":2,"q":"AWS","workspace_token":"wrkspc_abc123"})
list-saved-filtersinferredShow me the first page of reusable Cost Report filters in Vantage for workspace wrkspc_abc123.pass
list-saved-filters({"page":1,"workspace_token":"wrkspc_abc123"})
query-financial-commitment-report-costsdirectUse query-financial-commitment-report-costs to fetch cost data for financial commitment report fncl_cmnt_rprt_abc123 from 2024-03-01 through 2024-03-31, grouped by resource_account_id and service.pass
query-financial-commitment-report-costs({"financial_commitment_report_token":"fncl_cmnt_rprt_abc123","page":1,"start_date":"2024-03-01","end_date":"2024-03-31","groupings":["resource_account_id","service"]})
query-financial-commitment-report-costsinferredFor Vantage financial commitment report fncl_cmnt_rprt_abc123, show me which linked accounts' resources are actually consuming the commitments, broken down by resource account and commitment type. Use the report's saved date range and filters.pass
query-financial-commitment-report-costs({"financial_commitment_report_token":"fncl_cmnt_rprt_abc123","page":1,"groupings":["resource_account_id","commitment_type"]})
query-kubernetes-efficiency-report-costsdirectUse query-kubernetes-efficiency-report-costs to fetch August 2026 costs for kbnts_eff_rprt_abc123, grouped by cluster_id, namespace, and label:app, on page 2.pass
query-kubernetes-efficiency-report-costs({"kubernetes_efficiency_report_token":"kbnts_eff_rprt_abc123","start_date":"2026-08-01","end_date":"2026-08-31","groupings":["cluster_id","namespace","label:app"],"page":2,"limit":128})
query-kubernetes-efficiency-report-costsinferredFor Vantage Kubernetes efficiency report kbnts_eff_rprt_abc123, show daily workload cost, idle cost, and efficiency by namespace and team label using the report's saved date range and filter. Show page 1.pass
query-kubernetes-efficiency-report-costs({"kubernetes_efficiency_report_token":"kbnts_eff_rprt_abc123","date_bin":"day","groupings":["namespace","label:team"],"page":1,"limit":128})
remove-team-memberdirectUse remove-team-member to remove user usr_def456 from Vantage Team team_abc123.pass
remove-team-member({"team_token":"team_abc123","user_token":"usr_def456"})
pass
remove-team-member({"team_token":"team_abc123","user_token":"usr_def456"})
remove-team-memberinferredTake user usr_legacy789 off the Vantage Team team_abc123 without deleting the user.pass
remove-team-member({"team_token":"team_abc123","user_token":"usr_legacy789"})
pass
remove-team-member({"team_token":"team_abc123","user_token":"usr_legacy789"})
update-financial-commitment-reportdirectUse update-financial-commitment-report to rename fncl_cmnt_rprt_abc123 to Account Commitment Coverage and group it by resource_account_id and service.pass
update-financial-commitment-report({"financial_commitment_report_token":"fncl_cmnt_rprt_abc123","title":"Account Commitment Coverage","groupings":["resource_account_id","service"]})
update-financial-commitment-reportinferredChange Vantage financial commitment report fncl_cmnt_rprt_abc123 to be weekly, make on demand discountable, and group by cost type and commitment typepass
update-financial-commitment-report({"financial_commitment_report_token":"fncl_cmnt_rprt_abc123","date_bucket":"week","on_demand_costs_scope":"discountable","groupings":["cost_type","commitment_type"]})
update-kubernetes-efficiency-reportdirectUse update-kubernetes-efficiency-report to rename kbnts_eff_rprt_abc123 to Team Efficiency and group it by namespace and label:team.pass
update-kubernetes-efficiency-report({"kubernetes_efficiency_report_token":"kbnts_eff_rprt_abc123","title":"Team Efficiency","groupings":["namespace","label:team"]})
update-kubernetes-efficiency-reportinferredChange Vantage Kubernetes efficiency report kbnts_eff_rprt_abc123 to use weekly buckets over the last 30 days and rank rows by idle cost.pass
update-kubernetes-efficiency-report({"kubernetes_efficiency_report_token":"kbnts_eff_rprt_abc123","date_interval":"last_30_days","aggregated_by":"idle_cost","date_bucket":"week"})
update-saved-filterdirectUse update-saved-filter to rename svd_fltr_abc123 to Azure Spend and set its VQL filter to (costs.provider = 'azure').pass
update-saved-filter({"saved_filter_token":"svd_fltr_abc123","title":"Azure Spend","filter":"(costs.provider = 'azure')"})
update-saved-filterinferredChange the Vantage Cost Report filter svd_fltr_abc123 so its saved VQL is (costs.provider = 'aws').pass
update-saved-filter({"saved_filter_token":"svd_fltr_abc123","filter":"(costs.provider = 'aws')"})
update-teamdirectUse update-team to rename team_abc123 to Platform Engineering and clear its default Dashboard.pass
update-team({"team_token":"team_abc123","name":"Platform Engineering","default_dashboard_token":null})
pass
update-team({"team_token":"team_abc123","name":"Platform Engineering","default_dashboard_token":null})
update-teaminferredThe Vantage Team team_abc123 should now be called Cloud Efficiency.pass
update-team({"team_token":"team_abc123","name":"Cloud Efficiency"})
pass
update-team({"team_token":"team_abc123","name":"Cloud Efficiency"})