Execution Audit

Workflow Run #54

Full execution trace, ordered step results, duration metrics, and failure context.
Status
Failed
Trigger
Manual
Duration
6m 10s
Steps
3
Run Header

Execution Metadata

Workflow
Optimization Campaign
optimization-campaign
Schedule
Manual / None
Started
2026-03-31 22:14:14 UTC
Finished
2026-03-31 22:20:25 UTC
Summary
Workflow failed at step 'Create Experiment'.
Error
Microsoft.EntityFrameworkCore.DbUpdateException: An error occurred while saving the entity changes. See the inner exception for details.
 ---> Microsoft.Data.SqlClient.SqlException (0x80131904): A transport-level error has occurred when sending the request to the server. (provider: Session Provider, error: 19 - Physical connection is not usable)
   at System.Threading.Tasks.ContinuationResultTaskFromResultTask`2.InnerInvoke()
   at System.Threading.Tasks.Task.<>c.<.cctor>b__292_0(Object obj)
   at System.Threading.ExecutionContext.RunInternal(ExecutionContext executionContext, ContextCallback callback, Object state)
--- End of stack trace from previous location ---
   at System.Threading.ExecutionContext.RunInternal(ExecutionContext executionContext, ContextCallback callback, Object state)
   at System.Threading.Tasks.Task.ExecuteWithThreadLocal(Task& currentTaskSlot, Thread threadPoolThread)
--- End of stack trace from previous location ---
   at Microsoft.EntityFrameworkCore.Storage.RelationalCommand.ExecuteReaderAsync(RelationalCommandParameterObject parameterObject, CancellationToken cancellationToken)
   at Microsoft.EntityFrameworkCore.Storage.RelationalCommand.ExecuteReaderAsync(RelationalCommandParameterObject parameterObject, CancellationToken cancellationToken)
   at Microsoft.EntityFrameworkCore.Update.ReaderModificationCommandBatch.ExecuteAsync(IRelationalConnection connection, CancellationToken cancellationToken)
ClientConnectionId:38891e7a-2564-4526-858d-82ca0bceaa91
Error Number:-1,State:0,Class:20
   --- End of inner exception stack trace ---
   at Microsoft.EntityFrameworkCore.Update.ReaderModificationCommandBatch.ExecuteAsync(IRelationalConnection connection, CancellationToken cancellationToken)
   at Microsoft.EntityFrameworkCore.SqlServer.Update.Internal.SqlServerModificationCommandBatch.ExecuteAsync(IRelationalConnection connection, CancellationToken cancellationToken)
   at Microsoft.EntityFrameworkCore.Update.Internal.BatchExecutor.ExecuteAsync(IEnumerable`1 commandBatches, IRelationalConnection connection, CancellationToken cancellationToken)
   at Microsoft.EntityFrameworkCore.Update.Internal.BatchExecutor.ExecuteAsync(IEnumerable`1 commandBatches, IRelationalConnection connection, CancellationToken cancellationToken)
   at Microsoft.EntityFrameworkCore.Update.Internal.BatchExecutor.ExecuteAsync(IEnumerable`1 commandBatches, IRelationalConnection connection, CancellationToken cancellationToken)
   at Microsoft.EntityFrameworkCore.Storage.RelationalDatabase.SaveChangesAsync(IList`1 entries, CancellationToken cancellationToken)
   at Microsoft.EntityFrameworkCore.ChangeTracking.Internal.StateManager.SaveChangesAsync(IList`1 entriesToSave, CancellationToken cancellationToken)
   at Microsoft.EntityFrameworkCore.ChangeTracking.Internal.StateManager.SaveChangesAsync(StateManager stateManager, Boolean acceptAllChangesOnSuccess, CancellationToken cancellationToken)
   at Microsoft.EntityFrameworkCore.SqlServer.Storage.Internal.SqlServerExecutionStrategy.ExecuteAsync[TState,TResult](TState state, Func`4 operation, Func`4 verifySucceeded, CancellationToken cancellationToken)
   at Microsoft.EntityFrameworkCore.DbContext.SaveChangesAsync(Boolean acceptAllChangesOnSuccess, CancellationToken cancellationToken)
   at Microsoft.EntityFrameworkCore.DbContext.SaveChangesAsync(Boolean acceptAllChangesOnSuccess, CancellationToken cancellationToken)
   at BettingOdds.App.Projections.Services.Optimization.ProjectionExperimentRunner.ExecuteCandidateAsync(Int32 projectionExperimentId, String targetMetric, Int32 projectionExperimentCandidateId, Int32 projectionModelVersionId, String candidateLabel, IReadOnlyCollection`1 gameIds, CancellationToken ct) in C:\Users\Cesar\Documents\bettingodds2\BettingOdds\App\Projections\Services\Optimization\ProjectionExperimentRunner.cs:line 182
   at BettingOdds.App.Projections.Services.Optimization.ProjectionExperimentRunner.ExecuteCandidateAsync(Int32 projectionExperimentId, String targetMetric, Int32 projectionExperimentCandidateId, Int32 projectionModelVersionId, String candidateLabel, IReadOnlyCollection`1 gameIds, CancellationToken ct) in C:\Users\Cesar\Documents\bettingodds2\BettingOdds\App\Projections\Services\Optimization\ProjectionExperimentRunner.cs:line 183
   at BettingOdds.App.Projections.Services.Optimization.ProjectionExperimentRunner.<>c__DisplayClass4_0.<<RunExperimentAsync>b__0>d.MoveNext() in C:\Users\Cesar\Documents\bettingodds2\BettingOdds\App\Projections\Services\Optimization\ProjectionExperimentRunner.cs:line 87
--- End of stack trace from previous location ---
   at System.Threading.Tasks.Parallel.<>c__53`1.<<ForEachAsync>b__53_0>d.MoveNext()
--- End of stack trace from previous location ---
   at BettingOdds.App.Projections.Services.Optimization.ProjectionExperimentRunner.RunExperimentAsync(Int32 projectionExperimentId, CancellationToken ct) in C:\Users\Cesar\Documents\bettingodds2\BettingOdds\App\Projections\Services\Optimization\ProjectionExperimentRunner.cs:line 78
   at BettingOdds.App.Projections.Services.Optimization.ProjectionSearchOrchestrator.RunStructuredAdaptiveSearchAsync(String experimentName, String targetMetric, String rangeType, Int32 baseProjectionModelVersionId, IReadOnlyCollection`1 searchSpaces, Int32 candidateCount, IReadOnlyCollection`1 historySignals, IReadOnlyCollection`1 directionalPreferences, ProjectionSearchLanePolicy lanePolicy, String createdBy, Nullable`1 startUtc, Nullable`1 endUtc, IReadOnlyCollection`1 fixedProjectionModelVersionIds, CancellationToken ct) in C:\Users\Cesar\Documents\bettingodds2\BettingOdds\App\Projections\Services\Optimization\ProjectionSearchOrchestrator.cs:line 301
   at BettingOdds.App.Projections.Services.Optimization.Planning.StructuredAdaptiveProjectionSearchService.RunStructuredAdaptiveSearchAsync(StructuredAdaptiveSearchRequest request, CancellationToken ct) in C:\Users\Cesar\Documents\bettingodds2\BettingOdds\App\Projections\Services\Optimization\Planning\StructuredAdaptiveProjectionSearchService.cs:line 120
   at BettingOdds.App.Automation.CampaignExperiments.ProjectionOptimizationCampaignExperimentFactory.CreateExperimentAsync(Int64 projectionOptimizationCampaignRunId, String createdBy, CancellationToken ct) in C:\Users\Cesar\Documents\bettingodds2\BettingOdds\App\Automation\CampaignExperiments\ProjectionOptimizationCampaignExperimentFactory.cs:line 129
   at BettingOdds.App.Automation.Steps.CampaignExperiments.OptimizationCampaignCreateExperimentStepExecutor.ExecuteAsync(AutomationExecutionContext context, CancellationToken ct) in C:\Users\Cesar\Documents\bettingodds2\BettingOdds\App\Automation\Steps\CampaignExperiments\OptimizationCampaignCreateExperimentStepExecutor.cs:line 44
   at BettingOdds.App.Automation.Services.AutomationWorkflowRunner.RunManualAsync(AutomationManualRunRequest request, CancellationToken ct) in C:\Users\Cesar\Documents\bettingodds2\BettingOdds\App\Automation\Services\AutomationWorkflowRunner.cs:line 136
Step Timeline

Ordered Execution Trace

1
Select Campaign
optimization-campaign-select
Succeeded
Started 2026-03-31 22:14:14 UTC
Finished 2026-03-31 22:14:14 UTC
Duration 189 ms
Summary
Selected optimization campaign 'Minutes Calibration Sprint 01' (minutes-calibration-sprint-01).
2
Bootstrap Campaign Run
optimization-campaign-bootstrap
Succeeded
Started 2026-03-31 22:14:14 UTC
Finished 2026-03-31 22:14:15 UTC
Duration 362 ms
Summary
Bootstrapped campaign run 8 for 'Minutes Calibration Sprint 01'. Baseline model version = 45. Search anchor model version = 45.
3
Create Experiment
optimization-campaign-create-experiment
Failed
Started 2026-03-31 22:14:15 UTC
Finished 2026-03-31 22:20:25 UTC
Duration 6m 10s
Error
Microsoft.EntityFrameworkCore.DbUpdateException: An error occurred while saving the entity changes. See the inner exception for details.
 ---> Microsoft.Data.SqlClient.SqlException (0x80131904): A transport-level error has occurred when sending the request to the server. (provider: Session Provider, error: 19 - Physical connection is not usable)
   at System.Threading.Tasks.ContinuationResultTaskFromResultTask`2.InnerInvoke()
   at System.Threading.Tasks.Task.<>c.<.cctor>b__292_0(Object obj)
   at System.Threading.ExecutionContext.RunInternal(ExecutionContext executionContext, ContextCallback callback, Object state)
--- End of stack trace from previous location ---
   at System.Threading.ExecutionContext.RunInternal(ExecutionContext executionContext, ContextCallback callback, Object state)
   at System.Threading.Tasks.Task.ExecuteWithThreadLocal(Task& currentTaskSlot, Thread threadPoolThread)
--- End of stack trace from previous location ---
   at Microsoft.EntityFrameworkCore.Storage.RelationalCommand.ExecuteReaderAsync(RelationalCommandParameterObject parameterObject, CancellationToken cancellationToken)
   at Microsoft.EntityFrameworkCore.Storage.RelationalCommand.ExecuteReaderAsync(RelationalCommandParameterObject parameterObject, CancellationToken cancellationToken)
   at Microsoft.EntityFrameworkCore.Update.ReaderModificationCommandBatch.ExecuteAsync(IRelationalConnection connection, CancellationToken cancellationToken)
ClientConnectionId:38891e7a-2564-4526-858d-82ca0bceaa91
Error Number:-1,State:0,Class:20
   --- End of inner exception stack trace ---
   at Microsoft.EntityFrameworkCore.Update.ReaderModificationCommandBatch.ExecuteAsync(IRelationalConnection connection, CancellationToken cancellationToken)
   at Microsoft.EntityFrameworkCore.SqlServer.Update.Internal.SqlServerModificationCommandBatch.ExecuteAsync(IRelationalConnection connection, CancellationToken cancellationToken)
   at Microsoft.EntityFrameworkCore.Update.Internal.BatchExecutor.ExecuteAsync(IEnumerable`1 commandBatches, IRelationalConnection connection, CancellationToken cancellationToken)
   at Microsoft.EntityFrameworkCore.Update.Internal.BatchExecutor.ExecuteAsync(IEnumerable`1 commandBatches, IRelationalConnection connection, CancellationToken cancellationToken)
   at Microsoft.EntityFrameworkCore.Update.Internal.BatchExecutor.ExecuteAsync(IEnumerable`1 commandBatches, IRelationalConnection connection, CancellationToken cancellationToken)
   at Microsoft.EntityFrameworkCore.Storage.RelationalDatabase.SaveChangesAsync(IList`1 entries, CancellationToken cancellationToken)
   at Microsoft.EntityFrameworkCore.ChangeTracking.Internal.StateManager.SaveChangesAsync(IList`1 entriesToSave, CancellationToken cancellationToken)
   at Microsoft.EntityFrameworkCore.ChangeTracking.Internal.StateManager.SaveChangesAsync(StateManager stateManager, Boolean acceptAllChangesOnSuccess, CancellationToken cancellationToken)
   at Microsoft.EntityFrameworkCore.SqlServer.Storage.Internal.SqlServerExecutionStrategy.ExecuteAsync[TState,TResult](TState state, Func`4 operation, Func`4 verifySucceeded, CancellationToken cancellationToken)
   at Microsoft.EntityFrameworkCore.DbContext.SaveChangesAsync(Boolean acceptAllChangesOnSuccess, CancellationToken cancellationToken)
   at Microsoft.EntityFrameworkCore.DbContext.SaveChangesAsync(Boolean acceptAllChangesOnSuccess, CancellationToken cancellationToken)
   at BettingOdds.App.Projections.Services.Optimization.ProjectionExperimentRunner.ExecuteCandidateAsync(Int32 projectionExperimentId, String targetMetric, Int32 projectionExperimentCandidateId, Int32 projectionModelVersionId, String candidateLabel, IReadOnlyCollection`1 gameIds, CancellationToken ct) in C:\Users\Cesar\Documents\bettingodds2\BettingOdds\App\Projections\Services\Optimization\ProjectionExperimentRunner.cs:line 182
   at BettingOdds.App.Projections.Services.Optimization.ProjectionExperimentRunner.ExecuteCandidateAsync(Int32 projectionExperimentId, String targetMetric, Int32 projectionExperimentCandidateId, Int32 projectionModelVersionId, String candidateLabel, IReadOnlyCollection`1 gameIds, CancellationToken ct) in C:\Users\Cesar\Documents\bettingodds2\BettingOdds\App\Projections\Services\Optimization\ProjectionExperimentRunner.cs:line 183
   at BettingOdds.App.Projections.Services.Optimization.ProjectionExperimentRunner.<>c__DisplayClass4_0.<<RunExperimentAsync>b__0>d.MoveNext() in C:\Users\Cesar\Documents\bettingodds2\BettingOdds\App\Projections\Services\Optimization\ProjectionExperimentRunner.cs:line 87
--- End of stack trace from previous location ---
   at System.Threading.Tasks.Parallel.<>c__53`1.<<ForEachAsync>b__53_0>d.MoveNext()
--- End of stack trace from previous location ---
   at BettingOdds.App.Projections.Services.Optimization.ProjectionExperimentRunner.RunExperimentAsync(Int32 projectionExperimentId, CancellationToken ct) in C:\Users\Cesar\Documents\bettingodds2\BettingOdds\App\Projections\Services\Optimization\ProjectionExperimentRunner.cs:line 78
   at BettingOdds.App.Projections.Services.Optimization.ProjectionSearchOrchestrator.RunStructuredAdaptiveSearchAsync(String experimentName, String targetMetric, String rangeType, Int32 baseProjectionModelVersionId, IReadOnlyCollection`1 searchSpaces, Int32 candidateCount, IReadOnlyCollection`1 historySignals, IReadOnlyCollection`1 directionalPreferences, ProjectionSearchLanePolicy lanePolicy, String createdBy, Nullable`1 startUtc, Nullable`1 endUtc, IReadOnlyCollection`1 fixedProjectionModelVersionIds, CancellationToken ct) in C:\Users\Cesar\Documents\bettingodds2\BettingOdds\App\Projections\Services\Optimization\ProjectionSearchOrchestrator.cs:line 301
   at BettingOdds.App.Projections.Services.Optimization.Planning.StructuredAdaptiveProjectionSearchService.RunStructuredAdaptiveSearchAsync(StructuredAdaptiveSearchRequest request, CancellationToken ct) in C:\Users\Cesar\Documents\bettingodds2\BettingOdds\App\Projections\Services\Optimization\Planning\StructuredAdaptiveProjectionSearchService.cs:line 120
   at BettingOdds.App.Automation.CampaignExperiments.ProjectionOptimizationCampaignExperimentFactory.CreateExperimentAsync(Int64 projectionOptimizationCampaignRunId, String createdBy, CancellationToken ct) in C:\Users\Cesar\Documents\bettingodds2\BettingOdds\App\Automation\CampaignExperiments\ProjectionOptimizationCampaignExperimentFactory.cs:line 129
   at BettingOdds.App.Automation.Steps.CampaignExperiments.OptimizationCampaignCreateExperimentStepExecutor.ExecuteAsync(AutomationExecutionContext context, CancellationToken ct) in C:\Users\Cesar\Documents\bettingodds2\BettingOdds\App\Automation\Steps\CampaignExperiments\OptimizationCampaignCreateExperimentStepExecutor.cs:line 44
   at BettingOdds.App.Automation.Services.AutomationWorkflowRunner.RunManualAsync(AutomationManualRunRequest request, CancellationToken ct) in C:\Users\Cesar\Documents\bettingodds2\BettingOdds\App\Automation\Services\AutomationWorkflowRunner.cs:line 136
Execution Summary

Step Status Breakdown

Succeeded 2
Failed 1
Running 0
Skipped 0
Partial 0
Operator Notes

Reading This Run

Use the ordered timeline to identify the exact failure boundary.
Compare workflow summary vs step summary to isolate whether failure was orchestration-level or step-level.
Long durations usually indicate remote source latency or backfill-heavy sync execution.