Execution Audit
Workflow Run #54
Full execution trace, ordered step results, duration metrics, and failure context.
Status
Failed
Trigger
Manual
Duration
6m 10s
Steps
3
Run Header
Execution Metadata
Summary
Workflow failed at step 'Create Experiment'.
Error
Microsoft.EntityFrameworkCore.DbUpdateException: An error occurred while saving the entity changes. See the inner exception for details. ---> Microsoft.Data.SqlClient.SqlException (0x80131904): A transport-level error has occurred when sending the request to the server. (provider: Session Provider, error: 19 - Physical connection is not usable) at System.Threading.Tasks.ContinuationResultTaskFromResultTask`2.InnerInvoke() at System.Threading.Tasks.Task.<>c.<.cctor>b__292_0(Object obj) at System.Threading.ExecutionContext.RunInternal(ExecutionContext executionContext, ContextCallback callback, Object state) --- End of stack trace from previous location --- at System.Threading.ExecutionContext.RunInternal(ExecutionContext executionContext, ContextCallback callback, Object state) at System.Threading.Tasks.Task.ExecuteWithThreadLocal(Task& currentTaskSlot, Thread threadPoolThread) --- End of stack trace from previous location --- at Microsoft.EntityFrameworkCore.Storage.RelationalCommand.ExecuteReaderAsync(RelationalCommandParameterObject parameterObject, CancellationToken cancellationToken) at Microsoft.EntityFrameworkCore.Storage.RelationalCommand.ExecuteReaderAsync(RelationalCommandParameterObject parameterObject, CancellationToken cancellationToken) at Microsoft.EntityFrameworkCore.Update.ReaderModificationCommandBatch.ExecuteAsync(IRelationalConnection connection, CancellationToken cancellationToken) ClientConnectionId:38891e7a-2564-4526-858d-82ca0bceaa91 Error Number:-1,State:0,Class:20 --- End of inner exception stack trace --- at Microsoft.EntityFrameworkCore.Update.ReaderModificationCommandBatch.ExecuteAsync(IRelationalConnection connection, CancellationToken cancellationToken) at Microsoft.EntityFrameworkCore.SqlServer.Update.Internal.SqlServerModificationCommandBatch.ExecuteAsync(IRelationalConnection connection, CancellationToken cancellationToken) at Microsoft.EntityFrameworkCore.Update.Internal.BatchExecutor.ExecuteAsync(IEnumerable`1 commandBatches, IRelationalConnection connection, CancellationToken cancellationToken) at Microsoft.EntityFrameworkCore.Update.Internal.BatchExecutor.ExecuteAsync(IEnumerable`1 commandBatches, IRelationalConnection connection, CancellationToken cancellationToken) at Microsoft.EntityFrameworkCore.Update.Internal.BatchExecutor.ExecuteAsync(IEnumerable`1 commandBatches, IRelationalConnection connection, CancellationToken cancellationToken) at Microsoft.EntityFrameworkCore.Storage.RelationalDatabase.SaveChangesAsync(IList`1 entries, CancellationToken cancellationToken) at Microsoft.EntityFrameworkCore.ChangeTracking.Internal.StateManager.SaveChangesAsync(IList`1 entriesToSave, CancellationToken cancellationToken) at Microsoft.EntityFrameworkCore.ChangeTracking.Internal.StateManager.SaveChangesAsync(StateManager stateManager, Boolean acceptAllChangesOnSuccess, CancellationToken cancellationToken) at Microsoft.EntityFrameworkCore.SqlServer.Storage.Internal.SqlServerExecutionStrategy.ExecuteAsync[TState,TResult](TState state, Func`4 operation, Func`4 verifySucceeded, CancellationToken cancellationToken) at Microsoft.EntityFrameworkCore.DbContext.SaveChangesAsync(Boolean acceptAllChangesOnSuccess, CancellationToken cancellationToken) at Microsoft.EntityFrameworkCore.DbContext.SaveChangesAsync(Boolean acceptAllChangesOnSuccess, CancellationToken cancellationToken) at BettingOdds.App.Projections.Services.Optimization.ProjectionExperimentRunner.ExecuteCandidateAsync(Int32 projectionExperimentId, String targetMetric, Int32 projectionExperimentCandidateId, Int32 projectionModelVersionId, String candidateLabel, IReadOnlyCollection`1 gameIds, CancellationToken ct) in C:\Users\Cesar\Documents\bettingodds2\BettingOdds\App\Projections\Services\Optimization\ProjectionExperimentRunner.cs:line 182 at BettingOdds.App.Projections.Services.Optimization.ProjectionExperimentRunner.ExecuteCandidateAsync(Int32 projectionExperimentId, String targetMetric, Int32 projectionExperimentCandidateId, Int32 projectionModelVersionId, String candidateLabel, IReadOnlyCollection`1 gameIds, CancellationToken ct) in C:\Users\Cesar\Documents\bettingodds2\BettingOdds\App\Projections\Services\Optimization\ProjectionExperimentRunner.cs:line 183 at BettingOdds.App.Projections.Services.Optimization.ProjectionExperimentRunner.<>c__DisplayClass4_0.<<RunExperimentAsync>b__0>d.MoveNext() in C:\Users\Cesar\Documents\bettingodds2\BettingOdds\App\Projections\Services\Optimization\ProjectionExperimentRunner.cs:line 87 --- End of stack trace from previous location --- at System.Threading.Tasks.Parallel.<>c__53`1.<<ForEachAsync>b__53_0>d.MoveNext() --- End of stack trace from previous location --- at BettingOdds.App.Projections.Services.Optimization.ProjectionExperimentRunner.RunExperimentAsync(Int32 projectionExperimentId, CancellationToken ct) in C:\Users\Cesar\Documents\bettingodds2\BettingOdds\App\Projections\Services\Optimization\ProjectionExperimentRunner.cs:line 78 at BettingOdds.App.Projections.Services.Optimization.ProjectionSearchOrchestrator.RunStructuredAdaptiveSearchAsync(String experimentName, String targetMetric, String rangeType, Int32 baseProjectionModelVersionId, IReadOnlyCollection`1 searchSpaces, Int32 candidateCount, IReadOnlyCollection`1 historySignals, IReadOnlyCollection`1 directionalPreferences, ProjectionSearchLanePolicy lanePolicy, String createdBy, Nullable`1 startUtc, Nullable`1 endUtc, IReadOnlyCollection`1 fixedProjectionModelVersionIds, CancellationToken ct) in C:\Users\Cesar\Documents\bettingodds2\BettingOdds\App\Projections\Services\Optimization\ProjectionSearchOrchestrator.cs:line 301 at BettingOdds.App.Projections.Services.Optimization.Planning.StructuredAdaptiveProjectionSearchService.RunStructuredAdaptiveSearchAsync(StructuredAdaptiveSearchRequest request, CancellationToken ct) in C:\Users\Cesar\Documents\bettingodds2\BettingOdds\App\Projections\Services\Optimization\Planning\StructuredAdaptiveProjectionSearchService.cs:line 120 at BettingOdds.App.Automation.CampaignExperiments.ProjectionOptimizationCampaignExperimentFactory.CreateExperimentAsync(Int64 projectionOptimizationCampaignRunId, String createdBy, CancellationToken ct) in C:\Users\Cesar\Documents\bettingodds2\BettingOdds\App\Automation\CampaignExperiments\ProjectionOptimizationCampaignExperimentFactory.cs:line 129 at BettingOdds.App.Automation.Steps.CampaignExperiments.OptimizationCampaignCreateExperimentStepExecutor.ExecuteAsync(AutomationExecutionContext context, CancellationToken ct) in C:\Users\Cesar\Documents\bettingodds2\BettingOdds\App\Automation\Steps\CampaignExperiments\OptimizationCampaignCreateExperimentStepExecutor.cs:line 44 at BettingOdds.App.Automation.Services.AutomationWorkflowRunner.RunManualAsync(AutomationManualRunRequest request, CancellationToken ct) in C:\Users\Cesar\Documents\bettingodds2\BettingOdds\App\Automation\Services\AutomationWorkflowRunner.cs:line 136
Step Timeline
Ordered Execution Trace
1
Select Campaign
optimization-campaign-select
Summary
Selected optimization campaign 'Minutes Calibration Sprint 01' (minutes-calibration-sprint-01).
2
Bootstrap Campaign Run
optimization-campaign-bootstrap
Summary
Bootstrapped campaign run 8 for 'Minutes Calibration Sprint 01'. Baseline model version = 45. Search anchor model version = 45.
3
Create Experiment
optimization-campaign-create-experiment
Error
Microsoft.EntityFrameworkCore.DbUpdateException: An error occurred while saving the entity changes. See the inner exception for details. ---> Microsoft.Data.SqlClient.SqlException (0x80131904): A transport-level error has occurred when sending the request to the server. (provider: Session Provider, error: 19 - Physical connection is not usable) at System.Threading.Tasks.ContinuationResultTaskFromResultTask`2.InnerInvoke() at System.Threading.Tasks.Task.<>c.<.cctor>b__292_0(Object obj) at System.Threading.ExecutionContext.RunInternal(ExecutionContext executionContext, ContextCallback callback, Object state) --- End of stack trace from previous location --- at System.Threading.ExecutionContext.RunInternal(ExecutionContext executionContext, ContextCallback callback, Object state) at System.Threading.Tasks.Task.ExecuteWithThreadLocal(Task& currentTaskSlot, Thread threadPoolThread) --- End of stack trace from previous location --- at Microsoft.EntityFrameworkCore.Storage.RelationalCommand.ExecuteReaderAsync(RelationalCommandParameterObject parameterObject, CancellationToken cancellationToken) at Microsoft.EntityFrameworkCore.Storage.RelationalCommand.ExecuteReaderAsync(RelationalCommandParameterObject parameterObject, CancellationToken cancellationToken) at Microsoft.EntityFrameworkCore.Update.ReaderModificationCommandBatch.ExecuteAsync(IRelationalConnection connection, CancellationToken cancellationToken) ClientConnectionId:38891e7a-2564-4526-858d-82ca0bceaa91 Error Number:-1,State:0,Class:20 --- End of inner exception stack trace --- at Microsoft.EntityFrameworkCore.Update.ReaderModificationCommandBatch.ExecuteAsync(IRelationalConnection connection, CancellationToken cancellationToken) at Microsoft.EntityFrameworkCore.SqlServer.Update.Internal.SqlServerModificationCommandBatch.ExecuteAsync(IRelationalConnection connection, CancellationToken cancellationToken) at Microsoft.EntityFrameworkCore.Update.Internal.BatchExecutor.ExecuteAsync(IEnumerable`1 commandBatches, IRelationalConnection connection, CancellationToken cancellationToken) at Microsoft.EntityFrameworkCore.Update.Internal.BatchExecutor.ExecuteAsync(IEnumerable`1 commandBatches, IRelationalConnection connection, CancellationToken cancellationToken) at Microsoft.EntityFrameworkCore.Update.Internal.BatchExecutor.ExecuteAsync(IEnumerable`1 commandBatches, IRelationalConnection connection, CancellationToken cancellationToken) at Microsoft.EntityFrameworkCore.Storage.RelationalDatabase.SaveChangesAsync(IList`1 entries, CancellationToken cancellationToken) at Microsoft.EntityFrameworkCore.ChangeTracking.Internal.StateManager.SaveChangesAsync(IList`1 entriesToSave, CancellationToken cancellationToken) at Microsoft.EntityFrameworkCore.ChangeTracking.Internal.StateManager.SaveChangesAsync(StateManager stateManager, Boolean acceptAllChangesOnSuccess, CancellationToken cancellationToken) at Microsoft.EntityFrameworkCore.SqlServer.Storage.Internal.SqlServerExecutionStrategy.ExecuteAsync[TState,TResult](TState state, Func`4 operation, Func`4 verifySucceeded, CancellationToken cancellationToken) at Microsoft.EntityFrameworkCore.DbContext.SaveChangesAsync(Boolean acceptAllChangesOnSuccess, CancellationToken cancellationToken) at Microsoft.EntityFrameworkCore.DbContext.SaveChangesAsync(Boolean acceptAllChangesOnSuccess, CancellationToken cancellationToken) at BettingOdds.App.Projections.Services.Optimization.ProjectionExperimentRunner.ExecuteCandidateAsync(Int32 projectionExperimentId, String targetMetric, Int32 projectionExperimentCandidateId, Int32 projectionModelVersionId, String candidateLabel, IReadOnlyCollection`1 gameIds, CancellationToken ct) in C:\Users\Cesar\Documents\bettingodds2\BettingOdds\App\Projections\Services\Optimization\ProjectionExperimentRunner.cs:line 182 at BettingOdds.App.Projections.Services.Optimization.ProjectionExperimentRunner.ExecuteCandidateAsync(Int32 projectionExperimentId, String targetMetric, Int32 projectionExperimentCandidateId, Int32 projectionModelVersionId, String candidateLabel, IReadOnlyCollection`1 gameIds, CancellationToken ct) in C:\Users\Cesar\Documents\bettingodds2\BettingOdds\App\Projections\Services\Optimization\ProjectionExperimentRunner.cs:line 183 at BettingOdds.App.Projections.Services.Optimization.ProjectionExperimentRunner.<>c__DisplayClass4_0.<<RunExperimentAsync>b__0>d.MoveNext() in C:\Users\Cesar\Documents\bettingodds2\BettingOdds\App\Projections\Services\Optimization\ProjectionExperimentRunner.cs:line 87 --- End of stack trace from previous location --- at System.Threading.Tasks.Parallel.<>c__53`1.<<ForEachAsync>b__53_0>d.MoveNext() --- End of stack trace from previous location --- at BettingOdds.App.Projections.Services.Optimization.ProjectionExperimentRunner.RunExperimentAsync(Int32 projectionExperimentId, CancellationToken ct) in C:\Users\Cesar\Documents\bettingodds2\BettingOdds\App\Projections\Services\Optimization\ProjectionExperimentRunner.cs:line 78 at BettingOdds.App.Projections.Services.Optimization.ProjectionSearchOrchestrator.RunStructuredAdaptiveSearchAsync(String experimentName, String targetMetric, String rangeType, Int32 baseProjectionModelVersionId, IReadOnlyCollection`1 searchSpaces, Int32 candidateCount, IReadOnlyCollection`1 historySignals, IReadOnlyCollection`1 directionalPreferences, ProjectionSearchLanePolicy lanePolicy, String createdBy, Nullable`1 startUtc, Nullable`1 endUtc, IReadOnlyCollection`1 fixedProjectionModelVersionIds, CancellationToken ct) in C:\Users\Cesar\Documents\bettingodds2\BettingOdds\App\Projections\Services\Optimization\ProjectionSearchOrchestrator.cs:line 301 at BettingOdds.App.Projections.Services.Optimization.Planning.StructuredAdaptiveProjectionSearchService.RunStructuredAdaptiveSearchAsync(StructuredAdaptiveSearchRequest request, CancellationToken ct) in C:\Users\Cesar\Documents\bettingodds2\BettingOdds\App\Projections\Services\Optimization\Planning\StructuredAdaptiveProjectionSearchService.cs:line 120 at BettingOdds.App.Automation.CampaignExperiments.ProjectionOptimizationCampaignExperimentFactory.CreateExperimentAsync(Int64 projectionOptimizationCampaignRunId, String createdBy, CancellationToken ct) in C:\Users\Cesar\Documents\bettingodds2\BettingOdds\App\Automation\CampaignExperiments\ProjectionOptimizationCampaignExperimentFactory.cs:line 129 at BettingOdds.App.Automation.Steps.CampaignExperiments.OptimizationCampaignCreateExperimentStepExecutor.ExecuteAsync(AutomationExecutionContext context, CancellationToken ct) in C:\Users\Cesar\Documents\bettingodds2\BettingOdds\App\Automation\Steps\CampaignExperiments\OptimizationCampaignCreateExperimentStepExecutor.cs:line 44 at BettingOdds.App.Automation.Services.AutomationWorkflowRunner.RunManualAsync(AutomationManualRunRequest request, CancellationToken ct) in C:\Users\Cesar\Documents\bettingodds2\BettingOdds\App\Automation\Services\AutomationWorkflowRunner.cs:line 136
Execution Summary
Step Status Breakdown
Succeeded
2
Failed
1
Running
0
Skipped
0
Partial
0
Operator Notes
Reading This Run
Use the ordered timeline to identify the exact failure boundary.
Compare workflow summary vs step summary to isolate whether failure was orchestration-level or step-level.
Long durations usually indicate remote source latency or backfill-heavy sync execution.