Q15ML Solution Monitoring, Maintenance, and Security
A company has deployed an XGBoost prediction model in production to predict if a customer is likely to cancel a subscription. The company uses Amazon SageMaker Model Monitor to detect deviations in the F1 score. During a baseline analysis of model quality, the company recorded a threshold for the F1 score. After several months of no change, the model's F1 score decreases significantly. What could be the reason for the reduced F1 score?
← → navigate · a answer
Community votes
Discussion · 5
A 3
Selected Answer: A
Concept Drift: Means the statistical properties of the underlying data distribution change over time --> Decrease F1 score --> perform poorly on new data
A 3
Selected Answer: A
Concept Drift: Happens when the statistical properties of the data used for predictions change over time, making the model perform worse on current data.
Why Not the Other Options?
B. If the model complexity was insufficient, the issue would have shown up during the initial evaluation or baseline analysis, not after months of stable performance.
C. A data quality issue would have affected the model's performance right after deployment, not months later.
D. Incorrect labels during baseline calculation could produce an inaccurate baseline F1 score, but it wouldn't explain a significant drop after stable performance over months.
A 2
Selected Answer: A
Option A could be the only possible reason for drifting "after several months".
A 1
Selected Answer: A
A. Yes, concept drift is an evolution of data that invalidates the data model. It happens when the statistical properties of the target variable, which the model is trying to predict, change over time in unforeseen ways. This causes problems because the predictions become less accurate as time passes.
B. No, if it was the case the F1 would have been low since the begin and this is not justifying a change after months
C. No, same as B
D. No, incorrect labels in the baseline calculations would undermine F1 baseline value, but this is not explain a significant drop after months
A 1
Selected Answer: A
concept drift is an evolution of data that invalidates the data model. It happens when the statistical properties of the target variable, which the model is trying to predict, shift over time in ways that were not expected. This causes problems because the predictions get less accurate as time passes.