You Memorized the Exam
Asystole

Perfect on my training data.

You memorized the exam.
Let's meet a new patient.

Overfitting: learn the noise along with useful patterns;
Look brilliant on the training set, then new performance flattens.
Too much flexibility for the information there;
Your model learnt the seating chart and called it clinical flair.
Regularization can shrink those eager coefficients;
Careful development needs enough data and clear decisions.
Internal validation estimates the optimism in the show;
Bootstrap or cross-validation helps you see what you don't know.
Keep preprocessing and tuning inside the training fold;
Don't let the held-out outcomes tell the model what to hold.
Same patient's records spilling into both sides of a split?
Your algorithm recognised the face and called that clinical wit.
Predictors must be available when the prediction's due;
A discharge code won't help a forecast made before you knew.
Data leakage: future clues sneak back into the plan;
Your crystal ball is reading from tomorrow's discharge scan.

Can it rank? Do its risks agree
With outcomes in new people that it didn't get to see?
Freeze the model, test elsewhere, show uncertainty too;
You memorized the exam. Now who are you?
Can it rank? Do its risks agree
With outcomes in new people that it didn't get to see?
Freeze the model, test elsewhere, show uncertainty too;
You memorized the exam. Now who are you?

Discrimination: separate those with events from those without;
For a binary outcome at the stated time, work it out.
A-U-C: an event case meets a nonevent in the draw;
How often does the first rank higher? Ties get half the score.
Point-nine describes that ranking. For labels, choose a line;
Accuracy also shifts with case mix and threshold design.
Calibration asks whether predicted risks agree
With observed frequencies across the risks we see.
Among many given twenty percent risk for a year,
About twenty in a hundred should have events appear.
Check across the range; one overall match can deceive;
A good ranking can coexist with risks you shouldn't believe.
External validation: new data, frozen model, clear;
Different place or later time can show what travelled here.
Case mix and measurement can shift performance in the room;
Your training-set applause can't tell us how the next will do.

Can it rank? Do its risks agree
With outcomes in new people that it didn't get to see?
Freeze the model, test elsewhere, show uncertainty too;
You memorized the exam. Now who are you?
Can it rank? Do its risks agree
With outcomes in new people that it didn't get to see?
Freeze the model, test elsewhere, show uncertainty too;
You memorized the exam. Now who are you?

If you update on new data, test the update honestly;
Those same outcomes can't give untouched validation for free.
Check groups and settings; show intervals for the scores;
Calibration can drift after a model leaves its doors.
Clinical usefulness asks what happens when we act:
Benefits, missed cases, false alarms all enter that pact.
A strong score alone won't show improved patient care;
Evaluate the decisions and the consequences there.

Can it rank? Do its risks agree
With outcomes in new people that it didn't get to see?
Freeze the model, test elsewhere, show uncertainty too;
You memorized the exam. Now who are you?
Can it rank? Do its risks agree
With outcomes in new people that it didn't get to see?
Freeze the model, test elsewhere, show uncertainty too;
You memorized the exam. Now who are you?

My model's perfect.

Where?

In the title of my paper.
...All right. New patients. New test.