Tạo Callback ModelCheckpoint lưu weights Keras Python

Mô tả

Tạo callback ModelCheckpoint để tự động lưu weights trong training, cấu hình đường dẫn, tần suất, điều kiện lưu.

Code ví dụ đầy đủ

import tensorflow as tf
from tensorflow import keras
from tensorflow.keras import layers
import numpy as np
import os

def create_model():
    model = keras.Sequential([
        layers.Dense(64, activation='relu', input_shape=(784,)),
        layers.Dense(10, activation='softmax')
    ])
    model.compile(optimizer='adam', 
                  loss='sparse_categorical_crossentropy', 
                  metrics=['accuracy'])
    return model

# Data
x_train = np.random.rand(1000, 784).astype('float32')
y_train = np.random.randint(0, 10, 1000)
x_test = np.random.rand(200, 784).astype('float32')
y_test = np.random.randint(0, 10, 200)

# 1. CHECKPOINT PATH
checkpoint_path = "training_1/cp.ckpt"
checkpoint_dir = os.path.dirname(checkpoint_path)
os.makedirs(checkpoint_dir, exist_ok=True)

# 2. TẠO CALLBACK
cp_callback = keras.callbacks.ModelCheckpoint(
    filepath=checkpoint_path,
    save_weights_only=True,      # Chỉ weights
    save_best_only=False,        # Lưu mọi epoch
    monitor='val_loss',          # Theo dõi val_loss
    mode='min',                  # Minimize loss
    verbose=1
)

print("Set checkpoint path:", checkpoint_path)
print("Creating callback to save weights")

# 3. TRAIN VỚI CALLBACK
model = create_model()
print("Model is trained with new callback")
history = model.fit(
    x_train, y_train,
    epochs=10,
    validation_data=(x_test, y_test),
    callbacks=[cp_callback],
    verbose=1
)

# 4. LIỆT KÊ FILES CHECKPOINT
print(f"\nCheckpoint files in {checkpoint_dir}:")
for f in os.listdir(checkpoint_dir):
    filepath = os.path.join(checkpoint_dir, f)
    size = os.path.getsize(filepath)
    print(f"  {f} ({size} bytes)")

# 5. LATEST CHECKPOINT
latest = tf.train.latest_checkpoint(checkpoint_dir)
print(f"\nLatest checkpoint: {latest}")

Đầu ra checkpoint files

Checkpoint files in training_1:
  cp.ckpt.data-00000-of-00001
  cp.ckpt.index

# Nếu save_best_only=True, chỉ có 1 file best
# Nếu save_freq='epoch' với epoch format:
# cp-0001.ckpt.index, cp-0001.ckpt.data-...
# cp-0002.ckpt.index, cp-0002.ckpt.data-...
# ...

Các chế độ lưu (save_best_only + monitor)

Callback khác hữu ích

callbacks = [
    # Early stopping
    keras.callbacks.EarlyStopping(
        monitor='val_loss', patience=5, restore_best_weights=True
    ),
    # Reduce LR on plateau
    keras.callbacks.ReduceLROnPlateau(
        monitor='val_loss', factor=0.5, patience=3, min_lr=1e-6
    ),
    # TensorBoard logging
    keras.callbacks.TensorBoard(log_dir='logs'),
    # CSV Logger
    keras.callbacks.CSVLogger('training.log'),
    # Custom callback
    keras.callbacks.LambdaCallback(
        on_epoch_end=lambda epoch, logs: print(f"Epoch {epoch}: loss={logs['loss']:.4f}")
    )
]

Best Practices

1. Luôn dùng save_best_only=True cho production (chỉ giữ best model).
2. Kết hợp với EarlyStopping(restore_best_weights=True) để tự động rollback.
3. Dùng save_weights_only=True để tiết kiệm disk, load nhanh hơn.
4. Monitor val_loss (mode='min') an toàn hơn val_accuracy (có thể dao động).
5. Backup checkpoint định kỳ ra cloud storage cho training dài.