Duyệt dataset StackOverflow và xem mẫu dữ liệu với TensorFlow

TensorFlow text_dataset_from_directory tải dữ liệu văn bản có cấu trúc thư mục (mỗi class một folder).

Ví dụ: Load train/val/test, in sample

batch_size = 32
seed = 42

raw_train_ds = preprocessing.text_dataset_from_directory(
    train_dir, batch_size=batch_size,
    validation_split=0.25, subset='training', seed=seed
)

for text_batch, label_batch in raw_train_ds.take(1):
    for i in range(10):
        print("Question:", text_batch.numpy()[i][:100], '...')
        print("Label:", label_batch.numpy()[i])

# Validation set
raw_val_ds = preprocessing.text_dataset_from_directory(
    train_dir, batch_size=batch_size,
    validation_split=0.25, subset='validation', seed=seed
)

# Test set
test_dir = dataset_dir / 'test'
raw_test_ds = preprocessing.text_dataset_from_directory(
    test_dir, batch_size=batch_size
)

Kết quả

Iterating through the training data
Label 0 maps to csharp
Label 1 maps to java
Label 2 maps to javascript
Label 3 maps to python
The training parameters have been defined
Found 8000 files belonging to 4 classes. Using 6000 files for training.
Question: b'"my tester is going to the wrong constructor i am new to programming..."'
Label: 1
Question: b'"blank code slow skin detection this code changes the color space to lab..."'
Label: 3
... (8 câu hỏi khác)
The test dataset is being prepared
Found 8000 files belonging to 4 classes.

Giải thích

• validation_split=0.25 chia 25% train dir làm validation.
• Label tự động gán theo tên thư mục con (csharp, java, javascript, python).
• take(1) lấy batch đầu tiên để xem nhanh.

Nguồn: TensorFlow Text Tutorial