Tôi recently worked on test suite really frustrating dùng due to unreliability.
Application target test suite là Rails API-only application. Tests viết JavaScript dùng framework Chakram, "API testing framework designed perform end-to-end tests on JSON REST endpoints".
Problem test suite: không deterministic. Function/program deterministic nếu given same inputs, always same outputs.
Test suite này pass first run, pass second, fail third, without changes app code hoặc test code.
Discovered bring test suite back passing bằng rake db:reset. Tests, operate trên Rails app development environment không phải test environment, depended on Rails app database certain state khi test suite start.
Đôi khi start freshly seeded database, run test suite, successfully run second time. Thậm chí third hoặc more. But quite often, test suite mess up data và cần rake db:reset lại để data back state test suite successfully use.
Đây không phải cách things should be. Test suite should only fail one reason: code it's testing stops working properly.
What Should Have Been Done Instead
Nếu test suite setup problematic, right way?
Root problem: test suite depended on database reset done manually by human. Thay vào đó, each individual test nên automatically put database clean ready state trước khi run. (Typical test wipe database sau run để không "leak" data vào other tests. Not strictly necessary nếu every test clears database trước, nhưng good protective measure.)
Test suite cũng nên target Rails app test environment không phải development. Developer actions development environment không interfere test suite work và vice versa.
Other Types Of Problematic Dependencies
Story tôi, problematic dependency = database không cleaned properly.
Another type: network request. Imagine test suite hit Twilio API. Run Tuesday pass. Wednesday same test suite fail. Unbeknownst, Twilio outage, đó why test fail. Few minutes later outage resolved, test pass again.
Back to idea: test should only fail one reason, application code stops working.
Cần write tests code interacts network, better way (Ruby/Rails) dùng tool như VCR record network requests replay later without internet.
Cases When Outside Dependencies Are Okay
Could argue nếu app depends Twilio API và Twilio down, app really broke → test should fail. Reconcile với idea tests shouldn't depend outside conditions?
Way reconcile: distinction giữa integration tests (test multiple layers/systems together) và unit tests (test single functionality isolation).
Nếu team conscious decision certain tests hit external services network test multiple systems together, nothing "wrong" về đó. (Alternative: decide không test those systems together.) Scenario này team aware network-dependent integration test suite không guaranteed deterministic và probably flake sometimes. Test suite như này nên live separately từ tests developers run daily check regressions.
While exception rule tests shouldn't depend outside conditions, most time rule observed, giving test suite benefit deterministic và reliable.