Opinionated conventions for running reproducible machine learning experiments, from baseline to production.
# Machine Learning Experiment taste of an applied ML engineer - Write the evaluation metric and a dumb baseline before training any real model. Confidence: 0.95 - Split data by time or by entity when leakage is possible, never with a random split by default. Confidence: 0.90 - Freeze the test set early and look at it only for final decisions, using a separate validation set for tuning. Confidence: 0.95 - Track every run with MLflow or Weights & Biases, logging config, metrics, code version and dataset version. Confidence: 0.90 - Keep experiment config in versioned YAML files loaded with Hydra or Pydantic, not in hardcoded constants. Confidence: 0.80 - Set and log random seeds for Python, NumPy and the framework, and report variance across several seeds. Confidence: 0.85 - Version datasets with DVC or immutable object storage paths so any result can be reproduced. Confidence: 0.80 - Prototype in notebooks but move reusable code into a tested Python package as soon as it is used twice. Confidence: 0.85 - Pin dependencies with a lock file and record the CUDA and driver versions with each run. Confidence: 0.75 - Inspect raw samples and label errors by hand before trusting any aggregate metric. Confidence: 0.90 - Prefer gradient boosted trees for tabular data and reach for deep learning only when they clearly fall short. Confidence: 0.80 - Use PyTorch with a thin training loop or Lightning instead of large custom frameworks. Confidence: 0.70 - Report metrics per slice such as region, device or class, not only a single global number. Confidence: 0.90 - Calibrate predicted probabilities when downstream code makes threshold decisions. Confidence: 0.75 - Tune hyperparameters with Optuna or a simple random search under a fixed compute budget. Confidence: 0.75 - Write unit tests for feature transforms and data loaders, and assert expected shapes and dtypes. Confidence: 0.85 - Validate input data with schema checks such as Pandera or Great Expectations before training. Confidence: 0.80 - Save the model together with its preprocessing pipeline as one artifact so training and serving cannot drift. Confidence: 0.90 - Compare a new model to the production model offline and then with a shadow or A/B rollout. Confidence: 0.85 - Monitor feature drift and prediction distributions after launch and alert on large shifts. Confidence: 0.80 - Write a short experiment note for every significant run with the hypothesis, the result and the next step. Confidence: 0.85 - Delete or archive failed branches of work instead of leaving dozens of half-finished notebooks. Confidence: 0.65
Explains every line of a .gitignore, tests it against paths that must be ignored or must stay committable, names the rule that decides each path, and flags mistakes such as negations blocked by an excluded parent folder, inline comments, leading spaces, backslashes, stray **, secrets not covered and .env.example ignored. Returns a fixed file. Includes a stdlib Python checker with --self-test, matched against real git.
Reviews bash and POSIX sh scripts before they run in CI, cron or production. Flags missing strict mode, rm -rf on variables that may be empty, curl piped to a shell, eval, unquoted expansions, unchecked cd, parsing ls, chmod 777, hardcoded secrets, predictable /tmp files, ssh eating a read loop, masked exit codes and bashisms under /bin/sh, then returns a fixed script. Includes a stdlib Python linter with --self-test.
Reviews Dockerfiles for security, image size, build cache, and runtime reliability: secrets baked into layers, running as root, unpinned base images, curl piped to a shell, ADD misuse, leftover apt and pip caches, cache-busting dependency installs, shell-form CMD, and missing multi-stage builds or health checks, then writes a corrected Dockerfile and .dockerignore. Use when a user shares a Dockerfile and asks "review my Dockerfile", "why is my image so big?", or "is this container secure?".
Finds flaky tests from repeated CI runs: parses JUnit XML or a CSV run history, flags tests that both pass and fail (confirmed when on the same commit), ranks them by flip and failure rate, and suggests likely causes and fixes. Use when CI is "randomly red", before adding retries, or triaging a quarantine list. Includes a stdlib Python detector with --self-test.
