Transfer learning and domain adaptation exist because training data (the source domain) and deployment data (the target domain) rarely match. A model might learn from one website’s traffic and be used on another, or be trained on last year’s customer behaviour and deployed after a pricing change. In these cases, strong source validation can still hide target failures. Evaluation must therefore answer two questions: are predictions correct for the target task, and does the model remain reliable when the source and target distributions diverge? Many practitioners first encounter the vocabulary in an AI course in Kolkata, but the value comes from applying these metrics before and after deployment.

Define the target evaluation setup

Before choosing metrics, clarify what you can observe in the target domain.

• Labelled target data available: you can evaluate task performance directly.
• Few or no target labels: you will need proxy metrics (shift, alignment, confidence) plus limited spot-labelling.
• Multiple target segments: treat region, device type, time window, and customer cohort as separate targets and report per-segment outcomes.

Always compare against a baseline: the source-trained model evaluated on target data with no adaptation. Adaptation should beat this baseline consistently, not just on one convenient split.

Target task metrics are still the anchor

When you have labelled target data, task metrics remain the primary decision signal because they measure the real objective.

For classification:

• Use accuracy only when classes are balanced.
• Prefer macro-F1 or balanced accuracy when class imbalance changes between domains.
• For rare positives (fraud, defects), AUPRC is often more informative than AUROC.
• Review per-class recall and precision so average improvements do not hide a critical drop for a costly class.

For regression:

• MAE is interpretable and robust to outliers.
• RMSE penalises large errors more; use it when big misses are unacceptable.

Because adaptation can be sensitive to training dynamics, add bootstrapped confidence intervals and evaluate across multiple random seeds.

Divergence and representation alignment metrics

Task metrics tell you “how well,” but not “how different” the domains are. Divergence and alignment metrics quantify the gap and help explain why performance changes.

Useful choices include:

• Maximum Mean Discrepancy (MMD): distribution distance in a kernel space.
• Wasserstein distance: emphasises structured shifts.
• KL divergence or Jensen–Shannon divergence: compares probability distributions (for example, predicted class histograms).
• Population Stability Index (PSI): bin-based feature drift score for monitoring.
• Two-sample tests such as KS (continuous) and chi-square (categorical) for feature-level drift checks.

Compute these metrics not only on raw inputs, but also on learned embeddings (for example, the penultimate layer). A strong adaptation method may not remove raw drift, but it should reduce the effective source–target gap in representation space. Many teams start tracking embedding-level drift after an AI course in Kolkata, because it supports both method selection and early drift detection.

Guardrail metrics and evaluation protocols

Domain adaptation can backfire through negative transfer: it improves some segments while harming others, or it becomes confidently wrong. Add guardrails that surface these risks early.

• Target lift over baseline: report the delta between adapted and source-only models per segment, not only overall.
• Worst-segment performance: track the minimum score across segments (regions/devices/cohorts).
• Calibration: Expected Calibration Error (ECE) and Brier score test whether predicted probabilities match reality.
• Uncertainty behaviour: predictive entropy or probability margin should increase on unfamiliar examples.
• Selective prediction: measure performance as you abstain on low-confidence cases and route them to human review.

Make these metrics trustworthy with simple protocols: keep a clean labelled target test set that is never used for tuning, use time-aware splits when shift is temporal, and tune decision thresholds on target validation data rather than on the source domain. If you use pseudo-labels, audit their accuracy on a small labelled target subset. This checklist is often written down after an AI course in Kolkata to keep evaluation consistent across teams.

Conclusion

Evaluating domain adaptation is not a single-number exercise. Anchor decisions on target task metrics, quantify the domain gap with divergence and alignment measures, and add guardrails for worst-segment performance, calibration, and uncertainty. Whether you are formalising this process inside a product team or learning the foundations in an AI course in Kolkata, these criteria help you decide if adaptation truly improves real-world performance under distribution change.