Start with the question an explanation should answer
Explainable AI helps people inspect model behavior. A developer investigating a suspicious feature, a reviewer assessing a prediction, and a product owner comparing model versions need different views. Define the audience and question before choosing an explanation method or visualization.
An explanation does not establish that a prediction is correct, fair, clinically reliable, or compliant with every applicable requirement. Evaluate the model and the explanation separately. This guide provides an engineering workflow, not a claimed clinical case study or a promise of stakeholder confidence.
Local explanations and global summaries
A local explanation concerns one prediction under a defined method and configuration. A global summary aggregates behavior across selected examples. Write down which population the summary represents. A plot from a convenient sample does not necessarily represent production traffic or an important subgroup.
Keep the output being explained explicit: a class score, probability, regression value, or another quantity. Label the units and transformation. An attribution on a transformed score should not be presented as a percentage-point change in probability unless that interpretation is actually supported.
SHAP: document the baseline and explainer
The SHAP documentation describes a game-theoretic approach to explaining model output and provides different explainers and visualizations. Choose an explainer appropriate to the model and the question. The name SHAP alone does not specify the computation or its assumptions.
Record the background or reference data, feature representation, output scale, explainer configuration, and library version. A baseline defines the comparison behind an attribution. Changing it can change the explanation even when the model prediction stays the same. Choose reference data deliberately and explain its scope to the audience.
Check that feature labels map to the inputs the model actually consumed. Encoded categories, missing-value handling, transformed variables, and grouped text tokens can complicate interpretation. Preserve that mapping with the release artifact so a reviewer does not mistake an encoded feature for the original business concept.
LIME: evaluate the local surrogate
The LIME maintainer documentation describes explaining a prediction by perturbing an instance and fitting a sparse linear approximation nearby. This approximation concerns a neighborhood of the instance; it need not describe the model globally.
Record the perturbation procedure, neighborhood weighting, sample count, representation, random seed where applicable, and surrogate fit. Inspect whether generated samples are meaningful for the task. A neighborhood full of implausible feature combinations may answer a different question from the one a reviewer intended.
Repeat explanations with controlled configuration changes and inspect stability. If the highlighted features vary substantially while the prediction barely changes, investigate before presenting a single plot as definitive. Do not assume LIME is always faster than SHAP: runtime depends on the selected methods, model-call cost, sample count, and implementation.
Correlated features require careful interpretation
When features carry overlapping information, importance can be distributed in ways that depend on the method and assumptions. Removing one feature may not remove the underlying information because another feature carries a similar signal. Review feature relationships alongside the plot.
Use a concrete hypothesis and follow-up experiment. If a model appears to rely on a location code, investigate whether it proxies for a relevant factor, a data-processing artifact, or an inappropriate shortcut. A plausible narrative about the plot is not evidence that the narrative is true.
Predictive attribution is not a causal intervention
SHAP's causal-interpretation guide explains why exposing relationships learned by a predictive model does not make those relationships causal. Planning an intervention requires additional assumptions and evidence.
For example, a hypothetical retention model might use support activity to predict renewal. An attribution does not show that increasing support contacts would improve renewal. The activity might reflect engagement, product problems, or another shared cause. Treat an intervention recommendation as a separate question from explaining a prediction.
The same distinction matters for counterfactual explanations. A model-output change under altered inputs does not guarantee a feasible real-world change or an approved decision. Consider immutable features, constraints, dependent variables, and whether the proposed input combination is meaningful.
Attention and heatmaps are diagnostic views
Attention weights and image heatmaps can help investigate behavior, but their visual appeal does not establish faithfulness. The 2019 Jain and Wallace attention study examined this issue across neural NLP tasks and reported cases where different attention distributions yielded equivalent predictions. Its findings motivate validation; they should not be generalized into an untested claim about every architecture.
When using a visual diagnostic, identify the method and test what it responds to. Compare selected perturbations, alternative views, and known failure examples. A highlight on a medically plausible region does not prove that the model made a reliable clinical decision. Preserve the distinction between a diagnostic image and independently validated task performance.
Validate explanation quality
- Reproducibility: preserve model, preprocessing, inputs or controlled references, explainer settings, and versions.
- Fidelity: assess whether the explanation represents the relevant model behavior in its stated scope.
- Stability: inspect changes under controlled repetition and small meaningful input or configuration changes.
- Coverage: review important groups, edge cases, missing inputs, and incorrect predictions.
- Interpretation: test whether the intended audience understands the output and its limitations.
Agree on acceptance criteria for the product rather than claiming a universal score. Document failures and disagreements. Domain review can identify implausible explanations, but agreement with an expert is not a substitute for measuring fidelity to the model.
A hypothetical debugging workflow
Suppose a product-classification model seems accurate in a development dataset but fails on new suppliers. An explanation repeatedly highlights a background label rather than the product. This observation suggests a hypothesis about a shortcut; it does not yet prove the cause.
Review labeled examples, test relevant controlled changes, and evaluate on held-out supplier data. If the evidence supports the hypothesis, revise the data or model and repeat both task evaluation and explanation review. Preserve the investigation and its limitations. This is an illustrative workflow, not a reported TensorBlue client result.
Production explanations need traceability and access controls
Associate an explanation with the actual prediction, model version, preprocessing, explainer version, timestamp, and output scale. If a cached explanation uses a different model or reference set, make that visible or invalidate it. An attractive plot with missing provenance can mislead an incident review.
Explainability logs may contain sensitive inputs or derived information. Define access, redaction, retention, and deletion behavior with the responsible owners. Store enough controlled evidence for investigation without copying every payload into an unrestricted dashboard.
Measure explanation latency and cost independently from inference. Decide whether the product needs explanations synchronously, on demand, or in an offline review. A timeout should produce a clear explanation-unavailable state rather than an invented fallback reason. Precomputed results need freshness and version checks.
Keep governance and user communication explicit
Define who can review and challenge the prediction, which information they can inspect, and how corrections are handled. Assess applicable obligations with the responsible owners. Installing an explanation library does not complete a regulatory or fairness assessment.
Use understandable labels and show the limitations needed to interpret the result. Avoid implying that a ranked feature list is the complete reason for a real-world decision. Evaluate task quality, human workflow, and explanation quality together before expanding the use.
Explanation delivery checklist
- The audience, question, output scale, and scope are defined.
- The method, baseline or neighborhood, and configuration are recorded.
- Fidelity, stability, coverage, and human interpretation are evaluated.
- Predictive associations are distinguished from intervention claims.
- Logs, caches, access, and version compatibility are controlled.
- Model performance and explanation evidence are reviewed separately.
For surrounding engineering work, read our MLOps guide and healthcare implementation guide. Explore AI consulting and application monitoring, or bring a specific model, audience, and review workflow to a scoping discussion.