Cognixa Labs
AI Model Development

AI Model Interpretability and Explainability: Making Black Boxes Transparent

Comprehensive guide to understanding and explaining AI model decisions using SHAP, LIME, attention mechanisms, and other interpretability techniques.

By Cognixa Labs Research Team January 15, 2024 18 min read

As AI systems become more complex and influential in decision-making, understanding how they work becomes crucial. This guide covers modern techniques for interpreting and explaining AI model behavior.

SHAP (SHapley Additive exPlanations)

SHAP Implementation for Tabular Data

import shap
import pandas as pd
import numpy as np
from sklearn.ensemble import RandomForestClassifier
from sklearn.model_selection import train_test_split
import matplotlib.pyplot as plt

class SHAPExplainer:
    def __init__(self, model, X_train, feature_names=None):
        self.model = model
        self.X_train = X_train
        self.feature_names = feature_names or [f'feature_{i}' for i in range(X_train.shape[1])]
        self.explainer = None
        self.shap_values = None
    
    def create_explainer(self, explainer_type='tree'):
        """Create SHAP explainer based on model type"""
        if explainer_type == 'tree':
            self.explainer = shap.TreeExplainer(self.model)
        elif explainer_type == 'kernel':
            self.explainer = shap.KernelExplainer(
                self.model.predict_proba, 
                shap.sample(self.X_train, 100)
            )
        elif explainer_type == 'linear':
            self.explainer = shap.LinearExplainer(
                self.model, 
                self.X_train
            )
        else:
            raise ValueError(f"Unsupported explainer type: {explainer_type}")
    
    def explain_predictions(self, X_test):
        """Generate SHAP values for test data"""
        if self.explainer is None:
            self.create_explainer()
        
        self.shap_values = self.explainer.shap_values(X_test)
        return self.shap_values
    
    def plot_summary(self, X_test, max_display=20):
        """Create SHAP summary plot"""
        if self.shap_values is None:
            self.explain_predictions(X_test)
        
        shap.summary_plot(
            self.shap_values, 
            X_test, 
            feature_names=self.feature_names,
            max_display=max_display
        )
    
    def plot_waterfall(self, instance_idx, X_test):
        """Create waterfall plot for single prediction"""
        if self.shap_values is None:
            self.explain_predictions(X_test)
        
        shap.waterfall_plot(
            shap.Explanation(
                values=self.shap_values[instance_idx],
                base_values=self.explainer.expected_value,
                data=X_test[instance_idx],
                feature_names=self.feature_names
            )
        )
    
    def get_feature_importance(self, X_test):
        """Calculate global feature importance"""
        if self.shap_values is None:
            self.explain_predictions(X_test)
        
        importance = np.abs(self.shap_values).mean(axis=0)
        
        return pd.DataFrame({
            'feature': self.feature_names,
            'importance': importance
        }).sort_values('importance', ascending=False)

SHAP for Deep Learning Models

import torch
import torch.nn as nn
from captum.attr import IntegratedGradients, DeepLift, GradientShap

class DeepSHAPExplainer:
    def __init__(self, model, device='cpu'):
        self.model = model
        self.device = device
        self.model.eval()
    
    def explain_image_classification(self, images, target_class=None):
        """Explain image classification predictions"""
        # Use Integrated Gradients
        ig = IntegratedGradients(self.model)
        
        if target_class is None:
            # Get predicted class
            with torch.no_grad():
                outputs = self.model(images)
                target_class = outputs.argmax(dim=1)
        
        # Generate attributions
        attributions = ig.attribute(
            images, 
            target=target_class,
            n_steps=50
        )
        
        return attributions.detach().cpu().numpy()
    
    def explain_text_classification(self, input_ids, attention_mask, target_class=None):
        """Explain text classification predictions"""
        # Use DeepLift for text
        dl = DeepLift(self.model)
        
        if target_class is None:
            with torch.no_grad():
                outputs = self.model(input_ids, attention_mask=attention_mask)
                target_class = outputs.logits.argmax(dim=1)
        
        # Generate attributions
        attributions = dl.attribute(
            inputs=(input_ids, attention_mask),
            target=target_class
        )
        
        return attributions[0].detach().cpu().numpy()  # Return input_ids attributions
    
    def visualize_image_attributions(self, image, attributions, alpha=0.5):
        """Visualize image attributions as heatmap overlay"""
        import matplotlib.pyplot as plt
        
        # Normalize attributions
        attr_norm = np.abs(attributions).max(axis=0)
        attr_norm = (attr_norm - attr_norm.min()) / (attr_norm.max() - attr_norm.min())
        
        fig, (ax1, ax2, ax3) = plt.subplots(1, 3, figsize=(15, 5))
        
        # Original image
        ax1.imshow(image.transpose(1, 2, 0))
        ax1.set_title('Original Image')
        ax1.axis('off')
        
        # Attribution heatmap
        ax2.imshow(attr_norm, cmap='hot')
        ax2.set_title('Attribution Heatmap')
        ax2.axis('off')
        
        # Overlay
        ax3.imshow(image.transpose(1, 2, 0))
        ax3.imshow(attr_norm, cmap='hot', alpha=alpha)
        ax3.set_title('Attribution Overlay')
        ax3.axis('off')
        
        plt.tight_layout()
        plt.show()

LIME (Local Interpretable Model-agnostic Explanations)

LIME for Tabular Data

from lime import lime_tabular
from lime.lime_text import LimeTextExplainer
from lime.lime_image import LimeImageExplainer

class LIMEExplainer:
    def __init__(self, X_train, feature_names, class_names, mode='classification'):
        self.X_train = X_train
        self.feature_names = feature_names
        self.class_names = class_names
        self.mode = mode
        
        self.explainer = lime_tabular.LimeTabularExplainer(
            X_train,
            feature_names=feature_names,
            class_names=class_names,
            mode=mode
        )
    
    def explain_instance(self, instance, predict_fn, num_features=10):
        """Explain a single prediction"""
        explanation = self.explainer.explain_instance(
            instance,
            predict_fn,
            num_features=num_features
        )
        return explanation
    
    def get_explanation_data(self, explanation):
        """Extract explanation data as structured format"""
        exp_list = explanation.as_list()
        
        return pd.DataFrame({
            'feature': [item[0] for item in exp_list],
            'importance': [item[1] for item in exp_list]
        })
    
    def batch_explain(self, X_test, predict_fn, num_features=10):
        """Explain multiple instances"""
        explanations = []
        
        for i, instance in enumerate(X_test):
            exp = self.explain_instance(instance, predict_fn, num_features)
            exp_data = self.get_explanation_data(exp)
            exp_data['instance_id'] = i
            explanations.append(exp_data)
        
        return pd.concat(explanations, ignore_index=True)

LIME for Text Classification

class LIMETextExplainer:
    def __init__(self, class_names):
        self.class_names = class_names
        self.explainer = LimeTextExplainer(class_names=class_names)
    
    def explain_text(self, text, predict_fn, num_features=10):
        """Explain text classification prediction"""
        explanation = self.explainer.explain_instance(
            text,
            predict_fn,
            num_features=num_features
        )
        return explanation
    
    def visualize_text_explanation(self, explanation, save_path=None):
        """Create HTML visualization of text explanation"""
        html = explanation.as_html()
        
        if save_path:
            with open(save_path, 'w', encoding='utf-8') as f:
                f.write(html)
        
        return html
    
    def get_word_importances(self, explanation):
        """Extract word importance scores"""
        exp_list = explanation.as_list()
        
        return {
            'words': [item[0] for item in exp_list],
            'scores': [item[1] for item in exp_list]
        }

Attention Visualization for Transformers

import torch
from transformers import AutoTokenizer, AutoModel
import seaborn as sns
import matplotlib.pyplot as plt

class AttentionVisualizer:
    def __init__(self, model_name):
        self.tokenizer = AutoTokenizer.from_pretrained(model_name)
        self.model = AutoModel.from_pretrained(model_name, output_attentions=True)
        self.model.eval()
    
    def get_attention_weights(self, text):
        """Extract attention weights from transformer model"""
        inputs = self.tokenizer(text, return_tensors='pt', padding=True, truncation=True)
        
        with torch.no_grad():
            outputs = self.model(**inputs)
        
        # Get attention weights from all layers
        attentions = outputs.attentions  # Tuple of attention weights
        tokens = self.tokenizer.convert_ids_to_tokens(inputs['input_ids'][0])
        
        return attentions, tokens
    
    def visualize_attention_head(self, text, layer=0, head=0, figsize=(12, 8)):
        """Visualize attention weights for specific layer and head"""
        attentions, tokens = self.get_attention_weights(text)
        
        # Get attention weights for specified layer and head
        attention = attentions[layer][0, head].cpu().numpy()
        
        # Create heatmap
        plt.figure(figsize=figsize)
        sns.heatmap(
            attention,
            xticklabels=tokens,
            yticklabels=tokens,
            cmap='Blues',
            cbar=True
        )
        plt.title(f'Attention Weights - Layer {layer}, Head {head}')
        plt.xlabel('Key Tokens')
        plt.ylabel('Query Tokens')
        plt.xticks(rotation=45)
        plt.yticks(rotation=0)
        plt.tight_layout()
        plt.show()
    
    def get_attention_summary(self, text):
        """Get summary statistics of attention patterns"""
        attentions, tokens = self.get_attention_weights(text)
        
        summary = {
            'num_layers': len(attentions),
            'num_heads': attentions[0].shape[1],
            'sequence_length': len(tokens),
            'tokens': tokens
        }
        
        # Calculate attention entropy for each layer
        entropies = []
        for layer_attention in attentions:
            # Average across heads and compute entropy
            avg_attention = layer_attention.mean(dim=1)[0]  # [seq_len, seq_len]
            entropy = -(avg_attention * torch.log(avg_attention + 1e-8)).sum(dim=1).mean()
            entropies.append(entropy.item())
        
        summary['attention_entropies'] = entropies
        return summary
    
    def find_important_tokens(self, text, threshold=0.1):
        """Find tokens with high attention weights"""
        attentions, tokens = self.get_attention_weights(text)
        
        # Average attention across all layers and heads
        all_attention = torch.stack(attentions).mean(dim=(0, 1, 2))  # [seq_len]
        
        important_indices = (all_attention > threshold).nonzero().squeeze()
        important_tokens = [(tokens[i], all_attention[i].item()) for i in important_indices]
        
        return sorted(important_tokens, key=lambda x: x[1], reverse=True)

Gradient-based Explanations

from captum.attr import (
    Saliency, InputXGradient, GuidedBackprop, 
    IntegratedGradients, GradCAM
)

class GradientExplainer:
    def __init__(self, model, device='cpu'):
        self.model = model
        self.device = device
        self.model.eval()
    
    def saliency_maps(self, inputs, target_class):
        """Generate saliency maps using gradients"""
        saliency = Saliency(self.model)
        attribution = saliency.attribute(inputs, target=target_class)
        return attribution.detach().cpu().numpy()
    
    def integrated_gradients(self, inputs, target_class, baseline=None, steps=50):
        """Generate attributions using Integrated Gradients"""
        ig = IntegratedGradients(self.model)
        
        if baseline is None:
            baseline = torch.zeros_like(inputs)
        
        attribution = ig.attribute(
            inputs, 
            baselines=baseline,
            target=target_class,
            n_steps=steps
        )
        return attribution.detach().cpu().numpy()
    
    def guided_backprop(self, inputs, target_class):
        """Generate attributions using Guided Backpropagation"""
        gbp = GuidedBackprop(self.model)
        attribution = gbp.attribute(inputs, target=target_class)
        return attribution.detach().cpu().numpy()
    
    def grad_cam(self, inputs, target_class, target_layer):
        """Generate GradCAM heatmaps"""
        grad_cam = GradCAM(self.model, target_layer)
        attribution = grad_cam.attribute(inputs, target=target_class)
        return attribution.detach().cpu().numpy()
    
    def compare_methods(self, inputs, target_class, target_layer=None):
        """Compare different gradient-based explanation methods"""
        methods = {
            'Saliency': self.saliency_maps(inputs, target_class),
            'Integrated Gradients': self.integrated_gradients(inputs, target_class),
            'Guided Backprop': self.guided_backprop(inputs, target_class)
        }
        
        if target_layer is not None:
            methods['GradCAM'] = self.grad_cam(inputs, target_class, target_layer)
        
        return methods
    
    def visualize_comparisons(self, original_image, attributions_dict):
        """Visualize comparison of different explanation methods"""
        num_methods = len(attributions_dict)
        fig, axes = plt.subplots(1, num_methods + 1, figsize=(5 * (num_methods + 1), 5))
        
        # Original image
        axes[0].imshow(original_image.transpose(1, 2, 0))
        axes[0].set_title('Original')
        axes[0].axis('off')
        
        # Attribution visualizations
        for i, (method_name, attribution) in enumerate(attributions_dict.items()):
            # Normalize attribution
            attr_norm = np.abs(attribution).max(axis=0)
            attr_norm = (attr_norm - attr_norm.min()) / (attr_norm.max() - attr_norm.min())
            
            axes[i + 1].imshow(attr_norm, cmap='hot')
            axes[i + 1].set_title(method_name)
            axes[i + 1].axis('off')
        
        plt.tight_layout()
        plt.show()

Model-Agnostic Interpretability

from sklearn.inspection import permutation_importance
from sklearn.metrics import accuracy_score

class ModelAgnosticExplainer:
    def __init__(self, model, X_train, y_train, X_test, y_test):
        self.model = model
        self.X_train = X_train
        self.y_train = y_train
        self.X_test = X_test
        self.y_test = y_test
    
    def permutation_importance_analysis(self, n_repeats=10, random_state=42):
        """Calculate permutation importance for features"""
        result = permutation_importance(
            self.model, 
            self.X_test, 
            self.y_test,
            n_repeats=n_repeats,
            random_state=random_state,
            scoring='accuracy'
        )
        
        importance_df = pd.DataFrame({
            'feature': range(len(result.importances_mean)),
            'importance_mean': result.importances_mean,
            'importance_std': result.importances_std
        }).sort_values('importance_mean', ascending=False)
        
        return importance_df, result
    
    def partial_dependence_analysis(self, feature_indices, grid_resolution=100):
        """Calculate partial dependence for specified features"""
        from sklearn.inspection import partial_dependence
        
        pdp_results = {}
        
        for feature_idx in feature_indices:
            pd_result = partial_dependence(
                self.model,
                self.X_train,
                features=[feature_idx],
                grid_resolution=grid_resolution
            )
            
            pdp_results[feature_idx] = {
                'values': pd_result['values'][0],
                'average': pd_result['average'][0]
            }
        
        return pdp_results
    
    def feature_interaction_analysis(self, feature_pairs):
        """Analyze feature interactions using partial dependence"""
        from sklearn.inspection import partial_dependence
        
        interaction_results = {}
        
        for pair in feature_pairs:
            pd_result = partial_dependence(
                self.model,
                self.X_train,
                features=pair,
                grid_resolution=20
            )
            
            interaction_results[tuple(pair)] = {
                'values': pd_result['values'],
                'average': pd_result['average']
            }
        
        return interaction_results
    
    def global_surrogate_model(self, surrogate_model_class, **surrogate_params):
        """Train a global surrogate model for interpretability"""
        # Get predictions from black box model
        y_pred = self.model.predict(self.X_train)
        
        # Train surrogate model
        surrogate = surrogate_model_class(**surrogate_params)
        surrogate.fit(self.X_train, y_pred)
        
        # Evaluate surrogate fidelity
        surrogate_pred = surrogate.predict(self.X_test)
        original_pred = self.model.predict(self.X_test)
        
        fidelity = accuracy_score(original_pred, surrogate_pred)
        
        return surrogate, fidelity
    
    def anchors_explanation(self, instance, predict_fn, feature_names):
        """Generate anchor explanations (rule-based explanations)"""
        # Simplified anchor implementation
        # In practice, use the anchors library
        
        # Find stable features (features that don't change prediction when perturbed)
        stable_features = []
        original_pred = predict_fn([instance])[0]
        
        for i, feature_name in enumerate(feature_names):
            # Perturb feature
            perturbed_instance = instance.copy()
            perturbed_instance[i] *= 1.1  # 10% increase
            
            perturbed_pred = predict_fn([perturbed_instance])[0]
            
            if original_pred == perturbed_pred:
                stable_features.append((i, feature_name))
        
        return {
            'anchor_features': stable_features,
            'prediction': original_pred,
            'coverage': len(stable_features) / len(feature_names)
        }

Interpretability Evaluation Framework

class InterpretabilityEvaluator:
    def __init__(self, model, X_test, y_test):
        self.model = model
        self.X_test = X_test
        self.y_test = y_test
    
    def faithfulness_test(self, explanation_method, num_samples=100):
        """Test faithfulness of explanations by feature removal"""
        faithfulness_scores = []
        
        for i in range(min(num_samples, len(self.X_test))):
            instance = self.X_test[i:i+1]
            original_pred = self.model.predict_proba(instance)[0]
            
            # Get feature importance from explanation method
            importance = explanation_method(instance)
            
            # Remove top important features and measure prediction change
            top_features = np.argsort(np.abs(importance))[-5:]  # Top 5 features
            
            modified_instance = instance.copy()
            modified_instance[:, top_features] = 0  # Remove features
            
            modified_pred = self.model.predict_proba(modified_instance)[0]
            
            # Calculate prediction change
            pred_change = np.abs(original_pred - modified_pred).max()
            faithfulness_scores.append(pred_change)
        
        return np.mean(faithfulness_scores)
    
    def stability_test(self, explanation_method, num_perturbations=10, noise_level=0.1):
        """Test stability of explanations under input perturbations"""
        stability_scores = []
        
        for i in range(min(50, len(self.X_test))):
            instance = self.X_test[i:i+1]
            original_explanation = explanation_method(instance)
            
            perturbation_explanations = []
            for _ in range(num_perturbations):
                # Add noise to instance
                noise = np.random.normal(0, noise_level, instance.shape)
                perturbed_instance = instance + noise
                
                perturbed_explanation = explanation_method(perturbed_instance)
                perturbation_explanations.append(perturbed_explanation)
            
            # Calculate stability as correlation with original
            correlations = [
                np.corrcoef(original_explanation.flatten(), exp.flatten())[0, 1]
                for exp in perturbation_explanations
            ]
            
            stability_scores.append(np.mean(correlations))
        
        return np.mean(stability_scores)
    
    def comprehensiveness_test(self, explanation_method, percentiles=[10, 30, 50, 70, 90]):
        """Test comprehensiveness by removing features in order of importance"""
        results = {}
        
        for percentile in percentiles:
            prediction_changes = []
            
            for i in range(min(100, len(self.X_test))):
                instance = self.X_test[i:i+1]
                original_pred = self.model.predict_proba(instance)[0]
                
                # Get feature importance
                importance = explanation_method(instance)
                
                # Remove top percentile of features
                num_features_to_remove = int(len(importance) * percentile / 100)
                top_features = np.argsort(np.abs(importance))[-num_features_to_remove:]
                
                modified_instance = instance.copy()
                modified_instance[:, top_features] = 0
                
                modified_pred = self.model.predict_proba(modified_instance)[0]
                pred_change = np.abs(original_pred - modified_pred).max()
                
                prediction_changes.append(pred_change)
            
            results[percentile] = np.mean(prediction_changes)
        
        return results
    
    def generate_evaluation_report(self, explanation_methods):
        """Generate comprehensive evaluation report"""
        report = {}
        
        for method_name, method_func in explanation_methods.items():
            print(f"Evaluating {method_name}...")
            
            report[method_name] = {
                'faithfulness': self.faithfulness_test(method_func),
                'stability': self.stability_test(method_func),
                'comprehensiveness': self.comprehensiveness_test(method_func)
            }
        
        return report

Key Benefits and Applications

Trust and Transparency

  • Regulatory Compliance: Meet explainability requirements
  • Stakeholder Confidence: Build trust in AI decisions
  • Bias Detection: Identify unfair model behavior
  • Error Analysis: Understand model failures

Business Value

  • Decision Support: Provide reasoning for recommendations
  • Model Debugging: Improve model performance
  • Feature Engineering: Identify important variables
  • Risk Management: Understand model limitations

Implementation Best Practices

  1. Multiple Methods: Use complementary explanation techniques
  2. Evaluation Metrics: Assess explanation quality
  3. User-Centric Design: Tailor explanations to audience
  4. Continuous Monitoring: Track explanation consistency
  5. Documentation: Maintain explanation methodology records

Future Directions

  • Causal Explanations: Move beyond correlational explanations
  • Interactive Explanations: Dynamic user-driven exploration
  • Multimodal Explanations: Combine text, visual, and numerical explanations
  • Automated Explanation Generation: AI-generated natural language explanations

Conclusion

Model interpretability is essential for responsible AI deployment. By combining multiple explanation techniques and rigorous evaluation, we can build transparent and trustworthy AI systems that users can understand and rely upon.

References

  1. Lundberg, S. M., & Lee, S. I. (2017). A unified approach to interpreting model predictions. NIPS.
  2. Ribeiro, M. T., et al. (2016). "Why should I trust you?": Explaining the predictions of any classifier. KDD.
  3. Molnar, C. (2020). Interpretable machine learning. Lulu.com.
InterpretabilitySHAPLIMEExplainable AIModel Transparency
CX

Cognixa Labs Research Team

Independent AI research and benchmarking

The collective byline for benchmark releases, methodology notes, and research writing from Cognixa Labs.

Keep riding

Related from the archive