51 Ethical Dilemmas

An ethical dilemma arises when we have to make a choice between alternative actions that transgress a moral (ethical) principle. There often is no right\(\texttrademark\) answer, the data scientist must weigh competing principles, consider stakeholders, and justify their reasoning.

We face ethical dilemmas all the time. Deciding how to allocate our own limited resources, money or time, between ourselves and others. Do I keep the wallet I found or return it to its owner? A manager awards a project to a friend’s department, although that department is not best equipped to succeed. Do I report a colleague or superior for unethical or illegal behavior, knowing it could lead to a hostile work environment?

51.1 Examples from Data Science

Here are some examples of ethical dilemmas in data science.

Privacy vs. Public Health
A data scientist has access to location data that could track disease spread during a pandemic with high accuracy, potentially saving lives. But using this data means violating individuals’ privacy expectations and potentially enabling future surveillance.

The tension: Preventing harm and saving lives versus respect for autonomy and privacy.

Fairness Trade-offs
A lending algorithm shows that including zip code improves accuracy in predicting loan defaults, helping the company avoid losses and potentially offering lower rates to good customers. However, zip code correlates strongly with race due to historical redlining, meaning this creates disparate impact on minority communities.

The tension: Economic efficiency and accuracy versus non-discrimination. Removing the variable entirely may still leave proxy variables that encode the same information. But which communities should bear the cost of less accurate predictions?

Individual Fairness vs. Group Fairness
A hiring algorithm can be calibrated to ensure equal hiring rates across demographic groups (group fairness) or to ensure that similar individuals receive similar scores regardless of demographics (individual fairness). Mathematical research shows you often cannot achieve both simultaneously.

The tension: You must choose which conception of fairness to prioritize, and different groups may be advantaged by different choices. A highly qualified candidate from an over-represented group might be scored lower under group fairness constraints.

Transparency vs. Security
A criminal risk assessment tool would be more trustworthy if its full methodology were public, allowing scrutiny for bias. However, transparency might allow defendants to game the system, potentially undermining public safety.

The tension: Accountability and the right to understand decisions that affect you versus system integrity and community safety. How much opacity is justified to prevent manipulation?

Consent vs. Research Value
Researchers have access to a large medical dataset with incredible potential for discovering disease patterns and improving treatments. The data was collected with broad consent forms, but patients could not have anticipated these specific uses. Re-contacting thousands of patients for explicit consent is impractical and would make the research impossible.

The tension: Respect for informed consent versus beneficence and scientific progress. The original consent was legally valid but ethically ambiguous about future uses.

Whistleblowing Dilemma
A data scientist discovers their company’s algorithm is discriminating against protected groups in ways that violate policy, but leadership dismisses concerns. Speaking publicly could protect affected individuals but would breach confidentiality agreements, harm the company (and colleagues), and end the data scientist’s career.

The tension: Professional duty to the public versus loyalty to employer and colleagues, plus self-interest. Internal channels have failed—now what?

Accuracy vs. Interpretability
A deep learning model achieves 95% accuracy in diagnosing a serious disease, but doctors cannot understand its reasoning. A simpler, interpretable model achieves only 88% accuracy but doctors can verify its logic and catch errors.

The tension: Maximizing patient outcomes through accuracy versus maintaining human oversight and accountability. The black box model will save more lives statistically, but individual errors cannot be caught or explained.

Data Retention Dilemma
A platform collects user data that could exonerate innocent people if subpoenaed in criminal cases. However, retaining this data also makes it available to authoritarian governments, hackers, and future misuse. Deleting it protects privacy but might let injustice occur.

The tension: Potential benefit to some individuals versus systemic privacy risks to all users. No choice avoids all harm.

51.2 Resolving the Dilemma

Resolving an ethical dilemma is not an easy task—by definition there is no right\(\texttrademark\) answer, otherwise we would not face a dilemma. There are trade-offs and reasonable people can weigh points and principles differently. However, there are systematic approaches and frameworks that can help you work through the dilemma and find (approach) a solution. Systematically working through ethical reasoning is itself a skill.

In practice, being able to articulate and justify your reasoning is important, as is the understanding that first decisions are often revisited as consequences become clearer: iteration is not unusual.

Domain-Specific Considerations

Your subject matter domain might have established ethical principles that can help resolve a dilemma.

In healthcare the principle of “do no harm” and patient autonomy carry special weight. The model accuracy versus model interpretability dilemma might favor interpretability despite lower accuracy, because medical errors require explanation and learning.

In criminal justice due process and presumption of innocence are paramount. The transparency versus security case leans toward transparency because the right to understand accusations is fundamental.

In scientific research integrity and reproducibility matter greatly. The consent versus research value case might require genuine informed consent even if it is costly.

Check out the ethics guidelines of the profession—examples are at the beginning of this chapter.

Multi-Lens Analysis

Rather than relying on a single ethical theory, you can view the dilemma through different lenses and and see where they converge or conflict:

Consequentialist lens
What outcomes result from each choice? Who benefits and who is harmed? Can we quantify the magnitude of harms and benefits? This works well for the privacy versus public health dilemma—you might calculate lives saved versus privacy violations and their downstream consequences.

Deontological lens
What duties and rights are at stake? Are we treating people as ends in themselves or merely as means? The consent versus research value dilemma highlights this - even if research produces good outcomes, do we violate a fundamental duty to respect autonomy?

Virtue ethics lens
What would a person of good character do? What dispositions and habits does this choice cultivate? The whistleblowing dilemma benefits from this—it asks what kind of professional you want to be, beyond just calculating outcomes.

Care ethics lens
Who is vulnerable here? What do our relationships and responsibilities demand? This is particularly useful for cases involving power imbalances, like the fairness trade-offs where some groups bear disproportionate costs.

Stakeholder Analysis

Map out all affected parties and their interests:

Who has power in this situation?
Who is most vulnerable?
Whose voices are typically excluded from these decisions?
What are the short-term vs. long-term impacts on each group?

For the individual versus group fairness dilemma, this means explicitly considering: rejected applicants from different demographics, the company, successful applicants, society’s interest in equal opportunity, and future applicants who will be affected by precedent.

Practical Decision Framework

Here is a recommended step-by-step process.

Clarify the dilemma: Precisely identify which principles conflict and why both options seem problematic
Gather facts: What do we actually know and what is speculation? The fairness trade-off case requires understanding: Does zip code actually improve accuracy? By how much? Are there alternatives?
Identify stakeholders: Who is affected by the resolution of the dilemma and how?
Apply multiple ethical frameworks: Do not rely on just one lens.
Look for creative alternatives: Can the problem be reframed? In the privacy versus public health dilemma, can we use
technologies like differential privacy or federated learning that preserve privacy? The apparent dilemma might dissolve with technical innovation.
Consider harm reduction: If we cannot avoid all harm, how do we minimize it? In the transparency versus security case: Can we provide partial transparency (showing general methodology without implementation details)?
Make the decision defensible: Document reasoning, acknowledge trade-offs, specify conditions for revisiting
Build in accountability: How will we monitor impacts? What would trigger reconsideration?

Sometimes how you decide matters as much as what you decide:

Transparency and participation: Can affected stakeholders participate in the decision? For the data retention dilemma, should users be given a choice about whether their data is retained?

Reversibility: Can the decision be revisited as circumstances change or as we learn more? The accuracy versus interpretability choice might involve deploying the accurate model while investing in interpretability research.

Precedent consideration: What rule would you want universalized? If everyone in similar situations made this choice, what world would result?

51.3 Example: Accuracy versus Interpretability: An Ethical Analysis

Using the practical decision framework, let’s work through the accuracy versus interpretability dilemma. Here is the scenario: A deep learning model achieves 95% accuracy in diagnosing a serious disease, but doctors cannot understand its reasoning. A simpler, interpretable model achieves only 88% accuracy but doctors can verify its logic and catch errors.

Step 1: Clarify the dilemma

Conflicting principles:

Beneficence (maximize patient welfare) - The 95% model will correctly diagnose more patients, potentially saving lives through earlier or more accurate detection
Accountability and autonomy - Doctors need to understand diagnostic reasoning to exercise professional judgment, explain decisions to patients, catch errors, and maintain trust
Non-maleficence (do no harm) - Both models will make errors, but different types: the black box may make inexplicable errors; the interpretable model makes more errors but they are potentially catchable

Why both options are problematic:

Choosing the accurate model: We sacrifice oversight, explanation, and the ability to catch systematic errors. Patients cannot give truly informed consent if even doctors do no’t understand the diagnosis.
Choosing the interpretable model: We knowingly use an inferior tool, accepting that some patients will be misdiagnosed who would have been correctly diagnosed.

The core tension: This is fundamentally about whether to accept 7% more diagnostic errors in exchange for human understanding and oversight.

Step 2: Gather Facts

What we need to know:

About the models

What is the 7% difference in practical terms? If we are diagnosing 10,000 patients annually, that is 700 different outcomes.
What types of errors does each model make? Does the black box fail differently than the interpretable model?
On what populations were these models tested? Do accuracy rates hold across different demographics, disease stages, or presentations?
How confident are the models when they’re wrong? Do they flag uncertainty?

About the clinical context

What disease are we diagnosing? For cancer, 7% could mean hundreds of lives. For a less serious condition, the calculus changes.
What happens after diagnosis? Is there human review, confirmatory testing, or treatment that might catch errors downstream?
What are the consequences of false positives vs. false negatives for this condition?
How experienced are the doctors who will use this? Specialists might compensate differently than generalists.

About interpretability

How interpretable is the “interpretable” model really? Can doctors actually use the explanations meaningfully?
Has anyone tried to improve the black box’s interpretability? Are there techniques like SHAP values or counterfactual explanations that could help?
Can we quantify how often doctors catch errors in the interpretable model? Is the theoretical benefit of interpretability realized in practice?

Critical unknowns to investigate

Are there published studies on clinical decision-making with vs. without interpretable AI?
Have there been cases where black box medical AI made catastrophic errors that weren’t caught?
What do patients actually want when surveyed about this trade-off?

Step 3: Identify Stakeholders

Patients with the condition

Want accurate diagnosis (favors black box)
Want to understand their diagnosis and participate in decisions (favors interpretable)
Vulnerable to both false negatives (missed diagnosis) and false positives (unnecessary treatment)

Patients without the condition

Risk false positives from either model
Less directly affected but care about healthcare system trustworthiness

Doctors

Professional duty to understand their diagnostic tools
Legal liability for unexplainable errors
Need to maintain skills and judgment (might atrophy with black box reliance)
Want to provide best outcomes for patients

Healthcare institution

Legal and reputational risk from errors
Responsibility for systematic quality assurance
Financial pressures (misdiagnoses are costly)

Regulatory bodies

Ensuring medical device safety and efficacy
Setting precedents for AI in medicine

Future patients

Will be affected by the precedent we set
Benefit from our learning about AI deployment

The broader medical community

Building knowledge requires understanding why diagnoses succeed or fail
Setting norms for AI integration into medicine

Underrepresented groups

Often underrepresented in training data
More vulnerable to model failures
May benefit more from interpretability that reveals bias

Step 4: Apply Multiple Lenses

Consequentialist analysis

Pure outcome maximization would favor the 95% model; more correct diagnoses means better outcomes overall.

But we need to consider:

Second-order consequences: If doctors become deskilled or stop trusting the system after unexplainable errors, long-term outcomes worsen
Error distribution: If the black box’s 5% errors cluster in certain populations (e.g., rare presentations, minority groups), the aggregate numbers hide serious injustices
Systemic learning: Medical knowledge advances through understanding. Black boxes prevent learning from errors, potentially harming future patients
Trust erosion: One catastrophic, unexplainable error could undermine public trust in AI medicine broadly

Rule consequentialism asks: what rule, if generally followed, produces the best outcomes? A rule requiring interpretability in high-stakes medical decisions might produce better long-term outcomes even if short-term accuracy suffers.

Deontological analysis

Respect for persons: Patients have a right to understand medical decisions affecting them. Using a black box that even doctors cannot explain treats patients as mere recipients of care rather than autonomous decision-makers.

Professional duties: Doctors have a duty to exercise informed judgment. If they cannot understand the diagnostic reasoning, can they fulfill this duty? The Hippocratic tradition requires doctors to understand their craft.

Informed consent: True informed consent requires explaining the diagnosis and reasoning. “The computer says you have cancer but we do not know why” seems ethically problematic.

However, there is also a duty of care; using an inferior tool when a better one exists could violate the duty to provide best available care.

Virtue ethics

What kind of medical practice do we want to cultivate?

Wisdom and judgment: Relying on black boxes might erode the very virtues that make doctors trustworthy
Humility: The interpretable model acknowledges the limits of our understanding
Courage: Choosing the harder path of interpretability shows commitment to doing right rather than just doing well
But also: Stubbornly refusing better tools out of pride could itself be a vice

A virtuous doctor wants to understand and wants to help patients maximally. The dilemma is that these come into conflict.

Care ethics

Focuses on relationships, vulnerability, and context:

The doctor-patient relationship is built on trust and communication. Black box diagnosis strains this.
Patients are vulnerable; they depend on doctors to advocate for them. Can doctors advocate if they don’t understand?
Care ethics emphasizes particularity - seeing each patient as an individual. Does the black box allow for this, or does it impose a standardized, algorithmic approach?
But care also means attending to actual needs. If interpretability is a philosophical principle that patients do not actually care about, imposing it might be paternalistic.

Justice perspective

Distributive justice: How are the benefits and burdens of each choice distributed? If the black box’s errors cluster among already-disadvantaged groups, this amplifies injustice.
Procedural justice: Interpretability enables auditing for bias. Without it, we cannot ensure fair treatment across populations.
Epistemic justice: Patients and doctors have a right to medical knowledge. Black boxes create epistemic hierarchies where only AI developers understand diagnostic reasoning.

Step 5: Look for Creative Alternatives

Can we reframe the problem to escape the dilemma?

Hybrid approaches

Human-AI collaboration: Use the black box model as one input, but require doctors to make final decisions considering other factors. This preserves human judgment while leveraging AI accuracy.
Staged deployment: Use the interpretable model as primary tool, but flag borderline cases for the black box. Doctors review any disagreements. This captures much of the black box’s value while maintaining interpretability for most cases.
Confidence-based routing: When the interpretable model is highly confident, use it. When uncertain, consult the black box. This preserves interpretability where it matters most (clear-cut cases) while using black box for edge cases.

Technical innovations

Invest in interpretability research: Use the interpretable model now while aggressively pursuing explainable AI techniques. SHAP values, attention mechanisms, and other tools might bridge the gap.
Post-hoc interpretation: Deploy the black box but require it to generate explanations for each diagnosis. Even if not perfect, this provides some accountability.
Ensemble methods: Combine multiple interpretable models (stacking) to approach black box accuracy while maintaining transparency.

Process innovations

Error review boards: Use the black box but establish rigorous processes for investigating all errors to learn from them, even if individual predictions are not interpretable.
Patient choice: Offer patients the option - would you prefer the more accurate but unexplainable model, or the interpretable one? This respects autonomy.
Parallel deployment: Run both models, track outcomes over time, and let evidence guide the decision. This provides real-world data rather than theoretical accuracy rates.

Context-specific solutions

Risk stratification: Use interpretability requirements based on stakes. For screening tests with confirmatory follow-up, maybe black boxes are acceptable. For diagnoses that immediately determine treatment, require interpretability.

Step 6: Consider Harm Reduction

If we must choose, how do we minimize harm?

If choosing the black box (95% accuracy)

Mandatory human oversight: Require doctors to review all diagnoses and approve before treatment
Extensive testing on diverse populations: Ensure the 95% holds across demographics
Uncertainty flagging: Require the model to indicate confidence levels
Extensive documentation: Record all predictions and outcomes for eventual review
Clear escalation paths: Protocols for when doctors disagree with the model
Regular auditing: Check for systematic errors, bias, or drift
Patient notification: Inform patients that AI is involved and what this means
Liability clarity: Establish who is responsible when the AI errs
Continued research: Fund interpretability research with goal of eventually explaining the black box

If choosing the interpretable model (88% accuracy)

Continuous improvement: Actively work to close the accuracy gap
Selective black box use: Deploy the black box in cases where interpretable model is most uncertain
Second opinions: For serious diagnoses, require confirmation
Enhanced training: Help doctors maximize the value of interpretable explanations
Documentation: Record cases where interpretability helped catch errors to justify the choice
Patient communication: Explain that we prioritize transparent, accountable medicine even at some accuracy cost

Step 7: Make the Decision Defensible

Based on this analysis, here is a recommendation.

Recommendation

Start with a hybrid approach, weighted toward interpretability:

Deploy the interpretable model as the primary diagnostic tool
Use the black box as a “second opinion” that flags disagreements
When models disagree, require doctor review and judgment
Collect data on outcomes to validate this approach
Invest in interpretability research to close the accuracy gap

Justification

Why not pure black box

The medical context strongly privileges accountability, understanding, and the doctor-patient relationship
We cannot verify that 95% accuracy holds across all populations without understanding the model
Setting a precedent that accuracy trumps interpretability in medicine could be disastrous long-term
The 7% accuracy gain isn’t so large that it obviously overrides other concerns
Black box errors that we can’t explain or learn from could erode trust catastrophically

Why not pure interpretable model

7% more errors represents real harm to real patients
We have an obligation to use the best available tools if we can do so responsibly
Completely ignoring the black box’s insights seems wasteful

Why hybrid

Captures most of the black box’s value while maintaining human oversight
Preserves interpretability as the default while leveraging black box for difficult cases
Allows us to collect evidence about whether the theoretical accuracy advantage translates to better real-world outcomes
Respects both the duty to understand and the duty to provide best care
Provides a path forward: if the hybrid approach works well, we can adjust; if interpretability proves crucial, we have that; if accuracy is paramount, we learn that too

Key conditions that would change this recommendation are the following:

If the disease is immediately fatal without treatment and black box catches cases interpretable model misses, I would lean toward black box
If the black box’s 95% masks significant disparities across populations, I would stay with interpretable
If patients strongly prefer accuracy over explanation when surveyed, this shifts the calculus
If we are doing screening with confirmatory testing, the black box becomes more acceptable
If interpretability research makes the black box sufficiently explainable, the dilemma dissolves

Step 8: Build in Accountability

Monitoring and evaluation

Track outcomes for both models in real-world deployment
Disaggregated analysis by demographics, disease stage, and presentation type
Error analysis: catalog when and how each model fails
Doctor feedback: systematically collect physician experiences
Patient satisfaction: survey patients about their experience with the diagnostic process
Time studies: does the hybrid approach create unacceptable delays?

Governance structure

Ethics review board with patient representatives, doctors, AI experts, and ethicists
Quarterly reviews of outcomes data
Transparent reporting of errors and lessons learned
Clear escalation procedures for concerning patterns
Sunset clauses: re-evaluate the hybrid approach after 12-18 months based on evidence

Triggers for reconsideration

If black box catches >10% more cases than interpretable model in practice (not just in testing)
If we identify systematic bias in either model
If doctor-AI collaboration proves unworkable in practice
If interpretability research makes significant breakthroughs
If patient surveys reveal strong preferences we didn’t anticipate
If a serious error occurs that interpretability would have prevented (or vice versa)

Documentation requirements

Full methodological transparency about both models
Public registry of all diagnoses and outcomes (de-identified)
Open publication of findings
Accessible explanation of the approach for patients

Reflection on the Process

What this framework revealed

The dilemma is genuine—there is no clearly correct answer, and reasonable people can disagree
Context is crucial—the recommendation would change dramatically for different diseases, healthcare settings, or patient populations
False dichotomy—creative alternatives can partially dissolve the dilemma, even if tension remains
Values matter—different ethical frameworks point in different directions, revealing that the choice reflects priorities about what medicine should be
Empirical questions—many of the crucial considerations are empirical (Do doctors actually use interpretability? How accurate is the black box across populations?) and we should gather evidence rather than just philosophize or guess
Process has value—even without perfect resolution, working through this systematically identifies key considerations, stakeholders, and risks

What did we learn?

Ethical reasoning is systematic, not just intuition
Documentation and justification matter as much as the decision itself
Real-world ethical practice involves compromise, monitoring, and adjustment
Different contexts privilege different principles
Creative problem-solving can reduce (if not eliminate) ethical tensions

51.4 When Dilemmas Remain Unresolved

Sometimes you cannot eliminate the tension; you have to make a difficult choice and own it.

Acknowledge trade-offs explicitly and do not pretend the decision is cost-free. If you choose accuracy over interpretability, admit that you are accepting reduced accountability.

Compensate those disadvantaged by the resolution when possible. If the lending algorithm creates disparate impact, can the company invest in affected communities or provide alternative paths to credit?

Build in requirements to revisit decisions as technology, norms, or circumstances change. Be prepared to review and sunset clauses.

Institutional safeguards such as ethics review boards, external audits,
whistleblower protections, and diverse decision-making teams can catch blind spots.