Ethics and Governance in Data Science
Data science operates on people's data, affects people's lives, and encodes human decisions into automated systems. Ethics in data science is not an afterthought or a compliance checkbox -- it is a design requirement. Ruha Benjamin's concept of the "New Jim Code" names the reality that automated systems can reproduce and amplify existing social inequalities while appearing objective. This skill covers the principles, frameworks, and practices that make ethical data science concrete and actionable.
Agent affinity: benjamin (bias audit, fairness analysis, ethical review), nightingale (routing ethics queries), cairo (communicating ethical findings)
Concept IDs: data-privacy-consent, data-algorithmic-bias, data-data-ownership, data-responsible-practice
Sources of Bias
Where Bias Enters the Pipeline
Bias can enter at every stage of the data science workflow. It is not a property of algorithms alone -- it is a property of the system: data, design decisions, deployment context, and feedback loops.
| Stage | Bias type | Example |
|---|---|---|
| Problem formulation | Framing bias | Defining "success" as engagement maximizes addictive behavior |
| Data collection | Selection bias | Training a facial recognition system on predominantly light-skinned faces |
| Data labeling | Annotation bias | Labelers' cultural assumptions influence what counts as "toxic" speech |
| Feature engineering | Proxy bias | ZIP code encodes race due to residential segregation |
| Model training | Optimization bias | Minimizing overall error ignores disparate performance across subgroups |
| Evaluation | Metric bias | Reporting aggregate accuracy hides poor performance on minority groups |
| Deployment | Automation bias | Decision-makers defer to model output without scrutiny |
| Feedback loops | Amplification bias | Predictive policing increases patrols in targeted areas, generating more arrests, confirming the model |
Historical Bias vs. Representation Bias
- Historical bias: The world is unequal, and data reflects that inequality. A hiring model trained on historical decisions inherits past discrimination. Even a "perfect" model of biased reality produces biased outputs.
- Representation bias: The training data does not represent the deployment population. A speech recognition system trained on American English performs poorly on other dialects. This is not a bug in the algorithm -- it is a gap in the data.
Both are real. Neither is solved by "better algorithms" alone. The fix requires changes to data collection, problem formulation, and deployment monitoring.
Fairness Metrics
Impossibility Theorem
Chouldechova (2017) and Kleinberg, Mullainathan, and Raghavan (2016) independently proved that three natural fairness criteria cannot all be satisfied simultaneously when base rates differ between groups:
- Calibration: Among those predicted positive, the fraction truly positive is the same across groups.
- Equal false positive rate: The rate of incorrectly predicting positive is the same across groups.
- Equal false negative rate: The rate of incorrectly predicting negative is the same across groups.
When the base rate (actual positive rate) differs between groups, satisfying any two of these requires violating the third. This is not a technical limitation to be solved -- it is a value judgment about which type of error matters more. The choice must be made explicitly, not hidden inside a loss function.
Common Fairness Definitions
| Metric | Definition | When appropriate |
|---|---|---|
| Demographic parity | P(positive prediction) is equal across groups | When the prediction itself causes differential treatment |
| Equalized odds | TPR and FPR are equal across groups | When false positives and false negatives have different costs |
| Equal opportunity | TPR is equal across groups (weaker than equalized odds) | When false negatives are the primary concern (e.g., loan approval for qualified applicants) |
| Predictive parity | Precision is equal across groups | When the model's positive predictions trigger consequential actions |
| Individual fairness | Similar individuals receive similar predictions | When you can define a meaningful similarity metric |
| Counterfactual fairness | Prediction would be the same in a counterfactual world where the individual belonged to a different group | When causal reasoning is possible and the causal model is trusted |
Measuring Disparate Impact
The four-fifths rule (EEOC, 1978): if the selection rate for a protected group is less than 80% of the rate for the most-selected group, there is evidence of adverse impact.
Disparate impact ratio = (selection rate for protected group) / (selection rate for most-selected group)
If this ratio < 0.8, investigate. This is a screening heuristic, not a legal standard -- but it is widely used as a first check.
Privacy
Privacy Principles (GDPR Framework)
| Principle | Meaning | Practical implication |
|---|---|---|
| Lawfulness | Legal basis for processing | Document the legal basis (consent, legitimate interest, contract, etc.) |
| Purpose limitation | Collect for specified purposes only | Do not repurpose data without new consent or legal basis |
| Data minimization | Collect only what is necessary | Every field in the dataset should have a documented purpose |
| Accuracy | Keep data correct and current | Provide mechanisms for correction; audit data quality |
| Storage limitation | Do not keep data longer than needed | Define retention periods; delete when expired |
| Integrity and confidentiality | Protect against unauthorized access | Encryption, access controls, audit logs |
| Accountability | Demonstrate compliance | Documentation, impact assessments, designated roles |
Anonymization Techniques
| Technique | How it works | Limitation |
|---|---|---|
| Suppression | Remove identifying fields | Remaining fields may be quasi-identifiers |
| Generalization | Replace specific values with ranges (age 34 -> 30-39) | Reduces data utility |
| k-Anonymity | Every record is indistinguishable from at least k-1 others on quasi-identifiers | Vulnerable to homogeneity and background knowledge attacks |
| l-Diversity | Each equivalence class has at least l distinct sensitive values | Better than k-anonymity but still vulnerable |
| t-Closeness | Distribution of sensitive attribute in each class is close to overall distribution | Strong but complex to implement |
| Differential privacy | Add calibrated noise so no individual's inclusion changes the output significantly | Mathematical guarantee; degrades with composition |
Differential Privacy
Differential privacy (Dwork, 2006) provides a mathematical guarantee: the output of an analysis is approximately the same whether any individual is in the dataset or not.
The privacy budget epsilon controls the tradeoff: smaller epsilon = stronger privacy = more noise = less accuracy. Epsilon is spent with each query, and it does not regenerate -- this is the composition theorem.
Practical deployment: Apple (emoji usage), Google (Chrome usage), US Census (2020). Each chose an epsilon value that balanced utility and privacy for their specific context. There is no universally "correct" epsilon.
Consent
Informed Consent Requirements
- Purpose: What the data will be used for, in plain language.
- Scope: What data is collected and how long it is retained.
- Rights: How to access, correct, or delete data.
- Risks: Potential consequences of participation.
- Voluntariness: Participation is optional; no penalty for declining.
- Third parties: Whether data is shared and with whom.
Consent Models
| Model | Description | Strength | Weakness |
|---|---|---|---|
| Opt-in | User actively agrees before dat |