Entities
View all entities
Alleged: Apple developed an AI system deployed by Goldman-Sachs, which harmed Apple Card female users
and
Apple Card female credit applicants.
Incident Stats
Incident ID
92
Report Count
6
Incident Date
2019-11-11
Editors
Sean McGregor, Khoa Lam
Applied Taxonomies
CSETv0, CSETv1_Annotator-1, CSETv1_Annotator-2, CSETv1, GMF
CSETv0 Taxonomy Classifications
Taxonomy Details
Problem Nature
Indicates which, if any, of the following types of AI failure describe the incident: "Specification," i.e. the system's behavior did not align with the true intentions of its designer, operator, etc; "Robustness," i.e. the system operated unsafely because of features or changes in its environment, or in the inputs the system received; "Assurance," i.e. the system could not be adequately monitored or controlled during operation.
Specification
Physical System
Where relevant, indicates whether the AI system(s) was embedded into or tightly associated with specific types of hardware.
Software only
Level of Autonomy
The degree to which the AI system(s) functions independently from human intervention. "High" means there is no human involved in the system action execution; "Medium" means the system generates a decision and a human oversees the resulting action; "low" means the system generates decision-support output and a human makes a decision and executes an action.
High
Nature of End User
"Expert" if users with special training or technical expertise were the ones meant to benefit from the AI system(s)’ operation; "Amateur" if the AI systems were primarily meant to benefit the general public or untrained users.
Amateur
Public Sector Deployment
"Yes" if the AI system(s) involved in the accident were being used by the public sector or for the administration of public goods (for example, public transportation). "No" if the system(s) were being used in the private sector or for commercial purposes (for example, a ride-sharing company), on the other.
No
Data Inputs
A brief description of the data that the AI system(s) used or were trained on.
credit score, credit report, reported income
CSETv1 Taxonomy Classifications
Taxonomy Details
Incident Number
The number of the incident in the AI Incident Database.
92
AI Tangible Harm Level Notes
Notes about the AI tangible harm level assessment
There was a gender bias in the rates and credit limits offered by the Apple card. This results in financial harm based on gender.
Special Interest Intangible Harm
An assessment of whether a special interest intangible harm occurred. This assessment does not consider the context of the intangible harm, if an AI was involved, or if there is characterizable class or subgroup of harmed entities. It is also not assessing if an intangible harm occurred. It is only asking if a special interest intangible harm occurred.
yes
Date of Incident Year
The year in which the incident occurred. If there are multiple harms or occurrences of the incident, list the earliest. If a precise date is unavailable, but the available sources provide a basis for estimating the year, estimate. Otherwise, leave blank.Enter in the format of YYYY
2019
Date of Incident Month
The month in which the incident occurred. If there are multiple harms or occurrences of the incident, list the earliest. If a precise date is unavailable, but the available sources provide a basis for estimating the month, estimate. Otherwise, leave blank.Enter in the format of MM
11
Estimated Date
“Yes” if the data was estimated. “No” otherwise.
No
CSETv1_Annotator-1 Taxonomy Classifications
Taxonomy Details
Incident Number
The number of the incident in the AI Incident Database.
92
AI Tangible Harm Level Notes
Notes about the AI tangible harm level assessment
3.2 - Goldman Sachs, who developed the card, never explicitly said whether the algorithm was AI. General media consensus is that machine learning was very likely involved.
Notes (special interest intangible harm)
Input any notes that may help explain your answers.
Women with similar financial backgrounds, credit scores, and other personal details as male counterparts were assigned much lower credit limits.
Special Interest Intangible Harm
An assessment of whether a special interest intangible harm occurred. This assessment does not consider the context of the intangible harm, if an AI was involved, or if there is characterizable class or subgroup of harmed entities. It is also not assessing if an intangible harm occurred. It is only asking if a special interest intangible harm occurred.
yes
CSETv1_Annotator-2 Taxonomy Classifications
Taxonomy Details
Incident Number
The number of the incident in the AI Incident Database.
92
AI Tangible Harm Level Notes
Notes about the AI tangible harm level assessment
There was a gender bias in the rates and credit limits offered by the Apple card. This results in financial harm based on gender.
Special Interest Intangible Harm
An assessment of whether a special interest intangible harm occurred. This assessment does not consider the context of the intangible harm, if an AI was involved, or if there is characterizable class or subgroup of harmed entities. It is also not assessing if an intangible harm occurred. It is only asking if a special interest intangible harm occurred.
yes
Date of Incident Year
The year in which the incident occurred. If there are multiple harms or occurrences of the incident, list the earliest. If a precise date is unavailable, but the available sources provide a basis for estimating the year, estimate. Otherwise, leave blank.Enter in the format of YYYY
2019
Estimated Date
“Yes” if the data was estimated. “No” otherwise.
No
Multiple AI Interaction
“Yes” if two or more independently operating AI systems were involved. “No” otherwise.
no
Reports Timeline
How the law got it wrong with Apple Card
techcrunch.com
- View the original report at its source
- View the report at the Internet Archive
What started with a viral Twitter thread metastasized into a regulatory investigation of Goldman Sachs’ credit card practices after a prominent software developer called attention to differences in Apple Card credit lines for male and femal…
- View the original report at its source
- View the report at the Internet Archive
The algorithm responsible for credit decisions for the Apple Card is giving females lower credit limits than equally qualified males. Those are the allegations that began spreading as consumers took to social media with complaints about App…
- View the original report at its source
- View the report at the Internet Archive
When tech entrepreneur David Heinmeier Hansson recently took to Twitter saying the Apple Card gave him a credit limit that was 20 times higher than his wife's, despite the fact that she had a higher credit score, it may have been the first …
- View the original report at its source
- View the report at the Internet Archive
US regulators are investigating whether Apple’s credit card, launched in August, is biased against women. Software engineer David Heinemeier Hansson reported on social media that Apple had offered him a spending limit 20 times higher than h…
- View the original report at its source
- View the report at the Internet Archive
The possibility that Apple Card applicants were subject to gender bias opens a new frontier for the financial services sector in which regulators are largely absent, argues Karen Mills.
In late August, the Apple Card debuted with a minimali…
- View the original report at its source
- View the report at the Internet Archive
Advocates of algorithmic justice have begun to see their proverbial “days in court” with legal investigations of enterprises like UHG and Apple Card. The Apple Card case is a strong example of how current anti-discrimination laws fall short…
A "variant" is an incident that shares the same causative factors, produces similar harms, and involves the same intelligent systems as a known AI incident. Rather than index variants as entirely separate incidents, we list variations of incidents under the first similar incident submitted to the database. Unlike other submission types to the incident database, variants are not required to have reporting in evidence external to the Incident Database. Learn more from the research paper.
By textual similarity
Did our AI mess up? Flag the unrelated incidents
·
5 reports
·
2 reports
·
3 reports
Previous IncidentNext Incident
