Mitigating Bias in Algorithmic Systems—A Fish-eye View

An inventory of the initial repository was taken, to understand which domains (i.e., communities) had produced a critical mass of publications related to the mitigation of algorithmic biases. We focused on well-established domains within the information and computer sciences, which are investigating data and knowledge transformation and communication to the user. Based on the initial inventory, four domains emerged—machine learning (ML), human-computer interaction (HCI), recommender systems (RecSys), and information retrieval (IR)—to characterize the problems that are being addressed, as well as the solutions being proposed, across domains. Next, we provide additional justification of these four domains.

The widespread application of ML techniques, which in many cases are opaque, led to the issue of potential bias and discrimination of algorithmic systems and processes. Hence, the ML community and ML-related publications naturally emerged as an established area we needed to review. RecSys represents a specific application area and a domain that attracts significant research attention on algorithmic bias. Within RecSys, ML techniques are applied for reasoning on and exploiting user characteristics; thus, within this domain, many challenges have arisen surrounding potential bias and fairness. IR focuses on information delivery to users, often with the use of search and ranking algorithms that are opaque; thus, bias and fairness have long been researched. The above domains cover a substantial amount of applications where the risk of bias and discrimination in the reasoning process exists. Finally, HCI directly considers the end users and their perceptions when interacting directly or indirectly with different applications. In particular, understanding the potential bias, discrimination or fairness issues that might emerge when a user is interacting with information presented through an interface is considered of high importance. It should be noted that an “Other” domain emerged, through the initial repository, where we collected a number of articles published in emerging, cross-disciplinary communities or domains that are not represented in the above main categories. Through “Other” we were able to capture research published in other domains where a mass of publications related to bias, fairness and explainability did not (yet) exist, but important work was published, hence, making this a comprehensive review with applicability in areas other than the main domains that emerged.

Through the exercise of selecting the domains, a list of high-impact publication venues was created for each domain, as presented in Table 1. Also, note that some venues publish articles across domains. For instance, while ACM CSCW is generally aligned with the HCI community, some articles describing studies of recommender systems can be found there. Such cases are indicated with parentheses in Table 1.

The next step was to review each publication venue’s proceedings/published volumes during the 12-year period of 2008 through 2021, resulting in a high-recall search for relevant published articles. The key words used were: “accountability,” “bias,” “discrimination,” “fair(ness),” “explain(able),” and “transparen(cy).” In addition, the articles collected address a particular algorithmic process or system, or a class of system. In other words, articles of a more abstract or philosophical nature were excluded from the survey. Likewise, in the ML category, articles from AI venues (e.g., AAAI, IJCAI) that were not published in the respective ML track, have been excluded, as to focus on algorithmic, data-driven systems.

To explore the problem space described in the literature, we characterized, for each article, the system component(s) deemed to be problematic, as well as the attribute(s) whose values are affected by the bias.

The solution space discussed in the literature consists of three main steps in mitigating algorithmic bias, with each involving different stakeholders within each community. Next, we describe the stakeholders involved in the solution processes. Afterwards, we give a detailed overview of each of the three steps in bias mitigation.

Before describing each set of techniques in detail, we provide a summary of the problems and solutions documented within the four domains surveyed. Table 3 presents Bias Detection solutions, while Tables 4 and 5 present solutions for Fairness and Explainability Management. The distribution of problems addressed across the four domains illustrates the insights gained from our “fish-eye view.” As expected, the ML literature addresses problems concerning the training data, the algorithmic model and the system output. The RecSys and IR literature, as user-focused application areas, consider problems both inside and outside the system, while HCI naturally addresses the interactions between the user and the algorithmic system.

Similarly, we find that across domains, researchers are engaged in all three steps in bias mitigation—detection, fairness and explainability management. In the following sections, we provide a detailed overview of example approaches of the three steps, from across domains, and shall compare the techniques used.

The most common auditing approach used for bias detection involves humans (external testers, researchers, journalists or the end users) acting as the auditors of the system. In IR systems, researchers usually perform an audit by submitting queries to search engine(s) and analyzing the results. For instance, Vincent et al. [204] performed an audit on Google result pages, where six types of important queries (e.g., trending, expensive advertising) were analyzed. The goal was to examine the importance of user-generated content (UGC) on the Web, in terms of the quality of information that the search engines provide to users (i.e., if there was a bias in favor/penalizing such content). Similarly, Kay et al. [105], Magno et al. [132], and Otterbacher et al. [147] submit queries to image search engines to study the perpetuation of gender stereotypes, while Metaxa et al. [139] consider the impact of gender and racial representation in image search results for common occupations. They compare gender and racial composition of occupations to that reflected in image search and find evidence of deviations on both dimensions. They also compare the gender representation data with that collected earlier by Kay et al. [105], finding little improvement over time.

Another example of bias detection in a search engine via auditing is the work of Kilman-Silver et al. [112] who examine the influence of geolocation on Web search (Google) personalization. They collect and analyze Google results for 240 queries over 30 days from 59 different GPS coordinates, looking for systematic differences. In addition, Robertson et al. [169] audited Google search engine result pages (SERPs) collected by study participants for evidence of filter bubble effects. Participants in the study completed a questionnaire on their political leaning and used a browser extension allowing the researchers to collect their SERPs.

Kulshrestha et al. [119] propose an auditing technique where queries are submitted on Twitter, to measure bias on Twitter results as compared to search engines. The proposed technique considers both the input and output bias. Input bias allows the researchers to understand what a user would see if shown a set of random items relevant to her query. The output bias isolates the bias of the ranking mechanism. In addition, Johnson et al. [101] investigate the demographic bias detection in Twitter results using as an auditing technique, the retrieval of geotagged content using Twitter API. Another example where researchers are the auditors is the study of Edelman et al. [57] where the authors run an experiment to audit Airbnb applications to detect racial bias in ranked results, and more specifically, for African American names.

Another cluster of user-based studies in IR systems concerned the detection of perceived biases about search and/or during a search for information. In these studies, users are the auditors. For instance, Kodama et al. [113] assessed young people’s mental models of the Google search engine, through a drawing task. Many informants anthropomorphized Google, and a few focused on inferring its internal workings. The authors called for a better understanding of young people’s conceptions of search tools, to better design information literacy interventions and programs. In addition, Otterbacher et al. [148] described a study in which participants were the auditors for detecting perceived bias. They were shown image search results for queries on personal traits (e.g., “sensitive person,” “intelligent person”) and were asked to evaluate the results on a number of aspects, including the extent to which they were “biased.”

Auditing approaches using ML have also been widely used. A situational testing auditing approach has been proposed by Luong et al. [130], to detect discrimination against a specific group of individuals, using an ML algorithm. K-nearest neighbors was combined with the situation testing approach to identify a group of tuples with similar characteristics to a target individual. Zhang et al. [227] proposed an improvement over the method of Luong et al. [130], by engaging Causal Bayesian networks (CBNs), which are probabilistic graphical models used for reasoning and inference. For the development of a CBN, the causal structure of the dataset and the causal effect of each attribute on the decision are used to guide the identification of the similar tuples to a target individual. Robertson et al. [170], present an auditing approach in the form of an opaque algorithm, called “recursive algorithm interrogation” used for detecting bias in search engines. The auto-complete functions of Google and Bing are treated as opaque algorithms. They recursively submitted queries, and their resulting child queries, to create a network of the algorithm’s suggestions.

Hu et al. [92] audited Google SERPs snippets, for evidence of partisanship where the generation of snippets is an opaque process. Moreover, Le et al. [122] audit Google News Search for evidence of reinforcing a user’s presumed partisanship. Using a sock-puppet technique, the browser first visited a political Web page, and then continued on to conduct a Google news search. The results of the audit suggested significant reinforcement of inferred partisanship via personalization. In addition, Eslami et al. [62] use a cross-platform audit technique that analyzed online ratings of hotels across three platforms, to understand how users perceived and managed biases in reviews.

In the HCI literature, auditing often involves characterizing the behavior of the algorithm from a user perspective. For instance, in Matsangidou and Otterbacher [135], the authors consider the inferences on physical attractiveness made by image tagging algorithms on images of people. They audited the output of four image recognition APIs on standardized portraits of people across genders and races. In a more recent work [12], the authors use auditing to understand machine behaviors in proprietary image tagging algorithms. The authors created a highly controlled dataset of people images, imposed on gender-stereotyped backgrounds. Evaluating five proprietary algorithms, they find that in three, gender inference is hindered when a background is introduced. Of the two that “see” both backgrounds and gender, it is the one with output that is most consistent with human stereotyping processes that is superior in recognizing gender. Another example is the work of Eslami et al. [63], where the authors describe a qualitative study of online discussions about Yelp on the algorithm existence and opacity. The authors further enhanced the results by conducting 15 interviews with Yelp users who acted as auditors of the system, in an attempt to understand how the reviews filtering algorithm works.

Auditing approaches have also been used to detect bias in ML classification systems. For instance, in Reference [24], the authors (developers) audit three automated facial analysis algorithms to detect any gender inequalities in the classification results. They found that males were classified more accurately than females in all the three algorithms and that all the algorithms performed worst on darker female subjects.

Recently, automated methods for auditing have been introduced to detect gender or race bias in the context of online housing advertisements and search engine rankings. Asplund et al. [6] propose the use of controlled “Sock-puppet” auditing techniques, which are automated systems that mimic user behavior in offline audits. They use these techniques to investigate gender-based and race-based discrimination in the context of online housing advertisements and any bias in search-result ranking. The authors use the definition of disparate impact to consider both application systems as fair or not.

A common approach for discrimination discovery is to compute metrics to detect any direct/indirect discrimination of specific groups in the data. Examples include absolute measures, conditional measures or statistical tests [233]. Absolute measures define the magnitude of discrimination over a dataset by taking into account the protected characteristics and the predicted outcome. Statistical tests, rather than measuring the magnitude of discrimination, indicate its presence/absence at a dataset level. Conditional measures compute the magnitude of discrimination that cannot be explained by any non-protected characteristics of individuals. Fairness notions have also been used in many works as metrics for discrimination.

In Bellogin et al. [15], the authors detect statistical biases in the evaluation metrics used in recommender systems that affect the effectiveness of the recommendations. They found out that there is sparsity and popularity bias on the evaluation metrics. Many works focus on investigating the racial bias in advertising recommendations systems. For instance, Sweeney [192] investigates the racial bias in advertising recommendations by an ad server when searching for particular names in Google and Reuters search engines. She finds that ads for services providing criminal records on names were significantly more likely to be served if the name search was on a typically Black first name. Ali et al. [2], Speicher et al. [188] and Imana et al. [94] detected significantly skewed ad delivery on racial lines in Facebook ads for employment, financial services and housing. More specifically, in Reference [94], the authors first build an audience that allows them to infer the gender of the ad recipients on the platforms that do not provide ad delivery statistics along gender lines, i.e., Facebook, Linkedin. They use this audience to distinguish between skew in ad delivery due to protected categories from the skew due to differences in qualifications among people in the targeted audience. Indirectly, they measure the “equality of opportunity” fairness notion. Another example of bias detection in RecSys and online social networks is the work of Chackraborty et al. [33] who detect demographic bias in the input data of crowds in Twitter who make posts worthy of being recommended as trending. The bias is detected by comparing the characteristics of the trend promoters with the demographics of the general population of Twitter. Apart from demographic bias, political bias is very common in social networks. In Jiang et al. [98], the authors measure the fairness in social media contexts based on the fairness notions: demographic parity and the equalized odds. The authors detect political bias through content moderation. Bias in the social platform Facebook has also been assessed through reverse engineering of the Facebook API ranking algorithm using logistic regression in Reference [89]. More specifically, the authors identify the features of a post that would affect its odds of being selected. Sentiment analysis reveals that there are significant differences in the sentiment word usage between the selected and non-selected posts.

In IR systems, discrimination discovery is primarily used in user-focused studies. Weber and Castillo [208] conducted a study of user search habits, which involved a large-scale analysis of Web logs from Yahoo!. Using the logs, as well as users’ profile information and US-census information (e.g., average income within a given zip code), the authors were able to characterize the typical behaviors of various segments of the population and detect any discrimination related to the users’ sensitive demographic attributes. In a similar manner, Yom-Tov [221] used search query logs to characterize the differences in the way that users of different ages, genders, and income brackets, formulate health-related queries. His driving concern was the ability to discover users with similar profiles, according to their demographic information (user cohorts), who are looking for the same information, e.g., a health condition. Pal et al. [150] considered the identification of experts in the context of a question-answering community. Their analysis revealed that as compared to other users with less expertise, experts exhibited significant selection biases in their engagement with content. They proposed to exploit this bias in a probabilistic model, to identify both current and potential experts. A method to identify selection bias, IMITATE, has also been proposed in Dost et al. [56]. IMITATE investigates the dataset’s probability density, then adds generated points to smooth out the density and have it resemble a Gaussian, the most common density occurring in real-world applications.

In a study of information exposure on the Mendeley platform for sharing academic research, Thelwall and Maflahi [196] illustrated a home-country bias. Articles were significantly more likely to be read by users in the home country of the authors, as compared to users located in other countries. Chen et al. [35] investigated direct and indirect (implicit) gender-based discrimination in the context of resume search engines, by a system toward its users. Direct discrimination happens when the system explicitly uses the inferred gender or other attributes to rank candidates, while indirect discrimination is when the system unintentionally discriminates against users (indirectly via sensitive attributes). The results suggested that the system under review indirectly discriminates against females, however, it does not implicitly use gender as a parameter. Another method for detecting bias in search engine results involves the use of metrics that quantify bias in search engines [142]. A series of articles by Wikie et al. [211, 212, 213] and a paper of Bashir and Rauber [13] investigates the identification retrieval bias in IR systems. Bashir and Rauber study the relationship between query characteristics and document retrievability using the TREC Chemical Retrieval track. In Wilkie and Azzopardi [212], they examined the issue of fairness vs. performance. Wilkie and Azzopardi [211] consider specific measures of retrieval bias and the correlation to the system performance. Wilkie and Azzopardi [213] consider the issue of bias resulting from the process of pooling in the creation of test sets. A recent study [178] detects gender and race bias in the annotation process of training data of image databases used for facial analysis. The authors found that the majority of image databases rarely contain underlying source material for how those identities are defined. Further, when they are annotated with race and gender information, database authors rarely describe the process of annotation. Instead, classifications of race and gender are portrayed as insignificant, indisputable, and apolitical.

A set of works in HCI analyzes crowdsourced data from the OpenStreetMap to detect potential biases such as gender and geographic information bias [48, 161]. In a similar vein, two other studies run a crowdsourcing study to detect any bias on human versus algorithmic decision-making [11, 78]. Green and Chen [78] run a crowdsourcing study to examine the influence of algorithmic risk assessment to human decision-making, while Barlas et al. [11] compared human and algorithmic generated descriptions of people images in a crowdsourcing study in an attempt to identify what is perceived as fair when describing the depicted person. The execution of a crowdsourcing study for detecting bias has also been used in IR systems [59, 127].

Many works study the problem of bias detection in textual data using data mining methods concerning specific protected groups. The typical approach is to extract association or classification rules from the data and to evaluate these rules according to discrimination of protected groups [157, 171]. For instance, Datta et al. [49] analyse the gender discrimination in online advertising (Google ads) using ML techniques, to identify the gender-based ad serving patterns. Specifically, they train a classifier to learn differences in the served ads and to predict the corresponding gender. Similarly, Leavy et al. [123] detect gender bias in training textual data by identifying linguistic features that are gender-discriminative, according to gender theory and sociolinguistics. Zhao et al. [229] detect gender bias in coreference resolution systems. They introduce a new benchmark dataset, WinoBias, which focuses on gender bias. They also use a data augmentation approach that in combination with existing word-embedding debiasing techniques, removes the gender bias demonstrated in the data. Madaan et al. [131] detect gender discrimination in movies using knowledge graphs and word embeddings after analyzing the data (using, for example, mentions of each gender in movies, emotions of the actors during the movies, occupation of each gender in the movies, screen time, etc.) In a similar vein, Ferrer et al. [66] propose a data-driven approach to discover and categorize language bias encoded on the vocabulary of online communities in the Reddit platform. They use word embeddings to discover the most biased words toward protected attributes, apply k-means clustering combined with a semantic analysis system to label the clusters, and use sentiment analysis to further specify biased words. Rekabsaz and his colleagues [164] also explore the detection of societal bias in word-embedding models by utilizing the first-order co-occurrence relations between the word and the representative concept words. Islam et al. [95] introduce a collaborative filtering method to detect gender bias in social media. Their proposed method is called Neural Fair Collaborative Filtering (NFCF). They also use debiasing embeddings, and fairness interventions via penalty term.

A cluster of works in IR addresses the detection of bias such as age-based bias, and text-frequency and stylistic biases in sentiment classification [51, 163, 184]. Other examples of detecting bias in classifiers that use sentiment analysis are the existence of offensive language or stereotyping of sensitive attributes in automated hate speech detection algorithms [7, 50] and the detection of cultural biases at Wikipedia pages using sentiment analysis [27]. Shandilya et al. [181] also detect the under-representation of sensitive attributes in the summarization algorithms. Keyes [107] identified the problem of automatic gender recognition in HCI research and how the approaches followed until recently are discriminatory toward transgendered people. For systems to be fair, Keyes [107] proposed alternative methods and the development of more inclusive approaches in the gender inference process and evaluation. Apart from automatic gender recognition, an additional significant advancement in the field of HCI is that of data-driven personas. Salminen et al. [175] investigated the presence of demographic bias in automatically generated, data-driven personas. They discovered that the more personas they generated, the more diverse the sample became in terms of gender and age representation. Practitioners who use data generated personas should consider the possibility of unintentional bias in the data they use, that consequently is transferred to the personas they generate.

Multiple approaches have been proposed in ML that detect discrimination in the data or classifier. Choi et al. [40] discover and mine discrimination patterns that reveal if an individual is classified differently when some sensitive attributes were observed. The algorithm detects discrimination patterns in a Naive Bayes classifier using branch and bound search and removes them. It learns maximum likelihood parameters based on these parameters. Pedreshi et al. [157] use an opaque predictive model to extract frequent classification rules based on an inductive approach. Background knowledge is used to identify the groups to be detected as potentially discriminated. However, Zhang et al. [226] use a causal Bayesian network and a learning structure algorithm to identify the causal factors for discrimination. The direct causal effect of the protected variable on the dependent variable represents the sensitivity of the dependent variable to changes in the discrimination grounds while all other variables are held fixed. They also detect discrimination in the prediction/classification outcome by computing the classification error rate (error bias). In a more recent work, Zucker et al. [234] introduce a new domain-specific programming language, the Arbiter for ML practitioners. It allows users to make guarantees about the level of bias in any produced models.

The notion of divergence [153], which estimates the difference in classification performance measures, has also been proposed as a metric to identify data subgroups in which a classifier performs differently. Pastor et al. [154] introduce the DivExplorer, an interactive visual analytics tool that identifies algorithmic bias using the divergence notion. An interactive system to detect fairness issues in the classifiers has also been proposed in Reference [125]. The system is called DENOUNCER and it allows users to explore fairness issues for a given test dataset, considering different fairness notions. In addition, Nargesian et al. [143] detect the groups in the dataset that are unfairly treated by the classifier by developing an exploration-exploitation-based strategy. Their approach captures the cost and approximations of group distributions in the given dataset.

In IR systems, a common type of bias is the cognitive or perception bias that arises from the manner in which information is presented to users, in combination with the user’s own cognition and/or perception. For example, Jansen and Resnick [96] analyzed the behaviors of 56 participants engaged in e-commerce search tasks, with the goal of understanding users’ perceptions of sponsored versus un-sponsored (organic) Web links. The links suggested by the search engine were manipulated to control content and quality. Even controlling for these factors, it was shown that users have a strong preference for organic Web links. In a similar vein, Bar-Ilan et al. [10] conducted a user experiment to examine the effect of position in a search engine results page (SERPs). Across a variety of queries and synthetic orderings of the results, they demonstrated a strong placement bias; a result’s placement, along with a small effect on its source, is the main determinant of perceived quality. User perception is also examined in a study [139] where the authors consider people’s impressions of occupations and sense of belonging in a given field when shown search results with different proportions of women and people of color. They find that both axes of representation as well as people’s own racial and gender identities impact their experience of image search results. Gezici et al. [73] propose a new evaluation framework to measure bias in the content of SERPs (on political and controversial search topics) by measuring stance and ideological bias. They propose three novel fairness-aware measures of bias based on common IR utility-based evaluation measures.

Ryen White, of Microsoft Research, has published extensively on detecting users’ perception bias during and after a search, particularly when trying to find information to answer health-related queries. In an initial work [209], a user study focused on finding yes-no answers to medical questions, showed that pre-search beliefs influence users’ search behaviors. For instance, those with strong beliefs pre-search, are less likely to explore the results page, thus reinforcing the above-mentioned positioning bias. A follow-up study by White and Horvitz [210] looked more specifically at users’ beliefs on the efficiency of medical treatments, and how these beliefs could be influenced by a Web search. An example of searching for user perception bias in recommender systems was presented in Reference [172], where drivers’ perceptions of the Uber application were investigated, taking into consideration drivers’ profiles and their history performance.

Table 7 in the Supplementary Materials compares the methods used for bias detection across the research domains surveyed.

Table 7 in the Supplementary Materials compares the the methods used for bias detection across the research domains surveyed.

An approach that is used to mitigate bias in the input or training data is the removal of sensitive attributes that may be involved in discrimination. However, in some cases, the inclusion of sensitive characteristics in the data may be beneficial to the design of a fair model [235]. To handle this issue, some approaches remove information about the protected variables from the training data but they also transform it using data mining methods. For instance, Kamiran and Calders [102] use a naive Bayes classifier to generate rankings of each observation in the training data based on its probability of belonging to the desirable class category. The outcome variable in the training data is adjusted until there is no remaining association between the protected variable and the intended outcome variable. The drawback of this solution is that it is limited to a binary outcome variable and the transformed training data cannot be used with other outcome variables.

Calders and Verwer [26] eliminate these drawbacks via three algorithms that transform (i.e., re-weight) the training data based on an objective function that is minimized when the outcomes from a model that fit to the transformed data are independent of the protected variable. This class is also restricted to binary outcomes and protected variables. Data transformation approaches have also been proposed by Johndrow and Lum [100] and Zemel [224]. Johndrow et al. [100] suggest a statistical framework where the training data are transformed by mapping individuals to an input space that is mutually independent of specific groupings. In Reference [224], the authors encode the data by mapping each individual, represented as a data point in an input space, to a probability distribution in a new representation space. The aim of this is to hide sensitive information that can identify if the individual belongs to a protected group. Percy et al. [158] propose an approach to mitigate gender bias on gambling. The method uses gender data for training only, constructing separate models for each gender and combining trained models into an ensemble that does not require gender data once deployed.

Another frequent pre-processing technique is the use of directed acyclic graphs and causal reasoning that capture the dependencies between the features and their impact on the outcome. For instance, Zhang et al. [226] discover and prevent discrimination bias in decision support systems using a causal Bayesian network (BN) to identify pairs of tuples with similar characteristics from the dataset. Then, they generate a new dataset sampled from the learned BN. Cardoso et al. [121] also use a Bayesian network estimated from real-world data to generate biased data that are learned from real-world data. A data transformation method has also been applied to ensure fairness in RecSys [215]. The authors propose a new graph-model technique, the FairGo model, which ensures fairness for every recommender system by transforming the original embedding of user and items into a filtered embedding space based on the sensitive feature set. FairGo is model-agnostic and can be applied to multiple sensitive attributes.

Rather than adjusting/transforming the observations of the training data, other works use re-labeling. Cardoso et al. [121] propose the use of an auditing tool to repair the dataset by changing attribute labels. Kamiran and Calders [102] massage the data by swapping some of the labels in such a way that a positive outcome for the disadvantaged group is more likely and then they re-train the model. Feldman et al. [65] proposed the disparate impact removal solution approach that manipulates individual data dimensions in a way that depends on the protected attribute.

Similar techniques to data transformation, but that consider the selection of features, have been introduced in Reference [32, 173]. Salazar et al. [173] use a multi-objective optimization algorithm for feature construction. They use this approach to generate more features that lead to both high accuracy and fairness by applying human understandable transformations. Celis et al. [32] develop a novel approach that takes as input a visibly diverse control set of images of people and uses this set as part of a procedure to select a set of images of people in response to a query. The goal is to have a resulting set that is more visibly diverse in a manner that emulates the diversity depicted in the control set.

Other popular fairness pre-processing methods are the re-sampling methods that generate a balanced dataset that will not under- or over-represent a particular protected group [51, 53, 76, 101, 184]. A similar approach is used in RecSys where a re-sampling method is used to balance the neighborhoods before producing recommendations [25] or re-balance the input data according to the protected attributes (e.g., gender) to produce a fair training dataset [130]. A re-sampling method has also been used by Sharma et al. [183]. They use a data augmentation technique that adds synthetic data for removing bias in the data. The technique selectively adds only a subset of the synthetic points to create new augmented dataset to meet the fairness criteria while maintaining accuracy.

One category of in-processing approaches involves the use of an optimization method. Xiao et al. [217] suggest a multi-objective optimization framework optimizing fairness and social welfare simultaneously on group recommendation. The goal was to maximize the satisfaction of each group member while minimizing the unfairness between them. Results show that considering fairness in group recommendation can enhance the recommendation accuracy. A multi-objective optimization approach has also been proposed in Reference [144] for fair allocations using two criteria, maximum fairness and efficiency. They propose a dynamic programming algorithm to construct an appropriate Pareto set. Optimization approaches with fairness weights have also been used in recommender systems for two-sided marketplaces [138]. In that scenario, the developed systems should be fair on both the demand and supplier sides. The authors propose different recommendation policies that jointly optimize the relevance of recommendations to consumer (i.e., user) and fairness of representation of suppliers. Kusner et al. [120] focus on satisfying the counterfactual fairness as the notion of fairness. They capture the social biases that may arise toward individuals based on sensitive attributes. They provide optimization of fairness and prediction accuracy of the classifier using a causal model.

A second category of in-processing methods is the use of regularization methods. Yan and Howe [219] introduce the FairST, a fairness-aware demand prediction model for spatiotemporal urban applications. Two spatio-temporal fairness metrics have been introduced as a form of regularization to reduce discrimination in demographic groups. Kamishima et al. [103], also use a regularization approach that can be applied to any algorithmic classifier. They introduce a prejudice remover regularizer that enforces independence from sensitive attributes. Instead of applying a regularization method, Rezaeil et al. [165] mitigate bias detected in a classifier by re-building it and incorporating fairness constraints to the predictor. The method reshapes the predictions (output) for each group to satisfy the fairness constraints that consider the protected groups.

In IR and RecSys, in-processing approaches primarily explore the mitigation of bias in the ranking algorithms using learn-to-rank methods. For instance, Dai and colleagues [47] propose a novel framework, Adversarial Imitation Click Model (AICM), which is based on imitation learning to address the exposure bias in click-models. Click-models rely on learning-to-rank, by studying how users interact with a ranked list of items. In Reference [149], the authors consider both the selection and position bias in the rank-based results. They frame the problem as a counterfactual problem and adapt Heckerman’s (rank) approach by combining it with position bias correction methods to correct both the selection and position bias. Yang and Ai [220] propose a fair and unbiased ranking method named Maximal Marginal Fairness (MMF) for dynamic learning to rank, to achieve both fairness and relevance in top-k results. In a recent work [117], the authors introduce a fair rank aggregation framework for aggregating multiple rankings in a database, which can be applied to the databases of the ranking systems. It uses pairwise discordance to both compute closeness among consensus and base rankings and measure the advantage given to each group of candidates. Another fairness issue in ranking systems concerns the mitigation of bias in the PageRank algorithm. The authors in Reference [199] provide a parity-based definition of fairness that imposes constraints on the proportion of PageRank allocated to the members of each group. They validate the fairness notion of local and personalized PageRank fairness.

The most well-known post-processing method used in ML literature is the re-labeling of the decision outcome. Pedreschi et al. [157] alter the confidence of classification rules inferred by the CPAR algorithm, whereas Kamiran et al. [102] re-label the class that is predicted at the labels of a decision tree. In Reference [84], the authors propose a new fairness definition to optimally adjust any learned predictor to remove discrimination. Their framework constructs classifiers from any Bayes optimal regressor following a post-processing step that minimizes the loss in utility. Additionally, in the literature of rank-based systems (i.e., IR and RecSys), post-processing methods focus on the re-ranking of the recommended or search results. In Reference [126], the authors provide a re-ranking approach to mitigate the unfairness problem between active and inactive users by adding constraints over evaluation metrics. Experiments show that their approach improves group fairness of users in recommender systems, and also achieves better overall recommendation performance.

A re-ranking method has also been proposed by Karako and Manggala [104] who introduce a fairness-aware variation of Maximal Marginal Relevance (MMR). The proposed method incorporates fairness in a recommender or search system by choosing a sample of labeled images, based on gender when retrieving untagged images similar to an input image or query. Mitigation of gender bias on image tagging has been proposed in Reference [195] where the authors introduce the Guided Attention Image Captioning model (GAIC). The GAIC pipeline encourages the model to provide correct gender with high confidence when evidence in the image is obvious. When gender evidence is vague or occluded, GAIC describes the person with neutral words, such as “person,” In addition, Zehlike et al. [223] and Singh and Joachims [186] propose fair top-k ranking algorithms for RecSys that makes the recommendations subject to group fairness criteria and constraints.

“Other” works in ranking systems propose methods to achieve the fairness of the general ranking results, rather than focusing on the top-k ranking. Patro and colleagues [155] propose the FairRec algorithm, which validates exhibiting the desired two-sided fairness, both consumer and producer fairness, by mapping the fair recommendation problem to a fair allocation problem. Kuhlman et al. [118] use an auditing methodology FARE (Fair Auditing based on Rank Error) for error-based fairness assessment of the ranking results. They propose three error-based fairness criteria, which are rank-appropriate, to assess the correctness of the rankings. In addition, Kirnap et al. [110] estimate four fair ranking metrics by acquiring group membership annotations for a sample of documents in its corpus.

Fairness certification methods are used to certify the fairness of the system using some constraints [31, 52, 111, 225] or by introducing new fairness metrics such as FACE and FACT [108], feature-a priori fairness, feature-accuracy fairness, and feature-disparity fairness [79]. Wu et al. [216] propose a framework that uses many of these fairness metrics as convex constraints that are directly incorporated into the classifier. They first present a constraint-free criterion (derived from the training data) that guarantees that any learned classifier will be fair according to the specified fairness metric. Thus, when the criterion is satisfied, there is no need to add any fairness constraint into the classifiers. When the criterion is not satisfied, a constrained optimization problem is used to learn fair classifiers.

Hu et al. [93] propose a metric-free individual fairness based on the gradient contextual bandit algorithm that aims to maximize fairness. In Reference [72], the authors use multi-objective clustering algorithms to maximize both accuracy and fairness and to introduce diversity and transparency as constraints. Counterfactual fairness is another well-known metric used for certifying system fairness. In Reference [182], counterfactual explanations evaluate fairness with respect to a particular individual as well as the fairness of the model toward groups of individuals. They define the metric “burden” to evaluate group fairness. The burden is computed taking into consideration how much the fitness measure differs for counterfactuals generated for specific groups of individuals. Cruz Cortes et al. [46] use population fairness metrics: Predictive Parity and Error Rate Balance. They propose a simple agent-based model to detect any discrimination inequalities in an arrest-sentence system. Group fairness has been used as a definition in recommender systems for group recommendations as well. Kaya et al. [106] define a new metric for group fairness called Group Fairness Aware Recommendations (GFAR) considering the fairness of the top-N ranked items. GFAR defines top-N ranking as fair when the relevance of each of the top-N items, to the group members is “balanced” across the group members.

In information retrieval systems, researchers often focus on user evaluation to certify the fairness of the system. Mitra et al. [141] presented the first large-scale study of users’ interactions with the auto-complete function of Bing. Through an analysis of query logs, they found evidence of a position bias (i.e., users were more likely to engage with higher-ranked suggestions). They were also more likely to engage with auto-complete suggestions after having typed at least half of their query. In a follow-up study, Hofmann et al. [90] conducted an eye-tracking study with Bing users. In half of their queries, users were shown the ranked auto-complete suggestions while in the other half of queries, the suggestions were random. The authors confirmed the position bias in the auto-complete results, across both ranking conditions. They found that the quality of the auto-complete suggestions affected search behaviors; in the random setting users visited more pages to complete their search task. Another popular fairness certification method is simply to raise users’ awareness. Epstein et al. [61] develop solutions for the Search Engine Manipulation Effect (SEME), citing recent evidence of its impact on the views of undecided voters in the political context. In a large-scale online experiment with 3,600 users in 39 countries, they showed that manipulating the rankings in political searches can shift users’ expressed voting preferences by up to 39%. However, providing users with a “bias alert,” which informed them that “the current page of search rankings you are viewing appears to be biased in favor of [name of candidate],” reduced the shift to 22%. They found that this could be reduced even further when more detailed bias alerts were provided to users. Nonetheless, they reported that SEME cannot be completely eliminated with this type of intervention, and suggested instituting an “equal-time” rule such as that used in traditional media advertisements.

“Other” works use alternative approaches rather than the computation of specific metrics to certify an ML system. For instance, Fang et al. [64] certify the fairness of a classifier by constructing fairgroups, considering the feature importance to the decision variable. Individuals with similar features are grouped into clusters. This approach adopts the notion of fairness related to disparate impact, which affects individuals with at least one protected feature. In addition, Kilbertus et al. [109] provide fairness learning and certification without access to users’ sensitive data. To achieve this, they use an encrypted version of sensitive data, privacy constraints and decision verification by employing secure multi-party computation (MPC) methods. The use of techno-moral graphs for certifying ML algorithmic systems was also suggested in Reference [97]. The authors argue that a three-dimensional conceptual space can be used to map ML algorithmic projects in terms of the morality of their respective and constitutive ground-truth practices. Such techno-moral graphs may, in turn, serve as equipment for greater governance of ML algorithms and systems.

Woodruff et al. [214] explore, in a qualitative study, the perception of algorithmic fairness by populations that have been marginalized. In particular, they consider how race and low socioeconomic status was used in stereotyping and adapting services to those involved. Most participants were not aware of algorithmic unfairness even though they had experienced discrimination in their daily lives. Brown et al. [22] also present a qualitative study for understanding the public’s perspective on algorithmic decision-making in public services. They discovered that many participants mentioned discrimination and bias based on race, ethnicity, gender, location, and socioeconomic status. A descriptive approach for identifying the notion of perceived fairness for ML was suggested by Srivastava et al. [189]. They argued that the perceived fairness of the user is the most appropriate notion of algorithmic fairness. Their results show that the formal measurement, demographic parity, most closely matches the perceived fairness of the users and that in cases when the stakes are high, accuracy is more important than equality.

Perceived fairness on algorithmic decision-making is also explored in Reference [207] where the authors conduct an online experiment to better understand perceptions of fairness, focusing on three sets of factors: algorithm outcomes, algorithm development and deployment procedures, and individual differences. They find that people rate the algorithm as more fair when the algorithm predicts in their favor, even surpassing the negative effects of describing algorithms that are very biased against particular demographic groups. This effect is moderated by several variables, including participants’ education level, gender, and several aspects of the development procedure. These findings suggest that systems that evaluate algorithmic fairness through users’ feedback must consider the possibility of “outcome favorability” bias. In another study, the authors identify perception bias in borderline fact-checking messages [152]. The authors conduct both a quantitative and qualitative study by conducting semi-supervised user interviews to learn the user experience and perception of different fact-checking conditions. In a recent work [134], the authors introduce a network-centric fairness perception function that can be viewed as a local measure of individual fairness.

In addition, Maxwell et al. [136] investigated the influence of result diversification on users’ search behaviors. Diversification can reduce search engine biases by exposing users to a broader coverage of information on their topic of interest. A within-subject study with 51 users was performed, using the TREC AQUAINT collection. Two types of search tasks - ad hoc versus aspectual—are assigned to each user using a non-diversified IR system as well as a diversified system. Results indicated significant differences in users’ search behaviors between the two systems, with users executing more queries, but examining fewer documents when using the diversified system on the aspectual (i.e., more complex) task.

Table 8 in the Supplementary Materials provides a comparison of the Fairness Management solutions.

Table 8 in the Supplementary Materials provides a comparison of the Fairness Management solutions.

Model explainability techniques are primarily used to explain the process of an opaque ML model such as a neural network (NN) or a deep learning (DL) model. These techniques usually use a transparent model to mimic the model’s behavior and be interpretable by humans. For instance, some works use a decision tree to mimic the behavior of a non-transparent model [21, 44, 99, 115] and tree ensemble models [54, 74, 177, 193, 230]. The use of decision trees for explaining NN was first presented in Reference [44] where the TREPAN network implements the algorithmic process of the NN and returns the representations of the model. Chipman et al. [38] use decision trees as an interpetable predictor model for tree ensemble models by summarizing the forest of trees through clustering, and use the associated clusters as explanation models. A similar technique is the use of decision rules to explain a non-transparent model, for instance, by extracting rules from a trained model such as a NN, and then using the NN to refine existing rules [44, 99, 231].

More recent works use ontologies to represent and integrate knowledge to the model to enhance human understandability. An example is the recent extension of TREPAN [42] that uses and integrates knowledge in the form of ontologies in the decision tree extraction to enhance human understandability on decision trees. In addition, Ribeiro et al. [166] use ontologies to explain NN. They build small classifiers that map a neural network model’s internal state to concepts from an ontology, enabling the generation of symbolic justifications for the output of NN. An alternative approach has been proposed in Reference [19], where the authors present the Bayes-TREX framework, which uses Bayesian inference techniques to explain NN based on the whole dataset, not only the test data. Bayes-TREX takes as input the whole data and finds in-distribution examples that trigger various model behaviors across several contexts.

In addition to the aforementioned approaches for explainability of non-transparent ML algorithms, many articles, especially in the domain of recommender systems, propose some approaches for interpreting the ranking (recommender) algorithms. In such systems, the authors aim to provide explanations based on user opinions and evaluation of previous purchases, rather than on the analysis of the ranking algorithm [205]. The aim is to provide personalized explanations by selecting the most appropriate explanation style. Nunes et al. [145] presented a systematic review on explanations for recommendations in decision support systems where they proposed a taxonomy of concepts that are required for providing explanation. According to Tintarev and Masthoff [197], there are seven purposes for providing explanations in a recommender system: transparency, scrutability, trust, effectiveness, persuasiveness, satisfaction and efficiency. Park et al. [151] introduce the J-RECS, a recommendation model-agnostic method that generates personalized justifications based on various types of product and user data (e.g., purchase history and product attributes). Although most of the surveyed works in RecSys provide explanations based on user data, a recent work [71] propose some metrics for measuring explainability and transparency of the ranking algorithm.

A good example that connects explainability to fairness perception is the recent work of Anik et al. [5]. The authors explore the concept of data-centric explanations for ML systems that describe the training data to end users. They first investigate the potential utility of such an approach, including the information about training data that participants find most compelling. The authors also investigate reactions to the explanations across four system scenarios. Their results suggest that data-centric explanations have the potential to impact how users judge the trustworthiness of a system and to assist users in assessing fairness.

Outcome explainability approaches attempt to provide an interpretation for the outcome generated by the model. A recent work focuses on providing both local and pedagogical explanations for the output of ML models [133]. Pedagogical explanations are those that teach something about how the model works rather than attempting to represent it directly. Outcome explanations are divided into: visual explanations, local explanations, and feature relevance explanations techniques.

Local explanation approaches are the Local Interpretable Model-Agnostic Explanations (LIME) [167] and its variations [80, 168, 200]. The explanations in LIME are only provided through linear models and their respective feature importance. Anchors is another local explanation method proposed by Ribeiro et al. [168] that uses decision rules for explaining the model sufficiently. A post-hoc global explainability method has been proposed in Reference [9]. A SEPA framework has been introduced that incorporates post-hoc global explanation methods for image classification tasks. SEPA uses understandable semantic concepts (entities and attributes) that are obtained via crowd-sourcing from local interpretability saliency maps.

An example of feature relevant explanation approach is the ExplainD, a framework presented in Reference [160] for interpreting the outcome of any non-transparent model. ExplainD uses generative additive models (GAM) to weight the importance of the input features. A unified framework of the class of six existing additive feature importance methods, the SHAP (SHapley Additive exPlanations) has also been introduced in Reference [129]. SHAP assigns each feature an importance value for a particular prediction to interpret the predictions. According to Slack et al. [187], post-hoc explanation techniques that rely on the input, such as LIME and SHAP, are not reliable, since they do not take into consideration the bias in the model. In Reference [187], the authors proposed a scaffolding technique that scaffolds any biased classifier in a way that its input data remain biased but the generated post-hoc explanations do not reflect the underlying bias.

Other examples of feature relevant explanations include the approach used in Horne et al. [91] for explaining the spread of fake news and misinformation online. They used an AI assistance framework for providing these explanations to users. This was been shown to improve the user perception of bias and reliability on online news consumption. In another approach, Henelius et al. [87] search for a group of attributes of which the interactions affect the predictive performance of a given classifier, and they evaluate the importance of each group of attributes using the fidelity metric. In addition, Vidovic et al. [203] propose the measure of feature importance (MFI), which is model-agnostic and can be applied to any type of model. Feature-relevant explanations are also used in Reference [1], where the authors suggest the DIFF operator, a declarative operator that unifies explanation and feature selection queries with relational analytics workloads.

Another widely known category of outcome explainability approaches is the use of counterfactual explanations, which is a special case of feature-related explanations [182, 206]. In Reference [182], they propose the CERTIFAI model-agnostic technique that provides counterfactual explanations using a genetic algorithm. The user can use counterfactual explanations to understand the importance of the features. In Reference [206], the authors introduce Lewis, an open-source software that provides counterfactual explanations for the decision-making of an algorithm at the global, local and contextual level. For individuals negatively impacted by the algorithm’s decision, it provides actionable resources to change the outcome of the algorithm in the future. Visualization model-specific techniques are used to inspect the training process of a deep neural network (DNN) behavior on images [17, 218, 232]. In these works, a Saliency Mask (SM) is used as the interpretable local predictor, e.g., a part of an image. Similarly, Fong et al. [67] propose a framework of explanations as meta-predictors for explaining the outcome of deep learning models. The meta-predictor is a rule that predicts the response of the model to certain inputs such as highlighting the salient parts of an image. Another set of works use saliency masks to incorporate the DL network activation into their visualizations [180, 185, 230].

In RecSys, outcome explainability approaches are used to explain recommendations to the user. One category of techniques for RecSys are the ones that explain the latent factors that contribute to the decision outcome based on the collection of users’ interests and items’ characteristics such as Explicit Factor Models [228] and Tensor factorization [36]. Other approaches for explaining recommendations are based on the use of knowledge graphs that relate the items’ characteristics and users’ behavior, based on their past interactions with the items [29, 85]. Visual explanations have recently been used in RecSys to justify the recommendation process in combination with giving more control to the user in a specific context of an interactive social recommender system [198]. By conducting a user study, the authors investigate how the addition of user control and explainability affect the user perception, user experience, and user engagement. Based on the results, the best user experience happens when there is full explainability and control.

Table 9 in the Supplementary Materials provides a comparison of the solutions focusing on Explainability Management.

Table 8 in the Supplementary Materials provides a comparison of the solutions focusing on Explainability Management.