Characterizing Stereotypical Bias from Privacy-preserving Pre-Training - Arnold, Stefan; Gröbner, René; Schreiner, Annika (Proceedings of the Fifth Workshop on Privacy in Natural Language Processing, 2024)
Differential Privacy (DP) can be applied to raw text by exploiting the spatial arrangement of words in an embedding space. We investigate the implications of such text privatization on Language Models (LMs) and their tendency towards stereotypical associations. Since previous studies documented that linguistic proficiency correlates with stereotypical bias, one could assume that techniques for text privatization, which are known to degrade language modeling capabilities, would cancel out undesirable biases. By testing BERT models trained on texts containing biased statements primed with varying degrees of privacy, our study reveals that while stereotypical bias generally diminishes when privacy is tightened, text privatization does not uniformly equate to diminishing bias across all social domains. This highlights the need for careful diagnosis of bias in LMs that undergo text privatization.
CRIS-Team Scopus Importer:2024-09-27

Disentangling the Linguistic Competence of Privacy-Preserving BERT - Arnold, Stefan; Kemmerzell, Nils; Schreiner, Annika (Proceedings of the 6th BlackboxNLP Workshop: Analyzing and Interpreting Neural Networks for NLP, 2023)

Driving context into text-to-text privatization - Arnold, Stefan; Yesilbas, Dilara; Weinzierl, Sven (Proceedings of the 3rd Workshop on Trustworthy Natural Language Processing, 2023)
Metric Differential Privacy enables text-to-text privatization by adding calibrated noise to the vector of a word derived from an embedding space and projecting this noisy vector back to a discrete vocabulary using a nearest neighbor search. Since words are substituted without context, this mechanism is expected to fall short at finding substitutes for words with ambiguous meanings, such as bank. To account for these ambiguous words, we leverage a sense embedding and incorporate a sense disambiguation step prior to noise injection. We encompass our modification to the privatization mechanism with an estimation of privacy and utility. For word sense disambiguation on the Words in Context dataset, we demonstrate a substantial increase in classification accuracy by 6.05%.<br

Guiding text-to-text privatization by syntax - Arnold, Stefan; Yesilbas, Dilara; Weinzierl, Sven (Proceedings of the 3rd Workshop on Trustworthy Natural Language Processing, 2023)
Metric Differential Privacy is a generalization of differential privacy tailored to address the unique challenges of text-to-text privatization. By adding noise to the representation of words in the geometric space of embeddings, words are replaced with words located in the proximity of the noisy representation. Since embeddings are trained based on word co-occurrences, this mechanism ensures that substitutions stem from a common semantic context. Without considering the grammatical category of words, however, this mechanism cannot guarantee that substitutions play similar syntactic roles. We analyze the capability of text-to-text privatization to preserve the grammatical category of words after substitution and find that surrogate texts consist almost exclusively of nouns. Lacking the capability to produce surrogate texts that correlate with the structure of the sensitive texts, we encompass our analysis by transforming the privatization step into a candidate selection problem in which substitutions are directed to words with matching grammatical properties. We demonstrate a substantial improvement in the performance of downstream tasks by up to 4.66% while retaining comparative privacy guarantees.<br

Rethinking Pre-Training in Industrial Quality Control - Yesilbas, Dilara; Arnold, Stefan; Felker, Alex (17th International Conference on Wirtschaftsinformatik, WI 2022, 2022)
The application of machine learning is of high significance for quality control tasks in the manufacturing industry due to large volumes of machine-generated data. However, labeling data is costly and labor-intensive. In this study, we evaluate the role of manual labeling and the moderating effect of autoencoder-based pre-training in optical quality control using real-world industrial data. We observe that pre-training substantially elevates the classification accuracy for small amounts of labeled data. With increasing amounts of labeled data available during fine-tuning, however, we find diminishing returns, analogous to recent concerns raised in non-industrial applications.
CRIS-Team Scopus Importer:2024-12-16
Keywords: Autoencoder; Optical Quality Control; Printed Circuit Boards