Alexander Miserlis Hoyle,Rupak Sarkar,Pranav Goel,Philip Resnik
Abstract
Recently, the relationship between automated and human evaluation of topic models has been called into question. Method developers have staked the efficacy of new topic model variants on automated measures, and their failure to approximate human preferences places these models on uncertain ground. Moreover, existing evaluation paradigms are often divorced from real-world use.Motivated by content analysis as a dominant real-world use case for topic modeling, we analyze two related aspects of topic models that affect their effectiveness and trustworthiness in practice for that purpose: the stability of their estimates and the extent to which the model’s discovered categories align with human-determined categories in the data. We find that neural topic models fare worse in both respects compared to an established classical method. We take a step toward addressing both issues in tandem by demonstrating that a straightforward ensembling method can reliably outperform the members of the ensemble.
See Also
How to evaluate novel topic modeling method.Topic modeling revisited: New evidence on algorithm performance and quality metricsDeciphering Topic Modeling Evaluation in Webex Contact Center: Exploring Clusters and LabelHow do you use topic modeling for text summarization, classification, or clustering?- Anthology ID:
- 2022.findings-emnlp.390
- Volume:
- Findings of the Association for Computational Linguistics: EMNLP 2022
- Month:
- December
- Year:
- 2022
- Address:
- Abu Dhabi, United Arab Emirates
- Editors:
- Yoav Goldberg,Zornitsa Kozareva,Yue Zhang
- Venue:
- Findings
- SIG:
- Publisher:
- Association for Computational Linguistics
- Note:
- Pages:
- 5321–5344
- Language:
- URL:
- https://aclanthology.org/2022.findings-emnlp.390
- DOI:
- 10.18653/v1/2022.findings-emnlp.390
- Bibkey:
- Cite (ACL):
- Alexander Miserlis Hoyle, Rupak Sarkar, Pranav Goel, and Philip Resnik. 2022. Are Neural Topic Models Broken?. In Findings of the Association for Computational Linguistics: EMNLP 2022, pages 5321–5344, Abu Dhabi, United Arab Emirates. Association for Computational Linguistics.
- Cite (Informal):
- Are Neural Topic Models Broken? (Hoyle et al., Findings 2022)
- Copy Citation:
- PDF:
- https://aclanthology.org/2022.findings-emnlp.390.pdf
- Video:
- https://aclanthology.org/2022.findings-emnlp.390.mp4
PDFCiteSearchVideo
Export citation
- BibTeX
- MODS XML
- Endnote
- Preformatted
@inproceedings{hoyle-etal-2022-neural, title = "Are Neural Topic Models Broken?", author = "Hoyle, Alexander Miserlis and Sarkar, Rupak and Goel, Pranav and Resnik, Philip", editor = "Goldberg, Yoav and Kozareva, Zornitsa and Zhang, Yue", booktitle = "Findings of the Association for Computational Linguistics: EMNLP 2022", month = dec, year = "2022", address = "Abu Dhabi, United Arab Emirates", publisher = "Association for Computational Linguistics", url = "https://aclanthology.org/2022.findings-emnlp.390", doi = "10.18653/v1/2022.findings-emnlp.390", pages = "5321--5344", abstract = "Recently, the relationship between automated and human evaluation of topic models has been called into question. Method developers have staked the efficacy of new topic model variants on automated measures, and their failure to approximate human preferences places these models on uncertain ground. Moreover, existing evaluation paradigms are often divorced from real-world use.Motivated by content analysis as a dominant real-world use case for topic modeling, we analyze two related aspects of topic models that affect their effectiveness and trustworthiness in practice for that purpose: the stability of their estimates and the extent to which the model{'}s discovered categories align with human-determined categories in the data. We find that neural topic models fare worse in both respects compared to an established classical method. We take a step toward addressing both issues in tandem by demonstrating that a straightforward ensembling method can reliably outperform the members of the ensemble.",}Download as File
<?xml version="1.0" encoding="UTF-8"?><modsCollection xmlns="http://www.loc.gov/mods/v3"><mods ID="hoyle-etal-2022-neural"> <titleInfo> <title>Are Neural Topic Models Broken?</title> </titleInfo> <name type="personal"> <namePart type="given">Alexander</namePart> <namePart type="given">Miserlis</namePart> <namePart type="family">Hoyle</namePart> <role> <roleTerm authority="marcrelator" type="text">author</roleTerm> </role> </name> <name type="personal"> <namePart type="given">Rupak</namePart> <namePart type="family">Sarkar</namePart> <role> <roleTerm authority="marcrelator" type="text">author</roleTerm> </role> </name> <name type="personal"> <namePart type="given">Pranav</namePart> <namePart type="family">Goel</namePart> <role> <roleTerm authority="marcrelator" type="text">author</roleTerm> </role> </name> <name type="personal"> <namePart type="given">Philip</namePart> <namePart type="family">Resnik</namePart> <role> <roleTerm authority="marcrelator" type="text">author</roleTerm> </role> </name> <originInfo> <dateIssued>2022-12</dateIssued> </originInfo> <typeOfResource>text</typeOfResource> <relatedItem type="host"> <titleInfo> <title>Findings of the Association for Computational Linguistics: EMNLP 2022</title> </titleInfo> <name type="personal"> <namePart type="given">Yoav</namePart> <namePart type="family">Goldberg</namePart> <role> <roleTerm authority="marcrelator" type="text">editor</roleTerm> </role> </name> <name type="personal"> <namePart type="given">Zornitsa</namePart> <namePart type="family">Kozareva</namePart> <role> <roleTerm authority="marcrelator" type="text">editor</roleTerm> </role> </name> <name type="personal"> <namePart type="given">Yue</namePart> <namePart type="family">Zhang</namePart> <role> <roleTerm authority="marcrelator" type="text">editor</roleTerm> </role> </name> <originInfo> <publisher>Association for Computational Linguistics</publisher> <place> <placeTerm type="text">Abu Dhabi, United Arab Emirates</placeTerm> </place> </originInfo> <genre authority="marcgt">conference publication</genre> </relatedItem> <abstract>Recently, the relationship between automated and human evaluation of topic models has been called into question. Method developers have staked the efficacy of new topic model variants on automated measures, and their failure to approximate human preferences places these models on uncertain ground. Moreover, existing evaluation paradigms are often divorced from real-world use.Motivated by content analysis as a dominant real-world use case for topic modeling, we analyze two related aspects of topic models that affect their effectiveness and trustworthiness in practice for that purpose: the stability of their estimates and the extent to which the model’s discovered categories align with human-determined categories in the data. We find that neural topic models fare worse in both respects compared to an established classical method. We take a step toward addressing both issues in tandem by demonstrating that a straightforward ensembling method can reliably outperform the members of the ensemble.</abstract> <identifier type="citekey">hoyle-etal-2022-neural</identifier> <identifier type="doi">10.18653/v1/2022.findings-emnlp.390</identifier> <location> <url>https://aclanthology.org/2022.findings-emnlp.390</url> </location> <part> <date>2022-12</date> <extent unit="page"> <start>5321</start> <end>5344</end> </extent> </part></mods></modsCollection>
Download as File
%0 Conference Proceedings%T Are Neural Topic Models Broken?%A Hoyle, Alexander Miserlis%A Sarkar, Rupak%A Goel, Pranav%A Resnik, Philip%Y Goldberg, Yoav%Y Kozareva, Zornitsa%Y Zhang, Yue%S Findings of the Association for Computational Linguistics: EMNLP 2022%D 2022%8 December%I Association for Computational Linguistics%C Abu Dhabi, United Arab Emirates%F hoyle-etal-2022-neural%X Recently, the relationship between automated and human evaluation of topic models has been called into question. Method developers have staked the efficacy of new topic model variants on automated measures, and their failure to approximate human preferences places these models on uncertain ground. Moreover, existing evaluation paradigms are often divorced from real-world use.Motivated by content analysis as a dominant real-world use case for topic modeling, we analyze two related aspects of topic models that affect their effectiveness and trustworthiness in practice for that purpose: the stability of their estimates and the extent to which the model’s discovered categories align with human-determined categories in the data. We find that neural topic models fare worse in both respects compared to an established classical method. We take a step toward addressing both issues in tandem by demonstrating that a straightforward ensembling method can reliably outperform the members of the ensemble.%R 10.18653/v1/2022.findings-emnlp.390%U https://aclanthology.org/2022.findings-emnlp.390%U https://doi.org/10.18653/v1/2022.findings-emnlp.390%P 5321-5344
Download as File
Markdown (Informal)
[Are Neural Topic Models Broken?](https://aclanthology.org/2022.findings-emnlp.390) (Hoyle et al., Findings 2022)
- Are Neural Topic Models Broken? (Hoyle et al., Findings 2022)
ACL
- Alexander Miserlis Hoyle, Rupak Sarkar, Pranav Goel, and Philip Resnik. 2022. Are Neural Topic Models Broken?. In Findings of the Association for Computational Linguistics: EMNLP 2022, pages 5321–5344, Abu Dhabi, United Arab Emirates. Association for Computational Linguistics.
See Also
Coherence