research-article

Free access

Hierarchical reinforcement learning and hidden Markov models for task-oriented natural language generation

Authors:

Heriberto Cuay�huitlAuthors Info & Claims

HLT '11: Proceedings of the 49th Annual Meeting of the Association for Computational Linguistics: Human Language Technologies: short papers - Volume 2

Pages 654 - 659

Published: 19 June 2011 Publication History

Abstract

Surface realisation decisions in language generation can be sensitive to a language model, but also to decisions of content selection. We therefore propose the joint optimisation of content selection and surface realisation using Hierarchical Reinforcement Learning (HRL). To this end, we suggest a novel reward function that is induced from human data and is especially suited for surface realisation. It is based on a generation space in the form of a Hidden Markov Model (HMM). Results in terms of task success and human-likeness suggest that our unified approach performs better than greedy or random baselines.

References

[1]

Gabor Angeli, Percy Liang, and Dan Klein. 2010. A simple domain-independent probabilistic approach to generation. In Proceedings of the 2010 Conference on Empirical Methods in Natural Language Processing, EMNLP '10, pages 502--512.

Digital Library

[2]

Srinivas Bangalore and Owen Rambow. 2000. Exploiting a probabilistic hierarchical model for generation. In Proceedings of the 18th Conference on Computational Linguistics (ACL) - Volume 1, pages 42--48.

Digital Library

[3]

Regina Barzilay and Lillian Lee. 2002. Bootstrapping lexical choice via multiple-sequence alignment. In Proceedings of the 2002 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 164--171.

Digital Library

[4]

Anja Belz and Ehud Reiter. 2006. Comparing Automatic and Human Evaluation of NLG Systems. In Proc. of the European Chapter of the Association for Computational Linguistics (EACL), pages 313--320.

[5]

Anja Belz. 2008. Automatic generation of weather forecast texts using comprehensive probabilistic generation-space models. Natural Language Engineering, 1:1--26.

Digital Library

[6]

Heriberto Cuay�huitl, Steve Renals, Oliver Lemon, and Hiroshi Shimodaira. 2005. Human-Computer Dialogue Simulation Using Hidden Markov Models. In Proc. of ASRU, pages 290--295.

[7]

Nina Dethlefs and Heriberto Cuay�huitl. 2010. Hierarchical Reinforcement Learning for Adaptive Text Generation. Proceeding of the 6th International Conference on Natural Language Generation (INLG).

Digital Library

[8]

Nina Dethlefs, Heriberto Cuay�huitl, and Jette Viethen. 2011. Optimising Natural Language Generation Decision Making for Situated Dialogue. In Proc. of the 12th Annual SIGdial Meeting on Discourse and Dialogue.

Digital Library

[9]

Thomas G. Dietterich. 1999. Hierarchical Reinforcement Learning with the MAXQ Value Function Decomposition. Journal of Artificial Intelligence Research, 13:227--303.

Digital Library

[10]

Mary Ellen Foster and Jon Oberlander. 2006. Data-driven generation of emphatic facial displays. In Proc. of the European Chapter of the Association for Computational Linguistic (EACL), pages 353--360.

[11]

Andrew Gargett, Konstantina Garoufi, Alexander Koller, and Kristina Striegnitz. 2010. The GIVE-2 corpus of giving instructions in virtual environments. In LREC.

[12]

Michael A. K. Halliday and Ruqaiya Hasan. 1976. Cohesion in English. Longman, London.

[13]

Srinivasan Janarthanam and Oliver Lemon. 2010. Learning to adapt to unknown users: referring expression generation in spoken dialogue systems. In Proc. of the Annual Meeting of the Association for Computational Linguistics (ACL), pages 69--78.

Digital Library

[14]

Alexander Koller, Kristina Striegnitz, Donna Byron, Justine Cassell, Robert Dale, Johanna Moore, and Jon Oberlander. 2010. The first challenge on generating instructions in virtual environments. In M. Theune and E. Krahmer, editors, Empirical Methods on Natural Language Generation, pages 337--361, Berlin/Heidelberg, Germany. Springer.

Digital Library

[15]

Irene Langkilde and Kevin Knight. 1998. Generation that exploits corpus-based statistical knowledge. In Proceedings of the 36th Annual Meeting of the Association for Computational Linguistics (ACL), pages 704--710.

Digital Library

[16]

W J M Levelt and S Kelter. 1982. Surface form and memory in question answering. Cognitive Psychology, 14.

[17]

François Mairesse, Milica Gašić, Filip Jurčíček, Simon Keizer, Blaise Thomson, Kai Yu, and Steve Young. 2010. Phrase-based statistical language generation using graphical models and active learning. In Proc. of the Annual Meeting of the Association for Computational Linguistics (ACL), pages 1552--1561.

Digital Library

[18]

Alice H. Oh and Alexander I. Rudnicky. 2000. Stochastic language generation for spoken dialogue systems. In Proceedings of the 2000 ANLP/NAACL Workshop on Conversational systems - Volume 3, pages 27--32.

Digital Library

[19]

Martin J. Pickering and Simon Garrod. 2004. Toward a mechanistc psychology of dialog. Behavioral and Brain Sciences, 27.

[20]

L R Rabiner. 1989. A Tutorial on Hidden Markov Models and Selected Applications in Speech Recognition. In Proceedings of IEEE, pages 257--286.

[21]

Verena Rieser, Oliver Lemon, and Xingkun Liu. 2010. Optimising information presentation for spoken dialogue systems. In Proc. of the Annual Meeting of the Association for Computational Lingustics (ACL), pages 1009--1018.

Digital Library

[22]

Richard S Sutton and Andrew G Barto. 1998. Reinforcement Learning: An Introduction. MIT Press, Cambridge, MA, USA.

Digital Library

[23]

Menno van Zaanen. 2000. Bootstrapping syntax and recursion using alginment-based learning. In Proceedings of the Seventeenth International Conference on Machine Learning (ICML), pages 1063--1070, San Francisco, CA, USA.

Digital Library

[24]

Marilyn A. Walker, Diane J. Litman, Candace A. Kamm, and Alicia Abella. 1997. PARADISE: A framework for evaluating spoken dialogue agents. In Proc. of the Annual Meeting of the Association for Computational Linguistics (ACL), pages 271--280.

Digital Library

[25]

Michael White. 2004. Reining in CCG chart realization. In Proc. of the International Conference on Natural Language Generation (INLG), pages 182--191.

Cited By

Emam SMiller J(2018)Inferring Extended Probabilistic Finite-State Automaton Models from Software ExecutionsACM Transactions on Software Engineering and Methodology10.1145/319688327:1(1-39)Online publication date: 5-Jun-2018
https://dl.acm.org/doi/10.1145/3196883
Emam SMiller J(2015)Test Case Prioritization Using Extended DigraphsACM Transactions on Software Engineering and Methodology10.1145/278920925:1(1-41)Online publication date: 2-Dec-2015
https://dl.acm.org/doi/10.1145/2789209
Cuay�huitl HKruijff-Korbayov� IDethlefs N(2014)Nonstrict Hierarchical Reinforcement Learning for Interactive Systems and RobotsACM Transactions on Interactive Intelligent Systems10.1145/26590034:3(1-30)Online publication date: 14-Oct-2014
https://dl.acm.org/doi/10.1145/2659003
Show More Cited By

Index Terms

Hierarchical reinforcement learning and hidden Markov models for task-oriented natural language generation

Recommendations

Learning nonsingular phylogenies and hidden Markov models
STOC '05: Proceedings of the thirty-seventh annual ACM symposium on Theory of computing

In this paper, we study the problem of learning phylogenies and hidden Markov models. We call a Markov model nonsingular if all transition matrices have determinants bounded away from 0 (and 1). We highlight the role of the nonsingularity condition for ...
Coding with partially hidden Markov models
DCC '95: Proceedings of the Conference on Data Compression

Partially hidden Markov models (PHMM) are introduced. They are a variation of the hidden Markov models (HMM) combining the power of explicit conditioning on past observations and the power of using hidden states. (P)HMM may be combined with arithmetic ...
Topology estimation of hierarchical hidden Markov models for language models
NLDB'10: Proceedings of the Natural language processing and information systems, and 15th international conference on Applications of natural language to information systems

Estimation of topology of probabilistic models provides us with an important technique for many statistical language processing tasks. In this investigation, we propose a new topology estimation method for Hierarchical Hidden Markov Model (HHMM) that ...

Comments

Information & Contributors

Information

Published In

cover image DL Hosted proceedings

HLT '11: Proceedings of the 49th Annual Meeting of the Association for Computational Linguistics: Human Language Technologies: short papers - Volume 2

June 2011

765 pages

ISBN:9781932432886

General Chair:
Dekang Lin
Google

Publisher

Association for Computational Linguistics

United States

Publication History

Published: 19 June 2011

Qualifiers

Research-article

Acceptance Rates

Overall Acceptance Rate 240 of 768 submissions, 31%

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

9
Total Citations
View Citations
409
Total Downloads

Downloads (Last 12 months)21
Downloads (Last 6 weeks)4

Reflects downloads up to 22 Oct 2024

Other Metrics

View Author Metrics

Citations

Cited By

Emam SMiller J(2018)Inferring Extended Probabilistic Finite-State Automaton Models from Software ExecutionsACM Transactions on Software Engineering and Methodology10.1145/319688327:1(1-39)Online publication date: 5-Jun-2018
https://dl.acm.org/doi/10.1145/3196883
Emam SMiller J(2015)Test Case Prioritization Using Extended DigraphsACM Transactions on Software Engineering and Methodology10.1145/278920925:1(1-41)Online publication date: 2-Dec-2015
https://dl.acm.org/doi/10.1145/2789209
Cuay�huitl HKruijff-Korbayov� IDethlefs N(2014)Nonstrict Hierarchical Reinforcement Learning for Interactive Systems and RobotsACM Transactions on Interactive Intelligent Systems10.1145/26590034:3(1-30)Online publication date: 14-Oct-2014
https://dl.acm.org/doi/10.1145/2659003
Rieser VLemon OKeizer S(2014)Natural language generation as incremental planning under uncertaintyIEEE/ACM Transactions on Audio, Speech and Language Processing10.1109/TASL.2014.231527122:5(979-994)Online publication date: 1-May-2014
https://dl.acm.org/doi/10.1109/TASL.2014.2315271
Dethlefs NCuay�huitl HChu-Carroll J(2012)Comparing HMMs and Bayesian networks for surface realisationProceedings of the 2012 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies10.5555/2382029.2382135(636-640)Online publication date: 3-Jun-2012
https://dl.acm.org/doi/10.5555/2382029.2382135
Dethlefs NGardent CStriegnitz K(2011)The Bremen system for the GIVE-2.5 challengeProceedings of the 13th European Workshop on Natural Language Generation10.5555/2187681.2187734(284-289)Online publication date: 28-Sep-2011
https://dl.acm.org/doi/10.5555/2187681.2187734
Janarthanam SLemon OGardent CStriegnitz K(2011)The GRUVE challengeProceedings of the 13th European Workshop on Natural Language Generation10.5555/2187681.2187717(208-211)Online publication date: 28-Sep-2011
https://dl.acm.org/doi/10.5555/2187681.2187717
Rieser VKeizer SLiu XLemon OGardent CStriegnitz K(2011)Adaptive information presentation for spoken dialogue systemsProceedings of the 13th European Workshop on Natural Language Generation10.5555/2187681.2187698(102-109)Online publication date: 28-Sep-2011
https://dl.acm.org/doi/10.5555/2187681.2187698
Dethlefs NCuay�huitl HViethen JMoore JTraum DChai JPassonneau R(2011)Optimising natural language generation decision making for situated dialogueProceedings of the SIGDIAL 2011 Conference10.5555/2132890.2132901(78-87)Online publication date: 17-Jun-2011
https://dl.acm.org/doi/10.5555/2132890.2132901

View Options

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Get Access

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

Media

Figures

Other

Tables

View Table of Contents