article

Free access

Knowledge-Based Kernel Approximation

Authors:

Olvi L. Mangasarian,

Jude W. Shavlik,

Edward W. WildAuthors Info & Claims

The Journal of Machine Learning Research, Volume 5

Pages 1127 - 1141

Published: 01 December 2004 Publication History

Abstract

Prior knowledge, in the form of linear inequalities that need to be satisfied over multiple polyhedral sets, is incorporated into a function approximation generated by a linear combination of linear or nonlinear kernels. In addition, the approximation needs to satisfy conventional conditions such as having given exact or inexact function values at certain points. Determining such an approximation leads to a linear programming formulation. By using nonlinear kernels and mapping the prior polyhedral knowledge in the input space to one defined by the kernels, the prior knowledge translates into nonlinear inequalities in the original input space. Through a number of computational examples, including a real world breast cancer prognosis dataset, it is shown that prior knowledge can significantly improve function approximation.

References

[1]

G. Baudat and F. Anouar. Kernel-based methods and function approximation. In International Joint Conference on Neural Networks, pages 1244-1249, Washington, D.C., 2001.

[2]

V. Cherkassky and F. Mulier. Learning from Data - Concepts, Theory and Methods. John Wiley & Sons, New York, 1998.

Digital Library

[3]

F. Deutsch. Best Approximation in Inner Product Spaces. Springer-Verlag, Berlin, 2001.

[4]

H. Drucker, C. J. C. Burges, L. Kaufman, A. Smola, and V. Vapnik. Support vector regression machines. In M. C. Mozer, M. I. Jordan, and T. Petsche, editors, Advances in Neural Information Processing Systems -9-, pages 155-161, Cambridge, MA, 1997. MIT Press.

[5]

T. Evgeniou, M. Pontil, and T. Poggio. Regularization networks and support vector machines. In A. Smola, P. Bartlett, B. Sch�lkopf, and D. Schuurmans, editors, Advances in Large Margin Classifiers, pages 171-203, Cambridge, MA, 2000. MIT Press.

[6]

G. Fung, O. L. Mangasarian, and J. Shavlik. Knowledge-based nonlinear kernel classifiers. Technical Report 03-02, Data Mining Institute, Computer Sciences Department, University of Wisconsin, Madison, Wisconsin, March 2003a. ftp://ftp.cs.wisc.edu/pub/dmi/tech-reports/02-03.ps. Conference on Learning Theory (COLT 03) and Workshop on Kernel Machines, Washington D.C., August 24-27, 2003. Proceedings edited by M. Warmuth and B. Sch�olkopf, Springer Verlag, Berlin, 2003, 102-113.

[7]

G. Fung, O. L. Mangasarian, and J. Shavlik. Knowledge-based support vector machine classifiers. In Suzanna Becker, Sebastian Thrun, and Klaus Obermayer, editors, Advances in Neural Information Processing Systems 15, pages 521-528. MIT Press, Cambridge, MA, October 2003b. ftp://ftp.cs.wisc.edu/pub/dmi/tech-reports/01-09.ps.

[8]

G. Kuhlmann, P. Stone, R. Mooney, and J. Shavlik. Guiding a reinforcement learner with natural language advice: Initial results in robocup soccer. In Proceedings of the AAAI Workshop on Supervisory Control of Learning and Adaptive Systems, San Jose, CA, 2004.

[9]

Y.-J. Lee, O. L. Mangasarian, and W. H. Wolberg. Survival-time classification of breast cancer patients. Technical Report 01-03, DataMining Institute, Computer Sciences Department, University of Wisconsin, Madison, Wisconsin, March 2001. ftp://ftp.cs.wisc.edu/pub/dmi/tech-reports/01- 03.ps. Computational Optimization and Applications 25, 2003, 151-166.

Digital Library

[10]

R. Maclin and J. Shavlik. Creating advice-taking reinforcement learners. Machine Learning, 22, 1996.

Digital Library

[11]

O. L. Mangasarian. Nonlinear Programming. SIAM, Philadelphia, PA, 1994.

[12]

O. L. Mangasarian. Generalized support vector machines. In A. Smola, P. Bartlett, B. Sch�lkopf, and D. Schuurmans, editors, Advances in Large Margin Classifiers, pages 135-146, Cambridge, MA, 2000. MIT Press. ftp://ftp.cs.wisc.edu/math-prog/tech-reports/98-14.ps.

[13]

O. L. Mangasarian. Data mining via support vector machines, July 23-27, 2001. http://ftp.cs.wisc.edu/math-prog/talks/ifip3tt.ppt.

[14]

O. L. Mangasarian and D. R. Musicant. Large scale kernel regression via linear programming. Machine Learning, 46:255-269, 2002. ftp://ftp.cs.wisc.edu/pub/dmi/tech-reports/99-02.ps.

Digital Library

[15]

O. L. Mangasarian and L. L. Schumaker. Splines via optimal control. In I. J. Schoenberg, editor, Approximations with Special Emphasis on Splines, pages 119-156, New York, 1969. Academic Press.

[16]

O. L. Mangasarian and L. L. Schumaker. Discrete splines via mathematical programming. SIAM Journal on Control, 9:174-183, May 1971.

[17]

O. L. Mangasarian, W. N. Street, and W. H. Wolberg. Breast cancer diagnosis and prognosis via linear programming. Operations Research, 43(4):570-577, July-August 1995.

Digital Library

[18]

C. A. Micchelli and F. I. Utreras. Smoothing and interpolation in a convex subset of a hilbert space. SIAM Journal of Statistical Computing, 9:728-746, 1988.

Digital Library

[19]

P. M. Murphy and D. W. Aha. UCI machine learning repository, 1992. www.ics.uci.edu/~mlearn/MLRepository.html.

[20]

J. A. Nelder and R. Mead. A simplex method for function minimization. The Computer Journal, 7: 308-313, 1965.

[21]

B. Sch�lkopf, P. Simard, A. Smola, and V. Vapnik. Prior knowledge in support vector kernels. In M. Jordan, M. Kearns, and S. Solla, editors, Advances in Neural Information Processing Systems 10, pages 640-646, Cambridge, MA, 1998. MIT Press.

Digital Library

[22]

B. Sch�lkopf and A. Smola. Learning with Kernels. MIT Press, Cambridge, MA, 2002.

[23]

A. Smola and B. Sch�lkopf. On a kernel-based method for pattern recognition, regression, approximation and operator inversion. Algorithmica, 22:211-231, 1998.

[24]

P. Stone and R. Sutton. Scaling reinforcement learning toward robocup soccer. In Proceedings of the Eighteenth International Conference on Machine Learning (ICML'01), Williams, MA, 2001.

Digital Library

[25]

R. Sutton and A. Barto. Reinforcement Learning: An Introduction. MIT Press, Cambridge, MA, 1998.

Digital Library

[26]

V. N. Vapnik. The Nature of Statistical Learning Theory. Springer, New York, second edition, 2000.

Digital Library

[27]

V. N. Vapnik, S. E. Golowich, and A. Smola. Support vector method for function approximation, regression estimation and signal processing. In Neural Information Processing Systems Volume 9, pages 281-287, Cambridge, MA, 1997. MIT Press.

[28]

W. H. Wolberg, W. N. Street, D. N. Heisey, and O. L. Mangasarian. Computerized breast cancer diagnosis and prognosis from fine-needle aspirates. Archives of Surgery, 130:511-516, 1995.

Cited By

Liu XXu FZhang XLiu TJiang SChen RZhang ZYu YAgmon NAn BRicci AYeoh W(2023)How To Guide Your Learner: Imitation Learning with Active Adaptive Expert InvolvementProceedings of the 2023 International Conference on Autonomous Agents and Multiagent Systems10.5555/3545946.3598773(1276-1284)Online publication date: 30-May-2023
https://dl.acm.org/doi/10.5555/3545946.3598773
Chen SGao CZhang P(2022)Incorporation of Data-Mined Knowledge into Black-Box SVM for InterpretabilityACM Transactions on Intelligent Systems and Technology10.1145/354877514:1(1-22)Online publication date: 9-Nov-2022
https://dl.acm.org/doi/10.1145/3548775
Zhao JHei XShi ZDong LLiu YYan RLi X(2018)Regression learning based on incomplete relationships between attributesInformation Sciences: an International Journal10.1016/j.ins.2017.09.023422:C(408-431)Online publication date: 1-Jan-2018
https://dl.acm.org/doi/10.1016/j.ins.2017.09.023
Show More Cited By

Knowledge-Based Kernel Approximation
1. Computing methodologies
2. Mathematics of computing
  1. Mathematical analysis
    1. Functional analysis

Recommendations

Multiscale Approximation and Reproducing Kernel Hilbert Space Methods

We consider reproducing kernels $K:\Omega\times \Omega \to \mathbb{R}$ in multiscale series expansion form, i.e., kernels of the form $K\left(\boldsymbol{x},\boldsymbol{y}\right)=\sum_{\ell\in\mathbb{N}}\lambda_\ell\sum_{j\in I_\ell}\phi_{\ell,j}\left(\...
Approximation theorems on mapping properties of the classical kernel functions of complex analysis
Nonlinear Knowledge in Kernel Approximation

Prior knowledge over arbitrary general sets is incorporated into nonlinear kernel approximation problems in the form of linear constraints in a linear program. The key tool in this incorporation is a theorem of the alternative for convex functions that ...

Comments

Information & Contributors

Information

Published In

cover image The Journal of Machine Learning Research

The Journal of Machine Learning Research Volume 5, Issue

12/1/2004

1571 pages

ISSN:1532-4435

EISSN:1533-7928

Issue’s Table of Contents

Publisher

JMLR.org

Publication History

Published: 01 December 2004

Published in JMLR Volume 5

Qualifiers

Article

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

24
Total Citations
View Citations
544
Total Downloads

Downloads (Last 12 months)55
Downloads (Last 6 weeks)4

Reflects downloads up to 22 Oct 2024

Other Metrics

View Author Metrics

Citations

Cited By

Liu XXu FZhang XLiu TJiang SChen RZhang ZYu YAgmon NAn BRicci AYeoh W(2023)How To Guide Your Learner: Imitation Learning with Active Adaptive Expert InvolvementProceedings of the 2023 International Conference on Autonomous Agents and Multiagent Systems10.5555/3545946.3598773(1276-1284)Online publication date: 30-May-2023
https://dl.acm.org/doi/10.5555/3545946.3598773
Chen SGao CZhang P(2022)Incorporation of Data-Mined Knowledge into Black-Box SVM for InterpretabilityACM Transactions on Intelligent Systems and Technology10.1145/354877514:1(1-22)Online publication date: 9-Nov-2022
https://dl.acm.org/doi/10.1145/3548775
Zhao JHei XShi ZDong LLiu YYan RLi X(2018)Regression learning based on incomplete relationships between attributesInformation Sciences: an International Journal10.1016/j.ins.2017.09.023422:C(408-431)Online publication date: 1-Jan-2018
https://dl.acm.org/doi/10.1016/j.ins.2017.09.023
Shao YYe YWang YDeng N(2016)Extensive semi-quantitative regressionNeurocomputing10.1016/j.neucom.2016.08.073218:C(26-36)Online publication date: 19-Dec-2016
https://dl.acm.org/doi/10.1016/j.neucom.2016.08.073
Balasundaram SGupta D(2016)Knowledge-based extreme learning machinesNeural Computing and Applications10.1007/s00521-015-1961-527:6(1629-1641)Online publication date: 1-Aug-2016
https://dl.acm.org/doi/10.1007/s00521-015-1961-5
Knyazhansky MPlotkin T(2012)Knowledge Bases Over Algebraic ModelsInternational Journal of Knowledge Management10.4018/jkm.20120101028:1(22-39)Online publication date: 1-Jan-2012
https://dl.acm.org/doi/10.4018/jkm.2012010102
Zhu XMahule TDutta HArora SKargupta HBorne K(2012)Peer-to-peer distributed text classifier learning in PADMINIStatistical Analysis and Data Mining10.1002/sam.111555:5(446-462)Online publication date: 1-Oct-2012
https://dl.acm.org/doi/10.1002/sam.11155
Kunapuli GMaclin RShavlik J(2011)Advice refinement in knowledge-based SVMsProceedings of the 24th International Conference on Neural Information Processing Systems10.5555/2986459.2986652(1728-1736)Online publication date: 12-Dec-2011
https://dl.acm.org/doi/10.5555/2986459.2986652
Sun ZZhang ZWang H(2011)Consistency and error analysis of Prior-Knowledge-Based Kernel RegressionNeurocomputing10.1016/j.neucom.2011.06.00174:17(3476-3485)Online publication date: 1-Oct-2011
https://dl.acm.org/doi/10.1016/j.neucom.2011.06.001
Zhang DTian YShi Y(2011)A group of knowledge-incorporated multiple criteria linear programming classifiersJournal of Computational and Applied Mathematics10.1016/j.cam.2011.01.014235:13(3705-3717)Online publication date: 1-May-2011
https://dl.acm.org/doi/10.1016/j.cam.2011.01.014
Show More Cited By

View Options

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Get Access

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Article

Media

Figures

Other

Tables

View Issue’s Table of Contents