Article

Joint data streaming and sampling techniques for detection of super sources and destinations

Authors:

Abhishek Kumar,

Jun XuAuthors Info & Claims

IMC '05: Proceedings of the 5th ACM SIGCOMM conference on Internet measurement

Page 7

Published: 19 October 2005 Publication History

Publisher Site Get Access

Abstract

Detecting the sources or destinations that have communicated with a large number of distinct destinations or sources during a small time interval is an important problem in network measurement and security. Previous detection approaches are not able to deliver the desired accuracy at high link speeds (10 to 40 Gbps). In this work, we propose two novel algorithms that provide accurate and efficient solutions to this problem. Their designs are based on the insight that sampling and data streaming are often suitable for capturing different and complementary regions of the information spectrum, and a close collaboration between them is an excellent way to recover the complete information. Our first solution builds on the standard hash-based flow sampling algorithm. Its main innovation is that the sampled traffic is further filtered by a data streaming module which allows for much higher sampling rate and hence much higher accuracy. Our second solution is more sophisticated but offers higher accuracy. It combines the power of data streaming in efficiently estimating quantities associated with a given identity, and the power of sampling in collecting a list of candidate identities. The performance of both solutions are evaluated using both mathematical analysis and trace-driven experiments on real-world Internet traffic.

References

[1]

{1} B. H. Bloom. Space/time trade-offs in hash coding with allowable errors. CACM, 13(7):422-426, 1970.

Digital Library

[2]

{2} J. Carter and M. Wegman. Universal classes of hash functions. Journal of Computer and System Sciences, pages 143-154, 1979.

[3]

{3} N. Duffield and M. Grossglauser. Trajectory sampling for direct traffic observation. IEEE transaction of Networking , pages 280-292, June 2001.

Digital Library

[4]

{4} N. Duffield, C. Lund, and M. Thorup. Estimating flow distribution from sampled flow statistics. In Proc. ACM SIGCOMM, August 2003.

Digital Library

[5]

{5} C. Estan and G. Varghese. New Directions in Traffic Measurement and Accounting. In Proc. ACM SIGCOMM , August 2002.

Digital Library

[6]

{6} C. Estan and G. Varghese. Bitmap algorithms for counting active flows on high speed links. In Proc. ACM/SIGCOMM IMC, October 2003.

Digital Library

[7]

{7} W. Fang and L. Peterson. Inter-AS traffic patterns and their implications. In Proc. IEEE GLOBECOM, December 1999.

[8]

{8} N. Hohn and D. Veitch. Inverting sampled traffic. In Proc. ACM/SIGCOMM IMC, October 2003.

Digital Library

[9]

{9} J. Jung, B. Krishnamurthy, and M. Rabinovich. Flash crowds and denial of service attacks: Characterization and implications for cdn and web sites. In Proc. World Wide Web Conference, May 2002.

Digital Library

[10]

{10} R. Karp, S. Shenker, and C. Papadimitriou. A simple algorithm for finding frequent elements in streams and bags. ACM Transactions on Database Systems (TODS), 28:51-55, 2003.

Digital Library

[11]

{11} A. Kumar, M. Sung, J. Xu, and J. Wang. Data streaming algorithms for efficient and accurate estimation of flow size distribution. In Proc. ACM SIGMETRICS, 2004.

Digital Library

[12]

{12} A. Kumar, J. Xu, J. Wang, O. Spatschek, and L. Li. Space-Code Bloom Filter for Efficient per-flow Traffic Measurement. In Proc. IEEE INFOCOM, March 2004.

[13]

{13} R. Motwani and P. Raghavan. Randomized Algorithms . Cambridge University Press, 1995.

Digital Library

[14]

{14} CISCO Tech Notes. Cisco netflow. available at http://www.cisco.com/warp/public/732/netflow/ index.html.

[15]

{15} V. Paxon. An analysis of using reflectors for distributed denial-of-service attacks. Computer Communication Review, 2001.

Digital Library

[16]

{16} D. S. Phatak and T. Goff. A novel mechanism for data streaming across multiple IP links for improving throughput and reliability in mobile environments. In Proc. IEEE INFOCOM, June 2002.

[17]

{17} D. Plonka. Flowscan: A network traffic flow reporting and visualization tool. In Proc. USENIX LISA, 2000.

Digital Library

[18]

{18} M. Ramakrishna, E. Fu, and E. Bahcekapili. Efficient hardware hashing functions for high performance computers. IEEE Transactions on Computers, pages 1378-1381, 1997.

Digital Library

[19]

{19} M. Roesch. Snort-lightweight intrusion detection for networks. In Proc. USENIX Systems Administration Conference, 1999.

Digital Library

[20]

{20} S. Venkataraman, D. Song, P. Gibbons, and A. Blum. New streaming algorithms for fast detection of superspreaders. In Proc. NDSS, 2005.

[21]

{21} K.Y. Whang, B.T. Vander-zanden, and H.M. Taylor. A linear-time probabilistic counting algorithm for database applications. IEEE transaction of Database Systems, pages 208-229, June 1990.

Digital Library

[22]

{22} Y. Zhang, S. Singh, S. Sen, N. Duffield, and C. Lund. Online identification of hierarchical heavy hitters: Algorithms, evaluation, and application. In Proc. ACM/SIGCOMM IMC, October 2004.

Digital Library

[23]

{23} Q. Zhao, A. Kumar, J. Wang, and J. Xu. Data streaming algorithms for accurate and efficient measurement of traffic and flow matrices. In Proc. ACM SIGMETRICS , June 2005.

Digital Library

[24]

{24} Q. Zhao, A. Kumar, and J. Xu. Joint data streaming and sampling techniques for detection of super sources and destinations. In Technical Report, July 2005.

Cited By

Aftab MChau SShenoy P(2020)Efficient Online Classification and Tracking on Resource-constrained IoT DevicesACM Transactions on Internet of Things10.1145/33920511:3(1-29)Online publication date: 13-Jul-2020
https://dl.acm.org/doi/10.1145/3392051
Wang PWang XTao JZhang PGuan X(2019)Continuously Distinct Sampling over Centralized and Distributed High Speed Data StreamsIEEE Transactions on Parallel and Distributed Systems10.1109/TPDS.2018.286545230:2(300-314)Online publication date: 1-Feb-2019
https://dl.acm.org/doi/10.1109/TPDS.2018.2865452
Tang HWu YLi THan CGe JZhao X(2019)Efficient Identification of TOP-K Heavy Hitters over Sliding WindowsMobile Networks and Applications10.1007/s11036-018-1051-x24:5(1732-1741)Online publication date: 1-Oct-2019
https://dl.acm.org/doi/10.1007/s11036-018-1051-x
Show More Cited By

Index Terms

Joint data streaming and sampling techniques for detection of super sources and destinations

Recommendations

Stratified random sampling from streaming and stored data
Abstract
Stratified random sampling (SRS) is a widely used sampling technique for approximate query processing. We consider SRS on continuously arriving data streams and statically stored data sets. We present a tight lower bound showing that any streaming ...
Sampling streaming data with replacement

Simple random sampling is a widely accepted basis for estimation from a population. When data come as a stream, the total population size continuously grows and only one pass through the data is possible. Reservoir sampling is a method of maintaining a ...
Detection of Super Sources and Destinations in High-Speed Networks: Algorithms, Analysis and Evaluation

Detecting the sources or destinations that have communicated with a large number of distinct destinations or sources (i.e., large "fan-out" or "fan-in") during a small time interval is an important problem in network measurement and security. Previous ...

Comments

Information & Contributors

Information

Published In

cover image ACM Conferences

IMC '05: Proceedings of the 5th ACM SIGCOMM conference on Internet measurement

October 2005

389 pages

Sponsors

SIGCOMM: ACM Special Interest Group on Data Communication

Publisher

USENIX Association

United States

Publication History

Published: 19 October 2005

Check for updates

Qualifiers

Article

Acceptance Rates

Overall Acceptance Rate 277 of 1,083 submissions, 26%

Upcoming Conference

IMC '24

Sponsor:
sigcomm
sigcomm

ACM Internet Measurement Conference

November 4 - 6, 2024

Madrid , AA , Spain

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

13
Total Citations
View Citations
236
Total Downloads

Downloads (Last 12 months)3
Downloads (Last 6 weeks)0

Reflects downloads up to 22 Oct 2024

Other Metrics

View Author Metrics

Citations

Cited By

Aftab MChau SShenoy P(2020)Efficient Online Classification and Tracking on Resource-constrained IoT DevicesACM Transactions on Internet of Things10.1145/33920511:3(1-29)Online publication date: 13-Jul-2020
https://dl.acm.org/doi/10.1145/3392051
Wang PWang XTao JZhang PGuan X(2019)Continuously Distinct Sampling over Centralized and Distributed High Speed Data StreamsIEEE Transactions on Parallel and Distributed Systems10.1109/TPDS.2018.286545230:2(300-314)Online publication date: 1-Feb-2019
https://dl.acm.org/doi/10.1109/TPDS.2018.2865452
Tang HWu YLi THan CGe JZhao X(2019)Efficient Identification of TOP-K Heavy Hitters over Sliding WindowsMobile Networks and Applications10.1007/s11036-018-1051-x24:5(1732-1741)Online publication date: 1-Oct-2019
https://dl.acm.org/doi/10.1007/s11036-018-1051-x
Anderson DBevan PLang KLiberty ERhodes LThaler JUhlig SMaennel O(2017)A high-performance algorithm for identifying frequent items in data streamsProceedings of the 2017 Internet Measurement Conference10.1145/3131365.3131407(268-282)Online publication date: 1-Nov-2017
https://dl.acm.org/doi/10.1145/3131365.3131407
Liu YChen WGuan Y(2016)Identifying High-Cardinality Hosts from Network-Wide Traffic MeasurementsIEEE Transactions on Dependable and Secure Computing10.1109/TDSC.2015.242367513:5(547-558)Online publication date: 1-Sep-2016
https://dl.acm.org/doi/10.1109/TDSC.2015.2423675
Wellem TLai YChung WBrebner GBachmutsky ADas C(2015)A Software Defined Sketch System for Traffic MonitoringProceedings of the Eleventh ACM/IEEE Symposium on Architectures for networking and communications systems10.5555/2772722.2772755(197-198)Online publication date: 7-May-2015
https://dl.acm.org/doi/10.5555/2772722.2772755
Wellem TLi GLai YPrasanna VBrebner GKeslassy I(2014)Superspreader detection system on NetFPGA platformProceedings of the tenth ACM/IEEE symposium on Architectures for networking and communications systems10.1145/2658260.2661766(247-248)Online publication date: 20-Oct-2014
https://dl.acm.org/doi/10.1145/2658260.2661766
Cheng GTang Y(2013)Line speed accurate superspreader identification using dynamic error compensationComputer Communications10.1016/j.comcom.2013.05.00636:13(1460-1470)Online publication date: 1-Jul-2013
https://dl.acm.org/doi/10.1016/j.comcom.2013.05.006
Locher TMeyer auf der Heide FRajaraman R(2011)Finding heavy distinct hitters in data streamsProceedings of the twenty-third annual ACM symposium on Parallelism in algorithms and architectures10.1145/1989493.1989541(299-308)Online publication date: 4-Jun-2011
https://dl.acm.org/doi/10.1145/1989493.1989541
Guan XWang PQin T(2009)A new data streaming method for locating hosts with large connection degreeProceedings of the 28th IEEE conference on Global telecommunications10.5555/1811982.1812446(6421-6426)Online publication date: 30-Nov-2009
https://dl.acm.org/doi/10.5555/1811982.1812446
Show More Cited By

View Options

Get Access

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Media

Figures

Other

Tables

View Table of Contents