1. PART3 Data collection methodology and NM paradigms 1

2. 2 Mediation Device Functionality: How to Process Data Records  Filtering  Estimation from Sampling  Threshold Monitoring  Data Record Correlation and Enrichment  Flow De-Duplication  Data Record Formatting and Storage Outline

3. filtering 3 Two main areas of filtering exist at the mediation device: Filtering to reduce the volume of the data collection Filtering for application purposes

4. Filtering to reduce the volume of the data collection 4 Complex filtering for volume reduction is a mediation device task, because the implementation of process-intensive filters at the network element has a performance impact Ideally, the collection granularity and filtering functions at the device would allow for configuring exactly the data set that is required

5. Filtering 5 To reduce the performance impact of metering to a minimum:  Simple filters are implemented at the device, ideally in hardware instead of software operations.  Complex and CPU-intensive operations are realized at the mediation device.

6. Filtering for application purposes 6 Filtering for application purposes is based on the "divide and conquer" paradigm: collect data records once, but use them as input for multiple applications, such as capacity planning, billing, and security monitoring

7. Filtering example 7

8. Estimation from Sampling 8 This section is specific to NetFlow and applies only if NetFlow sampling is configured at the device. It is an important task, because the child population gathered by sampling must be adjusted for the estimation of the parent population and to deduce an approximation of the volume based on the sampling rate.

9. 9 A NetFlow mediation device estimates the absolute traffic volumes by renormalizing the volume of sampled traffic through multiplication with the meter's sampling frequency Example: If sampling is applied with a sampling rate of 1:100, the data records need to be multiplied by a factor of 100. Absolute traffic volume=data records * sampling factor Estimation from Sampling

10. 10 The example in the section "Filtering at the Network Device" defined three traffic classes: priority, business, and best effort. Priority traffic was fully collected and needs no adjustment. Business traffic was sampled with a rate of 1:100 and therefore needs to be multiplied by a factor of 100. Best-effort traffic had a sampling rate of 1:1000 and needs to be multiplied by 1000. Estimation from Sampling

11. 11 The sampling factor must be included in the exported data record, because otherwise the estimation from sampling is incorrect. Estimation from Sampling

12. Threshold Monitoring 12 It can be implemented at both the mediation device and the application server level It is not so relevant where the function is located, but which purpose it serves

13. Threshold Monitoring in application server level 13 A metering device for a traffic planning application might leave the monitoring of the threshold up to the application server, because they are not critical for planning purposes

14. Threshold Monitoring at mediation device 14 On the other hand, if metering is applied for security monitoring, a relevant feature is to set thresholds for the received traffic and monitor them in real time, because a reaction to an attack must occur quickly Exceeded thresholds can identify security issues, such as a denial-of-service (DoS) attack, in which a huge number of very small datagrams flood a network and eventually stop the services in the network

15. Data Aggregation 15 The concept of aggregation describes the task of reducing the granularity by identifying common criteria (key fields) and combining information from multiple records into a single record.

16. Aggregation concepts 16 Two different aggregation concepts exist: Aggregation of key fields Aggregation over time

17. Aggregation of key fields 17 Aggregation of common criteria is related specifically to accounting records, whereas aggregation over time can be applied to both accounting and performance records

18. Aggregation Example 18

19. 19

20. Data Record Correlation and Enrichment 20 Another task at the mediation layer is correlating information from different metering sources to enrich the data records. Example: Modifying a data record by correlating the record details with DNS information to replace an IP address with a username

21. Correlation benefit 21 Grouping information from different sources into a common data records is a clear benefit of upper- layer applications, such as billing, which can retrieve enriched data sets instead of very basic sets, which need correlation afterwards.

22. Flow De-Duplication 22 Duplicate records lead to inaccurate results at the application level; therefore, these duplications need to be eliminated.

23. Flow De-Duplication steps 23 The following steps are performed: Identify common flow parameters, such as source and destination address, port numbers, AS number, ToS/DHCP fields, and others Check the time-stamps Associate the information and eliminate duplicate flows

24. Data Record Formatting and Storage 24 Finally, the processed data records are stored in a database and made available to other applications. Records have to describe usage type details, such as keys and values, where a key links to an index in a database table.

25. Common data format 25 A common data format definition protects the NMS and OSS applications from the variety of accounting formats that are implemented at the device level

26. Records storage 26 The location where the records are stored can be a simple flat file system, where a new folder is created for each device and subfolder per interface, with separate text files for each aggregation scheme and interval. Alternatively, the data store can be a complex relational database system that provides sophisticated operations for local and remote access.

27. Record format 27 The records are implemented as XML schemas with self-defining field attributes, including five major attributes: who, what, where, when, and why  Who?— Responsible user ID  When?— Time when the usage took place  What?— Service description and consumed resources  Where?— Source and destination ID  Why?— Reason for reporting the event

28. Summery 28 Figure 2-27 summarizes all mediation device functions in a flow chart. As mentioned, some tasks, such as threshold monitoring and aggregation over time, can be applied at the upper-layer application level instead of the mediation device.

29. Figure 2-27 29

30. Abbreviations 30 Meaning DoSdenial-of-service ToSTerms of Service ASAutonomous System DHCPDynamic Host Configuration Protocol OSSOperations support systems XMLEXtensible Markup Language