1. Data collection methodology and NM paradigms
part3

2. Outline Mediation Device Functionality: How to Process Data Records
Filtering
Estimation from Sampling
Threshold Monitoring
Data Record Correlation and Enrichment
Flow De-Duplication
Data Record Formatting and Storage

3. filtering Two main areas of filtering exist at the mediation device:
Filtering to reduce the volume of the data collection
Filtering for application purposes

4. Filtering to reduce the volume of the data collection
Complex filtering for volume reduction is a mediation device task, because the implementation of process-intensive filters at the network element has a performance impact
Ideally, the collection granularity and filtering functions at the device would allow for configuring exactly the data set that is required

5. Filtering To reduce the performance impact of metering to a minimum:
Simple filters are implemented at the device, ideally in hardware instead of software operations.
Complex and CPU-intensive operations are realized at the mediation device.

6. Filtering for application purposes
Filtering for application purposes is based on the "divide and conquer" paradigm: collect data records once, but use them as input for multiple applications, such as capacity planning, billing, and security monitoring

7. Filtering example

8. Estimation from Sampling
This section is specific to NetFlow and applies only if NetFlow sampling is configured at the device.
It is an important task, because the child population gathered by sampling must be adjusted for the estimation of the parent population and to deduce an approximation of the volume based on the sampling rate.

9. Estimation from Sampling
A NetFlow mediation device estimates the absolute traffic volumes by renormalizing the volume of sampled traffic through multiplication with the meter's sampling frequency
Example: If sampling is applied with a sampling rate of 1:100, the data records need to be multiplied by a factor of 100.
Absolute traffic volume=data records * sampling factor

10. Estimation from Sampling
The example in the section "Filtering at the Network Device" defined three traffic classes: priority, business, and best effort. Priority traffic was fully collected and needs no adjustment. Business traffic was sampled with a rate of 1:100 and therefore needs to be multiplied by a factor of 100. Best-effort traffic had a sampling rate of 1:1000 and needs to be multiplied by 1000.

11. Estimation from Sampling
The sampling factor must be included in the exported data record, because otherwise the estimation from sampling is incorrect.

12. Threshold Monitoring
It can be implemented at both the mediation device and the application server level
It is not so relevant where the function is located, but which purpose it serves

13. Threshold Monitoring in application server level
A metering device for a traffic planning application might leave the monitoring of the threshold up to the application server, because they are not critical for planning purposes

14. Threshold Monitoring at mediation device
On the other hand, if metering is applied for security monitoring, a relevant feature is to set thresholds for the received traffic and monitor them in real time, because a reaction to an attack must occur quickly
Exceeded thresholds can identify security issues, such as a denial-of-service (DoS) attack, in which a huge number of very small datagrams flood a network and eventually stop the services in the network

15. Data Aggregation
The concept of aggregation describes the task of reducing the granularity by identifying common criteria (key fields) and combining information from multiple records into a single record.

16. Aggregation concepts Two different aggregation concepts exist:
Aggregation of key fields
Aggregation over time

17. Aggregation of key fields
Aggregation of common criteria is related specifically to accounting records, whereas aggregation over time can be applied to both accounting and performance records

18. Aggregation Example

20. Data Record Correlation and Enrichment
Another task at the mediation layer is correlating information from different metering sources to enrich the data records.
Example: Modifying a data record by correlating the record details with DNS information to replace an IP address with a username

21. Correlation benefit
Grouping information from different sources into a common data records is a clear benefit of upper-layer applications, such as billing, which can retrieve enriched data sets instead of very basic sets, which need correlation afterwards.

22. Flow De-Duplication
Duplicate records lead to inaccurate results at the application level; therefore, these duplications need to be eliminated.

23. Flow De-Duplication steps
The following steps are performed:
Identify common flow parameters, such as source and destination address, port numbers, AS number, ToS/DHCP fields, and others
Check the time-stamps
Associate the information and eliminate duplicate flows

24. Data Record Formatting and Storage
Finally, the processed data records are stored in a database and made available to other applications.
Records have to describe usage type details, such as keys and values, where a key links to an index in a database table.

25. Common data format
A common data format definition protects the NMS and OSS applications from the variety of accounting formats that are implemented at the device level

26. Records storage
The location where the records are stored can be a simple flat file system, where a new folder is created for each device and subfolder per interface, with separate text files for each aggregation scheme and interval. Alternatively, the data store can be a complex relational database system that provides sophisticated operations for local and remote access.

27. Record format
The records are implemented as XML schemas with self-defining field attributes, including five major attributes: who, what, where, when, and why
Who?— Responsible user ID
When?— Time when the usage took place
What?— Service description and consumed resources
Where?— Source and destination ID
Why?— Reason for reporting the event

28. Summery
Figure 2-27 summarizes all mediation device functions in a flow chart. As mentioned, some tasks, such as threshold monitoring and aggregation over time, can be applied at the upper-layer application level instead of the mediation device.

29. Figure 2-27

30. Abbreviations Meaning DoS denial-of-service ToS Terms of Service AS
Autonomous System
DHCP
Dynamic Host Configuration Protocol
OSS
Operations support systems
XML
EXtensible Markup Language