Concepts

1. System Overview

Friendly QoE Monitoring is a part of the Friendly IoT Platform, which graphically represents monitoring data of various types of IT equipment by means of its web UI. It is capable of measuring a quality of service of Data, VoIP and IPTV services at the subscriber’s endpoint, and enables monitoring of single devices or groups of devices.

The system consists of:

  • Management Portal — Responsible for definition and activation/deactivation of monitoring

  • Device (CPE) — Responsible for delivery of monitoring data to IoT ACS

  • IoT ACS (or Cluster) — Responsible for collection and storage of monitoring data

  • QoE Monitoring — Responsible for collection, organization, storage and presentation of monitoring data

QoE Monitoring can operate with a single IoT ACS node or an IoT ACS cluster (multiple nodes behind a load balancer).

2. Monitoring Types

There are three types of monitoring:

2.1. Dynamic Monitoring

Monitoring of a dynamic list of devices per ProductClassGroup (a unique combination of Manufacturer + ProductClass). Any new device that connects to the IoT ACS while a dynamic monitor is active for the same ProductClassGroup will be monitored automatically.

2.2. Dynamic Monitoring Based on Group Filter

Monitoring of a dynamic list of devices per ProductClassGroup and a custom group filter (e.g., firmware version). Only new devices that match the group filter criteria will be added to monitoring automatically.

2.3. Standard Monitoring

Monitoring of a fixed list of devices. Only the devices selected at the time of monitor activation will be monitored. New devices connecting to the IoT ACS will not be added automatically.

Each device can participate in any number of monitors. If the same parameter is monitored by both a dynamic and a fixed monitor and one is deactivated, monitoring continues via the remaining monitor. The resulting monitoring interval equals the interval of the last activated monitor.

3. Monitor Activation & Deactivation

3.1. Activation

  1. A monitor is activated via the Management Portal (Activate or Save and Activate)

  2. Each relevant device receives a QoE: set parameter notification task for monitored parameters (passive notification)

  3. Each relevant device receives a QoE: set periodic inform interval task, setting the PeriodicInformInterval to the monitoring interval

3.2. Deactivation

  1. A monitor is deactivated via the Management Portal

  2. Each relevant device receives a task to restore PeriodicInformInterval to the value from the provision or profile

  3. Each relevant device receives a task to set parameter notification to off

4. Data Collection & Processing

4.1. Data Flow

  1. Device reports data — Devices send Inform messages to IoT ACS at configured intervals

  2. IoT ACS collects data — Extracts monitored parameter values from Inform messages; sends Get Parameter Values for missing data

  3. IoT ACS stores data — Writes data to CSV files and database tables

  4. QoE Monitoring processes data — Reads data from IoT ACS, calculates KPI values, stores results in ClickHouse

  5. QoE Monitoring presents data — Displays KPI graphs, alarms, and reports via the web UI

4.2. Device Data Collection Modes

Devices fall into three categories based on passive notification support:

Device Type Behavior

Supports passive notification

Device includes changed parameter values in Inform messages. IoT ACS extracts data directly.

Does not support passive notification

IoT ACS sends Get Parameter Values on each PeriodicInformInterval. Device model must be listed in gpv_models.csv.

Partial notification support

Device includes some parameters in Inform. IoT ACS sends Get Parameter Values for missing parameters.

5. ClickHouse Database Structure

The ClickHouse database (ftacs_qoe_ui_data) stores all time-series data. Tables are organized by function. For detailed column descriptions, engine types, and partitioning, see Database > ClickHouse.

5.1. Monitoring Data

Table Description

cpe_data

Raw parameter values received from devices (timestamp, serial, name_id, value, periodic, location_id, orig_name_id)

kpi_data

Calculated KPI values displayed on graphs (timestamp, serial, kpi_id, value, periodic, kpi_value_num, name_id)

kpi_data_in_alarm

KPI data points that triggered an alarm, with severity level and group_id

cpe_monitor_history

Monitor start/stop timestamps per device and parameter

user_activity

User action audit log (username, activity type, session, description)

schema_version

Database schema version tracking

5.2. Alarm Data

Table Description

kpi_threshold_history_repl

Single device alarm history (threshold ID, serial, KPI, severity level, state)

serials_in_group_alarm

Devices participating in group alarms (group_id, kpi_id, serials array, count)

5.3. Diagnostic Data

Table Description

diag_speed_test

Upload/Download speed test results — FCC diagnostics (value, value_total, file_size, test_bytes, total_bytes, num_conn)

diag_ip_ping

IPPing diagnostic results — FCC diagnostics (success count, failure count, average response time)

diag_trace / diag_trace_hops

Trace route results with hop-by-hop details (error codes, host addresses, RTT times)

diag_udp_echo / diag_udp_echo_packet

UDP echo diagnostic results with individual packet data (send/receive timestamps, sequence numbers)

5.4. Wi-Fi Data

Table Description

wifi_cpe_data

Wi-Fi Neighboring Diagnostic results and channel change history (channel, changing state, threshold SSID/signal)

wifi_collisions

Detected Wi-Fi channel collisions (SSID, channel, signal strength)

wifi_cpe_diagnostic_sent

Wi-Fi diagnostic task tracking (sent/completed timestamps, gpv_sent flag)

5.5. User Experience Data

Table Description

user_exp_host

User experience host data (MAC, interface type, layer1/3, address, active status)

user_exp_assoc_device

User experience associated device data (MAC, RSSI, signal strength)

host_data

Connected hosts data — ETL (MAC, IPv4/IPv6, connectivity type, SSID, BSSID, RSSI)

router_data

Router device data — ETL (MAC, WAN IPv4/IPv6, SSIDs, BSSIDs, byte counters)

5.6. Cached Data from Main DB

Table Description

ft_cpe_info

Device product data (serial, group_id, product_class, manufacturer). Engine: ReplacingMergeTree.

ft_qoe_cpe_info

QoE CPE data (serial, MAC, periodic, last connection). Engine: Join(ANY, LEFT, serial).

ft_cust_device

Account information (location, telephone, user ID, coordinates, cust1-cust20). Engine: Join(ANY, LEFT, serial).

ft_cpe_domain_info

Domain data (device ID, ISP ID, serial, ISP name). Engine: Join(ANY, LEFT, serial).

5.7. Materialized Views

View Description

kpi_data_aggregated

Pre-aggregated KPI data in 5-minute intervals for Device Groups page (AggregatingMergeTree)

kpi_data_latest

Latest KPI value per device. Auto-updated on kpi_data inserts (ReplacingMergeTree)

cpe_data_latest

Latest raw parameter value per device. Auto-updated on cpe_data inserts (ReplacingMergeTree)

Tables with Join engine are populated from the Main DB via JDBC. Temporary tables (tmp_ft_*) with Memory engine are used during the sync process. Cached data is updated every 30 seconds; device data is cleaned up every 2 hours after removal.

6. Advanced Features

6.1. FCC Diagnostics

FCC diagnostics measure network throughput by running IPPing, Download, and Upload diagnostics on devices. Activated via Group Update from Management Portal with the Add results to monitoring data option.

Results are stored in ClickHouse tables (diag_speed_test, diag_ip_ping) and displayed as four KPI graphs:

  • DownloadDiagnostic — Download speed (Mbps)

  • UploadDiagnostic — Upload speed (Mbps)

  • IPPingDiagnostic — Average response time

  • SuccessCount — Ping success rate (%)

6.2. Wi-Fi Neighboring Diagnostic

Automatic Wi-Fi channel rescan, activated by monitoring the NeighboringWiFiDiagnostic.DiagnosticsState parameter. When channel collisions are detected and signal strength exceeds the configured threshold, IoT ACS initiates an automatic channel change.

Four KPI graphs are generated:

  • Wi-Fi collisions for 2.4GHz and 5GHz networks

  • Channel change history for 2.4GHz and 5GHz networks

6.3. Trace Diagnostic

Measures network path to a target host. Results (hop-by-hop data) are stored in ClickHouse (diag_trace, diag_trace_hops) with no visual representation in the QoE UI.

6.4. UDP Echo Diagnostic

Measures UDP packet round-trip performance. Results are stored in ClickHouse (diag_udp_echo, diag_udp_echo_packet) with no visual representation in the QoE UI.

7. Emergency QoE Stop

The emergency stop mechanism allows disabling QoE monitoring when the IoT ACS is overloaded. It is controlled via the force_qoe_stop.properties file placed in the IoT ACS configuration directory.

Parameter Effect when set to 1

force.qoe.stop.all

Blocks QoE processing, truncates parameter cache, deactivates all monitors, stops data collection

force.qoe.stop.requests_block

Blocks QoE processing, passes all requests to IoT ACS. No restart needed to resume.

force.qoe.stop.truncate_db

Truncates the qoe_cpe_parameter table and memory cache. Requires monitor restart.

force.qoe.stop.perform_tasks

Generates notification-off and PeriodicInformInterval-restore tasks. Requires monitor restart.

force.qoe.stop.update_monitoring_states

Changes all monitors from Running to Not Active. Requires monitor restart.

force.qoe.stop.inform_enable_white_list

Limits stored Inform parameters to mandatory + whitelist only. No restart needed.

force.qoe.stop.inform_white_list

Comma-separated list of parameter names to preserve (used with inform_enable_white_list)

The force_qoe_stop.properties file does not exist by default and must be created manually when needed.
← Back | Main Page