Architecture Documentation
System Overview
The FT Device Network Service is a Spring Boot application designed to collect, process, and store network-related data from devices. The system integrates with the ACS to receive device parameters, processes them through a structured pipeline, and stores calculated metrics in ClickHouse for efficient querying. The processed data is made available through REST APIs for monitoring dashboards.
Key Components
-
Parameter Ingestion: Receives device parameters from ACS through Hazelcast Queue (future Kafka integration)
-
Data Processing Pipeline: Processes device parameters when all required data sets are available
-
Data Storage: Persists processed metrics in ClickHouse for efficient analytics
-
REST API: Provides endpoints for retrieving device network metrics by serial number
Data Processing Workflow
-
Network devices send parameter data to the main ACS service
-
ACS records parameters in the cpe_parameters table
-
ACS forwards these parameters to this service via Hazelcast Queue (
networkDeviceParameterDataQueue) -
The service processes incoming messages when all required parameter types are available
-
Processed data is stored in ClickHouse for efficient querying
-
UI clients can request device data through the REST API using the device serial number
Required Parameters
The service requires the following parameter types to be present before processing can begin:
| Parameter Path | Description |
|---|---|
|
Information about connected hosts and clients |
|
Wi-Fi configuration, SSIDs, and connected clients |
|
Ethernet interface configuration and statistics |
|
Fiber interface configuration and statistics |
|
Mobile interface configuration and statistics |
|
DSL interface configuration and statistics |
|
IP interface configuration and addressing |
|
Basic device information including serial number and model |
When parameters are received, they are stored in the networkDeviceParameters cache until all required parameter types are available.
Once complete, the full processing pipeline begins.
Key Features
-
Integration with ACS service via Hazelcast Queue (future Kafka support)
-
Parameter caching until the complete data set is available
-
Efficient data storage using ClickHouse
-
RESTful API for data access by device serial number
-
Comprehensive device network metrics calculation
-
Formatted responses for monitoring dashboard UIs
-
Health monitoring and metrics
-
Swagger API documentation
Data Processing
Parameter Processing Pipeline
-
Parameter Ingestion: Parameters are received from the Hazelcast Queue and validated
-
Parameter Caching: Incomplete parameter sets are cached until all required types are available
-
Data Parsing: When complete, parameters are parsed using specialized parsers:
-
WifiParser: Processes WiFi configuration, SSIDs, and clients -
InterfacesParser: Processes network interfaces (Ethernet, IP, Optical, DSL) -
HostsParser: Processes connected host information -
DeviceInfoParser: Processes basic device information
-
-
Data Mapping: Parsed data is mapped to internal domain objects using specialized mappers:
-
NetworkDevicesMapper: Maps network client data -
WiFiStatusMapper: Maps WiFi band information -
InterfacesMapper: Maps interface data (Ethernet, IP, etc.) -
HostInfoMapper: Maps connected host information
-
-
Metric Calculation: Various network metrics and scores are calculated
-
Data Storage: Results are stored in ClickHouse tables