CPE Parameter Name Cache Re-architecture SDD
This document describes the re-architecture of the CPE parameter name cache, reducing Hazelcast memory consumption from over 90 GB to an estimated 20-25 GB by selectively caching only frequently used parameters.
1. Initial Problem
In the Bezeq environment, the Hazelcast server responsible for caching the cpe_parameter_name table required more than 90GB of RAM, with memory usage continuously growing.
1.1. What Is Cached?
The table cpe_parameter_name contains:
-
id, name, type, encrypted
Customer table size: approximately 100 million rows.
1.2. What Is Stored in This Table?
-
The table holds names of parameters and objects retrieved from customer devices over time.
-
The name field is unique.
-
The table is append-only; data is never deleted, only updated (typically the type field).
-
Names include instance numbers; each unique instance combination results in a new row.
Example:
Device.Hosts.Host.1.IPv4Address.1.IPAddress Device.Hosts.Host.1.IPv4Address.2.IPAddress Device.Hosts.Host.2.IPv4Address.1.IPAddress
These would result in three separate entries in the table.
1.3. Why Does ACS Cache This Table?
Parameter name lookup is required in 99% of device message processing, including Inform messages.
ACS stores actual device parameters in cpe_parameter (with name_id as a foreign key to cpe_parameter_name.id).
Without a cache, frequent DB lookups would significantly degrade performance.
2. Analysis of Parameter Tree Usage
A script was developed to log parameter tree branches and count entries.
Top branches by parameter count:
| Branch | Count |
|---|---|
Device.DHCPv4.Server.Pool..Client..Option.* |
22,785,478 |
Device.Hosts.Host.* |
27,507,743 |
Device.WiFi.AccessPoint..AssociatedDevice. |
7,975,572 |
Device.WiFi.DataElements. |
19,918,170 |
Device.WiFi.MultiAP.APDevice..Radio..AP.*.AssociatedDevice. |
7,846,689 |
InternetGatewayDevice.LANDevice..Hosts.Host. |
1,443,789 |
InternetGatewayDevice.LANDevice..WLANConfiguration..AssociatedDevice.* |
3,926,044 |
InternetGatewayDevice.WANDevice..WANConnectionDevice..WANPPPConnection..PortMapping. |
795,132 |
Totals:
-
Device. branch: 86,033,652 parameters
-
InternetGatewayDevice. branch: 6,164,965 parameters
Observation:
Most of these parameters are rarely used by ACS, particularly for Inform messages.
Only the first few instances (e.g., 1—10) tend to be actively used by the majority of devices.
3. Solution Overview
The core idea is to cache only frequently used parameters.
Directly switching to lazy loading (on-demand cache fill) is too risky, as it would cause a DB overload immediately after startup.
Chosen Approach:
-
Exclude large and rarely-used branches from the initial cache load.
-
Ensure dynamic on-demand loading of uncached parameters.
-
Automatically evict unused entries after a configurable idle period.
3.1. Implementation
An optional configuration file parameterNamesCacheRules.yaml must be added into "JBOSS_HOME/standalone/configuration" folder.
It defines cache loading behavior:
blacklist:
- pattern: "Device.DHCPv4.Server.Pool.*.Client.*.Option.**"
- pattern: "Device.Hosts.Host.**"
- pattern: "Device.WiFi.AccessPoint.*.AssociatedDevice.**"
- pattern: "Device.WiFi.DataElements.**"
- pattern: "Device.WiFi.MultiAP.APDevice.*.Radio.*.AP.*.AssociatedDevice.**"
- pattern: "InternetGatewayDevice.LANDevice.*.Hosts.Host.**"
- pattern: "InternetGatewayDevice.LANDevice.*.WLANConfiguration.*.AssociatedDevice.**"
- pattern: "InternetGatewayDevice.WANDevice.*.WANConnectionDevice.*.WANPPPConnection.*.PortMapping.**"Additional optional rules:
-
white-list: override blacklist exclusions (e.g., instance 1 and 2).
-
instance-limits: include parameters only up to a certain instance number.
-
global-instance-limit: global instance number limit for all branches.
Hazelcast configuration supports TTL for unused entries:
<map name="CpeParameter*">
<near-cache>
<time-to-live-seconds>600</time-to-live-seconds>
<eviction eviction-policy="NONE" size="5000"/>
</near-cache>
<max-idle-seconds>129600</max-idle-seconds> <!-- 36h -->
<eviction eviction-policy="NONE"/>
</map>The max-idle-seconds setting is critical: 36 hours corresponds to 1.5 inform periods for most customer devices.
5. Production Expectations
-
Estimated Hazelcast memory usage: 20—25 GB (approximately 25% of original)
-
Initial Hazelcast JVM settings: -Xms20G -Xmx90G
|
These estimates are based on the assumption that most parameters are not used frequently (e.g., not accessed in >36h). Only real-time monitoring in production can validate this. |