Architecture Evolution Plan: Monolith to Modular Services

Table of Contents

1. Executive Summary
2. Current Architecture (As-Is)
- 2.1. Key Problems in Current Architecture
3. Target Architecture (To-Be)
4. Transformation Roadmap
5. Phase Details
6. Cross-Phase Concerns
7. Risk Matrix
8. Decision Log
9. Phase Completion Checklist
10. Appendix A: Current Hazelcast Cache Inventory
11. Appendix B: Event Type Inventory

1. Executive Summary

This document describes the phased evolution of OneIoT ACS from a monolithic architecture to a modular, independently scalable service-oriented system. The plan addresses six strategic goals:

Decrease complexity — Break the 80+ Hazelcast cache maps, 162 SOAP methods, and circular module dependencies into bounded contexts
Prepare for test coverage — Introduce testable interfaces, contract tests, and per-module unit test isolation
Improve performance — Replace in-process event bus and Hazelcast with Kafka + Redis for predictable throughput
Better load balancing — Enable per-service scaling instead of scaling the entire monolith
Performance formula — Isolate services so capacity can be modeled as capacity = f(service_instances, scenario_throughput)
Stability — Each phase is independently deployable and reversible

2. Current Architecture (As-Is)

2.1. Key Problems in Current Architecture

Problem Description Severity

Problem	Description	Severity
God Objects	`ACSWebService` has 124 methods spanning all domains. `HazelcastCacheFactory` is a single factory for 80+ cache maps.	HIGH
Circular Dependencies	`ftdm-qoe` depends on `ftdm-update-group` and vice versa through transitive dependencies.	HIGH
No Module Isolation	Every module depends on `ftdm-orm` (50+ entities). No service interfaces — direct class coupling.	HIGH
In-Process Events	`DefaultAsyncEventBusImpl` is a `ConcurrentHashMap` + thread pool. Events are lost on crash. Cannot be consumed by external services.	MEDIUM
SOAP Node-to-Node	`SoapNodeNotifier` calls `BroadcastingWebService` via JAX-WS for cluster events — brittle and unscalable.	MEDIUM
Hazelcast Overloaded	Used simultaneously as cache, distributed lock manager, pub/sub message bus, and task executor.	HIGH
Untestable	Modules cannot be unit-tested in isolation due to shared Hazelcast state and tight coupling.	HIGH
No Performance Formula	All services run in one JVM — impossible to attribute CPU/memory to specific scenarios.	HIGH

God Objects

ACSWebService has 124 methods spanning all domains. HazelcastCacheFactory is a single factory for 80+ cache maps.

HIGH

Circular Dependencies

ftdm-qoe depends on ftdm-update-group and vice versa through transitive dependencies.

HIGH

No Module Isolation

Every module depends on ftdm-orm (50+ entities). No service interfaces — direct class coupling.

HIGH

In-Process Events

DefaultAsyncEventBusImpl is a ConcurrentHashMap + thread pool. Events are lost on crash. Cannot be consumed by external services.

MEDIUM

SOAP Node-to-Node

SoapNodeNotifier calls BroadcastingWebService via JAX-WS for cluster events — brittle and unscalable.

MEDIUM

Hazelcast Overloaded

Used simultaneously as cache, distributed lock manager, pub/sub message bus, and task executor.

HIGH

Untestable

Modules cannot be unit-tested in isolation due to shared Hazelcast state and tight coupling.

HIGH

No Performance Formula

All services run in one JVM — impossible to attribute CPU/memory to specific scenarios.

HIGH

3. Target Architecture (To-Be)

4. Transformation Roadmap

5. Phase Details

5.1. Phase 1: Shared Libraries & Clean Interfaces

5.1.1. Objective

Extract shared entities and DTOs into standalone libraries. Define service interface contracts (API JARs) for each domain module. Break circular dependencies.

5.1.2. Transformation Diagram

5.1.3. Changes

Change Details

Change	Details
Create `shared-models`	Extract from `ftdm-orm`: `Cpe`, `CpeParameter`, `CpeParameterName`, `ProductClassGroup`, `Isp`, `CPEPendingTask`, `CPECompletedTask`, `BaseEntity` and all entities used by 3+ modules. Keep JPA annotations — consumers use the same entities.
Create `*-api` interface JARs	For each domain: `events-api`, `update-group-api`, `qoe-api`, `provision-api`, `core-api`. Each JAR contains Java interfaces + DTOs only (no implementation).
Break `qoe` ↔ `update-group` cycle	`ftdm-qoe` depends on `update-group-api` (interface) instead of `ftdm-update-group` (implementation). Wire via Spring `@Autowired`.
Refactor `HazelcastCacheFactory`	Split into domain-specific cache interfaces: `DeviceCacheService`, `ParameterCacheService`, `SecurityCacheService`, `EventCacheService`. Hazelcast remains the implementation but is hidden behind interfaces.

Create shared-models

Extract from ftdm-orm: Cpe, CpeParameter, CpeParameterName, ProductClassGroup, Isp, CPEPendingTask, CPECompletedTask, BaseEntity and all entities used by 3+ modules. Keep JPA annotations — consumers use the same entities.

Create *-api interface JARs

For each domain: events-api, update-group-api, qoe-api, provision-api, core-api. Each JAR contains Java interfaces + DTOs only (no implementation).

Break qoe ↔ update-group cycle

ftdm-qoe depends on update-group-api (interface) instead of ftdm-update-group (implementation). Wire via Spring @Autowired.

Refactor HazelcastCacheFactory

Split into domain-specific cache interfaces: DeviceCacheService, ParameterCacheService, SecurityCacheService, EventCacheService. Hazelcast remains the implementation but is hidden behind interfaces.

5.1.4. Impact Assessment

Area Impact Description Risk

Area	Impact	Description	Risk
Codebase	HIGH	Every module’s POM changes; imports shift from `ftdm-orm` to `shared-models` + `*-api`	LOW — compile-time changes only
Runtime behavior	NONE	No behavioral change — same code runs, just reorganized	NONE
Build pipeline	MEDIUM	New Maven modules, changed build order, new artifact publishing	LOW
Database	NONE	No schema changes	NONE
Deployment	NONE	Still a single WAR — internal structure only	NONE

Codebase

HIGH

Every module’s POM changes; imports shift from ftdm-orm to shared-models + *-api

LOW — compile-time changes only

Runtime behavior

NONE

No behavioral change — same code runs, just reorganized

NONE

Build pipeline

MEDIUM

New Maven modules, changed build order, new artifact publishing

LOW

Database

NONE

No schema changes

NONE

Deployment

NONE

Still a single WAR — internal structure only

NONE

5.1.5. Test Coverage Plan

Test Type What to Cover Tools

Test Type	What to Cover	Tools
Compilation tests	Verify all modules compile with new dependency graph. Verify no missing imports after extraction.	`mvn compile` in CI for every module
Interface contract tests	Verify every `*-api` interface has at least one implementation. Verify method signatures match existing usage.	JUnit 5 — one test per interface that instantiates the implementation
Dependency graph validation	Assert no circular dependencies exist. Assert `*-api` modules have zero implementation dependencies.	Maven Enforcer Plugin (`<banCircularDependencies>`) + `maven-dependency-plugin:analyze`
Integration smoke tests	Full application startup with the new module structure. Verify all Spring beans wire correctly.	Spring Boot integration test (`@SpringBootTest` with `ApplicationContext` assertion)
Regression pack	Run all existing tests (if any) to confirm zero behavioral change.	Existing test suite

Compilation tests

Verify all modules compile with new dependency graph. Verify no missing imports after extraction.

mvn compile in CI for every module

Interface contract tests

Verify every *-api interface has at least one implementation. Verify method signatures match existing usage.

JUnit 5 — one test per interface that instantiates the implementation

Dependency graph validation

Assert no circular dependencies exist. Assert *-api modules have zero implementation dependencies.

Maven Enforcer Plugin (<banCircularDependencies>) + maven-dependency-plugin:analyze

Integration smoke tests

Full application startup with the new module structure. Verify all Spring beans wire correctly.

Spring Boot integration test (@SpringBootTest with ApplicationContext assertion)

Regression pack

Run all existing tests (if any) to confirm zero behavioral change.

Existing test suite

Key tests to add in Phase 1

// Dependency graph test (maven-enforcer or custom)
@Test void noCircularDependencies()  // Verify DAG in module graph

// Interface contract tests
@Test void eventServiceInterfaceHasImplementation()
@Test void updateGroupServiceInterfaceHasImplementation()
@Test void qoeServiceInterfaceHasImplementation()
@Test void provisionApiInterfaceHasImplementation()

// Startup smoke test
@SpringBootTest
@Test void applicationContextLoads()

// Cache interface tests
@Test void deviceCacheServiceReturnsCorrectTypes()
@Test void parameterCacheServiceReturnsCorrectTypes()

5.1.6. Rollback Strategy

All changes are compile-time only. Rollback = revert the Maven POM changes and restore direct dependencies. Zero runtime risk.

5.2. Phase 2: Device Flow Layering

5.2.1. Objective

Extract the common device communication pipeline into explicit, protocol-agnostic layers. Each protocol (TR-069, MQTT, USP) becomes a thin adapter over a shared pipeline.

5.2.2. Transformation Diagram

5.2.3. Changes

Change Details

Change	Details
Define `DeviceSession` interface	Protocol-agnostic session with methods: `getDeviceId()`, `getParameters()`, `sendRequest(CpeRequest)`, `receiveResponse()`, `getProtocolType()`. Each protocol implements this.
Extract `BootstrapHandler`	Consolidate `CommonOperations.createOrUpdateDevice()` — currently duplicated in TR-069 `InformHandler`, MQTT `MqttCpeService.registerDevice()`, USP `UspCpeRegistrationService.register()`.
Extract `TaskOrchestrator`	Consolidate `CpeTaskProcessor` + protocol-specific executors (`TR069ResponseHandler`, `MqttCpeTasksExecutor`, `UspQueuedTasksExecutor`) into a single orchestrator that uses `DeviceSession.sendRequest()`.
Extract `ParameterPersistor`	Consolidate `BulkSQLProcessorCpeParameter` + event firing (`CpeParameterChangedEvent`) into a standalone component.
Thin adapters	TR-069: Parse CWMP SOAP → `DeviceSession`. MQTT: Topic-based RPC → `DeviceSession`. USP: Protobuf → `DeviceSession`. Protocol-specific logic stays in adapters.

Define DeviceSession interface

Protocol-agnostic session with methods: getDeviceId(), getParameters(), sendRequest(CpeRequest), receiveResponse(), getProtocolType(). Each protocol implements this.

Extract BootstrapHandler

Consolidate CommonOperations.createOrUpdateDevice() — currently duplicated in TR-069 InformHandler, MQTT MqttCpeService.registerDevice(), USP UspCpeRegistrationService.register().

Extract TaskOrchestrator

Consolidate CpeTaskProcessor + protocol-specific executors (TR069ResponseHandler, MqttCpeTasksExecutor, UspQueuedTasksExecutor) into a single orchestrator that uses DeviceSession.sendRequest().

Extract ParameterPersistor

Consolidate BulkSQLProcessorCpeParameter + event firing (CpeParameterChangedEvent) into a standalone component.

Thin adapters

TR-069: Parse CWMP SOAP → DeviceSession. MQTT: Topic-based RPC → DeviceSession. USP: Protobuf → DeviceSession. Protocol-specific logic stays in adapters.

5.2.4. Impact Assessment

Area Impact Description Risk

Area	Impact	Description	Risk
Codebase	HIGH	Refactors 3 protocol modules + `ftdm-core`. New pipeline package with 6 handler classes.	MEDIUM — behavioral changes in hot path
Runtime behavior	LOW	Same logic, different call path. Must verify no regression in device session flows.	MEDIUM — integration risk
Build pipeline	LOW	New module or package within existing module	LOW
Database	NONE	No schema changes	NONE
Deployment	NONE	Still single WAR	NONE

Codebase

HIGH

Refactors 3 protocol modules + ftdm-core. New pipeline package with 6 handler classes.

MEDIUM — behavioral changes in hot path

Runtime behavior

LOW

Same logic, different call path. Must verify no regression in device session flows.

MEDIUM — integration risk

Build pipeline

LOW

New module or package within existing module

LOW

Database

NONE

No schema changes

NONE

Deployment

NONE

Still single WAR

NONE

5.2.5. Test Coverage Plan

Test Type What to Cover Tools

Test Type	What to Cover	Tools
Unit tests per pipeline stage	Each handler tested in isolation with mocked dependencies. Test `BootstrapHandler` with mock `DeviceSession`, assert `Cpe` entity created/updated correctly.	JUnit 5 + Mockito
Protocol adapter tests	Verify each adapter correctly translates protocol-specific messages to `DeviceSession` interface calls. Test CWMP Inform → `DeviceSession.getParameters()`, MQTT topic → `DeviceSession.sendRequest()`.	JUnit 5 + mock protocol messages
End-to-end device session tests	Full pipeline execution for each protocol: bootstrap → inform → profile → tasks → persist → finalize.	Integration tests with embedded database (H2) + mock Hazelcast
Regression: TR-069 flows	All existing TR-069 scenarios: first inform, periodic inform, connection request, firmware download, value change.	TR-069 emulator + integration test
Regression: MQTT flows	Device registration, parameter reporting, task execution via MQTT.	MQTT emulator + integration test
Regression: USP flows	USP registration, CRUD operations, subscription notifications.	USP emulator + integration test
Performance baseline	Measure inform processing time before/after refactoring. Must not regress.	JMH microbenchmarks on `BootstrapHandler` + `ParameterPersistor`

Unit tests per pipeline stage

Each handler tested in isolation with mocked dependencies. Test BootstrapHandler with mock DeviceSession, assert Cpe entity created/updated correctly.

JUnit 5 + Mockito

Protocol adapter tests

Verify each adapter correctly translates protocol-specific messages to DeviceSession interface calls. Test CWMP Inform → DeviceSession.getParameters(), MQTT topic → DeviceSession.sendRequest().

JUnit 5 + mock protocol messages

End-to-end device session tests

Full pipeline execution for each protocol: bootstrap → inform → profile → tasks → persist → finalize.

Integration tests with embedded database (H2) + mock Hazelcast

Regression: TR-069 flows

All existing TR-069 scenarios: first inform, periodic inform, connection request, firmware download, value change.

TR-069 emulator + integration test

Regression: MQTT flows

Device registration, parameter reporting, task execution via MQTT.

MQTT emulator + integration test

Regression: USP flows

USP registration, CRUD operations, subscription notifications.

USP emulator + integration test

Performance baseline

Measure inform processing time before/after refactoring. Must not regress.

JMH microbenchmarks on BootstrapHandler + ParameterPersistor

Key tests to add in Phase 2

// DeviceSession contract test
@Test void tr069AdapterImplementsDeviceSession()
@Test void mqttAdapterImplementsDeviceSession()
@Test void uspAdapterImplementsDeviceSession()

// Pipeline unit tests
@Test void bootstrapHandler_newDevice_createsEntity()
@Test void bootstrapHandler_existingDevice_updatesLastConnect()
@Test void informProcessor_detectsParameterChanges()
@Test void profileApplicator_firstConnect_appliesProfile()
@Test void profileApplicator_subsequentConnect_skips()
@Test void taskOrchestrator_pendingTasks_executesInOrder()
@Test void taskOrchestrator_noTasks_completesGracefully()
@Test void parameterPersistor_bulkSave_firesChangeEvents()

// End-to-end per protocol
@Test void tr069_fullBootstrapToFinalize()
@Test void mqtt_fullBootstrapToFinalize()
@Test void usp_fullBootstrapToFinalize()

// Performance regression
@Test void informProcessingTime_doesNotRegress()

5.2.6. Rollback Strategy

Pipeline handlers delegate to the same underlying services. Rollback = restore direct calls from protocol modules to CommonOperations/CpeTaskProcessor. Feature flag can toggle between old and new paths during transition.

5.3. Phase 3: SOAP to REST API Modernization

5.3.1. Objective

Decompose the 162-method SOAP god object into domain-specific REST controllers with OpenAPI documentation. Replace internal SOAP-based node-to-node communication.

5.3.2. Transformation Diagram

5.3.3. SOAP Method Decomposition

REST Controller Methods from SOAP Estimated Endpoints

REST Controller	Methods from SOAP	Estimated Endpoints
`DeviceController`	getCpeList, getCpeInfo, deleteCpe, getCpeBySerial, getCpeStatus, …	~25
`ParameterController`	getParameterValues, setParameterValues, getParameterNames, getParameterAttributes, …	~15
`EventController`	getEventMonitors, createEventMonitor, deleteEventMonitor, getHardcodedEvents, …	~12
`ProfileController`	getProfiles, createProfile, updateProfile, deleteProfile, getProfileConditions, …	~15
`UpdateGroupController`	getUpdateGroups, createUpdateGroup, activateUpdateGroup, getUpdateGroupStatus, …	~10
`QoEController`	getQoeMonitoringGroups, startQoeMonitoring, stopQoeMonitoring, getQoeData, …	~12
`SecurityController`	getBlacklist, addToBlacklist, getWhitelist, addToWhitelist, …	~8
`ProvisionController`	addPendingTask, getPendingTasks, getCompletedTasks, downloadFile, …	~15
`DiagnosticController`	runIpPing, runTraceroute, runDownloadDiag, runUploadDiag, …	~10
`ConfigController`	getConfigParameters, setConfigParameter, configureAccountInfoMapping, …	~8

DeviceController

getCpeList, getCpeInfo, deleteCpe, getCpeBySerial, getCpeStatus, …

~25

ParameterController

getParameterValues, setParameterValues, getParameterNames, getParameterAttributes, …

~15

EventController

getEventMonitors, createEventMonitor, deleteEventMonitor, getHardcodedEvents, …

~12

ProfileController

getProfiles, createProfile, updateProfile, deleteProfile, getProfileConditions, …

~15

UpdateGroupController

getUpdateGroups, createUpdateGroup, activateUpdateGroup, getUpdateGroupStatus, …

~10

QoEController

getQoeMonitoringGroups, startQoeMonitoring, stopQoeMonitoring, getQoeData, …

~12

SecurityController

getBlacklist, addToBlacklist, getWhitelist, addToWhitelist, …

ProvisionController

addPendingTask, getPendingTasks, getCompletedTasks, downloadFile, …

~15

DiagnosticController

runIpPing, runTraceroute, runDownloadDiag, runUploadDiag, …

~10

ConfigController

getConfigParameters, setConfigParameter, configureAccountInfoMapping, …

5.3.4. Impact Assessment

Area Impact Description Risk

Area	Impact	Description	Risk
Codebase	HIGH	~10 new REST controllers, DTO mapping layer, OpenAPI annotations. SOAP classes kept but deprecated.	LOW — additive change, SOAP still works
API consumers	HIGH	External clients must eventually migrate from SOAP to REST. Dual-run period required.	MEDIUM — client coordination needed
Internal communication	MEDIUM	`SoapNodeNotifier` / `BroadcastingWebService` replaced (prepare for Kafka in Phase 4)	MEDIUM
Database	NONE	No schema changes	NONE
Deployment	LOW	Additional REST endpoints exposed. May need reverse proxy config update.	LOW

Codebase

HIGH

~10 new REST controllers, DTO mapping layer, OpenAPI annotations. SOAP classes kept but deprecated.

LOW — additive change, SOAP still works

API consumers

HIGH

External clients must eventually migrate from SOAP to REST. Dual-run period required.

MEDIUM — client coordination needed

Internal communication

MEDIUM

SoapNodeNotifier / BroadcastingWebService replaced (prepare for Kafka in Phase 4)

MEDIUM

Database

NONE

No schema changes

NONE

Deployment

LOW

Additional REST endpoints exposed. May need reverse proxy config update.

LOW

5.3.5. Test Coverage Plan

Test Type What to Cover Tools

Test Type	What to Cover	Tools
REST controller unit tests	Each endpoint: valid request → expected response. Error cases: invalid input, missing auth, not found.	JUnit 5 + MockMvc (`@WebMvcTest`)
SOAP ↔ REST parity tests	For every migrated operation: call both SOAP and REST with identical input, assert identical output. This is the critical safety net.	Parameterized tests with both clients
OpenAPI contract tests	Validate that generated OpenAPI spec matches expected schema. Validate request/response examples.	Swagger Parser + JSON Schema validation
API versioning tests	Verify `/api/v1/` prefix works. Verify content negotiation (JSON).	Integration tests
Authentication tests	REST endpoints enforce same auth as SOAP. Test token/session propagation.	Spring Security test support (`@WithMockUser`)
Load tests	REST vs SOAP performance comparison. REST should be faster (no XML overhead).	JMeter or Gatling — same scenarios, both endpoints

REST controller unit tests

Each endpoint: valid request → expected response. Error cases: invalid input, missing auth, not found.

JUnit 5 + MockMvc (@WebMvcTest)

SOAP ↔ REST parity tests

For every migrated operation: call both SOAP and REST with identical input, assert identical output. This is the critical safety net.

Parameterized tests with both clients

OpenAPI contract tests

Validate that generated OpenAPI spec matches expected schema. Validate request/response examples.

Swagger Parser + JSON Schema validation

API versioning tests

Verify /api/v1/ prefix works. Verify content negotiation (JSON).

Integration tests

Authentication tests

REST endpoints enforce same auth as SOAP. Test token/session propagation.

Spring Security test support (@WithMockUser)

Load tests

REST vs SOAP performance comparison. REST should be faster (no XML overhead).

JMeter or Gatling — same scenarios, both endpoints

Key tests to add in Phase 3

// Parity tests (most critical)
@ParameterizedTest
@MethodSource("allMigratedOperations")
void soapAndRestReturnIdenticalResults(Operation op, Object input)

// REST controller tests
@WebMvcTest(DeviceController.class)
@Test void getDeviceList_returnsPagedResult()
@Test void getDeviceById_notFound_returns404()
@Test void deleteDevice_unauthorized_returns401()

// OpenAPI validation
@Test void openApiSpec_isValid()
@Test void openApiSpec_coversAllEndpoints()

// Performance comparison
@Test void restEndpoint_fasterThanSoap_forLargePayloads()

5.3.6. Rollback Strategy

Dual-run: SOAP endpoints remain active. If REST introduces issues, clients fall back to SOAP. Remove SOAP only after all consumers have migrated and a stabilization period has passed.

5.4. Phase 4: Kafka for Event Bus & Inter-Node Communication

5.4.1. Objective

Replace the in-process DefaultAsyncEventBusImpl and SOAP-based SoapNodeNotifier with Apache Kafka. Events become durable, replayable, and consumable by external services.

5.4.2. Transformation Diagram

5.4.3. Event Topic Mapping

Kafka Topic Event Classes Partition Key Volume

Kafka Topic	Event Classes	Partition Key	Volume
`acs.device.lifecycle`	`CpeCreatedEvent`, `CpeDeletedEvent`, `CpeDeletedBulkEvent`, `CpeStatusChangedEvent`, `BootEvent`, `BootstrapEvent`	`cpeId`	Medium
`acs.device.parameters`	`CpeParameterChangedEvent`, `CpeParameterCreatedEvent`, `CpeParameterDeletedEvent`, `CpeParameterMergedEvent`	`cpeId`	HIGH — fired on every inform
`acs.tasks`	`PendingTaskEvent`, `CpeTaskEvent`, `TaskStateBulkChanged`, `SetValueRollBackEvent`	`cpeId`	Medium
`acs.events.monitoring`	Event monitor trigger results, hardcoded event matches	`eventMonitorId`	Low-Medium
`acs.update-group`	`UpdateGroupActivationStarted/Finished`, `UpdateGroupInterrupted`	`updateGroupId`	Low
`acs.qoe`	`QoeCustomViewUpdateStarted/Finished`, QoE monitoring state	`qoeGroupId`	Low
`acs.config`	`ConfigurationParameterChangedEvent`, `SecurityConfigurationChangedEvent`	`configKey`	Very Low
`acs.cluster`	`DistributedPushEvent`, `DistributedDisconnectCpeEvent`, `DomainAssociatedWithCpeEvent`	`nodeAddr`	Low

acs.device.lifecycle

CpeCreatedEvent, CpeDeletedEvent, CpeDeletedBulkEvent, CpeStatusChangedEvent, BootEvent, BootstrapEvent

cpeId

Medium

acs.device.parameters

CpeParameterChangedEvent, CpeParameterCreatedEvent, CpeParameterDeletedEvent, CpeParameterMergedEvent

cpeId

HIGH — fired on every inform

acs.tasks

PendingTaskEvent, CpeTaskEvent, TaskStateBulkChanged, SetValueRollBackEvent

cpeId

Medium

acs.events.monitoring

Event monitor trigger results, hardcoded event matches

eventMonitorId

Low-Medium

acs.update-group

UpdateGroupActivationStarted/Finished, UpdateGroupInterrupted

updateGroupId

Low

acs.qoe

QoeCustomViewUpdateStarted/Finished, QoE monitoring state

qoeGroupId

Low

acs.config

ConfigurationParameterChangedEvent, SecurityConfigurationChangedEvent

configKey

Very Low

acs.cluster

DistributedPushEvent, DistributedDisconnectCpeEvent, DomainAssociatedWithCpeEvent

nodeAddr

Low

5.4.4. Impact Assessment

Area Impact Description Risk

Area	Impact	Description	Risk
Infrastructure	HIGH	New dependency: Apache Kafka cluster (3+ brokers for production). New operational concern.	MEDIUM — Kafka is well-understood but adds ops complexity
Codebase	MEDIUM	New `KafkaEventBusAdapter`. All 35+ event classes get serialization (Avro or JSON). Event handlers refactored to Kafka consumers.	MEDIUM — serialization edge cases
Runtime behavior	MEDIUM	Events become eventually-consistent (Kafka latency ~1-5ms). Previously in-process was near-instant.	LOW — acceptable for all current use cases
Ordering guarantees	LOW	Kafka guarantees order within a partition. Partition by `cpeId` ensures per-device ordering.	LOW
Node-to-node communication	HIGH	`SoapNodeNotifier` + `BroadcastingWebService` completely removed. Kafka handles all inter-node events.	MEDIUM — must verify all distributed event flows
Hazelcast scope reduction	MEDIUM	5 Hazelcast Topics (`UpdateGroup`, `QoeCustomView`, `ConfigSync`) removed. Hazelcast now only = cache + locks.	LOW

Infrastructure

HIGH

New dependency: Apache Kafka cluster (3+ brokers for production). New operational concern.

MEDIUM — Kafka is well-understood but adds ops complexity

Codebase

MEDIUM

New KafkaEventBusAdapter. All 35+ event classes get serialization (Avro or JSON). Event handlers refactored to Kafka consumers.

MEDIUM — serialization edge cases

Runtime behavior

MEDIUM

Events become eventually-consistent (Kafka latency ~1-5ms). Previously in-process was near-instant.

LOW — acceptable for all current use cases

Ordering guarantees

LOW

Kafka guarantees order within a partition. Partition by cpeId ensures per-device ordering.

LOW

Node-to-node communication

HIGH

SoapNodeNotifier + BroadcastingWebService completely removed. Kafka handles all inter-node events.

MEDIUM — must verify all distributed event flows

Hazelcast scope reduction

MEDIUM

5 Hazelcast Topics (UpdateGroup*, QoeCustomView*, ConfigSync) removed. Hazelcast now only = cache + locks.

LOW

5.4.5. Test Coverage Plan

Test Type What to Cover Tools

Test Type	What to Cover	Tools
Event serialization/deserialization	Every event class round-trips through Kafka serialization without data loss. Test all 35+ event types.	JUnit 5 + Kafka Avro/JSON serializer
Event ordering tests	Events for the same `cpeId` are consumed in publish order. Verify partition key strategy.	Embedded Kafka (`spring-kafka-test`) + ordered publish/consume assertions
Consumer group tests	Multiple ACS nodes in the same consumer group process events exactly once. New node joining rebalances correctly.	Embedded Kafka + multiple consumer instances
Event loss tests	Kill a node mid-publish. Verify uncommitted events are replayed on restart (at-least-once).	Integration test with simulated crashes
Latency benchmarks	Measure event publish-to-consume latency. Must be < 10ms for p99 within same datacenter.	Kafka metrics + custom latency tracking
Distributed event tests	Events previously handled by `SoapNodeNotifier` (push, disconnect) are correctly received by all nodes via Kafka.	Multi-instance integration test
Backward compatibility	During migration: local bus and Kafka run in parallel. Same event is handled by both. No duplicate processing.	Feature-flag test: toggle between local/kafka/dual
Load test	Sustained 10k events/second (simulating 10k device informs). Kafka keeps up, no consumer lag.	Kafka producer load test + consumer lag monitoring

Event serialization/deserialization

Every event class round-trips through Kafka serialization without data loss. Test all 35+ event types.

JUnit 5 + Kafka Avro/JSON serializer

Event ordering tests

Events for the same cpeId are consumed in publish order. Verify partition key strategy.

Embedded Kafka (spring-kafka-test) + ordered publish/consume assertions

Consumer group tests

Multiple ACS nodes in the same consumer group process events exactly once. New node joining rebalances correctly.

Embedded Kafka + multiple consumer instances

Event loss tests

Kill a node mid-publish. Verify uncommitted events are replayed on restart (at-least-once).

Integration test with simulated crashes

Latency benchmarks

Measure event publish-to-consume latency. Must be < 10ms for p99 within same datacenter.

Kafka metrics + custom latency tracking

Distributed event tests

Events previously handled by SoapNodeNotifier (push, disconnect) are correctly received by all nodes via Kafka.

Multi-instance integration test

Backward compatibility

During migration: local bus and Kafka run in parallel. Same event is handled by both. No duplicate processing.

Feature-flag test: toggle between local/kafka/dual

Load test

Sustained 10k events/second (simulating 10k device informs). Kafka keeps up, no consumer lag.

Kafka producer load test + consumer lag monitoring

Key tests to add in Phase 4

// Serialization round-trip (parameterized for all events)
@ParameterizedTest
@MethodSource("allEventClasses")
void eventSurvivesKafkaSerialization(Class<? extends Event> eventClass)

// Ordering
@Test void eventsForSameDevice_consumedInOrder()

// Exactly-once within consumer group
@Test void twoConsumersInGroup_eachEventProcessedOnce()

// Crash resilience
@Test void nodeKilledDuringPublish_eventsReplayedOnRestart()

// Latency
@Test void publishToConsume_p99Under10ms()

// Migration safety
@Test void dualMode_localAndKafka_noDoubleProcessing()

5.4.6. Rollback Strategy

Dual-run mode: during migration, both DefaultAsyncEventBusImpl and KafkaEventBusAdapter are active behind a feature flag. If Kafka issues arise, flag toggles back to local-only. Kafka producers can be disabled without code changes via configuration.

5.5. Phase 5: Redis Replacing Hazelcast

5.5.1. Objective

Replace all 80+ Hazelcast IMap caches with Redis. Replace distributed locks with Redisson. Remove Hazelcast dependency entirely.

5.5.2. Transformation Diagram

5.5.3. Cache Migration Strategy

Migrate in batches by risk level:

Batch Caches Rationale Risk

Batch	Caches	Rationale	Risk
Batch 1 (Low risk)	`ispCache`, `ispMetaDataCache`, `manufacturerByNameOuiCache`, `productClassIdByModelManufProtocolCache` — ISP and metadata caches (read-heavy, rarely change)	Low write frequency, easy to validate, used across many modules — good canary.	LOW
Batch 2 (Medium risk)	`profileCache`, `profileConditionCache`, `profileParameterCache`, `profileParameterAccessCache`, `profileUploadCache`, `profileCpeIdCache` — Profile caches	Read-heavy, updated on profile changes. Well-bounded domain.	LOW
Batch 3 (Medium risk)	`cpeBlackListSerial`, `CpeWhiteListSerialSet`, `CpeWhiteListSerialOnlyCreated`, `cpeWhiteListManufCache`, `cpeSerialCache` — Security caches	Security-critical but low-frequency updates.	MEDIUM
Batch 4 (Higher risk)	`CpeParameterIdNameCache`, `CpeParameterIdTypeCache`, `CpeParameterNameIdCache`, `CpeParameterNameIdsByTemplateCache` — Parameter name caches (hot path)	These are on the inform hot path with near-cache. Must benchmark Redis performance.	MEDIUM
Batch 5 (Higher risk)	`deviceIdBySerialCache`, `serialByDeviceIdCache`, `cpeIdLastConnection`, `cpeIdInSessionQN` — Device identity caches	Core device lookup. Must handle high concurrency.	MEDIUM
Batch 6 (Highest risk)	`LOCKERTaskId`, `LOCKERCpeId`, `LOCKERNextSessionCpeId`, `cpeIdsInCpeParamsModifyLock`, `qoeCpeSerialLock` — Distributed locks	Locks are correctness-critical. Redisson’s `RLock` (Redlock) must provide equivalent guarantees.	HIGH
Batch 7 (Final)	All remaining caches (event monitoring, QoE, task coordination, encryption) + `HazelcastInstance.getExecutorService()` replacement	Cleanup. Replace executor with local thread pool + Kafka for distributed dispatch.	MEDIUM

Batch 1 (Low risk)

ispCache, ispMetaDataCache, manufacturerByNameOuiCache, productClassIdByModelManufProtocolCache — ISP and metadata caches (read-heavy, rarely change)

Low write frequency, easy to validate, used across many modules — good canary.

LOW

Batch 2 (Medium risk)

profileCache, profileConditionCache, profileParameterCache, profileParameterAccessCache, profileUploadCache, profileCpeIdCache — Profile caches

Read-heavy, updated on profile changes. Well-bounded domain.

LOW

Batch 3 (Medium risk)

cpeBlackListSerial, CpeWhiteListSerialSet, CpeWhiteListSerialOnlyCreated, cpeWhiteListManufCache, cpeSerialCache — Security caches

Security-critical but low-frequency updates.

MEDIUM

Batch 4 (Higher risk)

CpeParameterIdNameCache, CpeParameterIdTypeCache, CpeParameterNameIdCache, CpeParameterNameIdsByTemplateCache — Parameter name caches (hot path)

These are on the inform hot path with near-cache. Must benchmark Redis performance.

MEDIUM

Batch 5 (Higher risk)

deviceIdBySerialCache, serialByDeviceIdCache, cpeIdLastConnection, cpeIdInSessionQN — Device identity caches

Core device lookup. Must handle high concurrency.

MEDIUM

Batch 6 (Highest risk)

LOCKERTaskId, LOCKERCpeId, LOCKERNextSessionCpeId, cpeIdsInCpeParamsModifyLock, qoeCpeSerialLock — Distributed locks

Locks are correctness-critical. Redisson’s RLock (Redlock) must provide equivalent guarantees.

HIGH

Batch 7 (Final)

All remaining caches (event monitoring, QoE, task coordination, encryption) + HazelcastInstance.getExecutorService() replacement

Cleanup. Replace executor with local thread pool + Kafka for distributed dispatch.

MEDIUM

5.5.4. Impact Assessment

Area Impact Description Risk

Area	Impact	Description	Risk
Infrastructure	HIGH	New dependency: Redis Cluster (3+ nodes). Remove: Hazelcast cluster. Net operational change is a wash.	MEDIUM — different ops model
Codebase	HIGH	Cache interface implementations change from Hazelcast IMap to Redis operations. ~80 cache sites.	MEDIUM — large change surface but mechanical
Performance	MEDIUM	Near-cache (local L1) lost — Redis is always network hop. May need local Caffeine L1 cache for hot maps.	MEDIUM — must benchmark
Distributed locks	HIGH	Redisson RLock vs Hazelcast IMap lock. Different failure modes, different lease policies.	HIGH — correctness critical
User Code Deployment	REMOVED	No more deploying `Cpe`, `QoeCpe`, `CpeTrace` to the Hazelcast cluster. Standard Redis serialization.	LOW — simplification
Serialization	MEDIUM	Hazelcast relied on Java serialization + deployed classes. Redis uses JSON/Protobuf — must define serialization for all cached types.	MEDIUM

Infrastructure

HIGH

New dependency: Redis Cluster (3+ nodes). Remove: Hazelcast cluster. Net operational change is a wash.

MEDIUM — different ops model

Codebase

HIGH

Cache interface implementations change from Hazelcast IMap to Redis operations. ~80 cache sites.

MEDIUM — large change surface but mechanical

Performance

MEDIUM

Near-cache (local L1) lost — Redis is always network hop. May need local Caffeine L1 cache for hot maps.

MEDIUM — must benchmark

Distributed locks

HIGH

Redisson RLock vs Hazelcast IMap lock. Different failure modes, different lease policies.

HIGH — correctness critical

User Code Deployment

REMOVED

No more deploying Cpe, QoeCpe, CpeTrace to the Hazelcast cluster. Standard Redis serialization.

LOW — simplification

Serialization

MEDIUM

Hazelcast relied on Java serialization + deployed classes. Redis uses JSON/Protobuf — must define serialization for all cached types.

MEDIUM

5.5.5. Test Coverage Plan

Test Type What to Cover Tools

Test Type	What to Cover	Tools
Cache operation tests	For each migrated cache: put/get/evict/ttl behave identically to Hazelcast. Test all 80+ caches.	JUnit 5 + Embedded Redis (Testcontainers)
Lock correctness tests	Two threads/processes competing for the same lock: only one wins. Lock auto-expires on holder crash. Reentrant lock works.	Testcontainers Redis + concurrent test threads
Near-cache replacement benchmarks	Hot path caches (`CpeParameter*`) with local Caffeine L1 + Redis L2. Measure hit rates and latency vs old Hazelcast near-cache.	JMH microbenchmarks
Serialization compatibility	All cached entity types serialize/deserialize correctly with the chosen Redis serializer.	Parameterized tests per entity type
Failover tests	Redis master failure: Sentinel promotes replica. Application reconnects. No data loss for persistent keys.	Testcontainers with Sentinel + simulated failure
Performance regression	End-to-end inform processing time. Must not regress > 5% vs Hazelcast.	Load test: 5k concurrent informs, measure p95 latency
Gradual migration tests	During migration: some caches on Hazelcast, some on Redis. System works correctly with mixed backends.	Integration test with feature flags per cache batch

Cache operation tests

For each migrated cache: put/get/evict/ttl behave identically to Hazelcast. Test all 80+ caches.

JUnit 5 + Embedded Redis (Testcontainers)

Lock correctness tests

Two threads/processes competing for the same lock: only one wins. Lock auto-expires on holder crash. Reentrant lock works.

Testcontainers Redis + concurrent test threads

Near-cache replacement benchmarks

Hot path caches (CpeParameter*) with local Caffeine L1 + Redis L2. Measure hit rates and latency vs old Hazelcast near-cache.

JMH microbenchmarks

Serialization compatibility

All cached entity types serialize/deserialize correctly with the chosen Redis serializer.

Parameterized tests per entity type

Failover tests

Redis master failure: Sentinel promotes replica. Application reconnects. No data loss for persistent keys.

Testcontainers with Sentinel + simulated failure

Performance regression

End-to-end inform processing time. Must not regress > 5% vs Hazelcast.

Load test: 5k concurrent informs, measure p95 latency

Gradual migration tests

During migration: some caches on Hazelcast, some on Redis. System works correctly with mixed backends.

Integration test with feature flags per cache batch

Key tests to add in Phase 5

// Per-cache correctness (parameterized)
@ParameterizedTest
@MethodSource("allCacheNames")
void cacheOperations_identicalToHazelcast(String cacheName)

// Lock correctness
@Test void distributedLock_mutualExclusion()
@Test void distributedLock_autoExpireOnCrash()
@Test void distributedLock_reentrant()

// Performance
@Test void parameterCacheLookup_p95Under2ms()
@Test void informProcessing_noRegressionVsHazelcast()

// Failover
@Test void redisMasterFailure_applicationRecovers()

5.5.6. Rollback Strategy

Batch-by-batch migration with feature flags. Each batch can independently fall back to Hazelcast. Both Hazelcast and Redis can run simultaneously during the transition. Full rollback = toggle all flags back to Hazelcast.

5.6. Phase 6: Service Extraction

5.6.1. Objective

Extract Events, Update Group, and QoE as independent services. Each gets its own deployment, scaling, and performance measurement.

5.6.2. Transformation Diagram

5.6.3. Service Boundaries

Service Owns (tables) Consumes (Kafka) Produces (Kafka)

Service	Owns (tables)	Consumes (Kafka)	Produces (Kafka)
Core ACS	`cpe`, `cpe_parameter`, `cpe_parameter_name`, `cpe_pending_task`, `profile_`, `isp`, `security_`	`acs.tasks` (completed), `acs.update-group` (status)	`acs.device.lifecycle`, `acs.device.parameters`, `acs.tasks`
Event Service	`event_monitor_`, `event_condition_`, `cpe_log`, `cpe_changed_parameter`	`acs.device.lifecycle`, `acs.device.parameters`, `acs.tasks`	`acs.events.monitoring`, notification delivery results
Update Group Service	`update_group_`, `ug_cpe_`, `ug_cpe_completed_his`	`acs.tasks` (task results), `acs.device.lifecycle` (device status)	`acs.update-group`, task commands → `acs.tasks`
QoE Service	`ftacs_qoe_ui.*` (all QoE tables), ClickHouse tables	`acs.device.parameters` (QoE metrics), `acs.device.lifecycle`	`acs.qoe` (monitoring state)

Core ACS

cpe, cpe_parameter, cpe_parameter_name, cpe_pending_task, profile_*, isp, security_*

acs.tasks (completed), acs.update-group (status)

acs.device.lifecycle, acs.device.parameters, acs.tasks

Event Service

event_monitor_*, event_condition_*, cpe_log, cpe_changed_parameter

acs.device.lifecycle, acs.device.parameters, acs.tasks

acs.events.monitoring, notification delivery results

Update Group Service

update_group_*, ug_cpe_*, ug_cpe_completed_his

acs.tasks (task results), acs.device.lifecycle (device status)

acs.update-group, task commands → acs.tasks

QoE Service

ftacs_qoe_ui.* (all QoE tables), ClickHouse tables

acs.device.parameters (QoE metrics), acs.device.lifecycle

acs.qoe (monitoring state)

5.6.4. Impact Assessment

Area	Impact	Description	Risk
Architecture	CRITICAL	Three new independent services. Service discovery, API gateway, distributed tracing needed.	HIGH — distributed system complexity
Database	HIGH	Table ownership split. May need schema separation or cross-service queries via API.	HIGH — data ownership boundaries
Deployment	HIGH	3 new deployable units. CI/CD pipelines, Docker images, health checks, monitoring per service.	MEDIUM — operational complexity
Performance	POSITIVE	Per-service scaling. Can scale Event Service independently during high-event periods.	LOW — this is the goal
Team organization	MEDIUM	Teams can now own individual services. Clear ownership boundaries.	LOW — organizational benefit
Observability	HIGH	Need distributed tracing (Jaeger/Zipkin), centralized logging, per-service dashboards.	MEDIUM — must be in place before extraction

Area

Impact

Description

Risk

Architecture

CRITICAL

Three new independent services. Service discovery, API gateway, distributed tracing needed.

HIGH — distributed system complexity

Database

HIGH

Table ownership split. May need schema separation or cross-service queries via API.

HIGH — data ownership boundaries

Deployment

HIGH

3 new deployable units. CI/CD pipelines, Docker images, health checks, monitoring per service.

MEDIUM — operational complexity

Performance

POSITIVE

Per-service scaling. Can scale Event Service independently during high-event periods.

LOW — this is the goal

Team organization

MEDIUM

Teams can now own individual services. Clear ownership boundaries.

LOW — organizational benefit

Observability

HIGH

Need distributed tracing (Jaeger/Zipkin), centralized logging, per-service dashboards.

MEDIUM — must be in place before extraction

5.6.5. Test Coverage Plan

Test Type	What to Cover	Tools
Service contract tests	Each service’s Kafka consumers/producers agree on schema. REST API contracts between services are verified.	Pact (consumer-driven contract testing) or Spring Cloud Contract
Service integration tests	Full flow: device informs → Core ACS publishes event → Event Service processes and sends notification. Verify end-to-end.	Docker Compose with all services + Kafka + Redis + DB
Service isolation tests	Each service starts independently. Each service handles Kafka unavailability gracefully (circuit breaker). Each service handles Redis unavailability gracefully.	Testcontainers with selective infrastructure
Data consistency tests	Events published by Core ACS are consumed exactly once by Event Service. No events lost during service restart.	Chaos testing: restart services during load
Performance per service	Establish per-service performance baselines. This is the performance formula enabler.	Per-service load tests + Grafana dashboards
Failover tests	One service goes down — others continue functioning. Kafka buffers events until service recovers. No data loss.	Chaos monkey-style testing
End-to-end regression	All existing functional test scenarios pass with the distributed architecture.	Full regression suite against Docker Compose environment

Test Type

What to Cover

Tools

Service contract tests

Each service’s Kafka consumers/producers agree on schema. REST API contracts between services are verified.

Pact (consumer-driven contract testing) or Spring Cloud Contract

Service integration tests

Full flow: device informs → Core ACS publishes event → Event Service processes and sends notification. Verify end-to-end.

Docker Compose with all services + Kafka + Redis + DB

Service isolation tests

Each service starts independently. Each service handles Kafka unavailability gracefully (circuit breaker). Each service handles Redis unavailability gracefully.

Testcontainers with selective infrastructure

Data consistency tests

Events published by Core ACS are consumed exactly once by Event Service. No events lost during service restart.

Chaos testing: restart services during load

Performance per service

Establish per-service performance baselines. This is the performance formula enabler.

Per-service load tests + Grafana dashboards

Failover tests

One service goes down — others continue functioning. Kafka buffers events until service recovers. No data loss.

Chaos monkey-style testing

End-to-end regression

All existing functional test scenarios pass with the distributed architecture.

Full regression suite against Docker Compose environment

Key tests to add in Phase 6

// Contract tests
@Test void coreACS_publishesDeviceLifecycleEvent_matchingEventServiceSchema()
@Test void eventService_consumesParameterChangedEvent_matchingCoreSchema()
@Test void updateGroupService_producesTaskCommand_matchingCoreSchema()

// Isolation tests
@Test void eventService_startsWithoutCoreACS()
@Test void eventService_handlesKafkaUnavailability_withCircuitBreaker()
@Test void updateGroupService_handlesRedisUnavailability_withFallback()

// Performance formula tests
@Test void coreACS_2instances_handles10kInformsPerMinute()
@Test void eventService_1instance_handles5kEventsPerMinute()
@Test void updateGroupService_1instance_handles1kUpdatesPerMinute()

// Chaos tests
@Test void eventServiceRestart_noEventsLost()
@Test void coreACSRestart_kafkaBuffersEvents_eventServiceCatchesUp()

5.6.6. Rollback Strategy

Each service extraction is independent. If Event Service extraction has issues, event processing falls back to the monolith (run both, toggle Kafka consumers). Services can be re-absorbed one at a time.

5.7. Phase 7: ClickHouse Expansion for Historical Data

5.7.1. Objective

Migrate high-volume historical tables from MySQL/Oracle to ClickHouse. Leverage existing ClickHouse integration (QoE) and extend to device logs, parameter changes, and completed tasks.

5.7.2. Transformation Diagram

5.7.3. Migration Plan per Table

Table Est. Rows/Day Current Size ClickHouse Engine Priority

Table	Est. Rows/Day	Current Size	ClickHouse Engine	Priority
`cpe_changed_parameter`	~500k-5M	Largest table	`MergeTree` partitioned by `toYYYYMM(created)`	1 (highest)
`cpe_log`	~100k-1M	Very large	`MergeTree` partitioned by `toYYYYMM(created)`	2
`cpe_completed_task`	~50k-500k	Large	`MergeTree` partitioned by `toYYYYMM(completed_at)`	3
`cpe_trace_log`	~10k-100k	Medium-Large	`MergeTree` partitioned by `toYYYYMM(created)`	4
`cpe_login`	~10k-50k	Medium	`MergeTree` partitioned by `toYYYYMM(login_time)`	5
`connection_request_failure`	~1k-10k	Small-Medium	`MergeTree` partitioned by `toYYYYMM(created)`	6
`error_log`	~1k-5k	Small-Medium	`MergeTree` partitioned by `toYYYYMM(created)`	7

cpe_changed_parameter

~500k-5M

Largest table

MergeTree partitioned by toYYYYMM(created)

1 (highest)

cpe_log

~100k-1M

Very large

MergeTree partitioned by toYYYYMM(created)

cpe_completed_task

~50k-500k

Large

MergeTree partitioned by toYYYYMM(completed_at)

cpe_trace_log

~10k-100k

Medium-Large

MergeTree partitioned by toYYYYMM(created)

cpe_login

~10k-50k

Medium

MergeTree partitioned by toYYYYMM(login_time)

connection_request_failure

~1k-10k

Small-Medium

MergeTree partitioned by toYYYYMM(created)

error_log

~1k-5k

Small-Medium

MergeTree partitioned by toYYYYMM(created)

5.7.4. Impact Assessment

Area Impact Description Risk

Area	Impact	Description	Risk
Query performance	HIGH (positive)	Analytical queries (aggregations, time-range filters) 10-100x faster. Device history pages load instantly.	LOW — performance improvement
Write performance	MEDIUM	Dual-write during transition adds latency. Kafka-based async write to ClickHouse mitigates this.	LOW — async via Kafka
Storage	HIGH (positive)	ClickHouse columnar compression reduces storage 5-10x. MySQL freed from large tables.	LOW — cost reduction
Application code	MEDIUM	DAO implementations switch datasource. Existing `*DaoJdbcImpl` pattern with `@Qualifier("clickhouseDataSource")` already established.	LOW — pattern exists
Reporting	HIGH (positive)	Grafana dashboards can query ClickHouse directly. New analytical capabilities.	LOW — additive
Data migration	MEDIUM	Historical data must be bulk-loaded into ClickHouse. May take hours for large datasets.	MEDIUM — one-time operation

Query performance

HIGH (positive)

Analytical queries (aggregations, time-range filters) 10-100x faster. Device history pages load instantly.

LOW — performance improvement

Write performance

MEDIUM

Dual-write during transition adds latency. Kafka-based async write to ClickHouse mitigates this.

LOW — async via Kafka

Storage

HIGH (positive)

ClickHouse columnar compression reduces storage 5-10x. MySQL freed from large tables.

LOW — cost reduction

Application code

MEDIUM

DAO implementations switch datasource. Existing *DaoJdbcImpl pattern with @Qualifier("clickhouseDataSource") already established.

LOW — pattern exists

Reporting

HIGH (positive)

Grafana dashboards can query ClickHouse directly. New analytical capabilities.

LOW — additive

Data migration

MEDIUM

Historical data must be bulk-loaded into ClickHouse. May take hours for large datasets.

MEDIUM — one-time operation

5.7.5. Test Coverage Plan

Test Type	What to Cover	Tools
Dual-write consistency	Same record appears in both MySQL and ClickHouse during transition. No data loss.	Integration test comparing row counts + sampling
Query parity tests	ClickHouse queries return same results as MySQL for identical time ranges and filters.	Parameterized tests: same query → both DBs → compare results
ClickHouse-specific query tests	Aggregation queries, time-series rollups, partition pruning work correctly.	JUnit + ClickHouse JDBC
Bulk migration test	Historical data migration completes without errors. Row counts match.	Migration script + validation query
Performance benchmarks	Measure query times for: last 24h device logs, parameter change history, completed task aggregation. Must be 10x+ faster than MySQL.	JMH or dedicated benchmark suite
Retention policy tests	ClickHouse TTL (if configured) correctly drops old partitions. No premature data loss.	Time-accelerated test with TTL

Test Type

What to Cover

Tools

Dual-write consistency

Same record appears in both MySQL and ClickHouse during transition. No data loss.

Integration test comparing row counts + sampling

Query parity tests

ClickHouse queries return same results as MySQL for identical time ranges and filters.

Parameterized tests: same query → both DBs → compare results

ClickHouse-specific query tests

Aggregation queries, time-series rollups, partition pruning work correctly.

JUnit + ClickHouse JDBC

Bulk migration test

Historical data migration completes without errors. Row counts match.

Migration script + validation query

Performance benchmarks

Measure query times for: last 24h device logs, parameter change history, completed task aggregation. Must be 10x+ faster than MySQL.

JMH or dedicated benchmark suite

Retention policy tests

ClickHouse TTL (if configured) correctly drops old partitions. No premature data loss.

Time-accelerated test with TTL

Key tests to add in Phase 7

// Dual-write consistency
@Test void dualWrite_recordAppearsInBothMysqlAndClickHouse()
@Test void dualWrite_kafkaConsumerLag_doesNotExceed1000()

// Query parity
@ParameterizedTest
@MethodSource("historicalQueries")
void queryReturnsIdenticalResults_mysqlVsClickHouse(String query)

// Performance
@Test void deviceLogQuery_last24h_under100ms_inClickHouse()
@Test void parameterChangeAggregation_lastMonth_under500ms()

// Migration
@Test void bulkMigration_allRowsCopied_noDataLoss()

5.7.6. Rollback Strategy

Dual-write means MySQL still has all data. If ClickHouse has issues, switch reads back to MySQL via datasource qualifier configuration. No data loss possible during dual-write period.

6. Cross-Phase Concerns

6.1. Infrastructure Evolution

6.2. Observability Requirements per Phase

Phase	Monitoring Additions	Dashboards
Phase 1-3	None — existing monitoring sufficient	Existing Grafana dashboards
Phase 4	Kafka broker metrics, consumer lag, topic throughput, event processing latency	New: Kafka dashboard, Event Bus dashboard
Phase 5	Redis memory usage, hit/miss ratios, lock contention, Redisson metrics	New: Redis dashboard. Update: replace Hazelcast dashboard
Phase 6	Per-service CPU/memory/GC, inter-service latency, circuit breaker states, distributed traces	New: per-service dashboards, service mesh dashboard, distributed trace viewer
Phase 7	ClickHouse query performance, insert rates, partition sizes, replication lag	New: ClickHouse dashboard, historical data dashboard

Phase

Monitoring Additions

Dashboards

Phase 1-3

None — existing monitoring sufficient

Existing Grafana dashboards

Phase 4

Kafka broker metrics, consumer lag, topic throughput, event processing latency

New: Kafka dashboard, Event Bus dashboard

Phase 5

Redis memory usage, hit/miss ratios, lock contention, Redisson metrics

New: Redis dashboard. Update: replace Hazelcast dashboard

Phase 6

Per-service CPU/memory/GC, inter-service latency, circuit breaker states, distributed traces

New: per-service dashboards, service mesh dashboard, distributed trace viewer

Phase 7

ClickHouse query performance, insert rates, partition sizes, replication lag

New: ClickHouse dashboard, historical data dashboard

6.3. Performance Formula (Phase 6 Goal)

Once services are extracted, capacity can be modeled per-service:

Total system capacity = min(
  core_acs_capacity(N_core, scenario),
  event_service_capacity(N_event, event_rate),
  update_group_capacity(N_ug, group_size),
  qoe_service_capacity(N_qoe, monitoring_rate)
)

Where:
  core_acs_capacity = N_core × informs_per_instance_per_minute
  event_service_capacity = N_event × events_per_instance_per_minute
  update_group_capacity = N_ug × devices_per_instance_per_minute
  qoe_service_capacity = N_qoe × metrics_per_instance_per_minute

Each variable is independently measurable and scalable. Bottleneck identification becomes trivial.

6.4. Skill Requirements

Phase	Team Skills Needed
Phase 1	Java, Maven multi-module projects, interface design
Phase 2	Design patterns (Strategy, Template Method), protocol knowledge (TR-069/MQTT/USP)
Phase 3	Spring MVC, REST API design, OpenAPI, HTTP security
Phase 4	Apache Kafka (producers, consumers, schemas, operations), event-driven architecture
Phase 5	Redis (data structures, clustering, Sentinel), Redisson, cache strategies (L1/L2)
Phase 6	Microservices patterns (service discovery, circuit breakers, distributed tracing), Docker, Kubernetes (optional), Pact/contract testing
Phase 7	ClickHouse (MergeTree engines, partitioning, SQL dialect), data migration tooling

Phase

Team Skills Needed

Phase 1

Java, Maven multi-module projects, interface design

Phase 2

Design patterns (Strategy, Template Method), protocol knowledge (TR-069/MQTT/USP)

Phase 3

Spring MVC, REST API design, OpenAPI, HTTP security

Phase 4

Apache Kafka (producers, consumers, schemas, operations), event-driven architecture

Phase 5

Redis (data structures, clustering, Sentinel), Redisson, cache strategies (L1/L2)

Phase 6

Microservices patterns (service discovery, circuit breakers, distributed tracing), Docker, Kubernetes (optional), Pact/contract testing

Phase 7

ClickHouse (MergeTree engines, partitioning, SQL dialect), data migration tooling

7. Risk Matrix

Risk Phase Probability Impact Mitigation

Risk	Phase	Probability	Impact	Mitigation
Module extraction breaks runtime wiring	1	Medium	Low	Zero behavioral change — compile-only. `@SpringBootTest` catches wiring issues.
Device flow refactoring introduces session bugs	2	Medium	High	Feature flag to toggle old/new path. Extensive protocol emulator tests. Run both paths in shadow mode before cutover.
SOAP client migration takes longer than expected	3	High	Medium	Dual-run indefinitely. No deadline to remove SOAP — it just costs maintenance.
Kafka adds latency to event processing	4	Low	Medium	Kafka within same DC is ~1-5ms. If needed, keep critical path events local and only replicate to Kafka asynchronously.
Redis lock semantics differ from Hazelcast	5	Medium	High	Extensive lock correctness tests. Use Redisson (battle-tested). Migrate locks last (Batch 6) after all caches are proven.
Near-cache loss degrades hot path	5	Medium	Medium	Add local Caffeine L1 cache with short TTL (5-30s) in front of Redis for `CpeParameter*` maps.
Distributed system complexity (cascading failures)	6	Medium	High	Circuit breakers (Resilience4j), bulkheads, retry with backoff. Kafka as buffer prevents cascading. Each service must function degraded when dependencies are down.
Data inconsistency between services	6	Medium	High	Event sourcing via Kafka ensures eventual consistency. Consumer idempotency is mandatory. Schema registry prevents breaking changes.
ClickHouse query compatibility	7	Low	Low	SQL dialect differences are small. Existing `*DaoJdbcImpl` pattern handles DB-specific queries. Test every query.
Team skill gaps	4-6	Medium	Medium	Training sprints before each infrastructure phase. Proof of concept in non-production first.

Module extraction breaks runtime wiring

Medium

Low

Zero behavioral change — compile-only. @SpringBootTest catches wiring issues.

Device flow refactoring introduces session bugs

Medium

High

Feature flag to toggle old/new path. Extensive protocol emulator tests. Run both paths in shadow mode before cutover.

SOAP client migration takes longer than expected

High

Medium

Dual-run indefinitely. No deadline to remove SOAP — it just costs maintenance.

Kafka adds latency to event processing

Low

Medium

Kafka within same DC is ~1-5ms. If needed, keep critical path events local and only replicate to Kafka asynchronously.

Redis lock semantics differ from Hazelcast

Medium

High

Extensive lock correctness tests. Use Redisson (battle-tested). Migrate locks last (Batch 6) after all caches are proven.

Near-cache loss degrades hot path

Medium

Add local Caffeine L1 cache with short TTL (5-30s) in front of Redis for CpeParameter* maps.

Distributed system complexity (cascading failures)

Medium

High

Circuit breakers (Resilience4j), bulkheads, retry with backoff. Kafka as buffer prevents cascading. Each service must function degraded when dependencies are down.

Data inconsistency between services

Medium

High

Event sourcing via Kafka ensures eventual consistency. Consumer idempotency is mandatory. Schema registry prevents breaking changes.

ClickHouse query compatibility

Low

SQL dialect differences are small. Existing *DaoJdbcImpl pattern handles DB-specific queries. Test every query.

Team skill gaps

4-6

Medium

Training sprints before each infrastructure phase. Proof of concept in non-production first.

8. Decision Log

Track key architectural decisions made during the evolution.

ID	Phase	Decision	Rationale
ADR-001	1	Use Java interfaces (not REST) for inter-module contracts within the monolith	Modules still run in the same JVM during Phases 1-5. REST overhead is unnecessary until Phase 6 extraction.
ADR-002	4	Use Kafka (not RabbitMQ) for event bus	Kafka provides durable, replayable, partitioned event log. Needed for event sourcing and audit trail. Higher throughput for parameter change events (~5M/day).
ADR-003	5	Use Redis (not Memcached) for cache replacement	Redis supports distributed locks (Redisson), pub/sub (backup), and data structures (Hash, Set). Memcached is cache-only.
ADR-004	5	Add Caffeine L1 cache for hot Redis maps	Near-cache loss from Hazelcast migration. Caffeine provides <1ms local reads with configurable TTL. Redis is L2 for cross-node consistency.
ADR-005	7	Use Kafka → ClickHouse pipeline (not direct dual-write)	Decouples write path. Application writes to Kafka; a dedicated consumer batch-inserts to ClickHouse. Better throughput, back-pressure handling, and retry.

Phase

Decision

Rationale

ADR-001

Use Java interfaces (not REST) for inter-module contracts within the monolith

Modules still run in the same JVM during Phases 1-5. REST overhead is unnecessary until Phase 6 extraction.

ADR-002

Use Kafka (not RabbitMQ) for event bus

Kafka provides durable, replayable, partitioned event log. Needed for event sourcing and audit trail. Higher throughput for parameter change events (~5M/day).

ADR-003

Use Redis (not Memcached) for cache replacement

Redis supports distributed locks (Redisson), pub/sub (backup), and data structures (Hash, Set). Memcached is cache-only.

ADR-004

Add Caffeine L1 cache for hot Redis maps

Near-cache loss from Hazelcast migration. Caffeine provides <1ms local reads with configurable TTL. Redis is L2 for cross-node consistency.

ADR-005

Use Kafka → ClickHouse pipeline (not direct dual-write)

Decouples write path. Application writes to Kafka; a dedicated consumer batch-inserts to ClickHouse. Better throughput, back-pressure handling, and retry.

9. Phase Completion Checklist

Use this checklist to verify each phase is complete before starting the next.

9.1. Phase 1 Checklist

shared-models module published and used by all dependent modules
*-api interface JARs exist for: events, update-group, qoe, provision, core
No circular Maven dependencies (enforcer plugin passes)
@SpringBootTest application context loads successfully
All existing tests pass
Maven dependency tree documented

9.2. Phase 2 Checklist

DeviceSession interface defined with TR-069, MQTT, USP implementations
Pipeline handlers: Bootstrap, Inform, Profile, Task, ParameterPersist, Finalizer
All three protocols pass end-to-end tests through new pipeline
Performance benchmark shows no regression
Feature flag allows fallback to old flow

9.3. Phase 3 Checklist

All 162 SOAP methods have REST equivalents
SOAP ↔ REST parity tests pass for 100% of endpoints
OpenAPI 3.0 spec generated and validated
BroadcastingWebService (internal SOAP) replaced or prepared for Phase 4
API versioning (/api/v1/) in place
Authentication works on REST endpoints

9.4. Phase 4 Checklist

Kafka cluster deployed and operational
All 35+ event types have Kafka topic mappings
KafkaEventBusAdapter replaces DefaultAsyncEventBusImpl
Hazelcast Topics (5) removed, replaced by Kafka topics
SoapNodeNotifier removed
Event ordering tests pass (per-device ordering)
Event loss test passes (at-least-once delivery)
Consumer lag monitoring in Grafana

9.5. Phase 5 Checklist

All 80+ Hazelcast IMap caches migrated to Redis
All distributed locks migrated to Redisson
HazelcastCacheFactory removed
Hazelcast dependency removed from POM
Caffeine L1 cache added for hot maps
Lock correctness tests pass
Performance regression test passes (< 5% degradation)
Redis monitoring dashboard in Grafana

9.6. Phase 6 Checklist

Event Service deployed as independent application
Update Group Service deployed as independent application
QoE Service deployed as independent application
Contract tests pass between all services
Each service has independent CI/CD pipeline
Distributed tracing operational (Jaeger/Zipkin)
Per-service Grafana dashboards
Performance formula established and validated
Chaos testing passed (service failure resilience)

9.7. Phase 7 Checklist

cpe_changed_parameter in ClickHouse with dual-write
cpe_log in ClickHouse with dual-write
cpe_completed_task in ClickHouse with dual-write
cpe_trace_log in ClickHouse with dual-write
Query parity tests pass (MySQL vs ClickHouse)
Performance benchmarks show 10x+ improvement for analytical queries
Retention/TTL policies configured
Grafana dashboards updated to query ClickHouse

10. Appendix A: Current Hazelcast Cache Inventory

Cache Name Module Access Pattern Migration Batch (Phase 5)

Cache Name	Module	Access Pattern	Migration Batch (Phase 5)
`ispCache`, `ispMetaDataCache`	ftdm-cache	Read-heavy	Batch 1
`manufacturerByNameOuiCache`	ftdm-cache	Read-heavy	Batch 1
`productClassIdByModelManufProtocolCache`	ftdm-cache	Read-heavy	Batch 1
`profileCache`, `profileConditionCache`, `profileParameterCache`	ftdm-profile	Read-heavy	Batch 2
`profileParameterAccessCache`, `profileUploadCache`, `profileCpeIdCache`	ftdm-profile	Read-heavy	Batch 2
`cpeBlackListSerial`, `CpeWhiteListSerialSet`, `CpeWhiteListSerialOnlyCreated`	ftdm-security	Read-heavy	Batch 3
`cpeWhiteListManufCache`, `cpeSerialCache`	ftdm-security	Read-heavy	Batch 3
`CpeParameterIdNameCache`, `CpeParameterIdTypeCache`	ftdm-cache	Read-heavy, near-cache	Batch 4
`CpeParameterNameIdCache`, `CpeParameterNameIdsByTemplateCache`	ftdm-cache	Read-heavy, near-cache	Batch 4
`deviceIdBySerialCache`, `serialByDeviceIdCache`	ftdm-cache	Read-heavy	Batch 5
`cpeIdLastConnection`, `cpeIdInSessionQN`	ftdm-cache	Read/Write	Batch 5
`LOCKERTaskId`, `LOCKERCpeId`, `LOCKERNextSessionCpeId`	ftdm-core	Lock	Batch 6
`cpeIdsInCpeParamsModifyLock`, `qoeCpeSerialLock`	ftdm-core, ftdm-qoe	Lock	Batch 6
`eventMonitoringByGroupCache`, `hardcodedEventCache`	ftdm-events	Read-heavy	Batch 7
`qoeCpeCache`, `qoeCpeCacheState`, `qoeMonitoringByGroupCache`	ftdm-qoe	Read/Write	Batch 7
`transactionsInProgressLong`, `cpeIdsNeedToBeWaitForTasks`	ftdm-core	Read/Write	Batch 7
`CpeProvisionEncryptionProgressCache`	ftdm-provision	Read/Write	Batch 7

ispCache, ispMetaDataCache

ftdm-cache

Read-heavy

Batch 1

manufacturerByNameOuiCache

ftdm-cache

Read-heavy

Batch 1

productClassIdByModelManufProtocolCache

ftdm-cache

Read-heavy

Batch 1

profileCache, profileConditionCache, profileParameterCache

ftdm-profile

Read-heavy

Batch 2

profileParameterAccessCache, profileUploadCache, profileCpeIdCache

ftdm-profile

Read-heavy

Batch 2

cpeBlackListSerial, CpeWhiteListSerialSet, CpeWhiteListSerialOnlyCreated

ftdm-security

Read-heavy

Batch 3

cpeWhiteListManufCache, cpeSerialCache

ftdm-security

Read-heavy

Batch 3

CpeParameterIdNameCache, CpeParameterIdTypeCache

ftdm-cache

Read-heavy, near-cache

Batch 4

CpeParameterNameIdCache, CpeParameterNameIdsByTemplateCache

ftdm-cache

Read-heavy, near-cache

Batch 4

deviceIdBySerialCache, serialByDeviceIdCache

ftdm-cache

Read-heavy

Batch 5

cpeIdLastConnection, cpeIdInSessionQN

ftdm-cache

Read/Write

Batch 5

LOCKERTaskId, LOCKERCpeId, LOCKERNextSessionCpeId

ftdm-core

Lock

Batch 6

cpeIdsInCpeParamsModifyLock, qoeCpeSerialLock

ftdm-core, ftdm-qoe

Lock

Batch 6

eventMonitoringByGroupCache, hardcodedEventCache

ftdm-events

Read-heavy

Batch 7

qoeCpeCache, qoeCpeCacheState, qoeMonitoringByGroupCache

ftdm-qoe

Read/Write

Batch 7

transactionsInProgressLong, cpeIdsNeedToBeWaitForTasks

ftdm-core

Read/Write

Batch 7

CpeProvisionEncryptionProgressCache

ftdm-provision

Read/Write

Batch 7

11. Appendix B: Event Type Inventory

Event Class Published By Kafka Topic (Phase 4)

Event Class	Published By	Kafka Topic (Phase 4)
`CpeCreatedEvent`	`CpeDaoJdbcImpl`	`acs.device.lifecycle`
`CpeDeletedEvent`, `CpeDeletedBulkEvent`	`CpeDaoJdbcImpl`	`acs.device.lifecycle`
`CpeStatusChangedEvent`	`CommonOperations`	`acs.device.lifecycle`
`BootEvent`, `BootstrapEvent`	`InformHandler`, `MqttCpeService`	`acs.device.lifecycle`
`CpeParameterChangedEvent`	`CpeParameterDaoJdbcImpl`	`acs.device.parameters`
`CpeParameterCreatedEvent`	`CpeParameterDaoJdbcImpl`	`acs.device.parameters`
`CpeParameterDeletedEvent`, `CpeParameterMergedEvent`	`CpeParameterDaoJdbcImpl`	`acs.device.parameters`
`PendingTaskEvent`, `CpeTaskEvent`	`CpeTaskProcessor`	`acs.tasks`
`TaskStateBulkChanged`	`CpeTaskProcessor`	`acs.tasks`
`SetValueRollBackEvent`	Task handlers	`acs.tasks`
`ConfigurationParameterChangedEvent`	`DMConfigurationParameter`	`acs.config`
`SecurityConfigurationChangedEvent`	Security module	`acs.config`
`DistributedPushEvent`	Protocol modules	`acs.cluster`
`DistributedDisconnectCpeEvent`	Protocol modules	`acs.cluster`
`IspEvent`, `DomainAssociatedWithCpeEvent`	Admin operations	`acs.config`
`CustomRpcCreatedEvent`, `DownloadFileCreatedEvent`	Provisioning	`acs.tasks`

CpeCreatedEvent

CpeDaoJdbcImpl

acs.device.lifecycle

CpeDeletedEvent, CpeDeletedBulkEvent

CpeDaoJdbcImpl

acs.device.lifecycle

CpeStatusChangedEvent

CommonOperations

acs.device.lifecycle

BootEvent, BootstrapEvent

InformHandler, MqttCpeService

acs.device.lifecycle

CpeParameterChangedEvent

CpeParameterDaoJdbcImpl

acs.device.parameters

CpeParameterCreatedEvent

CpeParameterDaoJdbcImpl

acs.device.parameters

CpeParameterDeletedEvent, CpeParameterMergedEvent

CpeParameterDaoJdbcImpl

acs.device.parameters

PendingTaskEvent, CpeTaskEvent

CpeTaskProcessor

acs.tasks

TaskStateBulkChanged

CpeTaskProcessor

acs.tasks

SetValueRollBackEvent

Task handlers

acs.tasks

ConfigurationParameterChangedEvent

DMConfigurationParameter

acs.config

SecurityConfigurationChangedEvent

Security module

acs.config

DistributedPushEvent

Protocol modules

acs.cluster

DistributedDisconnectCpeEvent

Protocol modules

acs.cluster

IspEvent, DomainAssociatedWithCpeEvent

Admin operations

acs.config

CustomRpcCreatedEvent, DownloadFileCreatedEvent

Provisioning

acs.tasks