LwM2M Scale Mode and Load Testing

Scale mode lets one emulator container register and hold thousands of LwM2M devices (multi-device runs with clientsCount x concurrentClients). It is off by default; with it off the emulator behaves exactly as before.

Enabling scale mode

Switch Default Description

FTEMUL_VIRTUAL_THREADS=true (environment) or -Dftemul.virtualThreads=true (JVM property)

off

Turns scale mode on. Read once at JVM start; changing it needs a restart. Either source set to true enables it.

-Dftemul.sharedPoolSize=<n>

mode ON: max(4096, planned devices); mode OFF: CPUs x 4

Size of the shared pool that runs every device’s Observe, firmware/software update and reboot work. Overrides both defaults.

When the mode is ON:

  • The emulator’s internal thread pools run on virtual threads. Per-device work runs on per-device executor views over one shared timer, so an idle device holds about 0 platform threads (measured locally: 0 platform threads per device against 6 to 7 with the mode off).

  • Each device still has its own UDP socket and its own serialization, so a response_delay or notify_delay sleep blocks only that device.

  • The startup log carries event=emulator-scale-mode virtualThreads=true shadow=…​ carriers=<n>. If the emulator cannot switch every pool, it refuses to start (event=emulator-scale-mode-refused) rather than run half-virtual.

  • Use -Djdk.virtualThreadScheduler.parallelism=<n> to change the number of carrier threads (default: the container’s CPUs).

When the mode is OFF (default), and for a single device in either mode, behaviour is unchanged.

A load run also needs a high file-descriptor limit (one per device, for example --ulimit nofile=65536:65536) and a fromPort below the container’s ephemeral range (for example 20000). The shipped default 56830 caps a container at about 8,700 devices.

Per-device credentials

Both options apply to the device’s security mode. They are read when a run is initialised (/api/init) and when a device starts; a change pushed to a running fleet does not alter the identity of devices that already started.

PSK per device

Set lwm2m.psk.per_device=true:

  • The PSK identity of each device is its endpoint name.

  • The key is the first 16 bytes of HMAC-SHA256(key = template PSK key, data = UTF-8 endpoint name). The template key is the hex-encoded PSK key configured on the connection, and must be at least 16 bytes.

  • A server can therefore provision every device from the formula alone.

  • Absent or false: every device uses the template identity and key, as before.

X.509 from a CA

With the X.509 security mode and lwm2m.x509.ca.keystore set, every device gets its own certificate minted at start (no EST involved).

Key Default Description

lwm2m.x509.ca.keystore

(unset)

Path, inside the container, of the CA PKCS#12 file (mount it read-only). Unset: the device uses the keystore-based credential as before.

lwm2m.x509.ca.alias

the store’s only key entry

Alias of the CA key entry.

lwm2m.x509.leaf.validity_days

365

Validity of each minted leaf, in days. A non-positive number, or one that ends beyond the supported date range, fails device creation.

lwm2m.x509.leaf.chain

leaf_and_ca

What the device presents: leaf_and_ca (leaf plus issuing CA) or leaf (leaf only). Anything else fails device creation.

FTEMUL_X509_CA_PASSWORD_FILE (environment)

(unset)

Path of a file holding the CA keystore password. This is the only password source: FTEMUL_X509_CA_PASSWORD is never read (it is reported as ignored with a warning), and the password is never a REST field and is never logged.

The minted leaf has CN equal to the device’s endpoint name, an EC P-256 key, a random serial, clientAuth extended key usage and is signed by the CA key. A self-signed CA is not presented as part of the chain, so leaf_and_ca then presents the leaf only. The certificate is kept per endpoint for the run: a reboot or wake keeps it, a new /api/init mints anew. Minting and EST enrollment on the same device are mutually exclusive and fail device creation with a clear message.

Mount the same CA into every container of a fleet: the server then needs one trust entry for all of them.

Transport knobs

Key Default Description

ack_timeout

10000

CoAP acknowledgment timeout in ms. Must be an integer of at least 1; 0, negative and non-numeric values are now refused at device creation (they were accepted before).

lwm2m.transport.protocol_stage_thread_cnt

1

Protocol-stage threads per device. Same validation (at least 1). It keeps its effect in scale mode: it sets how many tasks the device runs in parallel.

lwm2m.transport.receiver_thread_cnt

1

Receiver threads per device. Same validation (at least 1).

lwm2m.transport.sender_thread_cnt

1

Sender threads per device. Same validation (at least 1).

lwm2m.scale.dtls.receive_buffer

2048

Scale mode only: the largest inbound DTLS datagram in bytes. Range 1500 to 16490; any other value fails device creation. Read once per run at /api/init, never from a *_gui.properties file. Ignored when the mode is OFF.

A DTLS datagram larger than the limit is counted and logged as event=dtls-datagram-over-limit deviceId=…​ peer=<address:port> sizeAtLeast=<limit+1> limit=<limit> count=<n>: the first one per endpoint start, then every 1,000th. Raise the limit if a handshake flight (for example an X.509 chain) does not fit.

Devices with identical ack_timeout and thread-count values share one transport configuration (at most 16 distinct sets per JVM; further sets are private to the device, logged once as event=network-config-cap-reached).

Behaviour changes

  • WRITE-ATTRIBUTES sent to an instance that the server created (a server-Created instance) is now applied. It was dropped before.

  • Invalid transport knobs (see above) are refused when the device is created instead of being accepted.

Load testing

The repository ships a capacity harness under loadtest/. Its README.md is the reference for the knob table, verdicts and the host launch; it is not repeated here.

  • emulator-capacity.sh: capacity tool. Steps a fleet up (registered devices, threads, memory per device, the point where it breaks) for five arms: scale mode OFF and ON, each with no security, PSK and (ON only) X.509.

  • emucap-host.sh: host launcher. Runs all arms under one operations lock on the demo host, pinned off the QA CPUs, and tears down by label. Its window (WINDOW=any|night), give-up and hard-stop times are documented in loadtest/README.md, section "Demo host launch".

  • emucap-native-measure.sh: native measure tool. Runs the emulator jar directly (no container, loopback only) in scale mode and measures the live heap per device, with a PASS/FAIL verdict against the slice’s target.

Run the authoritative numbers on the demo host only; local runs are indicative.