QUESTION: How can I understand the high speed log and interpret the data to see how HMP Elements is performing?
The media engine runs on a 10 millisecond cycle. It receives RTP packets, processes and routes audio, creates outgoing audio, and queues packets for transmission. The important question is whether it consistently completes that work within each 10 ms cycle. Timing values shown in parentheses are normally in microseconds (e.g. 1,000 µs = 1 ms)
As a practical guideline:
- Less than 1 ms of occasional variation is generally normal.
- Occasional delays between 1 and 10 ms are worth noting but do not prove there is a problem.
- Repeated delays approaching 10 ms indicate that the server is struggling to keep up.
- Delays of 10 ms or more mean an entire media-processing interval has been consumed or missed.
- A 100 ms slip is severe and causes HmpElements to reset its media clock.
Some of the relevant entries are:
SleepRequested: … Delta: … Total Slew: …
This reports how accurately the main media thread woke up for its scheduled cycle.
-
-
- SleepRequested is the offset at the beginning of the wait.
- Delta is the change while sleeping or waiting.
- Total Slew is the most useful value: the final scheduling error when processing begins.
- A negative Total Slew means the media thread is late. For example, -3500 uSec means it started approximately 3.5 ms late.
-
HmpElements only writes this entry when the final offset is more than approximately 1 ms. An isolated entry is not necessarily a concern. A continuous series of negative values, especially values approaching or exceeding -10000 uSec, indicates CPU or operating-system scheduling delays.
**Full 100ms Slip*** Clock Reset
The media thread fell at least 100 ms behind and reset its clock instead of trying to catch up. Even a single occurrence is significant. Repeated occurrences strongly indicate host-level stalls or resource contention and can produce audible gaps, delayed audio, or packets being sent in bursts.
Spin Error: … (… uSec)
The short precision wait overshot its target. Occasional small values are not usually harmful, but repeated errors—or errors measured in milliseconds—indicate that the thread is not receiving predictable CPU time.
Router: … – Bus: … – setTime: …
This breaks down a slow media cycle:
-
-
- Router is audio generation, mixing, transcoding, and routing.
- Bus is RTP receive/send processing.
- setTime is the time used to wake related DSP and fax processing threads.
-
This line is written when the combined time exceeds 10 ms, so recurring entries are a strong indication that the server cannot consistently complete a media cycle on time.
The following Avail / Rec / Proc / Send and NST / PST / ND / PSIL / TCP / DTMF / Feeder / Misc lines provide a more detailed internal breakdown. Some fields may be zero depending on the enabled code path.
OutgoingRtpPacketManager::Sender() Proc > 10000: … Queues Processed: …
The RTP packet-sending thread either took more than 10 ms to process its work or found multiple batches queued. An occasional entry can occur during a short load spike. Repeated values over 10,000 µs or a growing queue count indicate that outgoing RTP packets are backing up.
PcapWorker::MainCodeSender – Proc > 10000 Or Intermittent Report
This entry is also printed periodically as a status report, so its presence alone is not an error. Check the actual value: less than 10,000 µs on an isolated periodic report is expected; repeated values above 10,000 µs are concerning.
The following entries indicate actual packet-loss conditions and should always be investigated:
-
- There are no EmptyOutgoingRtpPacketLists Available. All Packets this round will not be sent.
- Transmit Warning. Sent/Expected…
- Increasing Pcap stats Drop or ifdrop counters
- Socket errors followed by Locked out port
- Detected stream loss on port
- PacketReceived() Port Locked. Packet Dropped
- Audio buffering failures
One isolated delay does not normally identify a hardware problem. The strongest evidence is a sustained pattern of late media cycles, 10 ms processing overruns, sender queue buildup, packet-drop warnings, or 100 ms clock slips. These patterns usually point to host scheduling delays caused by CPU saturation, virtual-machine contention, power management, driver/DPC stalls, or other software competing for CPU time.