Testing Push Notification Delivery on Android Before Trusting Analytics
Push analytics usually report on the sender's intent. A campaign goes out, the dashboard counts it, and open rates are computed against a denominator that includes messages no phone ever displayed. On Android the gap between what FCM accepts and what a device receives is wide enough to distort product decisions, and it is measurable if you stop trusting the dashboard long enough to instrument the path yourself.
The Analytics Dashboard Lies First
Most push dashboards open with a delivery rate, and that number almost always means the message was accepted by the push service. Firebase Cloud Messaging returns a message ID when it takes a payload. Vendors that wrap FCM inherit that definition and rarely correct it, because a delivery rate near 100 percent makes the product look healthy.
The failures that matter are silent. A message deferred by Doze, blocked by a disabled notification channel, or aimed at a stale token produces no error at send time and no open at receive time. It simply vanishes from the funnel. Analytics tools built on top of FCM see the same gaps the platform does, so the dashboard and the SDK agree with each other while both disagree with the phone.
This is the same pattern this site has described in maintainer metrics that stay green while the underlying work degrades. The instrument reports on what it can see, and the operator mistakes that for the whole system.
Android Delivery Is a Pipeline Problem
FCM is the first hop, not the destination. Google's servers queue the payload, then hand it to the device through a persistent connection maintained by Google Play services. From there the message passes through the operating system's notification manager, which applies channel settings, importance levels, and per-app restrictions before anything is drawn on screen.
Doze and App Standby sit in the middle of that pipeline. When a phone is stationary and unplugged, Android defers background work and batches network access into maintenance windows. High-priority FCM messages are designed to wake the device, but the exemptions are narrower than most teams assume, and a message that arrives without the high-priority flag can wait hours.
Since Android 8, notification channels gate visibility. A user who muted a channel or dropped its importance to the lowest setting has effectively opted out of that category, and the app receives no signal that it happened. OEM builds add another layer. Several manufacturers ship battery managers that kill background processes more aggressively than stock Android, and their behavior changes between firmware releases with little documentation.
The order matters when you are debugging. A message can clear FCM, clear Play services, and still die at the notification manager because the channel was muted months ago. Working backward from the device log is the only reliable way to tell which stage ate it.
What FCM Actually Confirms
A successful FCM response means Google accepted the message for delivery. It does not confirm that a device received it, that the notification manager displayed it, or that a user saw it. Delivery receipts are not part of the standard FCM contract for Android, so there is no server-side callback that closes the loop.
Token rotation breaks targeting without warning. Registration tokens expire when an app is reinstalled, restored to a new device, or left idle long enough for the token to be recycled. Sends to a dead token may return an error like UNREGISTERED, but plenty of stale tokens fail quietly for a period before the service flags them.
Collapse keys cause a different kind of loss. When a sender sets a collapse key, FCM may replace an undelivered message with a newer one carrying the same key. That is correct behavior for a sync trigger and wrong for anything the user expects to see individually. A batch of five alerts can arrive as one, and the dashboard still counts five sends.
There is also a quota dimension that rarely shows up in dashboards. FCM throttles sends per app instance and per project under load, and a burst campaign can hit those limits while individual responses still look successful. If you fire a large batch at once, spread it over a window and watch the device-side receipt rate rather than the acceptance rate; the two diverge fastest exactly when volume is highest.
Instrumenting the Real Delivery Path
Receipt has to be logged on the device. A small handler in the app's messaging service that records message ID, timestamp, app version, and channel state at the moment of receipt gives you a ground truth the server cannot produce. Ship those events to your own backend rather than relying on an analytics SDK's automatic push tracking, which often fires on display rather than receipt.
Token refresh needs explicit tracking. Log every new token with the install ID and the reason if the platform exposes one, and reconcile those events against your send logs. A spike in token churn usually explains a drop in delivery that looks like a platform problem.
Channel state should be captured at send time, not at open time. Record the channel's importance and whether notifications are enabled for the app when the payload goes out, so you can separate a targeting failure from a user preference. The same discipline applies to certificate and key rotation on the server side, which a related piece on mTLS rotation without automation covers in the transport layer.
Finally, test on OEM builds. A Pixel running stock Android is the least representative device in your fleet. Samsung, Xiaomi, and Oppo handsets apply battery and autostart policies that stock Android does not, and those policies are where most silent drops originate.
Budget for the storage side too. Device receipt events arrive out of order, sometimes days late when a phone sat in a drawer, so your schema needs an event-time column separate from ingestion time and a dedupe key on message ID plus install ID. Without both, a delayed batch will double-count receipts and make a healthy campaign look like it over-delivered.
A Test Matrix That Finds Drops
Cover at least three OEM families in the device lab, and pick models that ship the manufacturers' own battery managers rather than near-stock builds. Run the same send script against each and compare device logs to the FCM diagnostics in the Firebase console. The console shows acceptance and error codes; only the device log shows receipt.
Exercise Doze deliberately. Put a test device on a table, unplug it, and leave it idle long enough for the screen to turn off and the maintenance windows to close. Send a normal-priority message and a high-priority message, then measure the gap between send time and receipt time on each. That gap is the number your product actually experiences.
Force token invalidation in staging by clearing app data or reinstalling, then send to the old token and record what comes back. Compare that response against what your production code does with it. Many apps log the error and keep the stale token in the database for weeks.
Run the matrix on a schedule rather than once. OEM battery policies shift with firmware updates, and a device that delivered reliably last quarter can start dropping after an over-the-air patch nobody announced. A weekly automated run against the same three handsets catches that drift before your users do.
Actions Before Trusting the Numbers
- Add a device-side receipt logger that writes message ID, timestamp, channel importance, and app version to your own backend before any analytics SDK sees the event.
- Audit your test fleet for OEM coverage and add at least two manufacturer builds with aggressive battery managers to the regular regression run.
- Treat every FCM acceptance count as a ceiling and label it that way in dashboards, so nobody reads it as confirmed delivery.
- Reconcile analytics open rates against raw device receipt events weekly, and investigate any campaign where the two diverge by more than a few points.
- Document notification channel behavior per Android version and OEM in your internal runbook, since the same channel setting can mean different things on different firmware.
- Store receipt events with both event time and ingestion time, keyed on message ID plus install ID, so late arrivals do not corrupt the delivery rate.
The cost is real. Device-side logging adds an endpoint, a schema, and a retention policy, and OEM testing multiplies the hardware you maintain. Teams that skip it are not measuring delivery. They are measuring their own optimism, and the dashboard will keep agreeing with them.