Anatomy of a production-only auth failure

The message every developer dreads arrived through Upwork: an app with 50,000+ users — a Bhagavad Gita reading app that people use daily — and Google Sign-In had stopped working in production. New users couldn't get in. The previous developer wasn't available. Every dev build worked perfectly.

"Works in dev, fails in prod" is a bug class with its own personality, and auth is its favourite habitat. This is the story of that rescue — what the root cause turned out to be, and the checklist I now run on every Flutter app that ships Google Sign-In.

Why auth breaks only in production

Most production-only failures come from the same underlying fact: the app you debug is not the app users install.

On Android with Google Sign-In, identity is anchored to the app's signing certificate. Google's servers only hand out auth tokens if the requesting app's package name and certificate SHA-1 fingerprint match a registered OAuth client. And here's the trap: with Play App Signing, Google re-signs your release with a key you don't hold locally. So there are at least three different certificates in play:

• your debug keystore (what your emulator builds use),
• your upload key (what you sign the bundle with),
• the app signing key (what Google actually signs the store build with).

Register the first two and forget the third, and you get exactly this symptom: every build a developer can produce works, and the one build users actually install fails. The failure is invisible from the codebase because it isn't in the codebase.

Reproduce first, theorise second

The first real step wasn't reading code — it was getting the failure in front of me. An internal-track build installed from the Play Store reproduced it immediately: account picker appears, user picks an account, then the silent failure — in the logs, the infamous status code 10, DEVELOPER_ERROR.

That error code narrowed the search from "anything" to "configuration": the app's signature or client registration doesn't match what Google expects. From there it was a methodical audit of the whole OAuth chain across the Google Cloud Console and Firebase:

1. Certificate fingerprints — which SHA-1s were actually registered, versus the fingerprint of the certificate Play uses to sign the delivered app (visible under Play Console → App integrity).
2. OAuth client credentials — whether the Android OAuth client in Google Cloud matched the live package name and certificate, and whether the config files shipped with the app were generated after those clients existed.
3. API enablement and permissions — the identity APIs the flow depends on being enabled for the correct Google Cloud project. Projects multiply over an app's lifetime; the credentials being checked and the project serving the app are not always the same one.

The audit found the mismatch in exactly that chain: the production signing certificate was never registered with the OAuth configuration, and adjacent credential and API-permission issues had accumulated around it — the kind of drift that happens when an app changes hands between developers over the years.

The fix is small; the verification isn't

Configuration fixes are one-line changes with system-wide blast radius, and this app had tens of thousands of people using it daily. So the fix went out the careful way:

• corrected the OAuth registration and refreshed the app's Firebase config,
• verified on an internal Play track — installed from the store, since that's the only build that proves anything,
• tested the full matrix: new Google sign-up, returning user, signed-out re-auth, plus account states we could simulate,
• then a staged rollout while watching sign-in success in the console before going to 100%.

Sign-in came back for the full user base, and the client left a five-star review. The whole engagement reinforced something I tell every client: rescues are mostly diagnosis. The edit took minutes; knowing which edit, and proving it safe for 50,000 people, was the job.

The checklist I now run on every release

If you ship Flutter with Google Sign-In, run this before launch and after any change of keys, developers, or Cloud projects:

1. Register all three SHA-1s — debug, upload, and the Play App Signing certificate — in Firebase, and re-download google-services.json afterwards. (Fingerprints are baked into that file's OAuth entries; registering a new one without refreshing the file does nothing.)
2. Test the store build, not the local build. An internal testing track exists precisely so you can install the Google-signed artifact before users do.
3. Know which Cloud project is real. Audit that OAuth clients, enabled APIs, and the Firebase project all agree — especially on inherited codebases.
4. Treat DEVELOPER_ERROR as a config diff, not a bug hunt. The answer is in a console, not in your Dart code.
5. After fixing auth config, roll out staged. Auth touches 100% of users; give yourself a checkpoint at 10%.

Production-only bugs feel mystical right up until you name the difference between the two environments. In auth, that difference is almost always identity: certificates, credentials, project bindings. Audit the chain end to end and the mystery evaporates.