System_server crashes leading to continious restarts (looping) after update

gernericuserxyz

Hi everybody,
I got a huge problem after (automatically) updating my Pixel 7 to 2024111800. My hope is that somebody can help out.

I already tried to find some advice on the GOS general Matrix chat, but did not get much information / replies there. Probably it is the wrong platform for such a detailed request...

On Thursday GOS updated to 2024111800. After the update, I rebooted as asked. I guess that the app optimizations went through, I'm not sure though since I usually don't sit next to the phone while its updating. The phone booted normally (Google splash screen and GraphneOS splash screen). After that I was asked to enter my SIM PIN and my device code. So far so good. A few seconds after logging in a notification with "system_server crash" appeared and the system slowed down tremendously. I just had enough time to open the message when the GrapheneOS splash screen appeared, the devices rebooted and I was promted to re-enter my device code, but not the SIM code for some reason. After logging in again multiple system_server crashed popped up, the system slowed down and rebooted after 15 seconds, tops. Bottom line is that the system is unusable in this state. I managed to capture a picture of two of the crash logs, since there is not enough time to export them all and the system is extremly laggy. It appears to be some sort of Java OutOfMemoryError. Instrestingly they don't have the same call stack. What I also noticed is that one app is missing in the dock. Might sound strange but I really have no clue which one it is. If I had to guess its the Phone application. Furthermore my combined weather + clock widget is missing. Please find two of the call stacks attached here: crash1, crash2 and crash3
On top of that every few reboots a kernel panic occurs. But its decreasing the system performance so bad that I have no chance to capture any picture of it before rebooting.

What I've tried so far is removing the SIM card, booting in safe mode (entered via the UI -> long press on restart button) and turning on Airplane Mode and turning off mic, location etc. None of that changed anything. System is still crashing in safe mode. So I guess its not a third party app that is causing that, right?

My main objective is to extract my data since the last backup is quite some time ago (I know, very bad).
After reading here some forum posts I'm planing to install/flash the previous build using adb. But if I understood it correctly from the documentation (GOS_docs) downgrading is not possible. So the only hope is to wait for a new build and to hope that this will fix that, right? While waiting for the latest build I have several questions. Maybe somebody can answer one or more of the following questions:

Is sideloading really possible when the bootloader is closed and USB debugging is turned off? Will it somehow affect my user data? What go wrong?
Doesn't the Pixel have two banks? Wouldn't it be possible to switch banks in order to boot the previous and for my device more stable build?
Woult you advice to report this to the official GitHub repo?

I would really appreciate any help and opinions on that matter!

Kind regards,

generic

de0u

gernericuserxyz Is sideloading really possible when the bootloader is closed and USB debugging is turned off?

https://grapheneos.org/usage#updates-sideloading

gernericuserxyz Will it somehow affect my user data?

It should not.

gernericuserxyz Doesn't the Pixel have two banks? Wouldn't it be possible to switch banks in order to boot the previous and for my device more stable build?

In general nothing that downgrades the OS or firmware to an older version can be done without wiping the user data partition. If you could revert to the previous OS, an adversary could do it too.

gernericuserxyz Would you advice to report this to the official GitHub repo?

If it crashes in very-different ways that might indicate a hardware problem, and unfortunately the best way to check is to wipe the user data partition to see if the problem goes away. Maybe gather some more crashes to see if there is a pattern?

gernericuserxyz

@de0u Thanks for your fast reply!

de0u In general nothing that downgrades the OS or firmware to an older version can be done without wiping the user data partition. If you could revert to the previous OS, an adversary could do it too.

So basically I have to wait for the next signed OTA update which I could try to sideload. Correct?

de0u If it crashes in very-different ways that might indicate a hardware problem, and unfortunately the best way to check is to wipe the user data partition to see if the problem goes away. Maybe gather some more crashes to see if there is a pattern?

The only pattern is that after a full (graceful) shutdown I have to provide the SIM PIN prior to the device code. Other than that its really kinda random. From what I observed is that the call stacks always state OOM as the error reason while the call stack and the included applications vary (as it can be seen inside the pictures). A hardware issues would be really bad, esspecially if that indicates that I can't sideload and backup my data afterwards. However I guess there is not much I can do but waiting for the next build and try to sideload.

de0u

gernericuserxyz So basically I have to wait for the next signed OTA update which I could try to sideload.

Yes.

But for users in the past who have either had a hardware problem or have had some bad misconfiguration of the user data partition, sideloading a new system image hasn't solved the problem. If the problem isn't in the system partition, getting a new system image probably won't resolve the problem -- unless the new system includes a fix targeted specifically at the particular problem. That requires a clear understanding of what's going wrong. Overall, sideloading an OTA image is fairly easy and should be safe, so there's no reason not to try it, but there is a fair chance it won't address your issue (and that trying again next time also won't).

de0u

gernericuserxyz The only pattern is that after a full (graceful) shutdown I have to provide the SIM PIN prior to the device code. Other than that its really kinda random. From what I observed is that the call stacks always state OOM as the error reason while the call stack and the included applications vary (as it can be seen inside the pictures).

Personally I'm not getting much from those stack traces. You mentioned kernel panics, which might provide more information. Might you be able to video-record a kernel panic from some other device and capture a frame?

Also, am I misreading the uptimes at the top of those crashes? For example, 325573 milliseconds would be 325 seconds or five minutes?

gernericuserxyz

de0u But for users in the past who have either had a hardware problem or have had some bad misconfiguration of the user data partition, sideloading a new system image hasn't solved the problem. If the problem isn't in the system partition, getting a new system image probably won't resolve the problem -- unless the new system includes a fix targeted specifically at the particular problem. That requires a clear understanding of what's going wrong. Overall, sideloading an OTA image is fairly easy and should be safe, so there's no reason not to try it, but there is a fair chance it won't address your issue (and that trying again next time also won't).

Since the behaviour started right after an update, I really hope it was that. Another point which contradicts user configurations or a third party app is, that even in safe mode the bevahiour isn't changing. A hardware issue could be though.

System_server_crash

de0u
Hi, I am not entirely sure if I can help here, but
I had a bug with similar symptoms.
I rebooted my phone after the update, and when I tried to enter my pin, the phone basically instantly rebooted. But it wasn't a normal reboot. It had the graphene OS logo instantly. No Google logo, and the reboot was much quicker. It then went in this semi-reboot circle or would let me type the password in, after which it instantly semi-rebooted again. In that state, the phone was really buggy, like not recognizing some taps. After leaving it around for a while and coming back. I tried to unlock it again, and it worked. I had like 20 notifications that said that the system_server has crashed.
But I am not sure if it is the same bug because the crash log doesn't look that similar other than both being a system_server crash. I unfortunately only saved one of the reports, so I can't say if the other reports looked more similar. (Or is there a way to view non saved crash reports?). I also made a video with another device on how it looked; if that would help, I could post it tomorrow.
Sorry if this is a different bug, but the symptoms are pretty similar.
Crash report:
``
type: crash

SystemUptimeMs: 52823
Process: system_server
Timestamp: 2024-11-21 17:49:39.618+0100
Build: google/shiba/shiba:15/AP3A.241105.007/2024111800:user/release-keys
Crash-Handler: com.android.internal.os.RuntimeInit$KillApplicationHandler
Loading-Progress: 1.0
Dropped-Count: 0

java.lang.UnsatisfiedLinkError: dlopen failed: library "libicing.so" not found
at java.lang.Runtime.loadLibrary0(Runtime.java:1081)
at java.lang.Runtime.loadLibrary0(Runtime.java:1003)
at java.lang.System.loadLibrary(System.java:1765)
at com.google.android.icing.IcingLibraryLoader.loadLibrary(IcingLibraryLoader.java:9)
at com.google.android.icing.IcingSearchEngineImpl.<clinit>(IcingSearchEngineImpl.java:41)
at com.google.android.icing.IcingSearchEngine.<init>(IcingSearchEngine.java:72)
at com.android.server.appsearch.external.localstorage.AppSearchImpl.<init>(AppSearchImpl.java:318)
at com.android.server.appsearch.external.localstorage.AppSearchImpl.create(AppSearchImpl.java:272)
at com.android.server.appsearch.AppSearchUserInstanceManager.createUserInstance(AppSearchUserInstanceManager.java:230)
at com.android.server.appsearch.AppSearchUserInstanceManager.getOrCreateUserInstance(AppSearchUserInstanceManager.java:102)
at com.android.server.appsearch.AppSearchManagerService.lambda$onUserUnlocking$1(AppSearchManagerService.java:361)
at com.android.server.appsearch.AppSearchManagerService.$r8$lambda$7li-QUZvrsDk5ZvsFzpgCXuGV7s(AppSearchManagerService.java:0)
at com.android.server.appsearch.AppSearchManagerService$$ExternalSyntheticLambda3.run(R8$$SyntheticClass:0)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:644)
at java.lang.Thread.run(Thread.java:1012)``

gernericuserxyz

de0u Personally I'm not getting much from those stack traces. You mentioned kernel panics, which might provide more information. Might you be able to video-record a kernel panic from some other device and capture a frame?

Also, am I misreading the uptimes at the top of those crashes? For example, 325573 milliseconds would be 325 seconds or five minutes?

Yes they occured sometime after a bunch of reboots. Its hard to capture them since the the device is super laggy and it's a gamble if the log opens before the system crashes. I will try on the weekend to capture one example.

I also noticed the very system uptime. It simply can't be. Yesterday I checked just for fun and measured the time. It took me less than one minute to get to the crash log while the log indicated roughly 3.5 minutes. Not sure whats going on there.

de0u

System_server_crash java.lang.UnsatisfiedLinkError: dlopen failed: library "libicing.so" not found

I don't know what's going wrong for you, but I suspect the key part of the crash is the part I quoted, and I don't think I've seen people report that before.

gernericuserxyz

System_server_crash I rebooted my phone after the update, and when I tried to enter my pin, the phone basically instantly rebooted. But it wasn't a normal reboot. It had the graphene OS logo instantly. No Google logo, and the reboot was much quicker. It then went in this semi-reboot circle or would let me type the password in, after which it instantly semi-rebooted again. In that state, the phone was really buggy, like not recognizing some taps. After leaving it around for a while and coming back. I tried to unlock it again, and it worked.

Thanks for your reply! :)
Apart from the last part this discription matches quite good. For me unfortunately it never recovers. I left the phone turned on without logging in (by that I mean without providing the passcode for the first time) the entire day yesterday. After entering the code unfortunately the same behaviour occured and the Graphene logo appeared.

Edit:
If you don't mind and don't leak anything private your video would be very interesting. I will also try to capture one on the weekend.

gernericuserxyz

Update: on Saturday I was able to get the phone out of that state by sideloading (using adb sideload) 2024112700 stable release. Today, so just a couple days later, it appeared again. First one system_server crash, followed by the GrapheneOS boot logo. After that even in lock screen the crashes come in like there is no tomorrow. After unlocking there is the same pattern: the device freezes withn 5 seconds tops. So far I could only spot system_server crashes and occasionally kernel panics. Unfortunately I was never able to caputre the kernel panics since it somehow takes longer to load the kernel panic crash logs.
I really don't know what is going on. Somehow I do not believe that this is some hardware defect since it was running for a couple days under heavy load without any flaws.

Is there any way to retrieve these logs in case I should be able to recover the phone once again?

de0u

gernericuserxyz Somehow I do not believe that this is some hardware defect since it was running for a couple days under heavy load without any flaws.

Unfortunately, given these symptoms, I suspect that a hardware problem is likely, and that the next steps to figure that out will need to be disruptive.

Maybe wait to see whether people agree with me or suggest a different path, but I would suggest:

If you can manage it, enable developer mode and enable OEM unlocking,
WIpe the user data partition (ouch!) by booting into recovery and doing a "factory reset",
If that doesn't solve the crashing, flash the device back to Google's OS and use it for at least a week,
If that doesn't solve the problem, try a repair shop (or warranty service, if available).

Since flashing an update didn't solve the problem, I doubt waiting and doing it again will do any better. The only other thing I can think of to try is to capture information about one or more of the kernel panics.

But overall I think the likelihood is fairly high that the device is failing.

gernericuserxyz

Hi @de0u,
thanks for your reply.
Another interesting bevahiour: the device automatically updated to 2024120200 followed by 2024120400 which both didn't change the behaviour -> system server crashes with OOM as stated root cause. Additionally some sporadic kernel panics.
I now wonder how the update could be triggered. Sure, I entered the device pin a couple of times in order to check the bevahiour, but is this really enough to trigger the updates? Once this process is started will it continue after an unexpected reboot due to e.g. power loss or a system crash? After noticing that, I turned back on airplane mode, which hopefully will lead the next sideloaded OTA to recover the phone once again.

Do I need developer mode and OEM unlocking in order to wipe user data partition? There is no change to get to that menu, even with several attempts. The device is freezing to fast.

@Moderation Could you please change the "Server_service" to "System_server" in the title? This is clearly not true.

de0u

gernericuserxyz the device automatically updated to 2024120200 followed by 2024120400 which both didn't change the behaviour -> system server crashes with OOM as stated root cause. Additionally some sporadic kernel panics.

If multiple updates have not improved the situation, it is not particularly likely that more updates will. OTA sideloading via adb is not more likely to improve the situation than OTA updates that happen OTA.

gernericuserxyz Do I need developer mode and OEM unlocking in order to wipe user data partition?

No. Follow online directions for booting into recovery and doing a "factory reset". Note that if the flash storage is sufficiently bad then a factory reset might permanently brick the device.

grapheneos-user

@gernericuserxyz
Did you solve this problem?

I have the same problem since today: Pixel 6 pro is instantly crashing after reboot.

de0u

grapheneos-user I have the same problem since today: Pixel 6 pro is instantly crashing after reboot.

Is it the same problem, meaning System Server crashes with stack traces implicating libicing.so? If not, starting a different thread would be good, stating as many details as possible. Unfortunately there are lots of reasons for a crash while booting, and the details matter.

grapheneos-user

de0u Thanks a lot. You are right. It's a different problem. I will do this. Sorry, for my false handling of the forum.