OpCache won't restart/stuck on restart_pending

[tm8747a]

I have a strange issue happening. Our server configuration is less common, we're using PHP on Windows over IIS using FastCGI, not sure if relevant to the problem or not. PHP version is 7.4.12. What happens is that running opcache_reset() on our prod servers just hangs. The opcache just sits in restart_pending = true and never seems to restart. I have not really found a pattern on if a specific thing/cached file causes it to lock up. I did setup another site on the same server, pointed it to an identical copy of PHP with the same configuration, and when I run the app on that site it opcache_reset() works fine. There are definitely more cached files on prod, somewhere around 1,300 vs 700 on the tests I've done, but the memory is nowhere near filled up (there is more free memory than used), number of cached keys is nowhere near the max, etc. I've found no other place where it fails (works on all dev machins, QA, stage, etc.), so am thinking it may be related to load? Has anybody experienced anything like this?

[gizmola]

I can only offer a couple of educated guesses.  It's real tough to try and debug something you can't replicate.  

opcache_reset uses a semaphore lock mechanism, and only does the reset when the php script exits.  If the script hangs or does not exit, it is possible it will be left in this state.   You also need to be sure that you don't have a race condition where there are 2 processes both trying to run opcache_reset -- neither of which will complete as they have deadlocked.

[tm8747a]

On 4/26/2021 at 2:26 PM, gizmola said:

I can only offer a couple of educated guesses.  It's real tough to try and debug something you can't replicate.  

opcache_reset uses a semaphore lock mechanism, and only does the reset when the php script exits.  If the script hangs or does not exit, it is possible it will be left in this state.   You also need to be sure that you don't have a race condition where there are 2 processes both trying to run opcache_reset -- neither of which will complete as they have deadlocked.

You may be onto something. I was able to try a reset after hours and it worked just fine. So I think the traffic is definitely the problem and there's a lock somewhere that cannot be released. I'll dig some more.