[BUGFIX] Keep the site running if the Redis server is unavailable - #35
Merged
Conversation
The connection to Redis is established in the constructor, and a RedisException thrown there was not caught. As LockFactory neither catches exceptions nor falls back to another strategy, an unavailable Redis server took down the whole frontend with a 503 error. Connection problems are now logged and handled by a fallback locking strategy (TYPO3's FileLockStrategy by default, configurable via "fallbackStrategy", null to skip locking entirely), both when the connection cannot be established and when it breaks during a request. An unreachable server is not contacted again for 30 seconds within the same PHP process, so a request does not run into the connection timeout for every single lock it creates. Set "gracefulDegradation" to false to get an exception instead, which is then a LockCreateException. Additionally, the logger is fetched lazily, as LockFactory instantiates locking strategies via "new" instead of makeInstance(), so logging a Redis problem ended in "Call to a member function critical() on null". Fixes #34 Fixes #27
|
@bmack Tested it in a customer environment using Thanks! |
Member
Author
|
@copilot resolve the merge conflicts in this pull request |
…gradation # Conflicts: # src/RedisLockingStrategy.php Co-authored-by: bmack <165630+bmack@users.noreply.github.com>
Contributor
Resolved by merging |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fixes #34, fixes #27.
Problem
The Redis connection is established in the constructor (
connectBackend()), and aRedisExceptionthrown there is not caught. Core does not cushion that either:LockFactory::createLocker()instantiates the highest-priority strategy withnew— no try/catch, no fallback to the next strategy — and it is called viaResourceMutexin thePrepareTypoScriptFrontendRenderingmiddleware. A Redis outage therefore takes down the whole frontend with a 503, which is what happened in #34.The same applies when the connection breaks while a request is running:
lock()then returnsfalse, the non-blockingacquire()throwsLockAcquireWouldBlockException, andResourceMutexkeeps polling untilmax_execution_timeis hit.Solution
Connection problems are logged and handled by a fallback locking strategy instead of ending the request:
gracefulDegradation(bool, defaulttrue) — survive an unavailable Redis server. Set tofalsefor the previous behaviour, which now throws aLockCreateExceptioninstead of a rawRedisException.fallbackStrategy(class name, defaultTYPO3\CMS\Core\Locking\FileLockStrategy) — the strategy used while Redis is unavailable. Set tonullto run without any locking in that case: the same page may then be generated by several processes in parallel, which is still better than a broken website. The same happens if the configured fallback strategy itself cannot be created.lock(),wait(),unlockAndSignal()), not only on connect.While at it: the logger is fetched lazily, because
LockFactoryinstantiates locking strategies vianewinstead ofmakeInstance(), so the logger is never injected and logging a Redis problem ended in "Call to a member function critical() on null" (#27).The README documents the new options, the behaviour on an outage and the previously undocumented
connectionTimeout/persistentConnectionoptions.Verification
Manually, against a TYPO3 v13 installation with phpredis 6.3 / PHP 8.5:
fallbackStrategy => null: no exception, locking is skipped.gracefulDegradation => false:LockCreateException.