Repository navigation
fuse: throttle foreground io-uring requests with the writeback cache - #241
hazhou-ddn wants to merge 1 commit into
Conversation
With the writeback cache, foreground requests can wait in userspace on DLM revokes. A revoke waits on page invalidation, which waits on writeback. If foreground requests held every entry of an io-uring queue, background writeback could not be sent and none of them would complete. When the connection has the writeback cache, each queue now keeps a share of its entries for background requests. Background requests may use a quarter of the entries, but at least 2. Foreground requests may use the rest, but at least 8. A foreground request over the limit waits on the new fuse_req_fg_queue and only moves to fuse_req_queue when a slot frees up. Background requests keep their own path to free entries and are held to their cap, except that each queue may always have one active background request. If a queue has fewer than 10 entries both minimums cannot hold, and the foreground minimum wins. The foreground count is released on completion, on commit errors and on entry teardown. Requests marked no_fg_limit skip the foreground limit. These are the page cache reads and writes (redfs_do_readfolio, redfs_send_readpages and redfs_send_write_pages) and the flush from write_inode. A page fault holds the inode range lock while its read is outstanding, and the revoke that is waiting for that range lock would never finish if the read were queued behind other foreground requests. The flush is sent by writeback itself. Without the writeback cache, dispatch is unchanged. Signed-off-by: Hai Zhong Zhou <hazhou@ddn.com>
|
Have you ever even checked how many requests you had waiting in such a hung situation? The waitqueue size is public in the connection data, so you can actually check if it is a request jam. This is definitely addressing the problem form the wrong end, but since I'm in no position to decline ... here we go. |
|
since we cannot change the branch on DDNStorage |
Thanks, Horst. I would like to add a new commit to address the problem, as we may have more addendum commits after doing more tests. |
Yes, the fuse requests were either blocked at redfsd client side or at redfs server side, just waiting for the invalidation to be done, but the invalidation cannot be done because no any free io-uring entries are available to service the invalidation and write-back. Yes, I have another redfsd userspace PR to be created to configure the io-uring depth... Actually, even for upstream, with more mature over time, they should also have such kind of foreground fuse workers throttling mechanisms, as this is obviously a kind of design shortage if one fuse requests are waiting for notify... |
|
Switch to create PR in my forked repo, to able to submit multiple commits in one PR.... so closing this one |
2 participants
FUSE IO-URING: add throttling mechanism for the fuse foreground worker threads to preserve the io-uring entries to the write-back background tasks, otherwise, no chance to complete the background tasks but the foreground worker threads are waiting on the background tasks, then form deadlock issues.
With the writeback cache, foreground requests can wait in userspace on DLM revokes. A revoke waits on page invalidation, which waits on writeback. If foreground requests held every entry of an io-uring queue, background writeback could not be sent and none of them would complete.
When the connection has the writeback cache, each queue now keeps a share of its entries for background requests. Background requests may use a quarter of the entries, but at least 2. Foreground requests may use the rest, but at least 8. A foreground request over the limit waits on the new fuse_req_fg_queue and only moves to fuse_req_queue when a slot frees up. Background requests keep their own path to free entries and are held to their cap, except that each queue may always have one active background request. If a queue has fewer than 10 entries both minimums cannot hold, and the foreground minimum wins. The foreground count is released on completion, on commit errors and on entry teardown.
Requests marked no_fg_limit skip the foreground limit. These are the page cache reads and writes (redfs_do_readfolio, redfs_send_readpages and redfs_send_write_pages) and the flush from write_inode. A page fault holds the inode range lock while its read is outstanding, and the revoke that is waiting for that range lock would never finish if the read were queued behind other foreground requests. The flush is sent by writeback itself. Without the writeback cache, dispatch is unchanged.