Skip to content

Speed up MAUI app sync (#208) - #209

Merged
fboucher merged 2 commits into
v-nextfrom
bug/208-slow-sync-sac9ok
Sep 30, 2026
Merged

fboucher merged 2 commits into
v-nextfrom
bug/208-slow-sync-sac9ok

Conversation

@fboucher

Copy link
Copy Markdown
Owner

Requested by Frank · project thread

Fixes #208

Before: every time the Posts page opened, the MAUI app downloaded all ~1500 posts, saved them one by one, made an extra HTTP call per unread post, and did all of it on the UI thread while re-rendering the progress bar for every post. The app froze during sync.

After: after the first sync, only posts changed since the last sync are downloaded and saved, all in one SQLite transaction. Sync runs off the UI thread, and the progress bar updates about every 250 ms. Posts deleted on the server are still removed locally, and HTML for read posts is still pruned.

How

  • API: new GET api/posts/ids returns only post ids (read and unread). It projects just the key columns, so it stays small.
  • Delta pull: PullAsync calls GetPostsModifiedAfter(lastSync ?? MinValue). Deletions are detected by comparing local posts with GetPostIds(). On the first sync the full list is already in hand, so the ids call is skipped. If the server is older and has no ids endpoint (404), sync falls back to the full list, so the app still works against an API that hasn't been redeployed yet.
  • Batched writes: pulled posts are saved with SavePostsAsync, which runs in one transaction and preserves IsPendingSync. Server-deleted posts are removed with the new RemovePostsAsync in one transaction. The end state is the same as the old DeletePostAsync + MarkSyncedAsync pair. Duplicate rows for one post (the API joins posts with notes) are de-duplicated.
  • Full post fetch: unread posts still get GetPost for the full details, but at most 5 requests run at once (Parallel.ForEachAsync).
  • Threading: SyncService.SyncAsync starts the work with Task.Run, which covers every caller (Posts page, Storage page, app resume, connectivity change). All progress and conflict handlers already marshal with InvokeAsync.
  • Progress throttling: pull and HTML-download progress events fire at most every 250 ms, plus the first and last events.
  • SyncHtmlAsync logic is unchanged apart from the progress throttle.

Tests

  • SyncServiceTests: existing pull tests updated for batched save. New tests cover delta pull with lastSync, deletion via ids, fallback when the ids endpoint is missing, first sync skipping the ids call, a 1500-post single-batch save, de-duplication, progress throttling, and bounded GetPost concurrency.
  • LocalDataServiceTests: RemovePostsAsync, and SavePostsAsync preserving the pending-sync flag.
  • DeltaApiTests: GET api/posts/ids returns read and unread posts and excludes deleted ones.
  • Locally, NoteBookmark.MauiApp.Tests passes (57/57). The API tests build but weren't run here because they need Docker for Azurite, so CI covers them. The MAUI app project itself wasn't built locally (no MAUI workload); SyncService is compiled through the test project.

🤖 Generated with Claude Code

https://claude.ai/code/session_01Kq68qA7Jv7puAF3Kc6GSFY


Generated by Claude Code

- Pull only posts modified since the last sync; detect server deletions
  with a new lightweight GET api/posts/ids endpoint (falls back to the
  full list on first sync or on servers without the endpoint).
- Save pulled posts in one SQLite transaction and remove deleted posts
  in one batch.
- Fetch full details for unread posts with at most 5 concurrent requests.
- Run sync on the thread pool and throttle progress events to ~250 ms.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Kq68qA7Jv7puAF3Kc6GSFY
@fboucher
fboucher marked this pull request as ready for review September 27, 2026 13:00
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Kq68qA7Jv7puAF3Kc6GSFY
@github-actions

Copy link
Copy Markdown

Code Coverage

Package Line Rate Branch Rate Health
NoteBookmark.AIServices 97% 64% ✔
NoteBookmark.Api 89% 80% ✔
NoteBookmark.BlazorApp 4% 10% ❌
NoteBookmark.Domain 93% 76% ✔
NoteBookmark.ServiceDefaults 96% 75% ✔
NoteBookmark.SharedUI 26% 33% ❌
Summary 58% (1164 / 2008) 50% (341 / 677) ❌

@fboucher
fboucher merged commit 73a2d76 into v-next Sep 30, 2026
1 check passed
@fboucher
fboucher deleted the bug/208-slow-sync-sac9ok branch September 30, 2026 15:44
fboucher added a commit that referenced this pull request Sep 30, 2026
* fix(#173): use Id ?? RowKey for checking and opening offline post HTML (#179)

* feat(#176): add sync progress bar and status updates for downloading offline post text (#180)

* should have done that long ago

* fix: resolve Android MAUI app crash when opening URLs (issue #177) (#181)

* feat: add up/down reordering for categories and notes in summary editor (issue #92) (#183)

Co-authored-by: Frank Boucher <2404846+fboucher@users.noreply.github.com>

* feat: add text size slider and fix text wrapping on offline reader page (issue #178) (#182)

* bumb version

* feat: implement async background post content extraction to save HTML in blob storage (#185)

Co-authored-by: fboucher-os <fboucheros+github@gmail.com>

* feat: improve sync html download, always display read post button, and remove ILocalHtmlCache (#186)

* feat: implement async background post content extraction to save HTML in blob storage

* feat: improve sync html download, always display read post button, and remove ILocalHtmlCache

* feat: configure protected Azure Function Parser API and key propagation (#187)

* feat: configure protected Azure Function Parser API and key propagation

* bump: version to 1.4.0

---------

Co-authored-by: fboucher-os <fboucheros+github@gmail.com>

* update missing env variable

* fix

* chore: bump version to 1.4.1

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

* fix(#190): update actions/cache to v5 for Node.js 24 compatibility in OpenCode triage workflow (#191)

* Update version to 1.4.2, add sync progress tracking, and reader page controls (#192)

* chore: bump version to 1.4.2

* feat: show detailed sync progress messages and progress bar on Posts page

* feat: add back button and text size slider at both top and bottom of reader page

* ci: include version in android apk filename (#193)

Co-authored-by: fboucher-os <fboucheros+github@gmail.com>

* Fix document sync stuck and improve sync progress and offline fallback (fixes #196) (#199)

* fix: constrain reader text size slider and add step buttons (fixes #195) (#198)

* feat: Add markdown saving for reading notes and bump version to 1.4.3

* feat: add MAUI storage tab (#200)

* feat: add MAUI storage tab for offline posts

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

* fix: refine MAUI storage controls

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

* fix: highlight storage actions

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

---------

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

* chore: bump application version to 1.4.3 (#201)

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

* fix(ci): fix timeout and rate limit failures in OpenCode Daily Triage workflow (#204) (#205)

* v1.4.3 - Enhances MAUI sync, local storage, and reader (#202)

* fix(#173): use Id ?? RowKey for checking and opening offline post HTML (#179)

* feat(#176): add sync progress bar and status updates for downloading offline post text (#180)

* should have done that long ago

* fix: resolve Android MAUI app crash when opening URLs (issue #177) (#181)

* feat: add up/down reordering for categories and notes in summary editor (issue #92) (#183)

Co-authored-by: Frank Boucher <2404846+fboucher@users.noreply.github.com>

* feat: add text size slider and fix text wrapping on offline reader page (issue #178) (#182)

* bumb version

* feat: implement async background post content extraction to save HTML in blob storage (#185)

Co-authored-by: fboucher-os <fboucheros+github@gmail.com>

* feat: improve sync html download, always display read post button, and remove ILocalHtmlCache (#186)

* feat: implement async background post content extraction to save HTML in blob storage

* feat: improve sync html download, always display read post button, and remove ILocalHtmlCache

* feat: configure protected Azure Function Parser API and key propagation (#187)

* feat: configure protected Azure Function Parser API and key propagation

* bump: version to 1.4.0

---------

Co-authored-by: fboucher-os <fboucheros+github@gmail.com>

* update missing env variable

* fix

* chore: bump version to 1.4.1

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

* fix(#190): update actions/cache to v5 for Node.js 24 compatibility in OpenCode triage workflow (#191)

* Update version to 1.4.2, add sync progress tracking, and reader page controls (#192)

* chore: bump version to 1.4.2

* feat: show detailed sync progress messages and progress bar on Posts page

* feat: add back button and text size slider at both top and bottom of reader page

* ci: include version in android apk filename (#193)

Co-authored-by: fboucher-os <fboucheros+github@gmail.com>

* Fix document sync stuck and improve sync progress and offline fallback (fixes #196) (#199)

* fix: constrain reader text size slider and add step buttons (fixes #195) (#198)

* feat: Add markdown saving for reading notes and bump version to 1.4.3

* feat: add MAUI storage tab (#200)

* feat: add MAUI storage tab for offline posts

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

* fix: refine MAUI storage controls

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

* fix: highlight storage actions

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

---------

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

* chore: bump application version to 1.4.3 (#201)

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

---------

Co-authored-by: fboucher-os <fboucheros+github@gmail.com>
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

* fix(ci): prevent timeout and rate limits in opencode daily triage (#204)

---------

Co-authored-by: Frank Boucher <2404846+fboucher@users.noreply.github.com>
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

* feat(ui): add dedicated quickadd popup and MinimalLayout for bookmarklet (#203) (#207)

* v1.4.3 - Enhances MAUI sync, local storage, and reader (#202)

* fix(#173): use Id ?? RowKey for checking and opening offline post HTML (#179)

* feat(#176): add sync progress bar and status updates for downloading offline post text (#180)

* should have done that long ago

* fix: resolve Android MAUI app crash when opening URLs (issue #177) (#181)

* feat: add up/down reordering for categories and notes in summary editor (issue #92) (#183)

Co-authored-by: Frank Boucher <2404846+fboucher@users.noreply.github.com>

* feat: add text size slider and fix text wrapping on offline reader page (issue #178) (#182)

* bumb version

* feat: implement async background post content extraction to save HTML in blob storage (#185)

Co-authored-by: fboucher-os <fboucheros+github@gmail.com>

* feat: improve sync html download, always display read post button, and remove ILocalHtmlCache (#186)

* feat: implement async background post content extraction to save HTML in blob storage

* feat: improve sync html download, always display read post button, and remove ILocalHtmlCache

* feat: configure protected Azure Function Parser API and key propagation (#187)

* feat: configure protected Azure Function Parser API and key propagation

* bump: version to 1.4.0

---------

Co-authored-by: fboucher-os <fboucheros+github@gmail.com>

* update missing env variable

* fix

* chore: bump version to 1.4.1

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

* fix(#190): update actions/cache to v5 for Node.js 24 compatibility in OpenCode triage workflow (#191)

* Update version to 1.4.2, add sync progress tracking, and reader page controls (#192)

* chore: bump version to 1.4.2

* feat: show detailed sync progress messages and progress bar on Posts page

* feat: add back button and text size slider at both top and bottom of reader page

* ci: include version in android apk filename (#193)

Co-authored-by: fboucher-os <fboucheros+github@gmail.com>

* Fix document sync stuck and improve sync progress and offline fallback (fixes #196) (#199)

* fix: constrain reader text size slider and add step buttons (fixes #195) (#198)

* feat: Add markdown saving for reading notes and bump version to 1.4.3

* feat: add MAUI storage tab (#200)

* feat: add MAUI storage tab for offline posts

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

* fix: refine MAUI storage controls

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

* fix: highlight storage actions

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

---------

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

* chore: bump application version to 1.4.3 (#201)

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

---------

Co-authored-by: fboucher-os <fboucheros+github@gmail.com>
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

* Fixes CI timeout and rate limits in triage workflow (#206)

* fix(#173): use Id ?? RowKey for checking and opening offline post HTML (#179)

* feat(#176): add sync progress bar and status updates for downloading offline post text (#180)

* should have done that long ago

* fix: resolve Android MAUI app crash when opening URLs (issue #177) (#181)

* feat: add up/down reordering for categories and notes in summary editor (issue #92) (#183)

Co-authored-by: Frank Boucher <2404846+fboucher@users.noreply.github.com>

* feat: add text size slider and fix text wrapping on offline reader page (issue #178) (#182)

* bumb version

* feat: implement async background post content extraction to save HTML in blob storage (#185)

Co-authored-by: fboucher-os <fboucheros+github@gmail.com>

* feat: improve sync html download, always display read post button, and remove ILocalHtmlCache (#186)

* feat: implement async background post content extraction to save HTML in blob storage

* feat: improve sync html download, always display read post button, and remove ILocalHtmlCache

* feat: configure protected Azure Function Parser API and key propagation (#187)

* feat: configure protected Azure Function Parser API and key propagation

* bump: version to 1.4.0

---------

Co-authored-by: fboucher-os <fboucheros+github@gmail.com>

* update missing env variable

* fix

* chore: bump version to 1.4.1

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

* fix(#190): update actions/cache to v5 for Node.js 24 compatibility in OpenCode triage workflow (#191)

* Update version to 1.4.2, add sync progress tracking, and reader page controls (#192)

* chore: bump version to 1.4.2

* feat: show detailed sync progress messages and progress bar on Posts page

* feat: add back button and text size slider at both top and bottom of reader page

* ci: include version in android apk filename (#193)

Co-authored-by: fboucher-os <fboucheros+github@gmail.com>

* Fix document sync stuck and improve sync progress and offline fallback (fixes #196) (#199)

* fix: constrain reader text size slider and add step buttons (fixes #195) (#198)

* feat: Add markdown saving for reading notes and bump version to 1.4.3

* feat: add MAUI storage tab (#200)

* feat: add MAUI storage tab for offline posts

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

* fix: refine MAUI storage controls

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

* fix: highlight storage actions

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

---------

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

* chore: bump application version to 1.4.3 (#201)

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

* fix(ci): fix timeout and rate limit failures in OpenCode Daily Triage workflow (#204) (#205)

* v1.4.3 - Enhances MAUI sync, local storage, and reader (#202)

* fix(#173): use Id ?? RowKey for checking and opening offline post HTML (#179)

* feat(#176): add sync progress bar and status updates for downloading offline post text (#180)

* should have done that long ago

* fix: resolve Android MAUI app crash when opening URLs (issue #177) (#181)

* feat: add up/down reordering for categories and notes in summary editor (issue #92) (#183)

Co-authored-by: Frank Boucher <2404846+fboucher@users.noreply.github.com>

* feat: add text size slider and fix text wrapping on offline reader page (issue #178) (#182)

* bumb version

* feat: implement async background post content extraction to save HTML in blob storage (#185)

Co-authored-by: fboucher-os <fboucheros+github@gmail.com>

* feat: improve sync html download, always display read post button, and remove ILocalHtmlCache (#186)

* feat: implement async background post content extraction to save HTML in blob storage

* feat: improve sync html download, always display read post button, and remove ILocalHtmlCache

* feat: configure protected Azure Function Parser API and key propagation (#187)

* feat: configure protected Azure Function Parser API and key propagation

* bump: version to 1.4.0

---------

Co-authored-by: fboucher-os <fboucheros+github@gmail.com>

* update missing env variable

* fix

* chore: bump version to 1.4.1

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

* fix(#190): update actions/cache to v5 for Node.js 24 compatibility in OpenCode triage workflow (#191)

* Update version to 1.4.2, add sync progress tracking, and reader page controls (#192)

* chore: bump version to 1.4.2

* feat: show detailed sync progress messages and progress bar on Posts page

* feat: add back button and text size slider at both top and bottom of reader page

* ci: include version in android apk filename (#193)

Co-authored-by: fboucher-os <fboucheros+github@gmail.com>

* Fix document sync stuck and improve sync progress and offline fallback (fixes #196) (#199)

* fix: constrain reader text size slider and add step buttons (fixes #195) (#198)

* feat: Add markdown saving for reading notes and bump version to 1.4.3

* feat: add MAUI storage tab (#200)

* feat: add MAUI storage tab for offline posts

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

* fix: refine MAUI storage controls

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

* fix: highlight storage actions

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

---------

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

* chore: bump application version to 1.4.3 (#201)

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

---------

Co-authored-by: fboucher-os <fboucheros+github@gmail.com>
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

* fix(ci): prevent timeout and rate limits in opencode daily triage (#204)

---------

Co-authored-by: Frank Boucher <2404846+fboucher@users.noreply.github.com>
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

---------

Co-authored-by: fboucher-os <fboucheros+github@gmail.com>
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

* feat(ui): add browser bookmarklet support for saving URLs (#203)

* feat(ui): add dedicated quickadd popup and MinimalLayout for bookmarklet (#203)

---------

Co-authored-by: Frank Boucher <2404846+fboucher@users.noreply.github.com>
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

* Speed up MAUI app sync (#208) (#209)

* Speed up MAUI sync: delta pull, batched saves, off-UI-thread (#208)

- Pull only posts modified since the last sync; detect server deletions
  with a new lightweight GET api/posts/ids endpoint (falls back to the
  full list on first sync or on servers without the endpoint).
- Save pulled posts in one SQLite transaction and remove deleted posts
  in one batch.
- Fetch full details for unread posts with at most 5 concurrent requests.
- Run sync on the thread pool and throttle progress events to ~250 ms.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Kq68qA7Jv7puAF3Kc6GSFY

* Restore CRLF line endings in LocalDataService and PostNoteClient

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Kq68qA7Jv7puAF3Kc6GSFY

---------

Co-authored-by: Claude <noreply@anthropic.com>

* chore: bump application version to 1.4.4

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

---------

Co-authored-by: fboucher-os <fboucheros+github@gmail.com>
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Co-authored-by: Claude <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants