Skip to content

K2GO-577 feat(share): move Kolibri channels with rsync, no tar - #789

Merged
luisguzman-adfa merged 17 commits into
mainfrom
feat/K2GO-577-kolibri-rsync
Nov 16, 2026
Merged

luisguzman-adfa merged 17 commits into
mainfrom
feat/K2GO-577-kolibri-rsync

Conversation

@luisguzman-adfa

Copy link
Copy Markdown
Collaborator

What changes

  • conex-core 0.4.1: Share moves a Kolibri channel with rsync. The sender module holds the channel DB and its available content files, in place: no export, no tar, no staging.
  • Receiver: the DB goes to a small work dir; the content files go straight into Kolibri storage. A file that already exists is skipped (its name is its md5). Temp and kept partial files stay in /library/.conex-partial; nothing is deleted in storage.
  • Link step with Kolibri's own commands: importchannel disk, scanforcontent --channel-import-mode none, then importcontent disk on a manifest of the nodes that arrived (sets admin_imported, copies nothing).
  • Pull liveness is the bytes on disk, not the parent rsync's I/O (also in the ZIM pull).
  • ADR-content-exchange.md enters the repo, with the K2GO-577 rsync transport amendment.

Why

  • Kolibri was exported to a temp, sent as a tar, extracted, then copied again: up to three full copies, and a drop restarted from zero. Kolibri's own network import does not resume and corrupts in-flight files after a sender drop. (K2GO-577)

Steps to QA

  • Baseline (main): share a Kolibri channel over Wi-Fi, cut the link mid-transfer, tap Retry. It starts again from 0; the receiver holds a tar and an extracted copy during the import.
  • Change: both boxes on conex-core 0.4.1, receiver on the PR 1 app build. Share a channel, cut mid-transfer, tap Retry. The files already received are kept; at the end every resource of the channel is available in Kolibri and no work or staging copy is left.

Verifications

  • conex-core unit tests: 251 OK.
  • OnePlus 7T -> OPPO, "Digital Awareness" (33 files, 259 MB): cut at 128 MB / 24 files, Retry kept them; 16/16 resources available and admin_imported, 33/33 files with md5 equal to the name, the channel lists in Kolibri. Evidence goes to the Jira ticket.
  • Deferred: a pull longer than 120 s on a device (the liveness fix is unit-tested); Kolibri content outside KOLIBRI_HOME/content.
  • Coordination: stacked on the PR 1 branch; merge after it.

Notes

  • Files of an abandoned pull stay in Kolibri storage on purpose: a later Retry reuses them (decision in the ADR).
  • No new UI strings.

The sender's close DELETE lost its token in the relay and got 401, so the
share stayed open after Done.
… place

A share session runs an rsync daemon module per ZIM (password = session token); the receiver
pulls straight into place and resumes from kept bytes. No staging copy. conex-core 0.4.0.
…elay

The receiver asks its box to pull a Share item the sender offers over rsync, relayed on the
hotspot network, and reports progress to the sender. A box that cannot pull uses the HTTP fetch.
…partials per pull

A stale module list could land last and keep a closed session's module serving.
Restart a dead share daemon on manifest or progress, anchor the pull rate after a resume, report
done with real bytes, and send progress off the import thread.
The sender serves the channel DB and its files in place; the receiver pulls the files straight
into Kolibri storage and links them with importchannel, scanforcontent and importcontent.
In a pull the parent rsync is the generator and moves almost nothing, so a long single-file
transfer looked stalled to the watchdog.
A pull orphaned by a core crash kept writing the partial, so a Retry could start a second writer
on the same file.
A partial the user will not resume no longer holds the disk until the 48 h sweep. A drop or a
Retry keeps it.
…sync

# Conflicts:
#	static/conex-core/CHANGELOG.md
@luisguzman-adfa
luisguzman-adfa merged commit 059c3b2 into main Nov 16, 2026
5 checks passed
@luisguzman-adfa
luisguzman-adfa deleted the feat/K2GO-577-kolibri-rsync branch November 16, 2026 22:45
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant