Skip to content
Draft
1 change: 1 addition & 0 deletions .gitignore
Original file line number Diff line number Diff line change
Expand Up @@ -64,3 +64,4 @@ yarn-debug.log*

# Worktrees
.worktrees/
/log/merge_duplicate_members/
20 changes: 20 additions & 0 deletions Makefile
Original file line number Diff line number Diff line change
Expand Up @@ -90,5 +90,25 @@ test: ## Run the test suite in parallel
bundle exec rake parallel:setup
bundle exec parallel_rspec spec/ -n 3

detect_duplicate_members: ## Detect members duplicated by the codebar auth flow
DB_NAME=$(DUMP_DB) bundle exec rake member:duplicates:detect

fix_duplicate_members: ## Dry-run merge of duplicate members (set APPLY=1 to execute)
DB_NAME=$(DUMP_DB) bundle exec rake member:duplicates:fix

verify_duplicate_members: ## Verify no codebar-auth duplicate members remain
DB_NAME=$(DUMP_DB) bundle exec rake member:duplicates:verify

detect_duplicate_members_production: ## Detect duplicates on production DB directly
@read -p "Connect to PRODUCTION database? [y/N] " ans && [ "$$ans" = "y" ] || exit 1
@DB_URL=$$(heroku config:get DATABASE_URL --app=$(DUMP_APP)) bundle exec rake member:duplicates:detect

fix_duplicate_members_production: ## Dry-run merge on production DB (APPLY=1 executes)
@read -p "This MODIFIES the PRODUCTION database. Continue? [y/N] " ans && [ "$$ans" = "y" ] || exit 1
@DB_URL=$$(heroku config:get DATABASE_URL --app=$(DUMP_APP)) bundle exec rake member:duplicates:fix

verify_duplicate_members_production: ## Verify no duplicates remain on production DB
@DB_URL=$$(heroku config:get DATABASE_URL --app=$(DUMP_APP)) bundle exec rake member:duplicates:verify

check: ## Run setup checks
bundle exec rake setup:check
117 changes: 117 additions & 0 deletions docs/merge_duplicate_members.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,117 @@
# Merge Duplicate Members

Temporary tool to detect and merge duplicate members created by the `/auth/codebar` GitHub sign-in flow (codebar/planner#2805).

## Background

When the codebar auth app was merged into planner on 2026-08-06, members with a GitHub account whose email differed from their `auth_services` record could not be matched automatically. The codebar auth flow created new accounts instead of linking to existing ones.

This tool finds those duplicates (by name, email, and auth UID heuristics) and merges their data into the original member.

## Prerequisites

A PostgreSQL dump of the production database, accessible as `codebar_production_dump`.

## Tasks

### Detect duplicates

```bash
make detect_duplicate_members
```

Dry run — lists all duplicate pairs found, with which detection strategies matched.

### Fix duplicates (dry run)

```bash
make fix_duplicate_members
```

Shows every step the merger would take, but wraps everything in a transaction that is rolled back. No data is modified.

### Fix duplicates (execute)

```bash
make fix_duplicate_members APPLY=1
```

Performs the merge for real inside a transaction.
Only high-confidence matches are merged (see below). To also merge low-confidence matches after review:

```bash
make fix_duplicate_members APPLY=1 INCLUDE_WEAK=1
```
A JSON log file is written to:

```
log/merge_duplicate_members/run_YYYYMMDDTHHMMSSZ.json
```

The path is printed after the run completes.

### Verify

```bash
make verify_duplicate_members
```

Re-detects duplicates and exits `0` when no high-confidence duplicates remain, or `1` with the remaining pairs.
Low-confidence matches are reported but do not fail the check.

## Detection strategies and confidence

Merges are gated by confidence. A shared name is **not** proof of the same person — two different members can register with the same name (e.g. members 31336 / 25796, who hold two different GitHub accounts). Only identity evidence merges by default.

| Strategy | Description | Confidence | Merged by default |
|----------|-------------|------------|-------------------|
| `email` | Exact case-insensitive match on email | high | yes |
| `manual` | Hard-coded override reviewed by a human | high | yes |
| `name+surname` | Exact case-insensitive match on both fields | low | no — review first |
| `first-name+uid-surname` | First name matches; duplicate has no surname, but the codebar auth UID contains the original’s surname | low | no — review first |
| `domain+local-part` | Non-generic domain; local parts overlap | low | no — review first |

`detect` lists all matches with a confidence column. `fix` merges only high-confidence matches; low-confidence matches are listed for manual review.
To merge a reviewed low-confidence pair, add it to `MANUAL_OVERRIDES` (preferred — it records the human decision) or re-run with `INCLUDE_WEAK=1` to merge all matches.

The tool also applies hard-coded manual overrides for edge cases the heuristics cannot detect.

## Safety properties

- **Dry run by default** — must pass `APPLY=1` to change data.
- **Idempotent** — re-running after a successful merge reports no duplicates.
- **Deactivates, not deletes** — duplicates are renamed to `duplicate.<id>.merged-into.<id>@codebar.io`, with all auth services and roles removed. Audit history is preserved in a `MemberNote`.
- **Logs every execution** — merge results written to a new per-run JSON file, even on failure. The log contains only opaque member IDs (`dup_id`, `orig_id`), strategies, and status. No names, emails, UIDs, or other PII. Safe to share or attach to PRs.

## Running in production

### Option A: local execution (no deploy required)

You can run the tool locally and connect directly to the Heroku production database using a connection string:

```bash
make detect_duplicate_members_production # lists duplicates from production
make fix_duplicate_members_production # dry-run against production
make fix_duplicate_members_production APPLY=1 # execute on production
make verify_duplicate_members_production # verify production
```

These targets fetch a fresh `DATABASE_URL` from Heroku each time and pass it via `DB_URL`. A confirmation prompt guards the detect and fix targets. A loud `WARNING: Connecting to REMOTE database …` message is printed when `DB_URL` points to a non-localhost host.

### Option B: run on Heroku dyno

Deploy the branch first, then run on Heroku:

```bash
heroku run rake member:duplicates:detect --app codebar-production
heroku run rake member:duplicates:fix APPLY=1 --app codebar-production
heroku run rake member:duplicates:verify --app codebar-production
```

## What to do if a false positive appears

Duplicate pairs can be added to the `MANUAL_OVERRIDES` array or excluded before execution. If in doubt, err on the side of not merging — the merge does not delete records, but it does permanently move associated data and deactivate the account.

---

*This is a temporary tool. Once all existing duplicates are resolved, it can be removed.*
Loading