Wasserman Lab BVL

Dev Environment Setup (Python / Django module)

git clone git@github.com:wassermanlab/variome.git
cd variome

1. Set up project dependencies using preferred method:

Option 1 - Uv (recommended)

• Install uv for dependency management (it will be available anywhere on your system, but won't collide with other projects): https://rye.astral.sh/guide/installation/
• NOTE: example commands in the rest of the document starting with "python" should start with "uv run" instead
• run:

uv sync

Option 2 - Pip

• Setup pip based on your OS and preferred Python environment solution (eg, venv or similar)
• run:

pip install -r requirements.lock

Option 3 - Conda and Pip

conda env create -f environment.yaml 
conda activate variome
pip install -r requirements.lock

2. Set up the database

Postgres is included as an example, but any database backend supported by Django will work

# mac/linux - with homebrew

brew install postgresql (if necessary)
brew services start postgresql (if necessary)

# windows - with chocolatey

choco install postgresql
net start postgresql

# If psql is not recognized, add PostgreSQL’s bin directory to your PATH environment variable.
# Example: C:\Program Files\PostgreSQL\15\bin

# Opening psql command line)

# mac / linux

psql

#windows

psql -U postgres

# commands to enter into psql command interface

CREATE DATABASE variome;
CREATE USER variome WITH PASSWORD 'variome';
GRANT ALL PRIVILEGES on DATABASE variome to variome;

3. Set up configuration files (.env)

Edit .env DB to match with your database environment, set timezone, other options

cp .env-sample .env

4. Check your installation:

python manage.py check

or, the UV equivalent:

uv run manage.py check

It should show something similar to this if it's working:

...connecting to postgresql://variome:variome@localhost:5432/variome...
System check identified no issues (0 silenced).

5. Load the test fixture data and create a superuser account

python manage.py migrate
python manage.py import_bvl
python manage.py createsuperuser

6. Install the local GRCh38 reference

Set REFERENCE_GENOME_PATH in .env to the desired FASTA filename (for example /srv/variome/reference/GRCh38.fa). The default is data/reference/GRCh38.fa. Then run:

python manage.py download_reference_genome

This streams the official Ensembl release 115 human GRCh38 primary assembly over HTTPS, decompresses it, and builds a samtools-compatible .fai index using pyfaidx. Allow about 4 GB of free disk space for the roughly 3.1 GB FASTA and index. No whole-genome load into memory is needed. Existing files are not overwritten; use a new path when replacing the reference.

If the configured directory is unwritable, the command downloads to the repository root instead and, after completion, tells you where to move both files. Ensure the Django service account can read the FASTA and its adjacent .fai. Genome files must not be committed to Git.

Authenticated sequence lookups use GET /api/sequence/region/1:1000..1001:1?coord_system_version=GRCh38 and return {"seq": "..."}. Coordinates are 1-based and inclusive; strand can be 1 or -1. chr prefixes and mitochondrial M/MT aliases are accepted. Only GRCh38 is supported, requests are limited to 200 bases, and unavailable reference files return HTTP 503 rather than falling back to an external service. The search UI uses this endpoint for its reference-allele warning.

Run the Django app

python manage.py runserver

(optional) - Run on a specific port. If you do, also specify BACKEND_ROOT=http://localhost:8888 in .env so the frontend can talk to the backend properly

python manage.py runserver 8888

Frontend setup

1. Install NodeJS and install the dependencies

cd frontend
npm install

2. Set up the frontend content files:

cp -r public-example content

Run the Frontend

cd frontend
npm install
npm run dev

It runs at localhost:3000 by default. You can press "o" and hit the enter key in the terminal window to open in a browser easily. Use "h" for more shortcuts

how to customize the content

You can edit the frontend/content files to customize according to branding, access policy information, and any other content.

Markdown format files in the /content folder are treated as full pages, available in the navigation sidebar. You can add more, rename them. To set the icons and sorting order, add them to the IconMap object in content/ContentConfiguration.js.

Markdown format files in /content/Home are individual, "hard-coded" page elements on the Home page, so you shouldn't add more or rename these.

config reference

PUBLIC_BVL - opens the database for public access, as a demonstration app. For this to work, a shared user account must also be created ( username: public_demo_user, email: public_demo_user@example.com )

Importing from a VCF file

Use the import_bvl_vcf management command to import data directly from VCF files into the database. Nuclear variants are read from --vcf and mitochondrial variants from --mt-vcf; one run imports both. With no arguments the development fixtures data/fixtures/vcf/nuclear-tiny.vcf and data/fixtures/vcf/mt.vcf are used. These are the canonical nuclear and unannotated mitochondrial fixtures. All mock VCFs also live in data/fixtures/vcf/. The mt-annotated.vcf fixture preserves mt.vcf and adds synthetic CSQ annotations at chrM:4440, with reordered fields and shared nuclear genes/transcripts for header-context edge-case tests; these annotations are artificial, not biological assertions. Mitochondrial records in --vcf (and non-mitochondrial records in --mt-vcf) are ignored with a warning; to import a combined VCF, pass the same file to both. Each table's CallFilter selects the relevant input(s) and produces one row iterator. Shared gene, variant and transcript annotation tables combine both inputs, retaining each file's own annotation header and deduplicating shared records. The importer runs once per table, so --delete replaces that table once rather than deleting the first file's rows when processing the second. Equivalent paths to the same file are read only once per filter. Tables whose corresponding input is absent are skipped.

Mitochondrial records (M, MT, chrM or chrMT) are imported into separate metadata and frequency tables. Both the IBVL tabular MT export and standard Variome VCF INFO/FORMAT metrics are supported. IDs use M consistently. The supplied data/fixtures/vcf/mt.vcf Mutect2 example is also supported, including sites with multiple alternate alleles. VEP CSQ annotations are optional; without them, gene/transcript annotations and CSQ-derived identifiers remain absent. Explicit MT cohort counts and frequencies are retained as supplied. Otherwise counts and frequencies are derived per ALT from valid sample FORMAT/AF: AF >= 0.95 is homoplasmic, 0 < AF < 0.95 is heteroplasmic. The denominator (AN) is the number of samples with a valid AF for that ALT, including AF=0; missing AF values are excluded, not assumed to be reference. This replaces diploid INFO/AN when deriving sample-based MT frequencies. Missing histogram and maximum values are also derived: when the VCF has no heteroplasmy_histogram or max_observed_heteroplasmy, they are derived per ALT allele from the sample FORMAT/AF heteroplasmy levels of carriers (10 bins of 0.1; the top bin includes 1.0). These sample-based values are not subject to gnomAD's heteroplasmy or quality filters and should not be treated as equivalently filtered. An optional preprocessed gnomAD mitochondrial TSV can be supplied with --mt-gnomad-tsv /path/to/gnomad_mt.tsv; only variants present in the VCF are imported. Its columns are chromosome, position, ref, alt, AN, AC_hom, AC_het, AF_hom, AF_het and max_observed_heteroplasmy. The variant page loads gnomAD frequencies dynamically for both nuclear and mitochondrial variants; it does not display the stored mitochondrial TSV values. Select an Example mt variant in the admin Settings editor to show its link on the home page.

Quickstart

python manage.py import_bvl_vcf --vcf /path/to/joint.vcf --mt-vcf /path/to/mt.vcf
python manage.py create_indices

Run python manage.py import_bvl_vcf --help for a full list of options and an explanation of the import summary.

Key options

Option	Default	Description
--vcf	data/fixtures/vcf/nuclear-tiny.vcf	Nuclear VCF file (plain text or .gz); '' skips it
--mt-vcf	data/fixtures/vcf/mt.vcf	Mitochondrial VCF file (plain text or .gz); '' or --no-mts skips it
--input-tsv-dir	data/fixtures	Path to the dir containing severities.tsv
--mts / --no-mts	enabled	Import mitochondrial metadata and cohort frequencies
--mt-gnomad-tsv	(none)	gnomAD mitochondrial frequency TSV to intersect with imported variants
--na	.	Value used to represent missing/null data
--out-chr	enabled	Prefix chromosome names with chr
--out-hyphens	enabled	Use hyphens in variant IDs (e.g. 1-100-A-G) if false, uses underscores instead
--cadd-threshold	20	cadd_intr will be classified as "Damaging" if cadd_score is greater than or equal to this number
--default-transcript-source	E	Fallback transcript source when unknown (E=Ensembl, R=RefSeq)
--ranges	(all)	Restrict processing to specific regions, e.g. 22:27010000-27020000,X:2702000-2802000
--convert-to-tsv	disabled	Write TSV files instead of importing into the database
--tsv-output-dir	data/vcf_output	Directory for TSV output (requires --convert-to-tsv)
--hash-compare	(none)	Directory of an existing TSV set to compare output hashes against (requires --convert-to-tsv)
--dry-run / -n	disabled	Parse and validate without writing to the database
--delete	disabled	Delete all existing data before importing
-v / -vv / -vvv	-vv	Output verbosity: summary only / normal / adds chromosome caching messages and timings
--no-genes, --no-variants, etc.	(all enabled)	Skip specific tables

Comparing output against a reference TSV set

When --convert-to-tsv is active, pass --hash-compare to verify that the generated TSV files match a previously known-good set ( this is useful for development of this project, ensuring consistency with legacy tsv-producting tools):

python manage.py import_bvl_vcf \
  --vcf /path/to/joint.vcf \
  --mt-vcf '' \
  --convert-to-tsv \
  --tsv-output-dir data/vcf_output \
  --hash-compare data/reference_tsvs

Per-file hash differences are reported in the log output.

Tracking dashboard

The tracking dashboard requires the library_access.view_tracking_dashboard permission. The active tab is stored in the URL (?tab=variant or ?tab=search), so refresh and browser back/forward navigation preserve the selected view.

Search queries have separate chr and pos columns. Filter by dates, validity, chromosome, case-insensitive query substring, and inclusive minimum/maximum result counts (including zero). Blank limits are unbounded. Click Apply search filters to apply the selected filters and reset pagination. Page size and date sorting are grouped separately from the search filters.

Running backend tests

All backend tests (middleware and VCF import filters) live in test/backend/variome_backend/.

First, install dev dependencies:

uv sync --dev

Run all backend tests once:

./test.sh -unit -all -v

Backend tests run through test.sh use an in-memory SQLite database, including when DB is set for local PostgreSQL development.

Install browser-test dependencies and Chromium once:

cd test/e2e && npm install && npx playwright install chromium

Run backend and browser tests with watching and press Enter to rerun on demand:

./test.sh

Use ./test.sh -e2e, ./test.sh -unit, ./test.sh -f test_search.SnvSearchTests, or ./test.sh -f home.spec.mjs to target a suite or test. E2E ports and timeout can be configured with E2E_BACKEND_PORT, E2E_FRONTEND_PORT, and -e TIMEOUT_MS.

attributions

example DNA image: https://commons.wikimedia.org/wiki/File:202104_Laboratory_instrument_dna.svg