Tidy up tools/, and harden the docs unpacking

tools/README.md

@@ -4,50 +4,30 @@ This directory contains various scripts used to manage www.pgadmin.org and pgAdm
 
 **Most are very system-specific; don't expect them to work for you!!**
 
-**Note:** The AWS CLI is expected to be installed at /usr/bin/aws.
+On the production web server, all scripts **must** be run as the *pgaupload* user.
 
-Om the production web server, all scripts **must** be run as the *pgaupload* user.
+## create_release.py
 
-## create-release.py
-
-This script is called by the Jenkins server as part of the release process. It creates all the required objects in the
-website database for a new release.
+This script is called as part of the release process, through the `create-release` verb of the publishing wrapper
+(`pkg/publish/pga-publish` in the pgadmin4 repo) which the GitHub Actions workflows reach over SSH. It creates all the
+required objects in the website database for a new release.
 
 ## load-docs.sh
 
-This script is called by the Jenkins server as part of the release process. It unpacks the docs for the version of
-pgAdmin specified on the command line, and loads them into the website database.
-
-## purge-cache.sh
-
-This script is called by the Jenkins server as part of the release process. It purges the entire Varnish cache that 
-sits in front of Nginx.
-
-# rebuild-apt-repo.sh
+This script is called as part of the release process, through the `load-docs` verb of the publishing wrapper. It
+unpacks the docs for the version of pgAdmin specified on the command line, and loads them into the website database.
 
-This script will rebuild the specified APT repo metadata, and re-sign everything. This is typically used if old 
-packages are manually removed.
+Because the tarball it unpacks is built elsewhere and lands in a tree that an upload key can write to, the script
+treats it as untrusted input and vets the member list before unpacking anything; the comments in the script explain
+what each check is there to stop.
 
-```bash
-./rebuild-apt-repo.sh focal
-```
-
-# rebuild-yum-repo.sh
-
-This script will rebuild the specified YUM repo metadata, and re-sign everything. This is typically used if old 
-packages are manually removed.
-
-```bash
-./rebuild-yum-repo.sh redhat rhel 8
-```
-
-## sync-ftp-to-s3.py
+## purge-cache.sh
 
-This script is called by the Jenkins server as part of the release process. It will copy all the packages to the S3 
-bucket that hosts https://pgadmin-archive.postgresql.org, and generate and sync HTML index pages for browsing.
+This script is called as part of the release process, through the `purge-cache` verb of the publishing wrapper. It
+purges the entire Varnish cache that sits in front of Nginx.
 
 ## update-docs.sh
 
-This script is typically called from cron. It updates the git checkout of pgAdmin at 
-/var/www/pgaweb/static/docs/pgadmin4-dev, and then rebuilds the HTML docs which are subsequently loaded into the 
-website database as the 'dev' docs.
+This script is run from cron on the web server, twice an hour, as the *pgaupload* user, and is not part of the release
+process. It updates the git checkout of pgAdmin at /var/www/pgaweb/static/docs/pgadmin4-dev, and then rebuilds the HTML
+docs which are subsequently loaded into the website database as the 'dev' docs.

tools/load-docs.sh

@@ -1,27 +1,127 @@
 #!/bin/bash
 
+set -e
+set -o pipefail
+
+DOCS_ROOT=/var/www/pgaweb/static/docs
+FTP_ROOT=/var/ftp/pgadmin4
+
 if [ $# -ne 1 ]
 then
   echo "usage: $0 <version number>"
   exit 1
 fi
 
+VERSION=$1
+
+# The version is interpolated into the tarball path and into the name of the
+# directory we unpack into, so its shape is checked before it is used for
+# either, because a value carrying a slash, a '..' or a shell metacharacter
+# would let whoever called us choose those paths rather than merely name a
+# release.
+if [[ ! ${VERSION} =~ ^[0-9]+\.[0-9]+(\.[0-9]+)?$ ]]
+then
+  echo "Invalid version number (${VERSION}); expected something like 9.18 or 9.18.1."
+  exit 1
+fi
+
+TARBALL=${FTP_ROOT}/v${VERSION}/docs/pgadmin4-${VERSION}-docs.tar.gz
+DOCS_DIR=pgadmin4-${VERSION}-docs
+
 # Check the docs file exists
-TARBALL=/var/ftp/pgadmin4/v${1}/docs/pgadmin4-${1}-docs.tar.gz
-if [ ! -f ${TARBALL} ]
+if [ ! -f "${TARBALL}" ]
 then
   echo "The source tarball (${TARBALL}) could not be found."
   exit 1
 fi
 
+# Everything below happens in a staging directory beside the live one, so that
+# a tarball which turns out to be bad, or an unpacking that dies part way
+# through, cannot leave the site serving a half replaced tree. It sits inside
+# DOCS_ROOT so that moving the result into place is a rename rather than a copy
+# of the best part of a gigabyte.
+STAGING=$(mktemp -d "${DOCS_ROOT}/.load-docs.XXXXXX")
+trap 'rm -rf "${STAGING}"' EXIT
+
+# The tarball is assembled by a CI runner and lands in a tree that an upload
+# key can write to, so it is untrusted input, and an archive decides for itself
+# where its contents are written. We therefore vet the member list before
+# unpacking anything, rather than trusting whatever a given tar happens to do
+# with a hostile archive, since that varies between implementations and
+# versions and is not a control we own. The two listings also fail the script
+# outright if the archive cannot be read from end to end.
+tar -tzvf "${TARBALL}" > "${STAGING}/listing.verbose"
+tar -tzf "${TARBALL}" > "${STAGING}/listing.names"
+
+# Every member must be a plain file or a directory, which rejects symlinks and
+# hard links whose targets could redirect a later member out of the docs tree,
+# along with devices and anything else that has no business in a docs tarball.
+# A listing line we cannot make sense of fails this test as well, which is the
+# right way round for it to fail.
+if cut -c1 "${STAGING}/listing.verbose" | sort -u | grep -qv '^[-d]$'
+then
+  echo "${TARBALL} holds members that are neither plain files nor directories."
+  exit 1
+fi
+
+# Every member must also sit under the one directory we expect, which rejects
+# absolute paths and '..' traversal, and confirms the archive carries the
+# single top level directory that the docloader below is pointed at.
+while read -r MEMBER
+do
+  case ${MEMBER} in
+    "${DOCS_DIR}"|"${DOCS_DIR}"/)
+      ;;
+    "${DOCS_DIR}"/*)
+      case ${MEMBER} in
+        */../*|*/..)
+          echo "${TARBALL} holds a member traversing out of ${DOCS_DIR} (${MEMBER})."
+          exit 1
+          ;;
+      esac
+      ;;
+    *)
+      echo "${TARBALL} holds a member outside ${DOCS_DIR} (${MEMBER})."
+      exit 1
+      ;;
+  esac
+done < "${STAGING}/listing.names"
+
+rm -f "${STAGING}/listing.verbose" "${STAGING}/listing.names"
+
 # Unpack the tarball
-cd /var/www/pgaweb/static/docs
-tar -zxvf ${TARBALL}
+tar -xzf "${TARBALL}" -C "${STAGING}" --no-same-owner --no-same-permissions
+
+# Confirm we got out what we expected, rather than assuming that a tar which
+# exited zero wrote the docs we are about to load and publish. The second check
+# is belt and braces over the member vetting above, and catches a tar that
+# creates something odd from an archive whose listing looked ordinary.
+if [ ! -f "${STAGING}/${DOCS_DIR}/index.html" ]
+then
+  echo "${TARBALL} did not unpack to ${DOCS_DIR} holding an index.html."
+  exit 1
+fi
+
+if [ -n "$(find "${STAGING}" ! -type d ! -type f -print -quit)" ]
+then
+  echo "${TARBALL} unpacked something that is neither a plain file nor a directory."
+  exit 1
+fi
+
+# Swap the new docs into place, keeping the old copy until the new one has
+# landed so that reloading a version we already hold does not leave a gap.
+if [ -e "${DOCS_ROOT}/${DOCS_DIR}" ]
+then
+  rm -rf "${DOCS_ROOT}/${DOCS_DIR}.replaced"
+  mv "${DOCS_ROOT}/${DOCS_DIR}" "${DOCS_ROOT}/${DOCS_DIR}.replaced"
+fi
+mv "${STAGING}/${DOCS_DIR}" "${DOCS_ROOT}/${DOCS_DIR}"
+rm -rf "${DOCS_ROOT}/${DOCS_DIR}.replaced"
 
 # Load 'em
 cd /var/www/pgaweb
 source /usr/share/python3/pginfra-virtualenv-django52-py3/bin/activate
-./manage.py docloader pgadmin4 ${1} pgadmin4-${1}-docs
+./manage.py docloader pgadmin4 "${VERSION}" "${DOCS_DIR}"
 
 # Clear the cache
 sudo varnishadm "ban req.url ~ ."