Treemap, bare-command pickers, jobs menu and schedules, background chart priming, 6 h cache, CI on Node 24

.github/workflows/ci.yml

@@ -17,7 +17,7 @@ jobs:
       matrix:
         python-version: ['3.x', '3.13.5'] # latest release, and the production version
     steps:
-      - uses: actions/checkout@v4
+      - uses: actions/checkout@v5
         with:
           fetch-depth: 0 # the base branch is needed to see what a pull request changes
       - id: changes
@@ -41,7 +41,7 @@ jobs:
           fi
           echo "docs_only=$docs_only" >> "$GITHUB_OUTPUT"
       - if: steps.changes.outputs.docs_only != 'true'
-        uses: actions/setup-python@v5
+        uses: actions/setup-python@v6
         with:
           python-version: ${{ matrix.python-version }}
           cache: pip
@@ -56,7 +56,7 @@ jobs:
     # production meets it. Then every module imports in it, and the container starts as compose.yaml starts it.
     runs-on: ubuntu-24.04
     steps:
-      - uses: actions/checkout@v4
+      - uses: actions/checkout@v5
         with:
           fetch-depth: 0 # the base branch is needed to see what a pull request changes
       - id: changes

Dockerfile

@@ -4,7 +4,7 @@
 FROM debian:trixie-slim
 
 RUN apt-get update \
-	&& apt-get install -y --no-install-recommends python3 python3-requests python3-croniter python3-dateutil \
+	&& apt-get install -y --no-install-recommends python3 python3-requests python3-dateutil \
 		python3-matplotlib python3-numpy python3-waitress python3-websocket python3-discord python3-nacl \
 		ca-certificates tzdata \
 	&& rm -rf /var/lib/apt/lists/*

README.md

@@ -4,48 +4,69 @@ finbot is a chat bot for Telegram, Slack and Discord that follows your [Sharesig
 portfolios and each chat's watchlist with Yahoo Finance data. It posts trades, price moves, earnings and ex-dividend
 dates, analyst rating changes and more on a schedule, and answers commands with reports and charts.
 
+If you don't have Sharesight, you can still add your holdings manually via the watchlist (`/watchlist add SYMBOL`).
+
 ## Features
-* Telegram, Slack and Discord, with no server to host: finbot fetches its updates itself (or, in HTTP mode, a network
-  posts them to your HTTPS server)
-* Sharesight trades as they happen, portfolio performance and holdings
-* Price moves past a threshold, earnings and ex-dividend dates, analyst rating changes, short interest, record highs
-* Stock lookup, price history charts and comparisons, rankings by P/E, PEG, market cap or analyst rating
-* Commands as messages (`.price`) or slash commands (`/price`), completed as they are typed where the network allows
-* Each chat's own scheduled jobs and watchlist, managed in the chat (`.jobs`, `.watchlist`)
-* Docker or a Debian host, hardened either way
+* Telegram, Slack and Discord (either the push or the pull method, depending on your environment)
+* Sharesight trades as they happen, portfolio performance comparisons
+* Charts with buttons to change the time period
+* Alerts of daily price movements past a threshold, earnings and ex-dividend dates, analyst rating changes, short
+  interest, record highs
+* Stock lookup and comparisons, rankings by P/E, PEG, market cap or analyst rating
+* Slash commands (`/price`), completed as they are typed where the network allows, or messages (`.price`), quicker to
+  type on a phone
+* Each chat's own scheduled jobs and watchlist, managed in the chat (`/jobs`, `/watchlist`)
+* Docker or a Debian host
 
 ![Screenshot of showing trade notifications on Slack](img/screenshot.png?raw=true "Screenshot showing trade notifications on Slack")
 
 ## Quick start
-With Docker:
+Run finbot with Docker or with Python. Both take the same keys, and both install every optional library, so each
+network works in either mode.
+
+### With Docker
+On a Linux host with Docker:
 1. `git clone https://github.com/robdevops/finbot /opt/finbot`
 2. `sudo /opt/finbot/util/docker_setup.sh`
 3. Set the keys of each network you use in `/opt/finbot.env` (below)
 4. `cd /opt/finbot && sudo docker compose up -d --build`
 5. `journalctl CONTAINER_NAME=finbot -f`
 
-To install on a Debian host with systemd instead, see [doc/install.md](doc/install.md).
+### With Python
+On Linux, macOS or WSL, with Python 3.12 or later:
+1. `git clone https://github.com/robdevops/finbot ~/finbot && cd ~/finbot`
+2. `python3 -m venv .venv && .venv/bin/pip install -r requirements.txt -r requirements-optional.txt` (on Debian and
+   Ubuntu, `sudo apt install python3-venv` first)
+3. `cp defaults/finbot.env ~/finbot.env && chmod 600 ~/finbot.env`, then set the keys of each network you use in it
+   (below)
+4. `.venv/bin/python util/check_config.py ~/finbot.env`
+5. `(set -a && . ~/finbot.env && exec .venv/bin/python bot.py)`, which logs to the terminal until Ctrl-C
 
-Each network needs a bot of its own, and finbot fetches its updates with no server (its pull mode):
-* **Telegram:** create the bot with [@BotFather](https://telegram.me/BotFather), set `telegramBotToken` and
-  `telegramChatID`, restart finbot ([guide](doc/telegram.md)).
+To run it as a service, under systemd on a Debian host, see [doc/install.md](doc/install.md).
+
+### The chat networks
+Each network needs a bot of its own. By default, finbot fetches its updates from each network itself (pull mode).
+Alternatively, if you prefer to run an HTTPS gateway (a reverse proxy on a domain with a valid certificate), HTTP mode
+subscribes the bot to the updates each network pushes to it (each guide's _HTTP mode_).
+* **Telegram:** create the bot with [@BotFather](https://telegram.me/BotFather), set `telegramBotToken`, restart
+  finbot ([guide](doc/telegram.md)).
 * **Slack:** create the app from the manifest that `util/slack_manifest.py` prints, add an app-level token, set
-  `slackBotToken`, `slackAppToken` and `slackChannelID`, restart finbot ([guide](doc/slack.md)).
-* **Discord:** create the application and its bot, turn on its Message Content intent, set `discordBotToken` and
-  `discordChannelID`, restart finbot ([guide](doc/discord.md)).
+  `slackBotToken` and `slackAppToken`, restart finbot ([guide](doc/slack.md)).
+* **Discord:** create the application and its bot, turn on its Message Content intent, set `discordBotToken`, restart
+  finbot ([guide](doc/discord.md)).
 
-Then send `.help` in the chat.
+Then send `/help` in a chat with the bot. Commands work as plain messages too (`.help`), which are quicker to type on a
+phone ([Commands](doc/commands.md#supported-commands)). From its first command, each chat has the default scheduled
+jobs, which `/jobs` switches on and off, adds to and runs ([Scheduling](doc/scheduling.md)).
 
 ## Documentation
 * [Commands and reports](doc/commands.md), with screenshots
 * [Scheduling](doc/scheduling.md): each chat's jobs, and the reports they can run
 * [Configuration](doc/configuration.md): `finbot.env` and `preferences.conf`, the data directory, Sharesight
 * [Caching](doc/caching.md)
-* Installing: [Docker](doc/docker.md) · [a Debian host with systemd](doc/install.md)
+* Installing: [Docker](doc/docker.md) · [Python, or a Debian host with systemd](doc/install.md)
 * Networks: [Telegram](doc/telegram.md) · [Slack](doc/slack.md) · [Discord](doc/discord.md)
 
-## Suggestions
-* Is my code doing something the hard way?
-* Is something important missing from the docs?
-
-Log an [issue](https://github.com/robdevops/finbot/issues)!
+## Issues
+* [Bug reports](https://github.com/robdevops/finbot/issues) welcomed
+* PRs are welcome, but not guaranteed to be merged

bot.py

@@ -76,10 +76,10 @@ def doubled(message):
 	return False
 
 def main(environ, start_response):
-	"""WSGI entry point. Never lets an exception escape. A failure handling an authenticated request is reported as one
-	line to the default channel and answered 200, so the service does not retry it into the chat again. Before
-	authentication a failure is only logged: a stranger must not be able to post into the chats. Either names the
-	network, not the path: a webhook's path is a secret (telegramOutgoingWebhook, slackOutgoingWebhook)."""
+	"""WSGI entry point. Never lets an exception escape. A failure handling an authenticated request is logged as an
+	error report (report_error) and answered 200, so the service does not retry it into the chat again; before
+	authentication, as a failed request, with a 500. Either names the network, not the path: a webhook's path is a
+	secret (telegramOutgoingWebhook, slackOutgoingWebhook)."""
 	try:
 		return handle(environ, start_response)
 	except Exception as e:
@@ -102,10 +102,11 @@ def handle(environ, start_response):
 	comes from that network (before its body is read, when the proof is a header), read the update (CONTENT_LENGTH
 	bytes, up to MAX_BODY) and take it (take): the network gets its answer at once."""
 	def respond(response):
-		"""Answer the request: a status line and a body (logged at INFO, naming the network, not the path). Work that
-		must wait for the answer starts once the server has sent it."""
+		"""Answer the request: a status line and a body (logged, naming the network, not the path: at DEBUG when it is
+		a success, the routine; else at INFO). Work that must wait for the answer starts once the server has sent it."""
 		network = environ.get(ROUTED) or 'no network'
-		log.info(f"{environ['REQUEST_METHOD']} from {network}: we answered {response.status}")
+		(log.debug if response.status.startswith('2') else log.info)(
+			f"{environ['REQUEST_METHOD']} from {network}: we answered {response.status}")
 		start_response(response.status, [('Content-type', 'application/json')])
 		return Sent([response.body], response.then) if response.then else [response.body]
 	def print_headers(say=log.info):
@@ -126,18 +127,18 @@ def print_headers(say=log.info):
 		return respond(inbound.FORBIDDEN)
 	length = environ.get('CONTENT_LENGTH') or '0'
 	if not length.isdigit() or int(length) > MAX_BODY:
-		log.warn(f"[{source.service}]: a body of {length!r} bytes refused unread")
+		log.warn(f"{source.service.capitalize()} a body of {length!r} bytes refused unread")
 		return respond(inbound.TOO_LARGE if length.isdigit() else inbound.BAD_REQUEST)
 	body = environ['wsgi.input'].read(int(length)) # the body and no more: a server need not end the stream there
 	try:
 		update = source.decode(body)
 	except ValueError:
-		log.warn(f"[{source.service}]: request body is not a JSON object")
+		log.warn(f"{source.service.capitalize()} request body is not a JSON object")
 		return respond(inbound.BAD_REQUEST)
 	if why := source.refuse(environ, body, update):
 		shown = {k: '<redacted>' if k == 'token' else v for k, v in update.items()} # Slack's verification token
 		log.warn("warning:", why)
-		log.warn(f"[{source.service}]: refused update", json.dumps(shown, indent=4, default=str))
+		log.warn(f"{source.service.capitalize()} refused update", json.dumps(shown, indent=4, default=str))
 		return respond(source.REFUSED)
 	environ[AUTHENTICATED] = True
 	return respond(take(source.network, update))
@@ -197,10 +198,11 @@ def take(network, update):
 	if isinstance(event, inbound.Suggest): # answered here and at once, while the command is typed
 		return network.suggestions(worker.suggest(f'{event.service}:{event.chat_id}', event.text))
 	if seen_before(network.service, network.update_id(update)):
-		log.info(f"[{network.service}]: update {network.update_id(update)} seen already, not run again")
+		log.info(f"{network.service.capitalize()} update {network.update_id(update)} seen already, not run again")
 		return inbound.OK
 	if isinstance(event, inbound.Message) and not event.interaction and doubled(event): # each interaction is answered
-		log.info(f"[{network.service}]: {event.user} sent {event.text!r} again within {DOUBLED} s, not run again")
+		log.info(log.cut(f"{network.service.capitalize()} {event.user} sent it again within {DOUBLED} s, not run "
+			"again: " + event.text))
 		return inbound.OK
 	answer = network.acknowledge(event)
 	if answer is not None:
@@ -211,10 +213,10 @@ def take(network, update):
 def dispatch(event):
 	"""Hand a message or a button press to lib/worker, in a thread of its own, which it returns."""
 	if isinstance(event, inbound.Press):
-		log.info(f"[{event.service.capitalize()} button]:", event.user, "chat", event.chat_id, event.data)
+		log.debug(f"{event.service.capitalize()} button", event.user, "chat", event.chat_id, event.data)
 		target = worker.process_callback
 	else:
-		log.info(f"[{event.service.capitalize()}]:", event.user, event.text)
+		log.info(log.cut(f"{event.service.capitalize()} {event.user} {event.text}"))
 		target = worker.process_request
 	thread = threading.Thread(target=target, args=(event,))
 	thread.start()
@@ -304,7 +306,7 @@ def http_server():
 		log.warn('WARN config:', line)
 	if os.getuid() == 0:
 		log.warn("Running as superuser. This is not recommended.")
-	migrate.run() # once: what was kept for all chats becomes the home chats', schedule.cron among it
+	migrate.run() # once, on an upgrade: what was kept for all chats becomes the old home chats'
 	scheduler = jobs.Scheduler()
 	scheduler.load()
 	render.preload()

defaults/finbot.env

@@ -1,4 +1,4 @@
-# finbot.env: THE STARTUP FILE. Where finbot runs, who it talks to, and where it reports.
+# finbot.env: THE STARTUP FILE. Where finbot runs, and who it talks to.
 # The bot never reads this file. Your launcher loads it into the bot's environment, and a change needs a RESTART:
 #   systemd:  EnvironmentFile=/opt/finbot.env             (in finbot.service)
 #   docker:   --env-file /opt/finbot.env
@@ -22,32 +22,26 @@ port=5000
 # How many minutes late a scheduled report missed while the bot was down may still fire
 scheduler_catchup=5
 
-# Telegram's home chat, a group or channel id: finbot's first start gives it the default jobs (.jobs), and errors are
-# reported there
-telegramChatID=
-
 # Sharesight API access
 sharesight_code=
 sharesight_client_id=
 sharesight_client_secret=
 
-# Chat networks. A Telegram bot token is enough for reports and commands: finbot fetches its updates (pull mode)
+# Chat networks. A Telegram bot token is enough for reports and commands: by default, finbot fetches its updates
+# (pull mode)
 telegramBotToken=
 slackBotToken=
-# Slack's home channel id, as Telegram's home chat above (needs slackBotToken)
-slackChannelID=
-# Slack's app-level token (xapp-) for commands without a server: finbot opens a socket to Slack (Socket Mode)
+# Slack's app-level token (xapp-), for Socket Mode, the default: finbot opens a socket to Slack for its updates
 slackAppToken=
-# Discord's bot token: reports, and commands over the gateway (pull mode); and its home channel id, as Telegram's
+# Discord's bot token: reports, and commands over the gateway API (pull mode, the default)
 discordBotToken=
-discordChannelID=
 
 # Chat networks: inbound over HTTPS (http mode). Telegram: set both to have it post its updates to your server
 # instead of finbot fetching them
 telegramOutgoingWebhook=
 telegramOutgoingToken=
 # Discord: the Interactions Endpoint URL and the app's public key, for slash commands and buttons over HTTPS instead of
-# the gateway
+# the gateway API
 discordOutgoingWebhook=
 discordPublicKey=
 slackOutgoingWebhook=

defaults/preferences.conf

@@ -40,18 +40,22 @@
 # Regional: where you are.
 # exchange_suffix: a ticker Yahoo does not know is tried again on your exchange, by Yahoo's suffix for it (AX: BHP is
 # tried as BHP.AX; NZ, L, TO and so on for others). Empty: a ticker is taken as written.
-# country_code and timezone: the reminder report's country and time zone; .jobs also shows each next run in timezone.
+# country_code: your country, for the default jobs' messages that are for one (AU: the tax year's reminders).
+# timezone: the clock of the times .jobs takes and shows, and a job's zone where it names none.
 #exchange_suffix=AX
 #country_code=AU
 #timezone=Australia/Melbourne
 
 # Caching and network, in seconds
 #cache=True
-#cache_seconds=43200
+#cache_seconds=21600
 #performance_cache_seconds=2400
 #http_timeout=18
 
 # Diagnostics. log_level: what the log shows, that level and those above it: ERROR, WARN, INFO (what finbot does,
-# and each request it sends) or DEBUG (each cache miss, hit and write too).
+# and each request it sends) or DEBUG (each cache miss, hit and write too). log_width: the most characters of a log
+# line that can run long (each request's method and URL, each chat message), cut there, ending in an ellipsis (…);
+# 0 cuts none. A chat message of several lines is logged on one line either way.
 #log_level=INFO
+#log_width=71
 #print_headers=False