Simplify and fix: shared HTTP session, command table, Yahoo error handling, USD market caps, min_analysts, consolidation sweep

CLAUDE.md

@@ -3,14 +3,19 @@
 ## Iterating (auto mode)
 The owner chats back and forth; every message that asks for a change gets a change.
 1. Make the change, run the tests (`.venv/bin/python -m unittest discover -s tests`), fix what breaks.
-2. Commit (with the attribution trailer) and **push to the work branch** (`git push -u origin <branch>`). Do this every iteration, so the branch always holds the latest working state.
+2. Commit (with the attribution trailer) and **push to the work branch** (`git push -u origin <branch>`). Do this every iteration, so the branch always holds the latest working state. If a PR from the work branch is open, update its title and description in the same iteration (see below).
 3. Reply briefly: what changed, test result, and the pushed commit's short hash.
 
 ## The work branch is the dev branch
 The work branch named in the session instructions (`claude/...`) is the persistent dev branch. The owner's test environment pulls it and restarts the service, so:
 - **Never rewrite the work branch's history, and never reset the branch after a merge**: no reset to `main`, no rebase, no amend of pushed commits, no force-push. A force-push breaks `git pull` on the test environment. After a merge, leave the branch as it is.
 - If the owner wants a fixed name such as `dev`, they will say so; only push to another branch with their explicit permission.
 
+## Pull requests from the work branch
+An open PR takes in every commit pushed to its branch, so a PR opened for one small change can merge far more under its old title (#35 was titled as a CI tweak and merged 14 review fixes). The merge commit on `main` keeps that title for good.
+- Before each push, check for an open PR from the work branch. If there is one, update its title and description so they cover every commit it now holds, not only the change it was opened for.
+- Never make such a PR mergeable (resolve its conflict, merge `main` into it, enable auto-merge) while its title or description is out of date.
+
 ## Facts
 - Python 3.13.5; tests are offline and need no network (`tests/`). CI runs them on the latest 3.x and 3.13.5, except for pull requests that change only README.md, CLAUDE.md or `img/*.png` (install, compile and tests are skipped; the allowlist in `.github/workflows/ci.yml` must never include a path a test reads).
 - Production runs the Debian system packages (including `python3-croniter`, for the scheduler in `lib/jobs.py`); `requirements.txt` is for pip/uv users and CI.

README.md

@@ -178,7 +178,7 @@ You can also specify a portfolio name to get trades for just that portfolio:
 
 `prices.py` sends premarket, midsession, and interday price movements if any holdings moved over a percentage threshold. This data is sourced from Yahoo! Finance and Sharesight. The default threshold is 5% but you can change it by setting `price_percent` in the `preferences.conf`, or by providing a number as argument when triggered through the chat bot. Decimal fractions are accepted.
 
-If `demote_volatile` is `true`, stocks with a [market cap](https://www.investopedia.com/terms/m/marketcapitalization.asp) under 150M, or a [beta](https://www.investopedia.com/terms/b/beta.asp) of > 1.5 and a market cap of under 1B will have their alert threshold (`price_percent`) doubled.
+If `demote_volatile` is `true`, stocks with a [market cap](https://www.investopedia.com/terms/m/marketcapitalization.asp) under US$150M, or a [beta](https://www.investopedia.com/terms/b/beta.asp) of > 1.5 and a market cap of under US$1B will have their alert threshold (`price_percent`) doubled. A market cap in another currency is converted at Yahoo's exchange rate.
 
 Config example:
 ```
@@ -207,21 +207,21 @@ Interactive trigger (pre-market):
 
 Interactive trigger (mid-session):
 ```
-.midsession [percent]
+.session [percent]
 ```
 ```
-@botname midsession [percent]
+@botname session [percent]
 ```
 
 Interactive trigger (interday).
 ```
 .price [percent]
 ```
 ```
-@botname [percent]
+@botname price [percent]
 ```
 
-For interday, can also specify the time period.
+For interday, can also specify the time period. A bare number is the percent; a period carries its unit: `7d`, `2w`, `3m`, `1y` or `ytd`. For example, `.price 3 7d` lists the moves of 3% or more over the past week.
 ```
 .price [percent] [period]
 ```
@@ -316,7 +316,7 @@ Interactive trigger:
 ### pe / forward pe / peg
 ![Forward PE report in Slack](img/finbot_fpe.png?raw=true "Forward PE report in Slack")
 
-List securities by P/E ratio, forward P/E ratio, or PEG ratio
+List securities by P/E ratio, forward P/E ratio, or PEG ratio. A forward P/E is an analysts' estimate, so its lists take only stocks covered by at least `min_analysts` analysts (7 by default; see `preferences.conf`).
 
 Interactive trigger:
 ```
@@ -347,7 +347,7 @@ Interactive trigger:
 ### Buy
 ![Buy ratings in Slack](img/finbot_buy.png?raw=true "Buy ratings in Slack")
 
-Show top recommended buys by Yahoo Finance analysts rating
+Show top recommended buys by Yahoo Finance analysts rating, among stocks rated by at least `min_analysts` analysts (7 by default; see `preferences.conf`)
 
 Interactive trigger:
 ```
@@ -360,7 +360,7 @@ Interactive trigger:
 ### Sell
 ![Rating report in Slack](img/finbot_sell.png?raw=true "Rating report in Slack")
 
-Show top recommended sells by Yahoo Finance analysts rating
+Show top recommended sells by Yahoo Finance analysts rating, among stocks rated by at least `min_analysts` analysts
 
 Interactive trigger:
 ```
@@ -373,14 +373,14 @@ Interactive trigger:
 ### Marketcap
 ![Marketcap report in Slack](img/finbot_marketcap.png?raw=true "Marketcap report in Slack")
 
-List the top (or bottom) stocks by market cap
+List the top (or bottom) stocks by market cap, in US dollars: market caps in other currencies are converted at Yahoo's exchange rates. With a symbol, its market cap in its own currency and in US dollars.
 
 Interactive trigger:
 ```
-.marketcap [top|bottom]
+.marketcap [top|bottom|SYMBOL]
 ```
 ```
-@botname marketcap [top|bottom]
+@botname marketcap [top|bottom|SYMBOL]
 ```
 
 ### Consensus rating changes
@@ -480,14 +480,15 @@ Copies preferences.conf and metadata files into the data directory (never overwr
 util/sync_defaults.py [-d DEST] --apply
 ```
 
-All 25 keys, read by the bot itself. A change takes effect on a reload.
+All 26 keys, read by the bot itself. A change takes effect on a reload.
 
 | Key | Default | Description |
 |---|---|---|
 | `past_days` | `28` | How many days back reports look |
 | `future_days` | `30` | How many days ahead the events calendar looks |
 | `price_percent` | `5` | Price alert threshold, percent |
 | `shorts_percent` | `15` | Short interest alert threshold, percent |
+| `min_analysts` | `7` | `.buy`, `.sell` and `.forward pe` list only stocks covered by at least this many analysts; with fewer, a consensus says little (thinly covered small caps are nearly all a perfect buy) |
 | `demote_volatile` | `True` | Volatile (small cap, high beta) stocks need a bigger move to alert |
 | `demote_leveraged` | `True` | Leveraged (2x, 3x) funds need a bigger move to alert |
 | `volatility_multiplier` | `2` | How much bigger a move volatile stocks need |

bot.py

@@ -17,6 +17,7 @@
 from lib import jobs
 from lib import charts
 from lib import sharesight
+from lib import net
 from lib import telegram
 from lib import webhook
 
@@ -36,7 +37,7 @@ def main(environ, start_response):
 			status = '200 OK'
 		else:
 			print(f"request to {environ.get('PATH_INFO')!r} failed:", webhook.error_line(e), file=sys.stderr)
-			traceback.print_exc()
+			print(net.redact(traceback.format_exc()), end='', file=sys.stderr)
 			status = '500 Internal Server Error'
 		try:
 			start_response(status, [('Content-type', 'text/plain')], sys.exc_info())
@@ -46,18 +47,14 @@ def main(environ, start_response):
 
 def handle(environ, start_response):
 	def print_body():
-		try:
-			print(f"inbound {uri} ", json.dumps(inbound, indent=4), file=sys.stderr)
-		except Exception as e:
-			print(e, "raw body: ", inbound, file=sys.stderr)
+		shown = {k: '<redacted>' if k == 'token' else v for k, v in inbound.items()} # Slack's verification token
+		print(f"inbound {uri} ", json.dumps(shown, indent=4, default=str), file=sys.stderr)
 	def print_headers():
-		for item in sorted(environ.items()):
-			print(item, file=sys.stderr)
+		for name, value in sorted(environ.items()):
+			print((name, '<redacted>' if name == 'HTTP_X_TELEGRAM_BOT_API_SECRET_TOKEN' else value), file=sys.stderr)
 	def reply(status, body=b''):
 		start_response(status, [('Content-type', 'application/json')])
 		return [body]
-	user=''
-	userRealName=''
 	print_headers() if config_print_headers else None
 	# route on the path first, and turn strangers away before reading the body
 	uri = environ.get('PATH_INFO', '')
@@ -83,7 +80,6 @@ def reply(status, body=b''):
 	if not isinstance(inbound, dict):
 		print(f"[{service}]: request body is not a JSON object", file=sys.stderr)
 		return reply('400 Bad Request', b'<h1>Bad Request</h1>')
-	#print_body() if debug else None
 	if service == 'telegram':
 		bot_name = botName
 		if "callback_query" in inbound: # chart period buttons
@@ -97,29 +93,23 @@ def reply(status, body=b''):
 			return reply('200 OK')
 		if "message" not in inbound:
 			return reply('200 OK', b'Unsupported')
-		message_id = str(inbound["message"]["message_id"])
-		chat_id = str(inbound["message"]["chat"]["id"])
-		if "username" in inbound["message"]["from"]:
-			user = userRealname = '@' + inbound["message"]["from"]["username"]
-			if len(inbound["message"]["from"]["first_name"]):
-				userRealName = inbound["message"]["from"]["first_name"]
+		msg = inbound["message"]
+		sender = msg["from"]
+		message_id = str(msg["message_id"])
+		chat_id = str(msg["chat"]["id"])
+		if "username" in sender:
+			user = '@' + sender["username"]
+			userRealName = sender["first_name"] or user # what to call them: the handle when the first name is blank
 		else:
-			user = userRealName = '@' + inbound["message"]["from"]["first_name"]
-		file_id = None
-		if "text" in inbound["message"]:
-			message = inbound["message"]["text"]
+			user = userRealName = '@' + sender["first_name"]
+		if "text" in msg:
+			message = msg["text"]
 			print("[Telegram]:", user, message, file=sys.stderr) if debug else None
-			#if message.startswith("Upcoming earnings for the next month") and user == telegram.getMe().get('username'):
-			#	telegram.pinChatMessage(message_id, chat_id)
-			# todo need to return sendMessage to get message_id and chat_id
-		elif "photo" in inbound["message"]:
-			message = ''
-			if "caption" in inbound["message"]:
-				message = inbound["message"]["caption"]
-			photo = inbound["message"]["photo"][-1]
-			file_id = photo["file_id"]
+		elif "photo" in msg:
+			message = msg.get("caption", '')
+			file_id = msg["photo"][-1]["file_id"]
 			print("[Telegram photo]:", user, file_id, message)
-		elif IGNORED_TELEGRAM_CONTENT & inbound["message"].keys():
+		elif IGNORED_TELEGRAM_CONTENT & msg.keys():
 			return reply('200 OK') # stickers, GIFs, voice notes and the like: nothing to answer, nothing worth logging
 		else:
 			print(f"[{service}]: unhandled: 'message' without 'text/photo'", file=sys.stderr)
@@ -157,18 +147,14 @@ def reply(status, body=b''):
 			user = userRealName = '<@' + event['user'] + '>' # ZXCVBN becomes <@ZXCVBN>
 			bot_name = '@' + bot_user # QWERTY becomes @QWERTY
 			chat_id = event['channel']
-			print(f"[{service}]:", user, message, sys.stderr) if debug else None
+			print(f"[{service}]:", user, message, file=sys.stderr) if debug else None
 			# this condition spawns a worker before returning
 		else:
 			print(f"[{service}]: unhandled 'type'", file=sys.stderr)
 			return reply('200 OK', b'<h1>Unhandled</h1>')
 
-	def runWorker():
-		worker.process_request(service, chat_id, user, message, bot_name, userRealName, message_id)
-
 	# process in a background thread so we don't keep the requesting client waiting
-	t = threading.Thread(target=runWorker)
-	t.start()
+	threading.Thread(target=worker.process_request, args=(service, chat_id, user, message, bot_name, userRealName, message_id)).start()
 
 	# Return an empty response to the client
 	return reply('200 OK')
@@ -179,7 +165,7 @@ def git_version(checkout=None):
 	"""Short hash of the checked-out commit, read straight from the .git directory: no git binary to run, and no "dubious
 	ownership" refusal when the service user does not own the checkout. 'unknown' when it cannot be read."""
 	try:
-		git = Path(checkout or os.path.dirname(os.path.abspath(__file__))) / '.git'
+		git = Path(checkout or REPO_DIR) / '.git'
 		if git.is_file(): # a linked worktree or submodule: .git is a file saying "gitdir: <path>"
 			git = (git.parent / git.read_text().split(':', 1)[1].strip()).resolve()
 		common = (git / (git / 'commondir').read_text().strip()).resolve() if (git / 'commondir').is_file() else git # refs of a worktree
@@ -206,8 +192,7 @@ def reload_all(*args):
 			print('config:', change, file=sys.stderr)
 		print(f"config: reloaded {config.preferences_path}, {len(changes)} change{'' if len(changes) == 1 else 's'}", file=sys.stderr)
 	except config.ConfigError as e:
-		for line in e.errors:
-			print('ERROR config:', line, file=sys.stderr)
+		config.print_errors(e)
 	if scheduler:
 		scheduler.reload()
 
@@ -216,8 +201,7 @@ def reload_all(*args):
 	try:
 		config.check_startup_environment()
 	except config.ConfigError as e:
-		for line in e.errors:
-			print('ERROR config:', line, file=sys.stderr)
+		config.print_errors(e)
 		sys.exit(1)
 	for line in config.startup_report():
 		print('config:', line, file=sys.stderr)

bot_send_stdin.py

@@ -2,20 +2,13 @@
 
 import sys
 import os
-from lib.config import *
-from lib import util
 from lib import webhook
 
 if os.isatty(sys.stdin.fileno()):
 	raise RuntimeError("No stdin input detected (not piped or redirected)")
 
 payload = [line.strip() for line in sys.stdin if line.strip()]
 
-if not webhooks:
-	print("Error: no services enabled in finbot.env", file=sys.stderr)
-	sys.exit(1)
-for service, url in webhooks.items():
-	if service == "telegram":
-		url = webhooks['telegram'] + "sendMessage?chat_id=" + config_telegramChatID
-	webhook.payload_wrapper(service, url, payload)
+for service, url, chat in webhook.default_chats():
+	webhook.send(service, url, chat, payload)