Skip to content

Add Timestral wrapper and fev-bench results (5M, 28M, 123M, 209M) - #202

Merged
shchur merged 2 commits into
autogluon:mainfrom
adrienpetralia:timestral/fev-bench
Oct 16, 2026
Merged

shchur merged 2 commits into
autogluon:mainfrom
adrienpetralia:timestral/fev-bench

Conversation

@adrienpetralia

Copy link
Copy Markdown
Contributor

Timestral is a family of time-series forecasting models from Mistral AI (weights, code).

This PR adds:

  • models/timestral/model.py and requirements.txt: wrapper around timestral.TimestralForecaster
  • benchmarks/fev_bench/results/timestral-{5m,28m,123m,209m}.csv
  • 4 entries in benchmarks/fev_bench/models.yaml

Results on the 100 fev-bench tasks:

model SQL skill MASE skill
Timestral-5M 46.19 34.54
Timestral-28M 48.42 37.27
Timestral-123M 50.18 39.19
Timestral-209M 50.39 39.49

To reproduce (one size):

uv run python models/evaluate.py -m timestral -b benchmarks/fev_bench/tasks.yaml -n Timestral-209M -k '{"model_path": "mistralai/Timestral-209M"}'

Notes:

  • Zero-shot: no fev-bench data in the training data (trained_on_datasets = []).
  • Past and known covariates are used. Categorical covariates are target-encoded per series, and static columns are not used.
  • The forecaster defaults are used, and the context is the full history up to 32768 steps.
  • Submission tests pass locally.

By submitting this pull request, I confirm that you can use, modify, copy, and redistribute this contribution, under the terms of your choice.

url: https://huggingface.co/mistralai/Timestral-123M
model_type: pretrained
zero_shot: true
commercial_use: false

@shchur shchur Oct 15, 2026 •

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I would prefer to only show 1 model variant to avoid crowding the leaderboard and distorting the win rates. Can you please set hidden: true for the 5m, 28m and 123m models (so only 209m is shown)?

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done, thanks! Only 209M is shown now.

@adrienpetralia
adrienpetralia requested a review from shchur October 16, 2026 10:13

@shchur shchur left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for the contribution and congrats on the release!

@shchur
shchur merged commit f9e2c7d into autogluon:main Oct 16, 2026
3 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants