Some of the download URL cannot be converted to signed URLs

@atweedie

We have a case where @atweedie is trying to download a result that he generated using the OST service. Whenever he downloads the result, we see in the logs the following URL that is converted to a signed URL: https://download.geohazards-tep.eu/44454d3235313539/collections/opensartoolkit-v2-2-1-mthcm-3fe4wr/items/S1A_IW_GRDH_1SDV_20241113T170607_20241113T170632_056539_06EEA8_B145-COG/assets/ost-ard-cog.

However, the location header seems to be missing, generating an error.

More detailed logs: https://grafana.apex.esa.int/goto/cfv1irzdx6akgd?orgId=1

Merge pull request #74 from ESA-APEx/improved_error_handling_signed_urls

@floeschau

After some additional logging, it seems that the initial download URL is returning a 404 error. The following error is logged through the dispatch-api.

404 Client Error: Not Found for url: https://download.geohazards-tep.eu/44454d3235313539/collections/opensartoolkit-v2-2-1-mthcm-3fe4wr/items/S1A_IW_GRDH_1SDV_20241113T170607_20241113T170632_056539_06EEA8_B145-COG/assets/ost-ard-cog

@floeschau / @fabricebrito - Do you guys have any idea why Adam is unable to download his results? I don't see any immediate changes to the dispatch that could have caused this.

@JanssenBrm

Hello @JanssenBrm,

The problem is not clear to me.

If you open the download link (https://download.geohazards-tep.eu/44454d3235313539/collections/opensartoolkit-v2-2-1-mthcm-3fe4wr/items/S1A_IW_GRDH_1SDV_20241113T170607_20241113T170632_056539_06EEA8_B145-COG/assets/ost-ard-cog) with a web browser, you should be redirected to the Terradue IAM sign-in page (if you aren't already signed in) and then to the actual signed HTTP URL for the S3 object.

This should work for any user who has Terradue/GEP credentials.I tried the link myself and downloaded the file ost-ard-cog.tif.

Can you please tell us what you are doing differently that leads to the problem on your side?

Thank you!
Frank

@floeschau

Hi @floeschau,

The problem seem like it's potentially linked to my GEP/Terradue account? At some point over a weekend a couple of months ago I was suddenly no longer able to correctly load a result via the dispatcher. This occurs to any old or new job that I run since then.

Bram can correct me if I'm wrong but it appears that the GEP has stopped returning the correct signed URLs for myself. (It certainly seems limited to just myself)

NOTE: It's not the download link that I have issue with (although that does result in a 403 for me) but the URL needed to stream the COG asset.

@atweedie

Hello @atweedie ,

Could you try the following commands from a shell, in order to exclude the possibility that there is an issue with your GEP account.

# Get token (set your credentials first)
GEP_USERNAME="<your-gep-username>"
GEP_PASSWORD="<your-gep-password>"
TOKEN=$(curl -s -X POST -d "client_id=processing-gep" -d "username=$GEP_USERNAME" -d "password=$GEP_PASSWORD" -d "grant_type=password" https://iam.terradue.com/realms/master/protocol/openid-connect/token | jq -r .access_token)

# Download file or show error message
[ "$TOKEN" == null ] && curl -s -X POST -d "client_id=processing-gep" -d "username=$GEP_USERNAME" -d "password=$GEP_PASSWORD" -d "grant_type=password" https://iam.terradue.com/realms/master/protocol/openid-connect/token || curl -Lv -o ost-ard-cog.tif -H "Authorization: Bearer $TOKEN" https://download.geohazards-tep.eu/44454d3235313539/collections/opensartoolkit-v2-2-1-mthcm-3fe4wr/items/S1A_IW_GRDH_1SDV_20241113T170607_20241113T170632_056539_06EEA8_B145-COG/assets/ost-ard-cog

If this test is succesful, the issue is with the Dispatcher API and I will provide another test for you to execute.

Please let us know the result.
Thank you

@floeschau

Hi @floeschau

I get the same 403 response as I do in the GE. I can also verify that I successfully retrieved my access token and contents of the JWT seem valid. (I am, of course, missing any context on specific claims I may need though)

{"detail":"STAC item 'opensartoolkit-v2-2-1-mthcm-3fe4wr' in collection 'S1A_IW_GRDH_1SDV_20241113T170607_20241113T170632_056539_06EEA8_B145-COG' cannot be accessed (HTTP status code: 403)"}

@atweedie

Thank you @atweedie
Could you please also share the JWT content you get ?

I wont post the full token here but these are the claims I currently get (with some values purposefully removed):

{
  "exp": 1790174076,
  "iat": 1790170476,
  "jti": "b108fabd-f87a-4ba4-9742-4810fd62ebdc",
  "iss": "https://iam.terradue.com/realms/master",
  "aud": [
    "account",
    "gep-ahf-marketplace"
  ],
  "sub": "d385872f-9191-4929-b7ac-a53f00c000e1",
  "typ": "Bearer",
  "azp": "processing-gep",
  "session_state": "2fc507ae-99ba-4f20-a87f-6872cad8ad67",
  "acr": "1",
  "allowed-origins": [
    "https://processing.geohazards-tep.eu"
  ],
  "resource_access": {
    "processing-gep": {
      "roles": [
        "ns:44454d3235313539:write"
      ]
    },
    "account": {
      "roles": [
        "manage-account",
        "manage-account-links",
        "view-profile"
      ]
    },
    "gep-ahf-marketplace": {
      "roles": [
        "marketplaceCustomer"
      ]
    }
  },
  "scope": "stac-permissions groups email openid profile",
  "sid": "2fc507ae-99ba-4f20-a87f-6872cad8ad67",
  "email_verified": true,
  "permissions": [
    "ns:44454d3235313539:write"
  ],
  "name": "#",
  "groups": [
    "44454d3235313539",
    "writers-processing",
    "default-roles-master",
    "offline_access",
    "uma_authorization"
  ],
  "preferred_username": "#",
  "given_name": "#",
  "family_name": "#",
  "email": "#"
}

@atweedie

Hello @atweedie, we updated your account settings and the download should work now. Can you please retry and let us know?

@floeschau

Hi @floeschau

The curl functionality is now working. It doesn't seem to have changed the experience within the GE however. I still get the same 403 if I try to follow the download link in the browser and the dispatcher is still not providing me with a 'view' url. This is the result returned form the dispatcher after running a new job.

{
    "id": "44454d3235313539-opensartoolkit-v2-2-1-6frdh",
    "description": "OGC API process result items for job 'opensartoolkit-v2-2-1-6frdh' of application 'opensartoolkit-v2-2-1'.",
    "stac_version": "1.0.0",
    "links": [],
    "stac_extensions": [],
    "title": "Results for opensartoolkit-v2-2-1",
    "type": "Collection",
    "assets": {
        "ost-ard-cog": {
            "href": "https://download.geohazards-tep.eu/44454d3235313539/collections/opensartoolkit-v2-2-1-6frdh-98hbwq/items/S1A_IW_GRDH_1SDV_20241113T170607_20241113T170632_056539_06EEA8_B145-COG/assets/ost-ard-cog",
            "type": "image/tiff; application=geotiff; profile=cloud-optimized",
            "title": "OST-processed ARD COG",
            "description": "OST ARD raster with 3 bands - (1) co-pol backscatter (dB), (2) cross-pol backscatter (dB), (3) derived co/cross ratio",
            "roles": [
                "data",
                "visual"
            ],
            "file:size": 57943547,
            "auth:refs": [
                "signed_url_auth"
            ],
            "alternate": {
                "https": {
                    "href": "https://download.geohazards-tep.eu/44454d3235313539/collections/opensartoolkit-v2-2-1-6frdh-98hbwq/items/S1A_IW_GRDH_1SDV_20241113T170607_20241113T170632_056539_06EEA8_B145-COG/assets/ost-ard-cog",
                    "alternate:name": "HTTPS"
                }
            }
        }
    },
    "license": "proprietary",
    "extent": {
        "spatial": {
            "bbox": [
                [
                    12.144953146710694,
                    42.03298062770812,
                    12.633995987385363,
                    42.29798363652338
                ]
            ]
        },
        "temporal": {
            "interval": [
                [
                    "2024-11-13T17:06:07Z",
                    "2024-11-13T17:06:32Z"
                ]
            ]
        }
    },
    "keywords": null,
    "providers": null,
    "summaries": null
}

@atweedie

Hello @atweedie,

Thank you for the update.

Regarding the download from the GE, have you tried signing out and in again or trying the download in a different browser or incognito/private window? It could be that with an old IdP session you still have the same insufficient permissions.

If this does not change the outcome, it would be useful for us to see the curl requests the web browser is doing and the related responses (from the Developer tools) when accessing the download URL.

Regarding the view URL, this seems to be a problem in the result collection generation. I will check why the link is missing. There should be an "xyz" link in the links array in the result collection with the tile server URL template.

Navigation Menu

Repository navigation

Some of the download URL cannot be converted to signed URLs #72

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Some of the download URL cannot be converted to signed URLs #72

Description

Activity

JanssenBrm commented on Aug 13, 2026

floeschau commented on Sep 16, 2026

atweedie commented on Sep 16, 2026

floeschau commented on Sep 17, 2026

atweedie commented on Sep 22, 2026

HerveCaumont commented on Sep 23, 2026

atweedie commented on Sep 23, 2026

floeschau commented on Sep 24, 2026

atweedie commented on Sep 24, 2026

floeschau commented on Sep 25, 2026

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions