How to access S-2 meta-data in AWS after bucket goes to Requester pays?

Yes, this service is part of Register of open data (RODA), AWS’ initiative. Access does not require any authentication and there are no charges attached.

Thanks a lot Grega,

will this be a service free of charge?

thanks in advance,
Marcela

The paging is not supported unfortunately (and it was not before) as this service was not really meant for crawling the data. If you are interested to get all the products in specific area, use OpenSearch.
http://opensearch.sentinel-hub.com/resto/api/collections/Sentinel2/describe.xml

@gmilcinski

In your example link above to list available products, the results are capped at 1,000 items. Reviewing the S3 docs [1], it seems like the “NextContinuationToken” value should be included as the request parameter ‘continuation-token’ in order to fetch the subsequent pages. When I tested it however, I always seem to get back the first page only. Is there something else needed to page the results or is the parameter maybe not being forwarded to S3 properly?

[1] https://docs.aws.amazon.com/AmazonS3/latest/API/v2-RESTBucketGET.html

Edit: I got it now, this is just for the meta data

I am new to this. You state you have implemented a service to provide access the metadata - where is this service? I can only link to the examples. I’m looking for tileInfo.jsons.

Service is available over standard web-interface, you simply need to combine an URL as described in the first post in this thread:

https://roda.sentinel-hub.com/sentinel-s2-l1c/tiles/16/R/BV/2018/4/16/0/tileInfo.json

Many thanks for your great work, Grega.
I cannot access to metadata.xml - e.g. https://roda.sentinel-hub.com/sentinel-s2-l1c/tiles/16/R/BV/2018/4/16/0/metadata.xml
The error is “503 ERROR The request could not be satisfied.”
Any solution?
Thaks a lot for your kind support.
Naoto

Hi Naoto,
I clicked the link you have provided and it works fine for me.
We have noticed that AWS Lambda sometimes simply does not work and we do not know, what is the reason for that. However, this happens rarely and few minutes later usually works fine.
Is the problem persisting?

Thanks a lot about the news.

But a have a doubt, maybe it is a problem because that changes. (This is a big question here)

Into my context, i get the access to the address in the first moment:
https://roda.sentinel-hub.com/sentinel-s2-l1c/tiles/29/S/PC/2018/8/23/0/

But in my process i need to do download file .jp2, so the complement here is:
https://roda.sentinel-hub.com/sentinel-s2-l1c/tiles/29/S/PC/2018/8/23/0/B8A.jp2

And download the file into my repository images is the end of my process, after… i will render it.

This scenario throw me a message error:

AccessDenied
Access Denied

Why this error code? I’m runnig my tests locally and i’m in in Europe region.

I appreciate any help.
Thanks,
Eder

Thanks a lot for your prompt reply!
Unfortunately, the problem remains and no access to “metadata.xml”
Because of this problem, the download command (e.g., sentinelhub.aws --tile T54HVH 2017-04-14 -e) doesn’t work.
I can access to “tileInfo.json” and “productInfo.json” but no access to “metadata.xml” and “preview.jpg”.
I will try alternative ways of downloading data.

Hi Eder,
as described in the first post in this thread, only meta-data are available over this interface. B8A.jp2 is not meta-data (see a list of available files on top).
For your case you need to use signed requests as described in this post, check options 1-4.

Hi,
I found that the last update of sentinelhub-py (vision = 2.4.1) on GitHub, and the config.json has changed from
"aws_base_url": "http://sentinel-s2-l1c.s3.amazonaws.com/"
to
"aws_metadata_base_url": "https://roda.sentinel-hub.com/"
and no “aws base url” anyway.

Is it means we cannot download S2 images as before by sentinelhub-py 2.4.1 because of these changes?

I have used the old version (2.0.1) to download image data (.jp2) successfully, which can replace the new version.

Regards

Hi @zheng.tang ,

you are correct - older versions of sentinelhub-py are now outdated and cannot download S2 images from AWS anymore. sentinelhub-py serves only as an interface for download and the data is not available at “http://sentinel-s2-l1c.s3.amazonaws.com/” anymore. Therefore I recommend upgrading the package.

By the way, sentinelhub-py 2.4.2 has just been released with a few improvements in handling jpeg 2000 images.

Hi @maleksandrov ,

That’s strange!

I am downloading images by using the old version, and the url is “http://sentinel-s2-l1c.s3.amazonaws.com/”

But I can’t download images by version 2.4.1, there’s some bugs with connecting to AWS S3 buckets, and the url is “https://roda.sentinel-hub.com”.

I assume the version 2.0.1 still works because you have set parameter "use_s3_l1c_bucket": true which then downloads everything from s3 bucket instead of “http://sentinel-s2-l1c.s3.amazonaws.com/”. In newer versions those metadata files which are available from “https://roda.sentinel-hub.com” are downloaded from there, hence slightly less requests to s3.

It is possible that newer versions are causing you problems because we changed library for reading Jpeg 2000 images from OpenCV to Pillow. Even though Pillow doesn’t raise errors during installation sometimes its capabilities to read Jpeg 2000 images don’t get installed properly (especially on Windows). Hence installation from wheel is recommended.

Anyway if you will still have some problems with the newest version of sentinelhub-py please create a new thread on this forum about it or raise an issue on Github. We will be happy to help you resolve the problem.

Hi @maleksandrov,

Yes, I set "use_s3_l1c_bucket" as true, and filled my aws id and key in "aws_access_key_id" and "aws_secret_access_key". Then I can download anything in S3.

In newer versions those metadata files which are available from “https://roda.sentinel-hub.com” are downloaded from there, hence slightly less requests to s3.

Is that means I can download metadata only in “https://roda.sentinel-hub.com”? Then I need to write a new program to access S3 to download jp2 images.

Regards

Is that means I can download metadata only in “https://roda.sentinel-hub.com”? Then I need to write a new program to access S3 to download jp2 images.

Everything else, jp2 images included, will still be downloaded from s3 just like before. Only those metadata files which are available at “https://roda.sentinel-hub.com” will be downloaded from there. This is all handled internally by sentinelhub-py, hence your code would still work without having to make any changes.

Hi @pablotcarreira,

I tried accessing the link you provided and it works fine. Is it possible that this service was temporarily unavailable when you tried to access the file?

Hi, when I try to download images, when sentinelhub.aws try to get the metadata I get the error:

503 ERROR
The request could not be satisfied.

Even if I access the example (https://roda.sentinel-hub.com/sentinel-s2-l1c/tiles/16/R/BV/2018/4/16/0/metadata.xml) directly from the browser I get the same error.

This happens in some cases but only in limited AWS regions. We do not yet know the reason for this error and are looking into it.
Usually though it works soon after.

Ok, thank you for the quick response. I’m accessing from Brazil and I’m still getting the error, I’ll try again later.
Edit: I tried accessing the address from an Ec2 instance in Ohio, Zona A and it worked.

Hi @pablotcarreira and @geoinformatics2018,
can you please try, if the access to meta-data now works from your location?
e.g.
https://roda.sentinel-hub.com/sentinel-s2-l1c/tiles/1/K/AT/2018/7/20/0/tileInfo.json
or

Thanks,
Grega

Hi Grega,

Thanks for your effort!
I can access to “tileInfo.json”, “productInfo.json”, and “preview.jpg” but still no access to “metadata.xml”.
I’m from Japan.

Naoto

Hi, tested at 14:25 utc for two scenes. tileInfo.json works and metadata.xml return Error 503.

Tests:

https://roda.sentinel-hub.com/sentinel-s2-l1c/tiles/16/R/BV/2018/4/16/0/tileInfo.json OK
https://roda.sentinel-hub.com/sentinel-s2-l1c/tiles/16/R/BV/2018/4/16/0/metadata.xml ERROR
https://roda.sentinel-hub.com/sentinel-s2-l1c/tiles/1/K/AT/2018/7/20/0/tileInfo.json OK
https://roda.sentinel-hub.com/sentinel-s2-l1c/tiles/1/K/AT/2018/7/20/0/metadata.xml ERROR

The Error is:

503 ERROR
The request could not be satisfied.
The Lambda function associated with the CloudFront distribution is invalid or doesn’t have the required permissions.
If you received this error while trying to use an app or access a website, please contact the provider or website owner for assistance.
If you provide content to customers through CloudFront, you can find steps to troubleshoot and help prevent this error by following steps in the CloudFront documentation (http://docs.aws.amazon.com/AmazonCloudFront/latest/DeveloperGuide/http-503-service-unavailable.html).

Generated by cloudfront (CloudFront)
Request ID: ba7JhQYbdPvUXe2osi0UDDhrwWiRLvibHlflNRAAZ_xv6kPZQg-k9g==