diff --git a/tutorials/euclid/1_Euclid_intro_MER_images.md b/tutorials/euclid/1_Euclid_intro_MER_images.md index 2a3a84ad..2d570c01 100644 --- a/tutorials/euclid/1_Euclid_intro_MER_images.md +++ b/tutorials/euclid/1_Euclid_intro_MER_images.md @@ -22,7 +22,17 @@ authors: - name: Vandana Desai --- -# Euclid Q1: MER Mosaics +# Euclid DR1: MER Mosaics + +```{danger} TODO before the DR1 release (remove this box when done) +**1. Remove the cell that points `Irsa.sia_url` to irsadev, once DR1 is public on the production IRSA services.** + +**2. Switch to, or add, `access_url` downloads once IRSA redirects the DR1 `access_url` links to the cloud (expected in about a week).** + +**3. Update the Kümmel et al. reference (journal, volume, arXiv ID), confirm it can be cited, and verify the DR1 DOI.** + +**4. Re-run the notebook against production IRSA and check the outputs and plots.** +``` +++ @@ -31,10 +41,13 @@ authors: +++ By the end of this tutorial, you will: -- understand the basic characteristics of Euclid Q1 MER mosaics; -- know how to download full MER mosaics; +- understand the basic characteristics of Euclid DR1 MER mosaics; +- understand the difference between wide and deep MER tiles; +- know how to identify the official wide survey mosaic when a tile has several mosaics per band; +- know how to access full MER mosaics from the cloud; - know how to make smaller cutouts of MER mosaics; - know how to use matplotlib to plot a grid of cutouts; +- know how to make a color composite from the Euclid VIS and NISP cutouts; - know how to identify sources in the cutouts and make basic measurements. +++ @@ -48,20 +61,37 @@ The primary science goals of Euclid are to better understand the composition and The Euclid mission is providing space-based imaging and spectroscopy as well as supporting ground-based imaging to achieve these primary goals. These data will be archived by multiple global repositories, including IRSA, where they will support transformational work in many areas of astrophysics. -Euclid Quick Release 1 (Q1) consists of consists of ~30 TB of imaging, spectroscopy, and catalogs covering four non-contiguous fields: -Euclid Deep Field North (22.9 sq deg), Euclid Deep Field Fornax (12.1 sq deg), Euclid Deep Field South (28.1 sq deg), and LDN1641. +Euclid Data Release 1 (DR1) comprises about 1900 sq deg of imaging, spectroscopy, and catalogs from the Euclid Wide Survey (EWS), which will eventually cover about 14,000 sq deg, and from the Euclid Deep Survey (EDS), which covers about 63 sq deg in the three Euclid Deep Fields (North, South, and Fornax). +DR1 also includes several photometric redshift and instrument calibration fields, such as COSMOS, CDFS, and AEGIS. +It follows Quick Release 1 (Q1), which covered four much smaller fields; all of the DR1 data, including the areas observed in Q1, have been reprocessed with updated pipelines and calibrations. -Among the data products included in the Q1 release are the Level 2 MER mosaics. -These are multiwavelength mosaics created from images taken with the Euclid instruments (VIS and NISP), as well as a variety of ground-based telescopes. -All of the mosaics have been created according to a uniform tiling on the sky, and mapped to a common pixel scale. +Among the data products included in DR1 are the Level 2 MER mosaics. +These are multiwavelength mosaics created from images taken with the Euclid instruments (VIS and NISP), as well as a variety of ground-based surveys (DES, CFIS, Pan-STARRS, and HSC-based surveys) that provide the u, g, r, i, and z bands. +All of the mosaics have been created according to a uniform tiling on the sky, and mapped to a common pixel scale of 0.1 arcsec. +The MER processing for DR1 is described in detail in Euclid Collaboration: Kümmel et al. (2026), see the [References](#references) section below. This notebook provides a quick introduction to accessing MER mosaics from IRSA. If you have questions about it, please contact the [IRSA helpdesk](https://irsa.ipac.caltech.edu/docs/help_desk.html). +++ +## Wide and deep MER tiles + +DR1 MER mosaics come in two kinds of tiles, which IRSA serves as two separate collections: + +* **Wide tiles** (`euclid_dr1_wide_DpdMerBksMosaic`): tile IDs start with `102`, about 32 x 32 arcmin, the same tiling as in Q1. +* **Deep tiles** (`euclid_dr1_deep_DpdMerBksMosaic`): tile IDs start with `101`, about 17 x 17 arcmin, covering the deep fields. + They stack all of the deep field observations (about 8 times the VIS exposure time of a wide tile), so they reach much fainter sources. + +The deep fields are also covered by wide tiles, and because they have been observed many times, these wide tiles have **several mosaics in every band**. +Only one is the official wide survey mosaic; the others are each built from a single pass over the deep field. +The `PPLANID` keyword in the FITS header tells them apart: the official mosaic has `DATA_RELEASE` in it, the others `DATA_RELEASE_DEEP`. + ++++ + ## Data volume -Each MER image is approximately 1.47 GB. Downloading can take some time. +Each wide MER mosaic is approximately 1.47 GB, and each deep MER mosaic is approximately 0.42 GB. +Downloading can take some time. +++ @@ -73,11 +103,11 @@ We rely on ``astroquery`` and ``sep`` features that have been recently added, so ```{code-cell} ipython3 # Uncomment the next line to install dependencies if needed. -# !pip install numpy 'astropy>=5.3' matplotlib 'astroquery>=0.4.10' 'sep>=1.4' fsspec +# !pip install numpy 'astropy>=5.3' matplotlib 'astroquery>=0.4.10' 'sep>=1.4' fsspec s3fs ``` ```{code-cell} ipython3 -import re +import json import numpy as np @@ -87,88 +117,139 @@ from matplotlib.patches import Ellipse from astropy.coordinates import SkyCoord from astropy.io import fits from astropy.nddata import Cutout2D -from astropy.utils.data import download_file +from astropy.stats import sigma_clipped_stats from astropy.visualization import ImageNormalize, PercentileInterval, AsinhStretch, ZScaleInterval, SquaredStretch from astropy.wcs import WCS from astropy import units as u from astroquery.ipac.irsa import Irsa +import s3fs import sep ``` -## 1. Search for multiwavelength Euclid Q1 MER mosaics that cover the star HD 168151 -Below we set a search radius of 10 arcsec and convert the name "HD 168151" into coordinates. + + +```{code-cell} ipython3 +# DR1 is not public yet, so we query IRSA's development server. +Irsa.sia_url = 'https://irsadev.ipac.caltech.edu/SIA' +``` + +## 1. Search for multiwavelength Euclid DR1 MER mosaics in the Euclid Deep Field North +Below we set a search radius of 10 arcsec around a position in the Euclid Deep Field North, an area covered by both deep and wide tiles. ```{code-cell} ipython3 search_radius = 10 * u.arcsec -coord = SkyCoord.from_name('HD 168151') +coord = SkyCoord(273.1670, 64.3687, unit='deg', frame='icrs') ``` -Use IRSA's Simple Image Access (SIA) API to search for all Euclid MER mosaics that overlap with the search region you have specified. We specify the euclid_DpdMerBksMosaic "collection" because it lists all of the multiwavelength MER mosaics, along with their associated catalogs. +Use IRSA's Simple Image Access (SIA) API to search for all Euclid MER mosaics that overlap with the search region you have specified. ```{tip} -The IRSA SIA collections can be listed using using the ``list_collections`` method, we can filter on the ones containing "euclid" in the collection name: +The IRSA SIA collections can be listed using the ``list_collections`` method, we can filter on the ones containing "euclid" in the collection name: Irsa.list_collections(filter='euclid') ``` -There are currently four collections available: +We search both DR1 MER collections described above, one for the deep and one for the wide tiles. +IRSA serves other Euclid data products, as well as the Q1 and Early Release Observations data, in further collections. +For more information, please read the [IRSA User Guide](https://irsa.ipac.caltech.edu/data/Euclid/docs/euclid_archive_at_irsa_user_guide.pdf). -* `'euclid_ero'` -- the Early Release Observations, for more information see [Euclid ERO documentation at IRSA](https://irsa.ipac.caltech.edu/data/Euclid/docs/overview_ero.html). -* `'euclid_DpdMerBksMosaic'` -- This is the collection of interest for us in this notebook, it contains all the Q1 MER mosaic data. For more information on MER mosaics, see the "Introduction" above. -* `'euclid_DpdVisCalibratedQuadFrame'` -- This collection contains the VIS instrument Calibrated images. Please note these data are not stacked, so there are four dither positions for every observation ID and two calibration frames. -* `'euclid_DpdNirCalibratedFrame'` -- This collection contains the NIR instrument Calibrated images. Please note these data are not stacked, so there are four dither positions for every observation ID. +```{code-cell} ipython3 +deep_table = Irsa.query_sia(pos=(coord, search_radius), collection='euclid_dr1_deep_DpdMerBksMosaic') +wide_table = Irsa.query_sia(pos=(coord, search_radius), collection='euclid_dr1_wide_DpdMerBksMosaic') +``` -For more information on the VIS and NIR calibrated frames, please read the [IRSA User Guide](https://irsa.ipac.caltech.edu/data/Euclid/docs/euclid_archive_at_irsa_user_guide.pdf) +These tables list all MER mosaic images available in this search position. These mosaics include the Euclid VIS, Y, J, H images, as well as ground-based telescopes which have been put on the same pixel scale. For more information, see the [Euclid documentation at IPAC](https://euclid.caltech.edu/page/euclid-faq-tech/). -In this notebook, we will focus on downloading and visualizing the MER mosaic data. +Note that there are various image types are returned as well, we filter out the `science` images from these: ```{code-cell} ipython3 -image_table = Irsa.query_sia(pos=(coord, search_radius), collection='euclid_DpdMerBksMosaic') +science_images = deep_table[deep_table['dataproduct_subtype'] == 'science'] +wide_science_images = wide_table[wide_table['dataproduct_subtype'] == 'science'] ``` -This table lists all MER mosaic images available in this search position. These mosaics include the Euclid VIS, Y, J, H images, as well as ground-based telescopes which have been put on the same pixel scale. For more information, see the [Euclid documentation at IPAC](https://euclid.caltech.edu/page/euclid-faq-tech/). +The tile ID is the first part of the `obs_id`: deep tile IDs start with `101` and wide tile IDs with `102`. +Adjacent tiles overlap, and our position happens to lie in the overlap of two deep tiles and of two wide tiles: -Note that there are various image types are returned as well, we filter out the `science` images from these: +```{code-cell} ipython3 +for name, images in [('Deep', science_images), ('Wide', wide_science_images)]: + tile_ids, n_vis = np.unique([obs_id[:9] for obs_id in images[images['energy_bandpassname'] == 'VIS']['obs_id']], return_counts=True) + print(f'{name} tiles and their number of VIS mosaics:', dict(zip(tile_ids.tolist(), n_vis.tolist()))) +``` + +Each deep tile has a single VIS mosaic, while the wide tiles have several. +For the deep tile, we keep the one whose center is closest to our position: ```{code-cell} ipython3 -science_images = image_table[image_table['dataproduct_subtype'] == 'science'] +tile_centers = SkyCoord(science_images['s_ra'], science_images['s_dec'], unit='deg') +deep_tileID = science_images['obs_id'][np.argmin(tile_centers.separation(coord))][:9] +science_images = science_images[[obs_id.startswith(deep_tileID) for obs_id in science_images['obs_id']]] + +print(f'Deep MER tile ID: {deep_tileID}') science_images ``` -## 2. Retrieve a Euclid Q1 MER mosaic image in the VIS bandpass +## 2. Locate the mosaics in the cloud -+++ - -### Let's first look at one example full image, the VIS image +IRSA hosts the DR1 MER mosaics in an AWS S3 bucket. +The `cloud_access` column of the search results gives the bucket name and file location of each mosaic, so let's extract the S3 file path from it: -Note that 'access_estsize' is in units of kb + ```{code-cell} ipython3 -filename = science_images[science_images['energy_bandpassname'] == 'VIS']['access_url'][0] -filesize = science_images[science_images['energy_bandpassname'] == 'VIS']['access_estsize'][0] / 1000000 -print(filename) +def get_s3_fpath(cloud_access): + cloud_info = json.loads(cloud_access) # converts str to dict + bucket_name = cloud_info['aws']['bucket_name'] + key = cloud_info['aws']['key'] -print(f'Please note this image is {filesize} GB. With 230 Mbps internet download speed, it takes about 1 minute to download.') -``` + return f'{bucket_name}/{key}' -```{code-cell} ipython3 -science_images + +science_images['s3_fpath'] = [get_s3_fpath(row['cloud_access']) for row in science_images] +wide_science_images['s3_fpath'] = [get_s3_fpath(row['cloud_access']) for row in wide_science_images] ``` -### Extract the tileID of this image from the filename +## 3. Identify the official wide survey VIS mosaic + +To find the official wide survey VIS mosaic, we read the `PPLANID` keyword from the header of each wide VIS mosaic returned by our search. +Reading only the header with a small block size avoids downloading the large image data. ```{code-cell} ipython3 -tileID = science_images[science_images['energy_bandpassname'] == 'VIS']['obs_id'][0][:9] +s3 = s3fs.S3FileSystem(anon=True) + + +def get_processing_plan(s3_fpath): + try: + with s3.open(s3_fpath, block_size=28800) as f: + return fits.Header.fromfile(f).get('PPLANID', '') + except FileNotFoundError: + # A few files listed in the search results may not be available in the bucket + return '' + -print(f'The MER tile ID for this object is : {tileID}') +wide_vis_images = wide_science_images[wide_science_images['energy_bandpassname'] == 'VIS'] +wide_vis_images['plan'] = [get_processing_plan(path) for path in wide_vis_images['s3_fpath']] + +wide_vis_images['official'] = ['-DATA_RELEASE-' in plan for plan in wide_vis_images['plan']] +wide_tileID = wide_vis_images[wide_vis_images['official']]['obs_id'][0][:9] + +wide_vis_images['obs_id', 'plan', 'official'] ``` -Retrieve the MER image -- note this file is about 1.46 GB +## 4. Retrieve a Euclid DR1 MER mosaic image in the VIS bandpass + ++++ + +### Let's first look at one example full image, the deep VIS image + +We open the full deep VIS mosaic directly from the S3 bucket. Note this file is about 0.42 GB. ```{code-cell} ipython3 -fname = download_file(filename, cache=True) -hdu_mer_irsa = fits.open(fname) +filename = science_images[science_images['energy_bandpassname'] == 'VIS']['s3_fpath'][0] +print(filename) + +hdu_mer_irsa = fits.open(f's3://{filename}', fsspec_kwargs={'anon': True}) print(hdu_mer_irsa.info()) header_mer_irsa = hdu_mer_irsa[0].header @@ -188,7 +269,7 @@ Have a look at the header information for this image. header_mer_irsa ``` -Lets extract just the primary image. +Lets extract just the primary image. This reads the full image from the cloud, so it may take a minute. ```{code-cell} ipython3 im_mer_irsa = hdu_mer_irsa[0].data @@ -196,11 +277,13 @@ im_mer_irsa = hdu_mer_irsa[0].data print(im_mer_irsa.shape) ``` -Due to the large field of view of the MER mosaic, let's cut out a smaller section (2"x2")of the MER mosaic to inspect the image +Due to the large field of view of the MER mosaic, let's cut out a smaller section (2'x2') around our search position to inspect the image ```{code-cell} ipython3 -plt.imshow(im_mer_irsa[0:1200,0:1200], cmap='gray', origin='lower', - norm=ImageNormalize(im_mer_irsa[0:1200,0:1200], interval=PercentileInterval(99.9), stretch=AsinhStretch())) +im_2arcmin = Cutout2D(im_mer_irsa, position=coord, size=2 * u.arcmin, wcs=WCS(header_mer_irsa)).data + +plt.imshow(im_2arcmin, cmap='gray', origin='lower', + norm=ImageNormalize(im_2arcmin, interval=PercentileInterval(99.9), stretch=AsinhStretch())) colorbar = plt.colorbar() ``` @@ -212,7 +295,7 @@ Uncomment the code below to plot an image of the entire field of view of the MER # colorbar = plt.colorbar() ``` -## 3. Create multiwavelength Euclid Q1 MER cutouts of a region of interest +## 5. Create multiwavelength Euclid DR1 MER cutouts of a region of interest +++ @@ -220,12 +303,6 @@ Uncomment the code below to plot an image of the entire field of view of the MER We'd like to take a look at the multiwavelength images of our object, but the full MER mosaics are very large, so we will inspect the multiwavelength cutouts. ``` -```{code-cell} ipython3 -urls = science_images['access_url'] - -urls -``` - Create an array with the instrument and filter name so we can add this to the plots. ```{code-cell} ipython3 @@ -237,58 +314,40 @@ science_images['filters'][science_images['filters']== 'VIS_VIS'] = "VIS" science_images['filters'] ``` -## 4. Define cutout parameters for a smaller region of interest +## 6. Define cutout parameters for a smaller region of interest ```{code-cell} ipython3 ######################## User defined section ############################ # Set the image cutout size im_cutout = 1.0 * u.arcmin -# Set the cutout center coordinates -# For now choose a random location on the image -# because the star itself is saturated -ra = 273.8667 -dec = 64.525 - -# Bright star position -# ra = 273.474451 -# dec = 64.397273 - -coords_cutout = SkyCoord(ra, dec, unit='deg', frame='icrs') +# Set the cutout center coordinates, here our search position +coords_cutout = coord ########################################################################## -# Iterate through each filter - -cutout_list = [] -for url in urls: - # Use fsspec to interact with the fits file without downloading the full file - hdu = fits.open(url, use_fsspec=True) - print(f"Opened {url}") - - # Store the header - header = hdu[0].header - - # Read in the cutout of the image that you want - cutout_data = Cutout2D(hdu[0].section, position=coords_cutout, size=im_cutout, wcs=WCS(hdu[0].header)) - - # Close the file - # hdu.close() +def make_cutout(s3_fpath): + # Read in only the cutout of the image that you want, without downloading the full file + with fits.open(f's3://{s3_fpath}', fsspec_kwargs={'anon': True}) as hdu: + header = hdu[0].header + cutout_data = Cutout2D(hdu[0].section, position=coords_cutout, size=im_cutout, wcs=WCS(header)) # Define a new fits file based on this smaller cutout, with accurate WCS based on the cutout size new_hdu = fits.PrimaryHDU(data=cutout_data.data, header=header) new_hdu.header.update(cutout_data.wcs.to_header()) + return new_hdu - # Append the cutout to the list - cutout_list.append(new_hdu) + +# Iterate through each filter +cutout_list = [make_cutout(s3_fpath) for s3_fpath in science_images['s3_fpath']] # Combine all cutouts into a single HDUList and display information final_hdulist = fits.HDUList(cutout_list) final_hdulist.info() ``` -## 5. Visualize multiwavelength Euclid Q1 MER cutouts +## 7. Visualize multiwavelength Euclid DR1 MER cutouts We need to determine the number of images for the grid layout, and then plot each cutout. @@ -316,47 +375,92 @@ plt.tight_layout() plt.show() ``` -## 6. Identify and measure sources in Euclid Q1 MER cutouts with sep +### Make a color composite -First we list all the filters so you can choose which cutout you want to extract sources on. We will choose VIS. +Because all MER mosaics share the same pixel grid, we can combine the deep cutouts directly into a false-color RGB image, with NISP H as red, NISP J as green, and VIS as blue. ```{code-cell} ipython3 -filt_index = np.where(science_images['filters'] == 'VIS')[0][0] +def scale_band(filt): + image_data = final_hdulist[np.where(science_images['filters'] == filt)[0][0]].data + return ImageNormalize(image_data, interval=PercentileInterval(99.5), stretch=AsinhStretch(), clip=True)(image_data) -img1 = final_hdulist[filt_index].data -``` -### Extract some sources from the cutout using sep (python package based on source extractor) +rgb = np.dstack([scale_band('NISP_H'), scale_band('NISP_J'), scale_band('VIS')]) -Following the sep tutorial, first create a background for the cutout -https://sep.readthedocs.io/en/stable/tutorial.html +plt.figure(figsize=(6, 6)) +plt.imshow(rgb, origin='lower') +plt.title('Deep tile: H (red), J (green), VIS (blue)') +plt.axis('off') +plt.show() +``` -Need to do some initial steps (swap byte order) with the cutout to prevent sep from crashing. Then create a background model with sep. +### Compare the deep and wide VIS cutouts + +Let's plot the same region in all of the VIS mosaics of the wide tile: first the official wide survey mosaic, then the mosaics built from single passes over the deep field, and finally the deep mosaic. +All of them are in the same units, so we display them with the same linear stretch from -2 to +5 times the sky noise of the official wide mosaic. +This makes the noise visible: the wide mosaics look grainy, while the deep mosaic is smoother and reveals more faint sources. +The sky noise of each mosaic, measured with sigma clipping, is given in the panel titles. ```{code-cell} ipython3 -img2 = img1.byteswap().view(img1.dtype.newbyteorder()) -c_contiguous_data = np.array(img2, dtype=np.float32) +filt_index = np.where(science_images['filters'] == 'VIS')[0][0] +deep_vis = final_hdulist[filt_index].data + +# Wide VIS mosaics of the tile with the official mosaic, official first, skipping files not available in the bucket +wide_tile_vis = wide_vis_images[[obs_id.startswith(wide_tileID) for obs_id in wide_vis_images['obs_id']]] +wide_tile_vis = wide_tile_vis[wide_tile_vis['plan'] != ''] +wide_tile_vis.sort('official', reverse=True) -bkg = sep.Background(c_contiguous_data) +images = [make_cutout(s3_fpath).data for s3_fpath in wide_tile_vis['s3_fpath']] + [deep_vis] +titles = (['Wide (official)'] + [f'Wide (single pass {i})' for i in range(1, len(wide_tile_vis))] + + [f'Deep tile {deep_tileID}']) -bkg_image = bkg.back() +noise = [sigma_clipped_stats(image_data)[2] for image_data in images] +norm = ImageNormalize(vmin=-2 * noise[0], vmax=5 * noise[0]) -plt.imshow(bkg_image, interpolation='nearest', cmap='gray', origin='lower') -plt.colorbar() +columns = 4 +rows = -(-len(images) // columns) +fig, axes = plt.subplots(rows, columns, figsize=(4 * columns, 4 * rows)) +axes = axes.flatten() +for ax, image_data, title, sigma in zip(axes, images, titles, noise): + ax.imshow(image_data, cmap='gray', origin='lower', norm=norm) + ax.set_title(f'{title}\nnoise = {sigma:.1e}') + ax.set_xticks([]) + ax.set_yticks([]) +for ax in axes[len(images):]: + fig.delaxes(ax) +plt.tight_layout() +plt.show() ``` -### Inspect the background rms as well +## 8. Identify and measure sources in Euclid DR1 MER cutouts with sep -```{code-cell} ipython3 -bkg_rms = bkg.rms() +We extract sources on the deep VIS cutout. -plt.imshow(bkg_rms, interpolation='nearest', cmap='gray', origin='lower') -plt.colorbar(); +```{code-cell} ipython3 +img1 = deep_vis ``` -### Subtract the background +### Extract some sources from the cutout using sep (python package based on source extractor) + +Following the sep tutorial, first create a background for the cutout +https://sep.readthedocs.io/en/stable/tutorial.html + +Need to do some initial steps (swap byte order) with the cutout to prevent sep from crashing. Then create a background model with sep, inspect it and its rms, and subtract it from the cutout. ```{code-cell} ipython3 +img2 = img1.byteswap().view(img1.dtype.newbyteorder()) +c_contiguous_data = np.array(img2, dtype=np.float32) + +# Use a background mesh (bw, bh) larger than our galaxy, so its light is not absorbed into the background +bkg = sep.Background(c_contiguous_data, bw=128, bh=128) + +fig, axes = plt.subplots(1, 2, figsize=(10, 4)) +for ax, image_data, title in zip(axes, [bkg.back(), bkg.rms()], ['Background', 'Background rms']): + im = ax.imshow(image_data, interpolation='nearest', cmap='gray', origin='lower') + ax.set_title(title) + fig.colorbar(im, ax=ax) +plt.show() + data_sub = img2 - bkg ``` @@ -383,7 +487,7 @@ sources_thr = sources[sources['flux'] > flux_threshold] print("Found", len(sources_thr), "objects above flux threshold") ``` -## 7. Review detected sources on the VIS cutout +## 9. Review detected sources on the VIS cutout We plot the VIS cutout with the sources detected overplotted with a red ellipse @@ -405,8 +509,16 @@ for i in range(len(sources_thr)): ax.add_artist(e) ``` +## References + + + +- Euclid Collaboration: Kümmel, M., et al. 2026, *Euclid Data Release (DR1): The MERge processing*, A&A, in prep. +- Euclid Collaboration: Romelli, E., Kümmel, M., Dole, H., et al. 2026, A&A, 711, A4 (the Q1 MER processing) +- Euclid Data Release 1, 2026, [doi:10.57780/esa-d079c51](https://doi.org/10.57780/esa-d079c51) + ## About this Notebook -**Updated**: 2025-03-31 +**Updated**: 2026-10-05 **Contact:** [the IRSA Helpdesk](https://irsa.ipac.caltech.edu/docs/help_desk.html) with questions or reporting problems. diff --git a/tutorials/euclid/2_Euclid_intro_MER_catalog.md b/tutorials/euclid/2_Euclid_intro_MER_catalog.md index 892b0990..1d825cc8 100644 --- a/tutorials/euclid/2_Euclid_intro_MER_catalog.md +++ b/tutorials/euclid/2_Euclid_intro_MER_catalog.md @@ -22,7 +22,13 @@ authors: - name: Vandana Desai --- -# Euclid Q1: MER Catalogs +# Euclid DR1: MER Catalogs + +```{danger} TODO before the DR1 release (remove this box when done) +**1. Remove the cell that points `Irsa.tap_url` and `Irsa.sia_url` to irsadev, once DR1 is public on the production IRSA services, then re-run the notebook against production and check the outputs and the plots.** + +**2. Update the Kümmel et al. reference (journal, volume, arXiv ID) and verify the DR1 DOI.** +``` +++ @@ -31,10 +37,14 @@ authors: +++ By the end of this tutorial, you will: -- Understand the basic characteristics of Euclid Q1 MER catalogs. +- Understand the basic characteristics of Euclid DR1 MER catalogs. +- Know which of the four DR1 MER catalog tables to use. - What columns are available in the MER catalog. -- How to query with ADQL in the MER catalog. -- How to make a simple color-magnitude diagram with the data. +- How to search the MER catalog around a position on the sky. +- How to use the MER quality columns to clean a catalog. +- How to select stars and compute their total fluxes in the Euclid and ground-based bands. +- How to look at the MER VIS images of catalog sources. +- How to query the MER catalog with ADQL and make a color-color diagram of stars. +++ @@ -47,14 +57,47 @@ The primary science goals of Euclid are to better understand the composition and The Euclid mission is providing space-based imaging and spectroscopy as well as supporting ground-based imaging to achieve these primary goals. These data will be archived by multiple global repositories, including IRSA, where they will support transformational work in many areas of astrophysics. -Euclid Quick Release 1 (Q1) consists of consists of ~30 TB of imaging, spectroscopy, and catalogs covering four non-contiguous fields: -Euclid Deep Field North (22.9 sq deg), Euclid Deep Field Fornax (12.1 sq deg), Euclid Deep Field South (28.1 sq deg), and LDN1641. +Euclid Data Release 1 (DR1) comprises about 1900 sq deg of imaging, spectroscopy, and catalogs from the Euclid Wide Survey (EWS), which will eventually cover about 14,000 sq deg, and from the Euclid Deep Survey (EDS), which covers about 63 sq deg in the three Euclid Deep Fields (North, South, and Fornax). +DR1 also includes several photometric redshift and instrument calibration fields, such as COSMOS, CDFS, and AEGIS. +It follows Quick Release 1 (Q1), which covered four much smaller fields; all of the DR1 data, including the areas observed in Q1, have been reprocessed with updated pipelines and calibrations. -Among the data products included in the Q1 release are the three MER catalogs: the final catalog, the morphology catalog, and the cutouts catalog. -This notebook provides an introduction to the MER final catalog. -Each entry is a single source with associated photometry from the multiwavelength MER Mosaics (VIS, Y, J, H, and any accompanying external ground-based measurements), along with other basic measurements, like size and shape. +Among the data products included in DR1 are the MER catalogs. +The MER processing detects sources on the VIS and NISP mosaics (see the MER mosaics notebook) and measures their photometry in the Euclid VIS, Y, J, H bands and in the accompanying ground-based u, g, r, i, z bands, along with other basic measurements, like size and shape. +This notebook provides an introduction to the MER final catalog, in which each entry is a single source. +In this notebook we follow a typical path through the MER catalog: we search for the sources around a position on the sky, clean the sample with the quality columns, select the stars, look at the spectral energy distributions and VIS images of a few of them, and finally make a color-color diagram of many stars. +The MER processing for DR1 is described in detail in Euclid Collaboration: Kümmel et al. (2026), see the [References](#references) section below. If you have questions about this notebook, please contact the [IRSA helpdesk](https://irsa.ipac.caltech.edu/docs/help_desk.html). + + ++++ + +## The MER catalog tables + +The DR1 MER catalog is split into four tables, according to how the data were observed and which MER tiling was used. +MER processes the sky in tiles: **wide** tiles (tile IDs starting with `102`, about 32 x 32 arcmin) and smaller **deep** tiles (tile IDs starting with `101`, about 17 x 17 arcmin), used for fields too deep to process as wide tiles. + +* `euclid_dr1_mer_catalogue_wide_survey` -- **the Euclid Wide Survey**, the standard-depth survey intended for cosmology (wide tiles). + **If you are looking for the standard Euclid survey, this is the table you want**, and it is the one used in this notebook. +* `euclid_dr1_mer_catalogue_deep_survey` -- **the Euclid Deep Survey**: the deep stacks of the Euclid Deep Fields North, South, and Fornax (deep tiles). +* `euclid_dr1_mer_catalogue_deep_mode` -- other fields deep enough to need the deep tiling, but not part of the Deep Survey: the Euclid auxiliary fields COSMOS, CDFS, and AEGIS. +* `euclid_dr1_mer_catalogue_wide_mode` -- other fields processed with the wide tiling that are neither part of the Wide Survey nor visits of the deep fields, for example some of the fields observed during the thermal stabilization period of the PSF calibration campaign, just before the main survey started (January/February 2024). + + + + +The deep fields are covered by both the Deep Survey (deep tiles) and the wide tiling, so the same object can appear in more than one table, measured on data of different depth. +Table 5 of Kümmel et al. (2026) lists all of the DR1 fields with their tiling and exposure times. + +The MER catalog from the earlier Quick Release 1 (Q1) is also available at IRSA, as the single table `euclid_q1_mer_catalogue`. +All of the Q1 areas have been reprocessed for DR1, so Q1 and DR1 fluxes should not be mixed (see the [notes on the DR1 MER photometry](#notes-on-the-dr1-mer-photometry) below). + +++ ## Imports @@ -65,20 +108,38 @@ We rely on ``astroquery`` features that have been recently added, so please make ```{code-cell} ipython3 # Uncomment the next line to install dependencies if needed -# !pip install numpy matplotlib 'astroquery>=0.4.10' +# !pip install numpy matplotlib astropy 'astroquery>=0.4.10' fsspec s3fs ``` ```{code-cell} ipython3 +import json + import numpy as np import matplotlib.pyplot as plt +import astropy.units as u +from astropy.coordinates import SkyCoord +from astropy.io import fits +from astropy.nddata import Cutout2D +from astropy.visualization import ImageNormalize, PercentileInterval, AsinhStretch +from astropy.wcs import WCS from astroquery.ipac.irsa import Irsa ``` -## 1. Download MER catalog from IRSA directly to this notebook + + +```{code-cell} ipython3 +# DR1 is not public yet, so we query IRSA's development server. +Irsa.tap_url = 'https://irsadev.ipac.caltech.edu/TAP' +Irsa.sia_url = 'https://irsadev.ipac.caltech.edu/SIA' +``` + +## 1. Find the MER catalog tables at IRSA +++ +The DR1 MER catalog is far too large to download, so we use IRSA's services to run queries on the server and retrieve only the rows and columns we need. + First, have a look at what Euclid catalogs are available. With the ``list_catalogs`` functionality, we'll receive a list of the name of the catalogs as well as their brief desciption. ```{code-cell} ipython3 @@ -87,8 +148,10 @@ Irsa.list_catalogs(filter='euclid') ### Choose the Euclid MER table +We use the Wide Survey table. To use another one of the four DR1 MER tables described above, change the table name here. + ```{code-cell} ipython3 -table_mer = 'euclid_q1_mer_catalogue' +table_mer = 'euclid_dr1_mer_catalogue_wide_survey' ``` ### Learn some information about the MER catalog: @@ -104,72 +167,342 @@ print(len(columns_info)) columns_info ``` + + ```{tip} -The MER catalog contains 476 columns, below are a few highlights: +Each DR1 MER catalog table contains 480 columns, below are a few highlights: - object_id -- flux_vis_psf, mer.flux_y_templfit,flux_j_templfit, mer.flux_h_templfit +- flux_detection_total: the total (Kron) flux in the detection band +- flux_y_templfit, flux_j_templfit, flux_h_templfit: template-fitting fluxes +- flux_vis_to_y_templfit, flux_vis_to_j_templfit, flux_vis_to_h_templfit: template-fitting fluxes measured on the VIS image degraded to the resolution of each NIR band - fwhm +- point_like_flag, point_like_prob, spurious_prob ``` +++ -### Define the following ADQL query to find the first 10k stars in the MER catalog +## 2. Search the MER catalog around a position + +We start with all of the sources within 0.5 deg of a position in the Wide Survey. +This area contains a few hundred thousand sources, so rather than retrieving all 480 columns, we write an ADQL query that returns only the columns we need for this notebook: +the positions and tile IDs, the quality columns, the star-galaxy classification, and the template-fitting fluxes in the Euclid and ground-based bands (see section 5). +The `CONTAINS` and `CIRCLE` functions select the sources in a circle on the sky. + + + +```{code-cell} ipython3 +coord = SkyCoord(ra=90.7299, dec=-45.3529, unit='deg') +search_radius = 0.5 * u.deg + +# Template-fitting fluxes in the Euclid NIR bands and in all of the ground-based bands +flux_cols = ([f'flux_{band}_templfit' for band in 'yjh'] + [f'flux_vis_to_{band}_templfit' for band in 'yjh'] + + [col for col in columns_info + if col.startswith('flux_') and '_ext_' in col and col.endswith('_templfit')]) +columns = (['object_id', 'ra', 'dec', 'tileid', 'vis_det', 'flux_detection_total', + 'point_like_flag', 'point_like_prob', 'spurious_prob', 'det_quality_flag', + 'flag_vis', 'flag_y', 'flag_j', 'flag_h'] + flux_cols) + +adql = (f"SELECT {', '.join(columns)} " + f"FROM {table_mer} " + "WHERE CONTAINS(POINT('ICRS', ra, dec), " + f"CIRCLE('ICRS', {coord.ra.deg}, {coord.dec.deg}, {search_radius.to_value(u.deg)})) = 1") + +# Alternatively, uncomment the query below to get 10,000 stars from anywhere in the table instead of all of the +# sources around a position. The sky map below assumes a search around a position, so skip it in that case. +# adql = f"SELECT TOP 10000 {', '.join(columns)} FROM {table_mer} WHERE point_like_flag = 1" +``` + +We run the query with the TAP service, which takes about a minute. + +```{code-cell} ipython3 +result = Irsa.query_tap(adql).to_table() + +# Keep only the sources with a positive total flux in the detection band, so that we can compute magnitudes +result = result[result['flux_detection_total'] > 0] +print(len(result)) +``` + +```{tip} +For a small search area, the `query_region` method of `astroquery` is a simpler alternative to writing ADQL, and it can return all of the columns, for example: + + Irsa.query_region(coordinates=coord, spatial='Cone', catalog=table_mer, radius=3 * u.arcmin, columns='*') +``` + +Let's look at the density of sources on the sky, as offsets from the center of the search. +The MER catalog is built tile by tile, and the map shows the tile structure, areas without sources (for example around bright stars), and the edge of the DR1 coverage. -Since we are just using the MER catalog alone, it does not have a column for classification. -We can use the `point_like_flag = 1` or `point_like_prob > 0.99` for stars. +```{code-cell} ipython3 +source_coords = SkyCoord(ra=result['ra'], dec=result['dec'], unit='deg') +offset_ra, offset_dec = coord.spherical_offsets_to(source_coords) +offset_ra = offset_ra.to_value(u.arcmin) +offset_dec = offset_dec.to_value(u.arcmin) + +limit = 1.05 * search_radius.to_value(u.arcmin) +fig, ax = plt.subplots(figsize=(7, 6)) +_, _, _, image = ax.hist2d(offset_ra, offset_dec, bins=np.linspace(-limit, limit, 120), cmap='Greys') +fig.colorbar(image, ax=ax, label='Number of sources per bin') +ax.add_patch(plt.Circle((0, 0), search_radius.to_value(u.arcmin), fill=False, linestyle='--', color='tab:blue')) +ax.set_xlim(limit, -limit) # RA increases to the left +ax.set_ylim(-limit, limit) +ax.set_aspect('equal') +ax.set_xlabel('RA offset (arcmin)') +ax.set_ylabel('Dec offset (arcmin)') +``` + +## 3. Clean the sample with the quality columns + +Before using the sources, we check the MER columns that identify sources whose detection or photometry may be unreliable: + +- `spurious_prob` is the probability that a source is a spurious detection, for example an artifact around a bright object. + `spurious_flag` is set for `spurious_prob > 0.5`, but this has proven too aggressive, and a threshold of 0.7 is suggested instead (see the [notes on the DR1 MER photometry](#notes-on-the-dr1-mer-photometry) below). +- `flag_vis`, `flag_y`, `flag_j`, and `flag_h` keep track of the flagged pixels (for example bad or saturated pixels) in the flag image of each band. +- `det_quality_flag` flags problems in the detection step that could corrupt the magnitude used for the star-galaxy separation. + +Nonzero flags do not necessarily mean that a source is unusable, but they are worth checking for your science case. -Set all the fluxes to be greater than 0 so the object is detected in all four Euclid MER mosaic images. + ```{code-cell} ipython3 -adql_stars = ("SELECT TOP 10000 mer.object_id, mer.ra, mer.dec, mer.flux_vis_psf, mer.fluxerr_vis_psf, " - "mer.flux_y_templfit,mer.fluxerr_y_templfit, mer.flux_j_templfit, mer.fluxerr_j_templfit, " - "mer.flux_h_templfit, mer.fluxerr_h_templfit, mer.point_like_prob, mer.extended_prob " - f"FROM {table_mer} AS mer " - "WHERE mer.flux_vis_psf > 0 " - "AND mer.flux_y_templfit > 0 " - "AND mer.flux_j_templfit > 0 " - "AND mer.flux_h_templfit > 0 " - "AND mer.point_like_flag = 1 ") +is_spurious = result['spurious_prob'] > 0.7 +print(f"spurious_prob > 0.5: {np.sum(result['spurious_prob'] > 0.5)} of {len(result)} sources") +print(f"spurious_prob > 0.7: {np.sum(is_spurious)} of {len(result)} sources") +print(f"flag_vis != 0: {np.sum(result['flag_vis'] != 0)} of {len(result)} sources") +print(f"det_quality_flag != 0: {np.sum(result['det_quality_flag'] != 0)} of {len(result)} sources") ``` -We can run the query with the TAP service, and then look at some of the results. +We remove the likely spurious sources from our sample: ```{code-cell} ipython3 -result_stars = Irsa.query_tap(adql_stars).to_table() -result_stars[:5] +clean_sources = result[~is_spurious] +print(len(clean_sources)) ``` -## 2. Make a color-magnitude diagram using the catalogs pulled from IRSA +## 4. Select the stars -- Convert from flux in uJy to magnitudes using the zero point correction -- Convert the error bars to magnitudes as well -- Plot the color-magnitude diagram +The MER catalog does not have a column for classification, but it provides `point_like_prob`, the probability that a VIS-detected source is point-like. +`point_like_flag = 1` selects sources with `point_like_prob > 0.96`, the threshold that reached a 99% star purity in simulations. +A lower `point_like_prob` threshold gives a more complete but less pure star sample. ```{code-cell} ipython3 -mag_y = -2.5 * np.log10(result_stars["flux_y_templfit"]) + 23.9 -mag_h = -2.5 * np.log10(result_stars["flux_h_templfit"]) + 23.9 +stars = clean_sources[clean_sources['point_like_flag'].filled(0) == 1] +print(len(stars)) +``` + +## 5. Look at the SEDs and VIS images of three stars -x = mag_y - mag_h # Y - H -y = mag_y +### Total fluxes in the Euclid and ground-based bands -xerr = (2.5 / np.log(10) * np.sqrt((result_stars["fluxerr_y_templfit"] / result_stars["flux_y_templfit"])**2 - + (result_stars["fluxerr_h_templfit"] / result_stars["flux_h_templfit"])**2)) -yerr = (2.5 / np.log(10) * (result_stars["fluxerr_y_templfit"] / result_stars["flux_y_templfit"])) +Besides the Euclid VIS, Y, J, and H bands, the MER catalog contains photometry measured on ground-based images in the u, g, r, i, and z bands. +Which ground-based surveys are available depends on the position on the sky, for example DECam in the south, or a combination of MegaCam, HSC, and Pan-STARRS in the north. +The catalog therefore has columns for all of the possible survey and band combinations, named `flux__ext__...`, and only some of them are filled at a given position. +Let's check which ground-based bands have measurements for our stars. -plt.errorbar(x, y, xerr=xerr, yerr=yerr, - fmt='o', markersize=1.5, ecolor='lightgrey', elinewidth=0.5, capsize=2) +```{code-cell} ipython3 +ext_flux_cols = [col for col in stars.colnames + if col.startswith('flux_') and '_ext_' in col and col.endswith('_templfit') + and not col.startswith('flux_vis_to_')] -plt.xlabel('Y-H') -plt.ylabel('Y') -# Note that these limits exclude a handful of points with large error bars. -plt.xlim(-2, 2) -plt.ylim(24, 16) -plt.title('10k Stars in MER catalog -- IRSA') +for col in ext_flux_cols: + n_measured = np.sum(~np.ma.getmaskarray(stars[col])) + print(f"{col}: {n_measured} of {len(stars)} stars") ``` +At this position, the ground-based photometry comes from DECam, in the g, r, i, and z bands. + +To get total fluxes, we follow the recipe of the [MER Photometry Cookbook](https://euclid.esac.esa.int/dr/q1/dpdd/merdpd/merphotometrycookbook.html). +The total flux of an object is measured with a Kron aperture in the detection band, which is VIS for VIS-detected objects (`vis_det = 1`), and is given in `flux_detection_total`. +The total flux in another band X, from Euclid or from the ground, is then obtained by scaling the detection-band total flux with the color measured by template fitting: + +$$F_{X,\mathrm{tot}} = \mathtt{flux\_detection\_total} \times \frac{\mathtt{flux\_X\_templfit}}{\mathtt{flux\_vis\_to\_X\_templfit}}$$ + +where `flux_vis_to_X_templfit` is the template-fitting flux measured on the VIS image degraded to the resolution of band X. +The cookbook gives an equivalent recipe with the aperture fluxes (`flux_X_nfwhm_aper / flux_vis_nfwhm_aper`). +The fluxes are in uJy, which we convert to AB magnitudes. + + + +```{code-cell} ipython3 +def total_mag(sources, band): + """Total AB magnitude of VIS-detected sources in a band, following the MER Photometry Cookbook.""" + flux = sources['flux_detection_total'] + if band != 'vis': + flux = flux * sources[f'flux_{band}_templfit'] / sources[f'flux_vis_to_{band}_templfit'] + return -2.5 * np.log10(flux) + 23.9 +``` + +### Choose three stars + +We pick three stars with positive fluxes in all bands, no flagged pixels in the Euclid bands, no detection problems, and of intermediate brightness. +To span the range of stellar colors, we take the stars closest to three points along the stellar locus in the VIS - J vs. J - H color-color diagram that we make below: a blue, an intermediate, and a red star. + +```{code-cell} ipython3 +# Approximate central wavelengths of the bands, in micron +band_wavelengths = {'g_ext_decam': 0.48, 'r_ext_decam': 0.64, 'vis': 0.72, 'i_ext_decam': 0.78, + 'z_ext_decam': 0.92, 'y': 1.08, 'j': 1.37, 'h': 1.77} + +has_all_bands = np.all([np.ma.filled(stars[f'flux_{band}_templfit'], 0) > 0 + for band in band_wavelengths if band != 'vis'], axis=0) +mag_vis = total_mag(stars, 'vis') +no_flags = ((stars['flag_vis'] == 0) & (stars['flag_y'] == 0) & (stars['flag_j'] == 0) & (stars['flag_h'] == 0) + & (stars['det_quality_flag'] == 0)) +candidates = stars[has_all_bands & no_flags & (mag_vis > 18.5) & (mag_vis < 21)] + +color_vis_j = total_mag(candidates, 'vis') - total_mag(candidates, 'j') +color_j_h = total_mag(candidates, 'j') - total_mag(candidates, 'h') +# (VIS - J, J - H) of a blue, an intermediate, and a red star on the stellar locus +locus_points = [(0.0, -0.2), (0.5, -0.03), (1.5, 0.03)] +three_stars = candidates[[np.argmin(np.hypot(color_vis_j - x, color_j_h - y)) for x, y in locus_points]] +three_stars['object_id', 'ra', 'dec', 'tileid'] +``` + +The MER `object_id` is built from the position of the source: its digits are the RA and the absolute value of the Dec, both in units of 1e-7 deg, and its sign is the sign of the Dec. +All of our sources are in the southern sky, so they have negative IDs. + +### Get VIS cutouts of the three stars + +The MER mosaics are available through IRSA's Simple Image Access (SIA) service and are hosted in the cloud (see the MER mosaics notebook for more details). +For each star, we search for the wide VIS mosaics at its position, and read a small cutout directly from the cloud, without downloading the full mosaic. +Each source in the MER catalog belongs to one tile, so we use the mosaic of the tile given by its `tileid`. + +```{code-cell} ipython3 +def get_s3_fpath(cloud_access): + cloud_info = json.loads(cloud_access) # converts str to dict + return f"{cloud_info['aws']['bucket_name']}/{cloud_info['aws']['key']}" + + +def vis_cutout(source, size=3 * u.arcsec): + position = SkyCoord(ra=source['ra'], dec=source['dec'], unit='deg') + sia_results = Irsa.query_sia(pos=(position, 1 * u.arcsec), collection='euclid_dr1_wide_DpdMerBksMosaic') + # The tile ID is the first part of the obs_id of the mosaic + is_vis_mosaic = ((sia_results['dataproduct_subtype'] == 'science') & (sia_results['energy_bandpassname'] == 'VIS') + & [obs_id.startswith(str(source['tileid'])) for obs_id in sia_results['obs_id']]) + mosaic = sia_results[is_vis_mosaic][0] + with fits.open(f"s3://{get_s3_fpath(mosaic['cloud_access'])}", fsspec_kwargs={'anon': True}) as hdu: + return Cutout2D(hdu[0].section, position=position, size=size, wcs=WCS(hdu[0].header)).data + + +cutouts = [vis_cutout(star) for star in three_stars] +``` + +### Plot the SEDs and the images + +```{code-cell} ipython3 +star_colors = ['tab:blue', 'tab:green', 'tab:red'] + +# The SEDs on the left, and the three VIS cutouts stacked in a narrow column on the right +fig = plt.figure(figsize=(12, 6)) +grid = fig.add_gridspec(3, 2, width_ratios=[4, 1]) +ax_sed = fig.add_subplot(grid[:, 0]) +for star, color in zip(three_stars, star_colors): + mags = [total_mag(star, band) for band in band_wavelengths] + ax_sed.plot(list(band_wavelengths.values()), mags, 'o-', color=color, label=f"object_id {star['object_id']}") +ax_sed.invert_yaxis() +ax_sed.set_xscale('log') +ax_sed.set_xticks(list(band_wavelengths.values()), ['g', 'r', 'VIS', 'i', 'z', 'Y', 'J', 'H']) +ax_sed.minorticks_off() +ax_sed.set_xlabel('Band (approximate central wavelength on a log scale)') +ax_sed.set_ylabel('Total AB magnitude') +ax_sed.legend() + +for i, (star, cutout, color) in enumerate(zip(three_stars, cutouts, star_colors)): + ax = fig.add_subplot(grid[i, 1]) + norm = ImageNormalize(cutout, interval=PercentileInterval(99.9), stretch=AsinhStretch()) + ax.imshow(cutout, origin='lower', cmap='gray', norm=norm) + ax.set_title(f"VIS = {total_mag(star, 'vis'):.1f}", color=color, fontsize='medium') + ax.set_xticks([]) + ax.set_yticks([]) + for spine in ax.spines.values(): + spine.set_edgecolor(color) + spine.set_linewidth(3) +fig.tight_layout() +``` + +All three stars are point sources in the VIS images, but their SEDs have very different shapes. +The bluest star is hot, with a nearly flat SED in AB magnitudes, while the cooler stars are much fainter in the blue bands than in the near-infrared. +A color-color diagram captures these differences for many stars at once. + +## 6. Make a color-color diagram of the stars + +Finally, we make a color-color diagram of all of the stars in our search area. +Some sources have no measurement in a band, for example where a tile is only partially covered (see the [notes on the DR1 MER photometry](#notes-on-the-dr1-mer-photometry) below). +These fluxes are masked in the table, and faint sources can also have negative fluxes, so we keep only the stars with positive fluxes in all of the bands we need. + +```{code-cell} ipython3 +cc_bands = ['j', 'h', 'g_ext_decam', 'r_ext_decam'] +# Masked fluxes are filled with 0, so they are removed as well +has_all_fluxes = np.all([np.ma.filled(stars[f'flux_{band}_templfit'], 0) > 0 for band in cc_bands], axis=0) +many_stars = stars[has_all_fluxes] +print(f"{len(many_stars)} of {len(stars)} stars have VIS, J, H, g, and r fluxes") + +color_x = total_mag(many_stars, 'vis') - total_mag(many_stars, 'j') +color_y = total_mag(many_stars, 'j') - total_mag(many_stars, 'h') +color_g_r = total_mag(many_stars, 'g_ext_decam') - total_mag(many_stars, 'r_ext_decam') +``` + +We plot the stars as points colored by their ground-based g - r color, add contours of the density of stars, and mark the three stars from above. + +```{code-cell} ipython3 +fig, ax = plt.subplots(figsize=(7, 6)) +image = ax.scatter(color_x, color_y, c=color_g_r, s=5, alpha=0.6, cmap='coolwarm', vmin=0, vmax=1.6, edgecolor='none') +fig.colorbar(image, ax=ax, label='g - r (DECam)') + +# Contours of the star counts in coarser bins, which smooths them +counts, xedges, yedges = np.histogram2d(color_x, color_y, + bins=[np.arange(-0.75, 2.6, 0.12), np.arange(-0.6, 0.65, 0.05)]) +ax.contour((xedges[:-1] + xedges[1:]) / 2, (yedges[:-1] + yedges[1:]) / 2, counts.T, + levels=[5, 15, 40], colors='black', linewidths=0.8) + +for star, color in zip(three_stars, star_colors): + ax.scatter(total_mag(star, 'vis') - total_mag(star, 'j'), total_mag(star, 'j') - total_mag(star, 'h'), + s=250, marker='*', color=color, edgecolor='black', linewidth=1.5, zorder=3) + +ax.set_xlim(-0.75, 2.5) +ax.set_ylim(-0.6, 0.6) +ax.set_xlabel('VIS - J') +ax.set_ylabel('J - H') +ax.set_title(f'{len(many_stars)} stars in the DR1 MER Wide Survey catalog -- IRSA') +``` + +The stars follow the stellar locus, a narrow sequence in color-color space set mainly by their temperature. +From hot to cool stars, both colors first become redder together, then J - H stays nearly constant while VIS - J keeps increasing: for the coolest stars, the M dwarfs, molecular absorption shapes the near-infrared spectrum and J - H no longer tracks temperature. +The longer wavelength baseline of VIS - J keeps separating these stars, and our three stars lie at different positions along the locus. +The ground-based g - r color, which also traces temperature, changes smoothly along the locus, from the blue, hot stars to the red, cool ones. + + + +## Notes on the DR1 MER photometry + +Kümmel et al. (2026) describe the DR1 MER photometry and its known limitations in detail. +The most important points for using the catalog are: + +* **NIR calibration offsets**: the Euclid Y, J, and H fluxes have offsets of a few to about 13 percent with respect to Gaia, depending on the photometric method. The offsets are similar in the three bands, so NIR colors are more reliable than absolute NIR magnitudes. +* **Partial coverage**: some tiles are only partially covered, at the survey borders or in individual bands, for example after solar proton events. Requiring a detection in all bands therefore removes sources in these areas. +* **Spurious sources**: `spurious_flag` is set for `spurious_prob > 0.5`, which has proven too aggressive; a threshold of 0.7 is suggested instead. The spurious probability is optimized for wide data. + +```{note} +If you have used the Euclid Quick Release 1 (Q1) MER catalog: the DR1 PSFs are no longer normalized to unit flux. +This corrects the large offsets seen in Q1 for bands with extended PSFs (such as Pan-STARRS i), but means that DR1 fluxes differ systematically from the Q1 ones, so the two releases should not be mixed. +Colors measured with aperture (A-PHOT) or template-fitting (T-PHOT) photometry are largely unaffected. +``` + + + ++++ + +## References + + + +- Euclid Collaboration: Kümmel, M., et al. 2026, *Euclid Data Release (DR1): The MERge processing*, A&A, in prep. +- Euclid Collaboration: Romelli, E., Kümmel, M., Dole, H., et al. 2026, A&A, 711, A4 (the Q1 MER processing) +- Euclid Data Release 1, 2026, [doi:10.57780/esa-d079c51](https://doi.org/10.57780/esa-d079c51) + ## About this Notebook -**Updated**: 2025-04-09 +**Updated**: 2026-10-23 **Contact:** [the IRSA Helpdesk](https://irsa.ipac.caltech.edu/docs/help_desk.html) with questions or reporting problems.