Skip to content

HERMES_GR_PREPROC job fail with auto-MONARCH

@ctena, I'm trying to run an experiment, but autosubmit gets stuck at the HERMES_GR_PREPROC step.

It seems the issue is related to the temporal_coords.nc file. You can see the full error in: a980_20221215_000_HERMES_GR_PREPROC.20250507150318.err

  File "/gpfs/scratch/bsc32/bsc143869/a980/HERMESv3/sources/HERMESv3_GR/hermesv3_gr/hermes.py", line 147, in run
    self.grid = select_grid(self.comm, self.options)
  File "/gpfs/scratch/bsc32/bsc143869/a980/HERMESv3/sources/HERMESv3_GR/hermesv3_gr/modules/grids/grid.py", line 49, in select_grid
    date = HermesGr(Config()).main()
  File "/gpfs/scratch/bsc32/bsc143869/a980/HERMESv3/sources/HERMESv3_GR/hermesv3_gr/hermes.py", line 69, in __init__
    grid = RotatedGrid(arguments.domain_type, arguments.vertical_description, arguments.output_timestep_num,
  File "/gpfs/scratch/bsc32/bsc143869/a980/HERMESv3/sources/HERMESv3_GR/hermesv3_gr/modules/grids/grid_rotated.py", line 80, in __init__
    self.grid = select_grid(self.comm, self.options)
  File "/gpfs/scratch/bsc32/bsc143869/a980/HERMESv3/sources/HERMESv3_GR/hermesv3_gr/modules/grids/grid.py", line 49, in select_grid
    total_area = self.get_cell_area()
  File "/gpfs/scratch/bsc32/bsc143869/a980/HERMESv3/sources/HERMESv3_GR/hermesv3_gr/modules/grids/grid.py", line 235, in get_cell_area
    grid = RotatedGrid(arguments.domain_type, arguments.vertical_description, arguments.output_timestep_num,
  File "/gpfs/scratch/bsc32/bsc143869/a980/HERMESv3/sources/HERMESv3_GR/hermesv3_gr/modules/grids/grid_rotated.py", line 80, in __init__
    total_area = self.get_cell_area()
  File "/gpfs/scratch/bsc32/bsc143869/a980/HERMESv3/sources/HERMESv3_GR/hermesv3_gr/modules/grids/grid.py", line 235, in get_cell_area
    dataset = Dataset(self.coords_netcdf_file, mode='r')
  File "src/netCDF4/_netCDF4.pyx", line 2353, in netCDF4._netCDF4.Dataset.__init__
    dataset = Dataset(self.coords_netcdf_file, mode='r')
  File "src/netCDF4/_netCDF4.pyx", line 2353, in netCDF4._netCDF4.Dataset.__init__
  File "src/netCDF4/_netCDF4.pyx", line 1963, in netCDF4._netCDF4._ensure_nc_success
  File "src/netCDF4/_netCDF4.pyx", line 1963, in netCDF4._netCDF4._ensure_nc_success
OSError: [Errno -101] NetCDF: HDF error: b'/gpfs/scratch/bsc32/bsc143869/a980/HERMESv3/auxiliar_files/d01/temporal_coords.nc'
OSError: [Errno -101] NetCDF: HDF error: b'/gpfs/scratch/bsc32/bsc143869/a980/HERMESv3/auxiliar_files/d01/temporal_coords.nc'

The file in question is: /gpfs/scratch/bsc32/bsc143869/a980/HERMESv3/auxiliar_files/d01/temporal_coords.nc It doesn't appear to be correct, the coastlines shift when viewing the cell_area variable, which seems suspicious.

Canceling the job and rerunning it without deleting anything seems to make the trick, and the rest of the experiment runs.

FYI: @mguevara