INT           Dashboard       Workflows       Jobs       AWT       Sites       Storages       Docs       Login

Jobsub ID 458509.0@justin-prod-sched01.dune.hep.ac.uk

Jobsub ID458509.0@justin-prod-sched01.dune.hep.ac.uk
Workflow TestingYes
Workflow ID1
Stage ID1
User nameamcnab@fnal.gov
RequestedProcessors1
GPUNo
RSS bytes1073741824 (1024 MiB)
Wall seconds limit3600 (1 hours)
Submitted time2025-11-13 22:59:55
SiteNL_NIKHEF
EntryVIRGO_NL_NIKHEF_juk_multicore
Last heartbeat2025-11-14 00:13:51
From worker nodeHostnamewn-snel-032.farm.nikhef.nl
cpuinfoAMD EPYC 7H12 64-Core Processor
OS releaseScientific Linux release 7.9 (Nitrogen)
Processors1
RSS bytes1073741824 (1024 MiB)
Wall seconds limit129600 (36 hours)
GPU
Inner Apptainer?True
Job statestalled
Started2025-11-13 23:01:18
Input files
Outputting started2025-11-14 00:13:51
Output files
Finished2025-11-14 00:21:16
List job events     (HTCondor job logs unavailable)

Jobscript log (last 10,000 characters)

oRename: checking if file exists davs://dtn14.nersc.gov:1094/global/cfs/cdirs/m3249/dune/RSE/testpro/10/37/awt-1763074884-J8PXSD6Dne
DEBUG:root:put: Attempt 1
DEBUG:root:gfal.NoRename: uploading file from awt-1763074884-J8PXSD6Dne to davs://dtn14.nersc.gov:1094/global/cfs/cdirs/m3249/dune/RSE/testpro/10/37/awt-1763074884-J8PXSD6Dne
INFO:root:Successful upload of temporary file. davs://dtn14.nersc.gov:1094/global/cfs/cdirs/m3249/dune/RSE/testpro/10/37/awt-1763074884-J8PXSD6Dne
DEBUG:root:skip_upload_stat=False
DEBUG:root:stat: pfn=davs://dtn14.nersc.gov:1094/global/cfs/cdirs/m3249/dune/RSE/testpro/10/37/awt-1763074884-J8PXSD6Dne
DEBUG:root:gfal.NoRename: getting stats of file davs://dtn14.nersc.gov:1094/global/cfs/cdirs/m3249/dune/RSE/testpro/10/37/awt-1763074884-J8PXSD6Dne
DEBUG:root:Filesize: Expected=26 Found=26
DEBUG:root:Checksum: Expected=5d9e06cf Found=5d9e06cf
DEBUG:root:gfal.NoRename: closing protocol connection
DEBUG:root:Upload done.
INFO:root:Successfully uploaded file awt-1763074884-J8PXSD6Dne
DEBUG:urllib3.connectionpool:Starting new HTTPS connection (1): dune-rucio.fnal.gov:443
/cvmfs/dune.opensciencegrid.org/products/dune/rucio/v38_1_0/NULL/lib/python3.9/site-packages/urllib3/connectionpool.py:1061: InsecureRequestWarning: Unverified HTTPS request is being made to host 'dune-rucio.fnal.gov'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings
  warnings.warn(
DEBUG:urllib3.connectionpool:https://dune-rucio.fnal.gov:443 "POST /traces/ HTTP/1.1" 404 207
DEBUG:urllib3.connectionpool:Resetting dropped connection: dune-rucio.fnal.gov
DEBUG:urllib3.connectionpool:https://dune-rucio.fnal.gov:443 "PUT /replicas HTTP/1.1" 200 0
DEBUG:urllib3.connectionpool:https://dune-rucio.fnal.gov:443 "POST /dids/testpro/awt-uploads-202545/dids HTTP/1.1" 201 7
DEBUG:urllib3.connectionpool:Starting new HTTPS connection (1): dune-rucio.fnal.gov:443
DEBUG:urllib3.connectionpool:https://dune-rucio.fnal.gov:443 "POST /replicas/list HTTP/1.1" 200 None
DEBUG:urllib3.connectionpool:Starting new HTTPS connection (1): dune-rucio.fnal.gov:443
DEBUG:urllib3.connectionpool:https://dune-rucio.fnal.gov:443 "GET /dids/testpro/awt-uploads-202545/files HTTP/1.1" 200 None
--- Upload try 1/1
--- Rucio upload 1/1 returns 0
--- Replica check try 1/1
--- Dataset awt-uploads-202545 check try 1/1
Traceback (most recent call last):
  File "/cvmfs/dune.opensciencegrid.org/products/dune/rucio/v38_1_0/NULL/lib/python3.9/site-packages/urllib3/response.py", line 761, in _update_chunk_length
    self.chunk_left = int(line, 16)
ValueError: invalid literal for int() with base 16: b''

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/cvmfs/dune.opensciencegrid.org/products/dune/rucio/v38_1_0/NULL/lib/python3.9/site-packages/urllib3/response.py", line 444, in _error_catcher
    yield
  File "/cvmfs/dune.opensciencegrid.org/products/dune/rucio/v38_1_0/NULL/lib/python3.9/site-packages/urllib3/response.py", line 828, in read_chunked
    self._update_chunk_length()
  File "/cvmfs/dune.opensciencegrid.org/products/dune/rucio/v38_1_0/NULL/lib/python3.9/site-packages/urllib3/response.py", line 765, in _update_chunk_length
    raise InvalidChunkLength(self, line)
urllib3.exceptions.InvalidChunkLength: InvalidChunkLength(got length b'', 0 bytes read)

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/cvmfs/dune.opensciencegrid.org/products/dune/python_requests/v2_25_0/NULL/lib/python3/site-packages/requests/models.py", line 753, in generate
    for chunk in self.raw.stream(chunk_size, decode_content=True):
  File "/cvmfs/dune.opensciencegrid.org/products/dune/rucio/v38_1_0/NULL/lib/python3.9/site-packages/urllib3/response.py", line 624, in stream
    for line in self.read_chunked(amt, decode_content=decode_content):
  File "/cvmfs/dune.opensciencegrid.org/products/dune/rucio/v38_1_0/NULL/lib/python3.9/site-packages/urllib3/response.py", line 857, in read_chunked
    self._original_response.close()
  File "/cvmfs/larsoft.opensciencegrid.org/products/python/v3_9_13/Linux64bit+3.10-2.17/lib/python3.9/contextlib.py", line 137, in __exit__
    self.gen.throw(typ, value, traceback)
  File "/cvmfs/dune.opensciencegrid.org/products/dune/rucio/v38_1_0/NULL/lib/python3.9/site-packages/urllib3/response.py", line 461, in _error_catcher
    raise ProtocolError("Connection broken: %r" % e, e)
urllib3.exceptions.ProtocolError: ("Connection broken: InvalidChunkLength(got length b'', 0 bytes read)", InvalidChunkLength(got length b'', 0 bytes read))

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/cvmfs/dune.opensciencegrid.org/products/dune/justin/01.05.02/NULL/bin/justin-rucio-upload", line 215, in <module>
    for file in filesGen:
  File "/cvmfs/dune.opensciencegrid.org/products/dune/rucio/v38_1_0/NULL/lib/python3.9/site-packages/rucio/client/baseclient.py", line 418, in _load_json_data
    for line in response.iter_lines():
  File "/cvmfs/dune.opensciencegrid.org/products/dune/python_requests/v2_25_0/NULL/lib/python3/site-packages/requests/models.py", line 797, in iter_lines
    for chunk in self.iter_content(chunk_size=chunk_size, decode_unicode=decode_unicode):
  File "/cvmfs/dune.opensciencegrid.org/products/dune/python_requests/v2_25_0/NULL/lib/python3/site-packages/requests/models.py", line 756, in generate
    raise ChunkedEncodingError(e)
requests.exceptions.ChunkedEncodingError: ("Connection broken: InvalidChunkLength(got length b'', 0 bytes read)", InvalidChunkLength(got length b'', 0 bytes read))
'justin-rucio-upload --rse T3_US_NERSC --protocol davs --scope testpro --dataset awt-uploads-202545 awt-1763074884-J8PXSD6Dne --timeout 1200' returns 1


subject   : /C=UK/O=eScience/OU=Manchester/L=HEP/CN=justin-jobs-production.dune.hep.ac.uk/CN=3855747338/CN=176307487853
issuer    : /C=UK/O=eScience/OU=Manchester/L=HEP/CN=justin-jobs-production.dune.hep.ac.uk/CN=3855747338
identity  : /C=UK/O=eScience/OU=Manchester/L=HEP/CN=justin-jobs-production.dune.hep.ac.uk/CN=3855747338
type      : RFC compliant proxy
strength  : 2048 bits
path      : /home/awt-proxy.pem
timeleft  : 167:12:28
key usage : Digital Signature, Key Encipherment, Key Agreement
=== VO dune extension information ===
VO        : dune
subject   : /C=UK/O=eScience/OU=Manchester/L=HEP/CN=justin-jobs-production.dune.hep.ac.uk
issuer    : /DC=org/DC=incommon/C=US/ST=Illinois/O=Fermi Research Alliance/CN=voms1.fnal.gov
attribute : /dune/Role=Production/Capability=NULL
attribute : /dune/Role=NULL/Capability=NULL
timeleft  : 147:38:12
uri       : voms1.fnal.gov:15042

===== Results =====

Download/upload commands:
xrdcp --force --nopbar --verbose $read_pfn downloaded.txt
echo '{"namespace":"testpro","name":"FILENAME","size":0}' >tmp.json
metacat file declare --json -f tmp.json "dune:all"
justin-rucio-upload --rse $rse_name --protocol $write_protocol --scope testpro --dataset awt-uploads-202545 --timeout 1200 FILENAME
Use the wrapper job link on the page for the job on the justIN Dashboard to find the full log file, with errors from these commands

Each line: $JUSTIN_SITE_NAME $rse_name $download_retval $upload_retval $read_pfn $write_protocol
==awt== NL_NIKHEF DUNE_CA_SFU 0 0 root://lcg-dunese1.sfu.computecanada.ca:1094//dune/RSE/testpro/bb/7f/awt-download-2023-03-07-01.txt davs
==awt== NL_NIKHEF DUNE_CERN_EOS 0 0 root://eospublic.cern.ch:1094//eos/experiment/neutplatform/protodune/dune/testpro/bb/7f/awt-download-2023-03-07-01.txt davs
==awt== NL_NIKHEF DUNE_ES_PIC 0 0 root://xrootd.pic.es:1094/pnfs/pic.es/data/dune/RSE/testpro/bb/7f/awt-download-2023-03-07-01.txt davs
==awt== NL_NIKHEF DUNE_FR_CCIN2P3_DISK 0 0 root://ccxrootdegee.in2p3.fr:1094/pnfs/in2p3.fr/data/dune/disk/testpro/bb/7f/awt-download-2023-03-07-01.txt davs
==awt== NL_NIKHEF DUNE_IT_INFN_CNAF 0 0 root://xrootd-archive.cr.cnaf.infn.it:1096//dune/testpro/bb/7f/awt-download-2023-03-07-01.txt davs
==awt== NL_NIKHEF DUNE_UK_GLASGOW 0 0 root://cephc02.gla.scotgrid.ac.uk:1094//cephfs/dune/RSE/testpro/bb/7f/awt-download-2023-03-07-01.txt davs
==awt== NL_NIKHEF DUNE_UK_LANCASTER_CEPH 0 0 root://xgate.hec.lancs.ac.uk:1094//cephfs/grid/dune/testpro/bb/7f/awt-download-2023-03-07-01.txt davs
==awt== NL_NIKHEF DUNE_UK_MANCHESTER_CEPH 0 0 root://meitner.tier2.hep.manchester.ac.uk:1094//cephfs/experiments/dune/RSE/testpro/bb/7f/awt-download-2023-03-07-01.txt davs
==awt== NL_NIKHEF DUNE_US_BNL_SDCC 0 0 root://dcdndoor.sdcc.bnl.gov:1094//pnfs/sdcc.bnl.gov/data/dune/RSE/testpro/bb/7f/awt-download-2023-03-07-01.txt davs
==awt== NL_NIKHEF DUNE_US_FNAL_DISK_STAGE 0 0 root://fndca1.fnal.gov:1094/pnfs/fnal.gov/usr/dune/persistent/staging/testpro/bb/7f/awt-download-2023-03-07-01.txt davs
==awt== NL_NIKHEF FNAL_DCACHE 0 99 root://fndca1.fnal.gov:1094/pnfs/fnal.gov/usr/dune/tape_backed/dunepro//other/awt-staging/awt-download-2023-03-07-01.txt_1749841165 davs
==awt== NL_NIKHEF NIKHEF 0 0 root://dune.dcache.nikhef.nl:1094/pnfs/nikhef.nl/data/dune/generic/rucio/testpro/bb/7f/awt-download-2023-03-07-01.txt davs
==awt== NL_NIKHEF PRAGUE 0 0 root://se1.farm.particle.cz:1094//dune/RSE/testpro/bb/7f/awt-download-2023-03-07-01.txt davs
==awt== NL_NIKHEF QMUL 0 99 root://xrootd1.esc.qmul.ac.uk:1094//dune/RSE/testpro/bb/7f/awt-download-2023-03-07-01.txt davs
==awt== NL_NIKHEF RAL-PP 0 0 root://mover.pp.rl.ac.uk:1094/pnfs/pp.rl.ac.uk/data/dune/testpro/bb/7f/awt-download-2023-03-07-01.txt davs
==awt== NL_NIKHEF RAL_ECHO 0 0 root://xrootd.echo.stfc.ac.uk:1094/dune:/protodune/RSE/testpro/bb/7f/awt-download-2023-03-07-01.txt davs
==awt== NL_NIKHEF SURFSARA 0 0 root://otter12.grid.surfsara.nl:21094/pnfs/grid.sara.nl/data/dune/disk/RSE/testpro/bb/7f/awt-download-2023-03-07-01.txt davs
==awt== NL_NIKHEF T3_US_NERSC 0 1 root://dtn14.nersc.gov:1094//global/cfs/cdirs/m3249/dune/RSE/testpro/bb/7f/awt-download-2023-03-07-01.txt davs
justIN time: 2025-12-19 01:33:36 UTC       justIN version: 01.06.rc4