INT           Dashboard       Workflows       Jobs       AWT       Sites       Storages       Docs       Login

Jobsub ID 274230.0@justin-prod-sched02.dune.hep.ac.uk

Jobsub ID274230.0@justin-prod-sched02.dune.hep.ac.uk
Workflow TestingYes
Workflow ID1
Stage ID1
User nameamcnab@fnal.gov
RequestedProcessors1
GPUNo
RSS bytes1073741824 (1024 MiB)
Wall seconds limit3600 (1 hours)
Submitted time2025-12-10 11:57:42
SiteUS_PuertoRico
EntryCMS_T3_US_PuertoRico_UPRM
Last heartbeat2025-12-10 12:51:45
From worker nodeHostnamesandy-0-2.local
cpuinfoIntel(R) Xeon(R) CPU E5-1620 0 @ 3.60GHz
OS releaseScientific Linux release 7.9 (Nitrogen)
Processors1
RSS bytes1073741824 (1024 MiB)
Wall seconds limit171000 (47 hours)
GPU
Inner Apptainer?True
Job stateoutputting_failed
Started2025-12-10 11:59:40
Input files
JobscriptExit code0
Real time45m (2729s)
CPU time0m (28s = 1%)
Max RSS bytes49213440 (46 MiB)
Outputting started2025-12-10 12:45:11
Output files
Finished2025-12-10 12:51:45
List job events     Cached HTCondor job logs

Jobscript log (last 10,000 characters)

t recent call last):
  File "/cvmfs/dune.opensciencegrid.org/products/dune/metacat/v4_1_2/NULL/bin/metacat", line 8, in <module>
    sys.exit(main())
  File "/cvmfs/dune.opensciencegrid.org/products/dune/metacat/v4_1_2/NULL/lib/python3.9/site-packages/metacat/ui/metacat_ui.py", line 202, in main
    cli.run(sys.argv, argv0="metacat")
  File "/cvmfs/dune.opensciencegrid.org/products/dune/metacat/v4_1_2/NULL/lib/python3.9/site-packages/metacat/ui/cli/cli.py", line 216, in run
    self._run(command, context, argv, usage_on_error)
  File "/cvmfs/dune.opensciencegrid.org/products/dune/metacat/v4_1_2/NULL/lib/python3.9/site-packages/metacat/ui/cli/cli.py", line 211, in _run
    return interp._run(pre_command + word, context, rest, usage_on_error = usage_on_error)
  File "/cvmfs/dune.opensciencegrid.org/products/dune/metacat/v4_1_2/NULL/lib/python3.9/site-packages/metacat/ui/cli/cli.py", line 211, in _run
    return interp._run(pre_command + word, context, rest, usage_on_error = usage_on_error)
  File "/cvmfs/dune.opensciencegrid.org/products/dune/metacat/v4_1_2/NULL/lib/python3.9/site-packages/metacat/ui/cli/cli.py", line 112, in _run
    return self(command, context, opts, args)
  File "/cvmfs/dune.opensciencegrid.org/products/dune/metacat/v4_1_2/NULL/lib/python3.9/site-packages/metacat/ui/metacat_file.py", line 200, in __call__
    response = list(client.declare_files(f"{dataset_namespace}:{dataset_name}", 
  File "/cvmfs/dune.opensciencegrid.org/products/dune/metacat/v4_1_2/NULL/lib/python3.9/site-packages/metacat/webapi/webapi.py", line 825, in declare_files
    out = self.post_json(url, lst)
  File "/cvmfs/dune.opensciencegrid.org/products/dune/metacat/v4_1_2/NULL/lib/python3.9/site-packages/metacat/webapi/webapi.py", line 215, in post_json
    response = self.send_request("post", uri_suffix, data=data, headers=headers, stream=True)
  File "/cvmfs/dune.opensciencegrid.org/products/dune/metacat/v4_1_2/NULL/lib/python3.9/site-packages/metacat/webapi/webapi.py", line 154, in send_request
    self.LastResponse = response = self.retry_request(method, url, headers=headers, **args)
  File "/cvmfs/dune.opensciencegrid.org/products/dune/metacat/v4_1_2/NULL/lib/python3.9/site-packages/metacat/webapi/webapi.py", line 133, in retry_request
    response = requests.post(url, timeout=self.Timeout, **args)
  File "/cvmfs/dune.opensciencegrid.org/products/dune/python_requests/v2_25_0/NULL/lib/python3/site-packages/requests/api.py", line 119, in post
    return request('post', url, data=data, json=json, **kwargs)
  File "/cvmfs/dune.opensciencegrid.org/products/dune/python_requests/v2_25_0/NULL/lib/python3/site-packages/requests/api.py", line 61, in request
    return session.request(method=method, url=url, **kwargs)
  File "/cvmfs/dune.opensciencegrid.org/products/dune/python_requests/v2_25_0/NULL/lib/python3/site-packages/requests/sessions.py", line 542, in request
    resp = self.send(prep, **send_kwargs)
  File "/cvmfs/dune.opensciencegrid.org/products/dune/python_requests/v2_25_0/NULL/lib/python3/site-packages/requests/sessions.py", line 655, in send
    r = adapter.send(request, **kwargs)
  File "/cvmfs/dune.opensciencegrid.org/products/dune/python_requests/v2_25_0/NULL/lib/python3/site-packages/requests/adapters.py", line 516, in send
    raise ConnectionError(e, request=request)
requests.exceptions.ConnectionError: HTTPSConnectionPool(host='metacat.fnal.gov', port=9443): Max retries exceeded with url: /dune_meta_prod/app/data/declare_files?dataset=dune:all (Caused by NewConnectionError('<urllib3.connection.HTTPSConnection object at 0x2b1fc8551af0>: Failed to establish a new connection: [Errno 110] Connection timed out'))
metacat file declare returns 1
GFAL_CONFIG_DIR:    GFAL_PLUGIN_DIR: 
justin-rucio-upload attempt 1
DEBUG:root:Num. of files that upload client is processing: 1
DEBUG:dogpile.cache.region:No value present for key: "host_to_choose_choice['https://dune-rucio.fnal.gov']"
DEBUG:dogpile.lock:NeedRegenerationException
DEBUG:dogpile.lock:no value, waiting for create lock
DEBUG:dogpile.lock:value creation lock <dogpile.cache.region.CacheRegion._LockWrapper object at 0x2b5b2adffa30> acquired
DEBUG:dogpile.cache.region:No value present for key: "host_to_choose_choice['https://dune-rucio.fnal.gov']"
DEBUG:dogpile.lock:Calling creation function for not-yet-present value
DEBUG:dogpile.cache.region:Cache value generated in 0.000 seconds for key(s): "host_to_choose_choice['https://dune-rucio.fnal.gov']"
DEBUG:dogpile.lock:Released creation lock
DEBUG:urllib3.connectionpool:Starting new HTTPS connection (1): dune-rucio.fnal.gov:443
DEBUG:urllib3.connectionpool:https://dune-rucio.fnal.gov:443 "GET /rses/?expression=T3_US_NERSC HTTP/1.1" 200 None
DEBUG:urllib3.connectionpool:Starting new HTTPS connection (1): dune-rucio.fnal.gov:443
DEBUG:urllib3.connectionpool:https://dune-rucio.fnal.gov:443 "GET /rses/T3_US_NERSC HTTP/1.1" 200 1240
DEBUG:root:Input validation done.
INFO:root:Preparing upload for file awt-1765367983-LVjlJ5d3Rv
DEBUG:urllib3.connectionpool:https://dune-rucio.fnal.gov:443 "GET /rses/T3_US_NERSC/attr/ HTTP/1.1" 200 139
DEBUG:root:wan domain is used for the upload
DEBUG:root:Registering file
DEBUG:urllib3.connectionpool:https://dune-rucio.fnal.gov:443 "GET /accounts/dunepro/scopes/ HTTP/1.1" 200 870
DEBUG:root:Trying to create dataset: testpro:awt-uploads-202549
DEBUG:urllib3.connectionpool:https://dune-rucio.fnal.gov:443 "POST /dids/testpro/awt-uploads-202549 HTTP/1.1" 500 323
--- Upload try 1/1
--- Rucio upload 1/1 fails: An unknown exception occurred.
Details: no error information passed (http status code: 500)
--- Exit with 99
'justin-rucio-upload --rse T3_US_NERSC --protocol davs --scope testpro --dataset awt-uploads-202549 awt-1765367983-LVjlJ5d3Rv --timeout 1200' returns 99


subject   : /C=UK/O=eScience/OU=Manchester/L=HEP/CN=justin-jobs-production.dune.hep.ac.uk/CN=378614481/CN=176536798044
issuer    : /C=UK/O=eScience/OU=Manchester/L=HEP/CN=justin-jobs-production.dune.hep.ac.uk/CN=378614481
identity  : /C=UK/O=eScience/OU=Manchester/L=HEP/CN=justin-jobs-production.dune.hep.ac.uk/CN=378614481
type      : RFC compliant proxy
strength  : 2048 bits
path      : /home/awt-proxy.pem
timeleft  : 167:14:30
key usage : Digital Signature, Key Encipherment, Key Agreement
=== VO dune extension information ===
VO        : dune
subject   : /C=UK/O=eScience/OU=Manchester/L=HEP/CN=justin-jobs-production.dune.hep.ac.uk
issuer    : /DC=org/DC=incommon/C=US/ST=Illinois/O=Fermi Research Alliance/CN=voms2.fnal.gov
attribute : /dune/Role=Production/Capability=NULL
attribute : /dune/Role=NULL/Capability=NULL
timeleft  : 158:35:52
uri       : voms2.fnal.gov:15042

===== Results =====

Download/upload commands:
xrdcp --force --nopbar --verbose $read_pfn downloaded.txt
echo '{"namespace":"testpro","name":"FILENAME","size":0}' >tmp.json
metacat file declare --json -f tmp.json "dune:all"
justin-rucio-upload --rse $rse_name --protocol $write_protocol --scope testpro --dataset awt-uploads-202549 --timeout 1200 FILENAME
Use the wrapper job link on the page for the job on the justIN Dashboard to find the full log file, with errors from these commands

Each line: $JUSTIN_SITE_NAME $rse_name $download_retval $upload_retval $read_pfn $write_protocol
==awt== US_PuertoRico DUNE_CA_SFU 0 99 root://lcg-dunese1.sfu.computecanada.ca:1094//dune/RSE/testpro/bb/7f/awt-download-2023-03-07-01.txt davs
==awt== US_PuertoRico DUNE_CERN_EOS 0 99 root://eospublic.cern.ch:1094//eos/experiment/neutplatform/protodune/dune/testpro/bb/7f/awt-download-2023-03-07-01.txt davs
==awt== US_PuertoRico DUNE_ES_PIC 0 99 root://xrootd.pic.es:1094/pnfs/pic.es/data/dune/RSE/testpro/bb/7f/awt-download-2023-03-07-01.txt davs
==awt== US_PuertoRico DUNE_FR_CCIN2P3_DISK 0 99 root://ccxrootdegee.in2p3.fr:1094/pnfs/in2p3.fr/data/dune/disk/testpro/bb/7f/awt-download-2023-03-07-01.txt davs
==awt== US_PuertoRico DUNE_IT_INFN_CNAF 0 99 root://xrootd-archive.cr.cnaf.infn.it:1096//dune/testpro/bb/7f/awt-download-2023-03-07-01.txt davs
==awt== US_PuertoRico DUNE_UK_GLASGOW 0 99 root://cephc02.gla.scotgrid.ac.uk:1094//cephfs/dune/RSE/testpro/bb/7f/awt-download-2023-03-07-01.txt davs
==awt== US_PuertoRico DUNE_UK_LANCASTER_CEPH 0 0 root://xgate.hec.lancs.ac.uk:1094//cephfs/grid/dune/testpro/bb/7f/awt-download-2023-03-07-01.txt davs
==awt== US_PuertoRico DUNE_UK_MANCHESTER_CEPH 0 0 root://meitner.tier2.hep.manchester.ac.uk:1094//cephfs/experiments/dune/RSE/testpro/bb/7f/awt-download-2023-03-07-01.txt davs
==awt== US_PuertoRico DUNE_US_BNL_SDCC 0 0 root://dcdndoor.sdcc.bnl.gov:1094//pnfs/sdcc.bnl.gov/data/dune/RSE/testpro/bb/7f/awt-download-2023-03-07-01.txt davs
==awt== US_PuertoRico DUNE_US_FNAL_DISK_STAGE 0 0 root://fndca1.fnal.gov:1094/pnfs/fnal.gov/usr/dune/persistent/staging/testpro/bb/7f/awt-download-2023-03-07-01.txt davs
==awt== US_PuertoRico FNAL_DCACHE 0 99 root://fndca1.fnal.gov:1094/pnfs/fnal.gov/usr/dune/tape_backed/dunepro//other/awt-staging/awt-download-2023-03-07-01.txt_1749841165 davs
==awt== US_PuertoRico NIKHEF 0 0 root://dune.dcache.nikhef.nl:1094/pnfs/nikhef.nl/data/dune/generic/rucio/testpro/bb/7f/awt-download-2023-03-07-01.txt davs
==awt== US_PuertoRico PRAGUE 0 0 root://se1.farm.particle.cz:1094//dune/RSE/testpro/bb/7f/awt-download-2023-03-07-01.txt davs
==awt== US_PuertoRico QMUL 0 0 root://xrootd1.esc.qmul.ac.uk:1094//dune/RSE/testpro/bb/7f/awt-download-2023-03-07-01.txt davs
==awt== US_PuertoRico RAL-PP 0 0 root://mover.pp.rl.ac.uk:1094/pnfs/pp.rl.ac.uk/data/dune/testpro/bb/7f/awt-download-2023-03-07-01.txt davs
==awt== US_PuertoRico RAL_ECHO 0 99 root://xrootd.echo.stfc.ac.uk:1094/dune:/protodune/RSE/testpro/bb/7f/awt-download-2023-03-07-01.txt davs
==awt== US_PuertoRico SURFSARA 0 99 root://otter12.grid.surfsara.nl:21094/pnfs/grid.sara.nl/data/dune/disk/RSE/testpro/bb/7f/awt-download-2023-03-07-01.txt davs
==awt== US_PuertoRico T3_US_NERSC 0 99 root://dtn14.nersc.gov:1094//global/cfs/cdirs/m3249/dune/RSE/testpro/bb/7f/awt-download-2023-03-07-01.txt davs
justIN time: 2025-12-19 01:32:02 UTC       justIN version: 01.06.rc4