Jobsub ID 104575.47@justin-prod-sched02.dune.hep.ac.uk
| Jobsub ID | 104575.47@justin-prod-sched02.dune.hep.ac.uk |
| Workflow ID | 531 |
| Stage ID | 1 |
| User name | timm@fnal.gov |
| HTCondor Group | group_dune.prod_mcsim |
| Requested | Processors | 1 |
| GPU | No |
| RSS bytes | 2097152000 (2000 MiB) |
| Wall seconds limit | 80000 (22 hours) |
| Submitted time | 2024-11-19 19:39:33 |
| Site | CA_SFU |
| Entry | DUNE_CA_SFU_lcg-ce3 |
| Last heartbeat | 2024-11-19 20:59:27 |
| From worker node | Hostname | cdr571.int.cedar.computecanada.ca |
| cpuinfo | Intel(R) Xeon(R) CPU E5-2683 v4 @ 2.10GHz |
| OS release | Scientific Linux release 7.9 (Nitrogen) |
| Processors | 1 |
| RSS bytes | 2097152000 (2000 MiB) |
| Wall seconds limit | 84598 (23 hours) |
| GPU | |
| Inner Apptainer? | True |
| Job state | outputting_failed |
| Allocator name | justin-allocator-int.dune.hep.ac.uk |
| Started | 2024-11-19 20:12:50 |
| Input files | protodune-sp:np04_raw_run005769_0013_dl2.root
|
| Jobscript | Exit code | 0 |
| Real time | 4m (247s) |
| CPU time | 0m (0s = 0%) |
| Max RSS bytes | 42983424 (40 MiB) |
| Outputting started | 2024-11-19 20:16:57 |
| Output files | |
| Finished | 2024-11-19 20:59:27 |
| List job events Wrapper job log |
Jobscript log (last 10,000 characters)
ilic.36@cdr571.int.cedar.computecanada.ca&tried=172.16.122.151,172.16.122.151,172.16.122.151,172.16.122.151,172.16.122.151,lcg-dunese1.sfu.computecanada.ca&triedrc=srverr,srverr,srverr,srverr,srverr,srverr&xrd.logintoken=org.dcache.door=172.16.122.151:1094, mode: 00, flags: kXR_open_read kXR_async kXR_retstat ).
[2024-11-19 12:14:56.192369 -0800][Info ][AsyncSock ] [lcg-dunese1.sfu.computecanada.ca:1094.0] TLS hand-shake done.
[2024-11-19 12:14:56.238167 -0800][Error ][XRootDTransport ] [lcg-dunese1.sfu.computecanada.ca:1094.0] Authentication with gsi failed: login failed
[2024-11-19 12:15:56.400232 -0800][Info ][XRootD ] [lcg-dunese1.sfu.computecanada.ca:1094] Retrying request: kXR_open (file: /dune/RSE/protodune-sp/3b/c6/np04_raw_run005769_0013_dl2.root?org.dcache.uuid=d3151917-821e-4a19-b5c9-652dc82c3dd9&org.dcache.xrootd.client=nilic.36@cdr571.int.cedar.computecanada.ca&tried=172.16.122.151,172.16.122.151,172.16.122.151,172.16.122.151,172.16.122.151,lcg-dunese1.sfu.computecanada.ca,lcg-dunese1.sfu.computecanada.ca&triedrc=srverr,srverr,srverr,srverr,srverr,srverr,srverr&xrd.logintoken=org.dcache.door=172.16.122.151:1094, mode: 00, flags: kXR_open_read kXR_async kXR_retstat ).
[2024-11-19 12:15:56.414352 -0800][Info ][AsyncSock ] [lcg-dunese1.sfu.computecanada.ca:1094.0] TLS hand-shake done.
[2024-11-19 12:15:56.460777 -0800][Error ][XRootDTransport ] [lcg-dunese1.sfu.computecanada.ca:1094.0] Authentication with gsi failed: login failed
[2024-11-19 12:16:57.335064 -0800][Warning][XRootD ] [lcg-dunese1.sfu.computecanada.ca:1094] Redirect limit has been reached for message kXR_open (file: /dune/RSE/protodune-sp/3b/c6/np04_raw_run005769_0013_dl2.root?org.dcache.uuid=b58ddcd3-5943-4da3-b3c9-d4869b5919d8&org.dcache.xrootd.client=nilic.36@cdr571.int.cedar.computecanada.ca&tried=172.16.122.151,172.16.122.151,172.16.122.151,172.16.122.151,172.16.122.151,lcg-dunese1.sfu.computecanada.ca,lcg-dunese1.sfu.computecanada.ca,lcg-dunese1.sfu.computecanada.ca,172.16.122.151,172.16.122.151,172.16.122.151,172.16.122.151,172.16.122.151,172.16.122.151,172.16.122.151,172.16.122.151,172.16.122.151,172.16.122.151,172.16.122.151&triedrc=srverr,srverr,srverr,srverr,srverr,srverr,srverr,srverr,srverr,srverr,srverr,srverr,srverr,srverr,srverr,srverr,srverr,srverr,srverr&xrd.logintoken=org.dcache.door=172.16.122.151:1094, mode: 00, flags: kXR_open_read kXR_async kXR_retstat ), the last known error is: [FATAL] TLS error
Run: [FATAL] Redirect limit has been reached: (source)
[2024-11-19 12:16:57.335440 -0800][Warning][XRootD ] Redirect trace-back:
[2024-11-19 12:16:57.335440 -0800][Warning][XRootD ] 0. Redirected from: root://lcg-dunese1.sfu.computecanada.ca:1094//dune/RSE/protodune-sp/3b/c6/np04_raw_run005769_0013_dl2.root to: root://172.16.122.151:33133/
[2024-11-19 12:16:57.335440 -0800][Warning][XRootD ] 1. Retrying: root://lcg-dunese1.sfu.computecanada.ca:1094//dune/RSE/protodune-sp/3b/c6/np04_raw_run005769_0013_dl2.root
[2024-11-19 12:16:57.335440 -0800][Warning][XRootD ] 2. Redirected from: root://lcg-dunese1.sfu.computecanada.ca:1094//dune/RSE/protodune-sp/3b/c6/np04_raw_run005769_0013_dl2.root to: root://172.16.122.151:33133/
[2024-11-19 12:16:57.335440 -0800][Warning][XRootD ] 3. Retrying: root://lcg-dunese1.sfu.computecanada.ca:1094//dune/RSE/protodune-sp/3b/c6/np04_raw_run005769_0013_dl2.root
[2024-11-19 12:16:57.335440 -0800][Warning][XRootD ] 4. Redirected from: root://lcg-dunese1.sfu.computecanada.ca:1094//dune/RSE/protodune-sp/3b/c6/np04_raw_run005769_0013_dl2.root to: root://172.16.122.151:33133/
[2024-11-19 12:16:57.335440 -0800][Warning][XRootD ] 5. Retrying: root://lcg-dunese1.sfu.computecanada.ca:1094//dune/RSE/protodune-sp/3b/c6/np04_raw_run005769_0013_dl2.root
[2024-11-19 12:16:57.335440 -0800][Warning][XRootD ] 6. Redirected from: root://lcg-dunese1.sfu.computecanada.ca:1094//dune/RSE/protodune-sp/3b/c6/np04_raw_run005769_0013_dl2.root to: root://172.16.122.151:33133/
[2024-11-19 12:16:57.335440 -0800][Warning][XRootD ] 7. Retrying: root://lcg-dunese1.sfu.computecanada.ca:1094//dune/RSE/protodune-sp/3b/c6/np04_raw_run005769_0013_dl2.root
[2024-11-19 12:16:57.335440 -0800][Warning][XRootD ] 8. Redirected from: root://lcg-dunese1.sfu.computecanada.ca:1094//dune/RSE/protodune-sp/3b/c6/np04_raw_run005769_0013_dl2.root to: root://172.16.122.151:33133/
[2024-11-19 12:16:57.335440 -0800][Warning][XRootD ] 9. Retrying: root://lcg-dunese1.sfu.computecanada.ca:1094//dune/RSE/protodune-sp/3b/c6/np04_raw_run005769_0013_dl2.root
[2024-11-19 12:16:57.335440 -0800][Warning][XRootD ] 10. Retrying: root://lcg-dunese1.sfu.computecanada.ca:1094//dune/RSE/protodune-sp/3b/c6/np04_raw_run005769_0013_dl2.root
[2024-11-19 12:16:57.335440 -0800][Warning][XRootD ] 11. Retrying: root://lcg-dunese1.sfu.computecanada.ca:1094//dune/RSE/protodune-sp/3b/c6/np04_raw_run005769_0013_dl2.root
[2024-11-19 12:16:57.335440 -0800][Warning][XRootD ] 12. Retrying: root://lcg-dunese1.sfu.computecanada.ca:1094//dune/RSE/protodune-sp/3b/c6/np04_raw_run005769_0013_dl2.root
[2024-11-19 12:16:57.335440 -0800][Warning][XRootD ] 13. Redirected from: root://lcg-dunese1.sfu.computecanada.ca:1094//dune/RSE/protodune-sp/3b/c6/np04_raw_run005769_0013_dl2.root to: root://172.16.122.151:33133/
[2024-11-19 12:16:57.335440 -0800][Warning][XRootD ] 14. Retrying: root://lcg-dunese1.sfu.computecanada.ca:1094//dune/RSE/protodune-sp/3b/c6/np04_raw_run005769_0013_dl2.root
[2024-11-19 12:16:57.335440 -0800][Warning][XRootD ] 15. Redirected from: root://lcg-dunese1.sfu.computecanada.ca:1094//dune/RSE/protodune-sp/3b/c6/np04_raw_run005769_0013_dl2.root to: root://172.16.122.151:33133/
[2024-11-19 12:16:57.335440 -0800][Warning][XRootD ] 16. Retrying: root://lcg-dunese1.sfu.computecanada.ca:1094//dune/RSE/protodune-sp/3b/c6/np04_raw_run005769_0013_dl2.root
[2024-11-19 12:16:57.335440 -0800][Warning][XRootD ] 17. Redirected from: root://lcg-dunese1.sfu.computecanada.ca:1094//dune/RSE/protodune-sp/3b/c6/np04_raw_run005769_0013_dl2.root to: root://172.16.122.151:33133/
[2024-11-19 12:16:57.335440 -0800][Warning][XRootD ] 18. Retrying: root://lcg-dunese1.sfu.computecanada.ca:1094//dune/RSE/protodune-sp/3b/c6/np04_raw_run005769_0013_dl2.root
[2024-11-19 12:16:57.335440 -0800][Warning][XRootD ] 19. Redirected from: root://lcg-dunese1.sfu.computecanada.ca:1094//dune/RSE/protodune-sp/3b/c6/np04_raw_run005769_0013_dl2.root to: root://172.16.122.151:33133/
[2024-11-19 12:16:57.335440 -0800][Warning][XRootD ] 20. Retrying: root://lcg-dunese1.sfu.computecanada.ca:1094//dune/RSE/protodune-sp/3b/c6/np04_raw_run005769_0013_dl2.root
[2024-11-19 12:16:57.335440 -0800][Warning][XRootD ] 21. Redirected from: root://lcg-dunese1.sfu.computecanada.ca:1094//dune/RSE/protodune-sp/3b/c6/np04_raw_run005769_0013_dl2.root to: root://172.16.122.151:33133/
[2024-11-19 12:16:57.335440 -0800][Warning][XRootD ] 22. Retrying: root://lcg-dunese1.sfu.computecanada.ca:1094//dune/RSE/protodune-sp/3b/c6/np04_raw_run005769_0013_dl2.root
[2024-11-19 12:16:57.335440 -0800][Warning][XRootD ] 23. Redirected from: root://lcg-dunese1.sfu.computecanada.ca:1094//dune/RSE/protodune-sp/3b/c6/np04_raw_run005769_0013_dl2.root to: root://172.16.122.151:33133/
[2024-11-19 12:16:57.335440 -0800][Warning][XRootD ] 24. Retrying: root://lcg-dunese1.sfu.computecanada.ca:1094//dune/RSE/protodune-sp/3b/c6/np04_raw_run005769_0013_dl2.root
[2024-11-19 12:16:57.335440 -0800][Warning][XRootD ] 25. Redirected from: root://lcg-dunese1.sfu.computecanada.ca:1094//dune/RSE/protodune-sp/3b/c6/np04_raw_run005769_0013_dl2.root to: root://172.16.122.151:33133/
[2024-11-19 12:16:57.335440 -0800][Warning][XRootD ] 26. Retrying: root://lcg-dunese1.sfu.computecanada.ca:1094//dune/RSE/protodune-sp/3b/c6/np04_raw_run005769_0013_dl2.root
[2024-11-19 12:16:57.335440 -0800][Warning][XRootD ] 27. Redirected from: root://lcg-dunese1.sfu.computecanada.ca:1094//dune/RSE/protodune-sp/3b/c6/np04_raw_run005769_0013_dl2.root to: root://172.16.122.151:33133/
[2024-11-19 12:16:57.335440 -0800][Warning][XRootD ] 28. Retrying: root://lcg-dunese1.sfu.computecanada.ca:1094//dune/RSE/protodune-sp/3b/c6/np04_raw_run005769_0013_dl2.root
[2024-11-19 12:16:57.335440 -0800][Warning][XRootD ] 29. Redirected from: root://lcg-dunese1.sfu.computecanada.ca:1094//dune/RSE/protodune-sp/3b/c6/np04_raw_run005769_0013_dl2.root to: root://172.16.122.151:33133/
[2024-11-19 12:16:57.335440 -0800][Warning][XRootD ] 30. Retrying: root://lcg-dunese1.sfu.computecanada.ca:1094//dune/RSE/protodune-sp/3b/c6/np04_raw_run005769_0013_dl2.root
[2024-11-19 12:16:57.335440 -0800][Warning][XRootD ] 31. Redirected from: root://lcg-dunese1.sfu.computecanada.ca:1094//dune/RSE/protodune-sp/3b/c6/np04_raw_run005769_0013_dl2.root to: root://172.16.122.151:33133/
[2024-11-19 12:16:57.335440 -0800][Warning][XRootD ] 32. Retrying: root://lcg-dunese1.sfu.computecanada.ca:1094//dune/RSE/protodune-sp/3b/c6/np04_raw_run005769_0013_dl2.root
[2024-11-19 12:16:57.335440 -0800][Warning][XRootD ] 33. Redirected from: root://lcg-dunese1.sfu.computecanada.ca:1094//dune/RSE/protodune-sp/3b/c6/np04_raw_run005769_0013_dl2.root to: root://172.16.122.151:33133/
[2024-11-19 12:16:57.335440 -0800][Warning][XRootD ] 34. Retrying: root://lcg-dunese1.sfu.computecanada.ca:1094//dune/RSE/protodune-sp/3b/c6/np04_raw_run005769_0013_dl2.root
sha1sum: tmp: No such file or directory
Processed protodune-sp:np04_raw_run005769_0013_dl2.root at root://lcg-dunese1.sfu.computecanada.ca:1094//dune/RSE/protodune-sp/3b/c6/np04_raw_run005769_0013_dl2.root for Workflow 531, Stage 1
