: executable file not found

[Pages:4]Orchestrator - Bug #46704 container_linux.go:349: "exec: \"stat\": executable file not found

07/24/2020 08:15 PM - Sebastian Wagner

Status:

Can't reproduce

% Done:

0%

Priority:

Normal

Assignee:

Category:

cephadm (binary)

Target version:

Source:

Q/A

Affected Versions:

Tags:

ceph-qa-suite:

Backport:

Pull request ID:

Regression:

No

Crash signature (v1):

Severity:

3 - minor

Crash signature (v2):

Reviewed:

Description

Description: rados:cephadm/smoke-roleless/{distro/ubuntu_18.04_podman start}

53591/

2020-07-24T18:11:43.836 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Jul 24 18:11:43 smithi 187 bash[12265]: cluster 2020-07-24T18:11:43.342367+0000 mon.smithi053 (mon.0) 263 : cluster [WRN]

overall HEALTH_WARN failed to probe daemons or devices; Reduced data availability: 1 pg inactive 2020-07-24T18:11:44.617 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Jul 24 18:11:44 smithi 053 bash[12319]: audit 2020-07-24T18:11:44.284616+0000 mon.smithi053 (mon.0) 264 : audit [DBG] fro m='mgr.14169 172.21.15.53:0/1343570546' entity='mgr.smithi053.erbylj' cmd=[{"prefix": "mon metadat a", "id": "smithi187"}]: dispatch 2020-07-24T18:11:44.828 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Jul 24 18:11:44 smithi 187 bash[12265]: audit 2020-07-24T18:11:44.284616+0000 mon.smithi053 (mon.0) 264 : audit [DBG] fro m='mgr.14169 172.21.15.53:0/1343570546' entity='mgr.smithi053.erbylj' cmd=[{"prefix": "mon metadat a", "id": "smithi187"}]: dispatch 2020-07-24T18:11:45.434 INFO:journalctl@ceph.mon.smithi053.smithi053.stdout:Jul 24 18:11:45 smithi 053 bash[12319]: cluster 2020-07-24T18:11:44.292523+0000 mgr.smithi053.erbylj (mgr.14169) 69 : clu ster [DBG] pgmap v39: 1 pgs: 1 unknown; 0 B data, 0 B used, 0 B / 0 B avail 2020-07-24T18:11:45.748 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Jul 24 18:11:45 smithi 187 bash[12265]: cluster 2020-07-24T18:11:44.292523+0000 mgr.smithi053.erbylj (mgr.14169) 69 : clu ster [DBG] pgmap v39: 1 pgs: 1 unknown; 0 B data, 0 B used, 0 B / 0 B avail 2020-07-24T18:11:46.700 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Jul 24 18:11:46 smithi 187 bash[12265]: cephadm 2020-07-24T18:11:45.601455+0000 mgr.smithi053.erbylj (mgr.14169) 70 : cep hadm [ERR] cephadm exited with an error code: 1, stderr:INFO:cephadm:Deploy daemon crash.smithi187

... 2020-07-24T18:11:46.701 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Jul 24 18:11:46 smithi 187 bash[12265]: INFO:cephadm:Non-zero exit code 127 from /usr/bin/podman run --rm --net=host --ip c=host -e CONTAINER_IMAGE=quay.ceph.io/ceph-ci/ceph:fa7e6da8476de2ac5aeaed8c68a1d4c3a29747ff -e NO DE_NAME=smithi187 --entrypoint stat quay.ceph.io/ceph-ci/ceph:fa7e6da8476de2ac5aeaed8c68a1d4c3a297 47ff -c %u %g /var/lib/ceph 2020-07-24T18:11:46.701 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Jul 24 18:11:46 smithi 187 bash[12265]: INFO:cephadm:stat:stderr Error: container_linux.go:349: starting container proces s caused "exec: \"stat\": executable file not found in $PATH": OCI runtime command not found error

Related issues: Related to Orchestrator - Bug #46529: cephadm: error removing storage for con...

Resolved

History #1 - 07/27/2020 03:28 PM - Sebastian Wagner - Description updated

12/15/2021

1/4

#2 - 07/27/2020 03:32 PM - Sebastian Wagner funny thing is: it was eventually deployed:

2020-07-24T18:15:41.726 INFO:teuthology.orchestra.run.smithi053.stdout:NAME

HOST

STA

TUS

REFRESHED AGE VERSION

IMAGE NAME

IMAGE ID

CONTAINER ID

2020-07-24T18:15:41.726 INFO:teuthology.orchestra.run.smithi053.stdout:alertmanager.smithi053 smithi053 run

ning (3m) 74s ago 5m 0.20.0

docker.io/prom/alertmanager:v0.20.0

0881eb8f169f b323d85a7e14

2020-07-24T18:15:41.727 INFO:teuthology.orchestra.run.smithi053.stdout:crash.smithi053

smithi053 run

ning (5m) 74s ago 5m 16.0.0-3805-gfa7e6da8476 quay.ceph.io/ceph-ci/ceph:fa7e6da8476de2ac5aeaed8c68a1

d4c3a29747ff 34453d280d95 943deeae7f0f

2020-07-24T18:15:41.727 INFO:teuthology.orchestra.run.smithi053.stdout:crash.smithi187

smithi187 run

ning (3m) 76s ago 3m 16.0.0-3805-gfa7e6da8476 quay.ceph.io/ceph-ci/ceph:fa7e6da8476de2ac5aeaed8c68a1

d4c3a29747ff 34453d280d95 e82a1d365979

2020-07-24T18:15:41.727 INFO:teuthology.orchestra.run.smithi053.stdout:grafana.smithi053

smithi053 run

ning (4m) 74s ago 4m 6.6.2

docker.io/ceph/ceph-grafana:6.6.2

a0dce381714a 6dd87560770a

2020-07-24T18:15:41.727 INFO:teuthology.orchestra.run.smithi053.stdout:mgr.smithi053.erbylj

smithi053 run

ning (6m) 74s ago 6m 16.0.0-3805-gfa7e6da8476 quay.ceph.io/ceph-ci/ceph:fa7e6da8476de2ac5aeaed8c68a1

d4c3a29747ff 34453d280d95 5e647371b21e

2020-07-24T18:15:41.727 INFO:teuthology.orchestra.run.smithi053.stdout:mgr.smithi187.sayfjr

smithi187 run

ning (3m) 76s ago 3m 16.0.0-3805-gfa7e6da8476 quay.ceph.io/ceph-ci/ceph:fa7e6da8476de2ac5aeaed8c68a1

d4c3a29747ff 34453d280d95 3a7af862b07b

2020-07-24T18:15:41.728 INFO:teuthology.orchestra.run.smithi053.stdout:mon.smithi053

smithi053 run

ning (6m) 74s ago 6m 16.0.0-3805-gfa7e6da8476 quay.ceph.io/ceph-ci/ceph:fa7e6da8476de2ac5aeaed8c68a1

d4c3a29747ff 34453d280d95 a851a0174c0a

2020-07-24T18:15:41.728 INFO:teuthology.orchestra.run.smithi053.stdout:mon.smithi187

smithi187 run

ning (4m) 76s ago 4m 16.0.0-3805-gfa7e6da8476 quay.ceph.io/ceph-ci/ceph:fa7e6da8476de2ac5aeaed8c68a1

d4c3a29747ff 34453d280d95 212157f82e4f

2020-07-24T18:15:41.728 INFO:teuthology.orchestra.run.smithi053.stdout:node-exporter.smithi053 smithi053 run

ning (4m) 74s ago 4m 0.18.1

docker.io/prom/node-exporter:v0.18.1

e5a616e4b9cf 6a9e5b90a6c8

2020-07-24T18:15:41.728 INFO:teuthology.orchestra.run.smithi053.stdout:node-exporter.smithi187 smithi187 run

ning (3m) 76s ago 3m 0.18.1

docker.io/prom/node-exporter:v0.18.1

e5a616e4b9cf 2511820d0f1d

2020-07-24T18:15:41.728 INFO:teuthology.orchestra.run.smithi053.stdout:osd.0

smithi053 run

ning (3m) 74s ago 3m 16.0.0-3805-gfa7e6da8476 quay.ceph.io/ceph-ci/ceph:fa7e6da8476de2ac5aeaed8c68a1

d4c3a29747ff 34453d280d95 4f8c603a608a

2020-07-24T18:15:41.729 INFO:teuthology.orchestra.run.smithi053.stdout:osd.1

smithi053 run

ning (3m) 74s ago 3m 16.0.0-3805-gfa7e6da8476 quay.ceph.io/ceph-ci/ceph:fa7e6da8476de2ac5aeaed8c68a1

d4c3a29747ff 34453d280d95 5120d6b4f625

2020-07-24T18:15:41.729 INFO:teuthology.orchestra.run.smithi053.stdout:osd.2

smithi053 run

ning (2m) 74s ago 2m 16.0.0-3805-gfa7e6da8476 quay.ceph.io/ceph-ci/ceph:fa7e6da8476de2ac5aeaed8c68a1

d4c3a29747ff 34453d280d95 888e18a6219b

2020-07-24T18:15:41.729 INFO:teuthology.orchestra.run.smithi053.stdout:osd.3

smithi053 run

ning (2m) 74s ago 2m 16.0.0-3805-gfa7e6da8476 quay.ceph.io/ceph-ci/ceph:fa7e6da8476de2ac5aeaed8c68a1

d4c3a29747ff 34453d280d95 cd9bd84f03f4

2020-07-24T18:15:41.729 INFO:teuthology.orchestra.run.smithi053.stdout:osd.4

smithi187 run

ning (2m) 76s ago 2m 16.0.0-3805-gfa7e6da8476 quay.ceph.io/ceph-ci/ceph:fa7e6da8476de2ac5aeaed8c68a1

d4c3a29747ff 34453d280d95 1a201c910a91

2020-07-24T18:15:41.730 INFO:teuthology.orchestra.run.smithi053.stdout:osd.5

smithi187 run

ning (112s) 76s ago 114s 16.0.0-3805-gfa7e6da8476 quay.ceph.io/ceph-ci/ceph:fa7e6da8476de2ac5aeaed8c68a1

d4c3a29747ff 34453d280d95 e352455cc168

2020-07-24T18:15:41.730 INFO:teuthology.orchestra.run.smithi053.stdout:osd.6

smithi187 run

ning (94s) 76s ago 96s 16.0.0-3805-gfa7e6da8476 quay.ceph.io/ceph-ci/ceph:fa7e6da8476de2ac5aeaed8c68a1

d4c3a29747ff 34453d280d95 5dab9bcc6712

2020-07-24T18:15:41.730 INFO:teuthology.orchestra.run.smithi053.stdout:osd.7

smithi187 run

ning (77s) 76s ago 79s 16.0.0-3805-gfa7e6da8476 quay.ceph.io/ceph-ci/ceph:fa7e6da8476de2ac5aeaed8c68a1

d4c3a29747ff 34453d280d95 b1a5a18ebc04

2020-07-24T18:15:41.730 INFO:teuthology.orchestra.run.smithi053.stdout:prometheus.smithi053

smithi053 run

ning (3m) 74s ago 4m 2.18.1

docker.io/prom/prometheus:v2.18.1

de242295e225 15f6c91c440b

2020

12/15/2021

2/4

#3 - 08/21/2020 12:12 PM - Deepika Upadhyay

2020-08-21T02:46:48.316 INFO:journalctl@ceph.mon.c.smithi040.stdout:Aug 21 02:46:48 smithi040 bash[8791]: time ="2020-08-21T02:46:48Z" level=error msg="unable to remove container 93488819ddcce8e2873eb8ab7665ce8761a15f308a 2b5c05a9d579bb4983ed38 after failing to start and attach to it" 2020-08-21T02:46:48.367 INFO:journalctl@ceph.mon.c.smithi040.stdout:Aug 21 02:46:48 smithi040 bash[8791]: Erro r: container_linux.go:345: starting container process caused "exec: \"/usr/bin/ceph-mon\": stat /usr/bin/cephmon: no such file or directory" 2020-08-21T02:46:48.368 INFO:journalctl@ceph.mon.c.smithi040.stdout:Aug 21 02:46:48 smithi040 bash[8791]: : OC I runtime error

/a/yuriw-2020-08-20_00:20:21-rados-wip-yuri7-testing-2020-08-19-2051-octopus-distro-basic-smithi/5360912/teuthology.log

#4 - 08/21/2020 03:47 PM - Sebastian Wagner - Related to Bug #46529: cephadm: error removing storage for container "...-mon": remove /var/lib/containers/storage/overlay/.../merged: device or resource busy added

#5 - 08/21/2020 03:48 PM - Sebastian Wagner Deepika Upadhyay wrote:

[...] /a/yuriw-2020-08-20_00:20:21-rados-wip-yuri7-testing-2020-08-19-2051-octopus-distro-basic-smithi/5360912/teuthology.log

Added to 5363117, 5362886, 5362916, 5362899, 5363031, 5363001

#6 - 09/02/2020 05:57 PM - Deepika Upadhyay Sebastian Wagner wrote:

Deepika Upadhyay wrote: [...] /a/yuriw-2020-08-20_00:20:21-rados-wip-yuri7-testing-2020-08-19-2051-octopus-distro-basic-smithi/5360912/teuthology.log

12/15/2021

3/4

Added to 5363117, 5362886, 5362916, 5362899, 5363031, 5363001 looks specific to: os_type: centos os_version: '7.6'

#7 - 05/27/2021 11:29 AM - Sebastian Wagner - Status changed from New to Can't reproduce

12/15/2021

4/4

Powered by TCPDF ()

 ................
................

In order to avoid copyright disputes, this page is only a partial summary.

Google Online Preview   Download