I have velero installed on two different EKS clusters in the same region. All velero installation configurations are same and installed using same charts and credentials, and uses same S3 bucket with same IAM policies and role.
However, while velero on cluster-1 works fine. Velero on cluster-2 throw following error while making or restoring backups.
I0229 18:28:30.285113 1 request.go:690] Waited for 1.030005747s due to client-side throttling, not priority and fairness, request: GET:https://182.20.0.2:443/apis/admissionregistration.k8s.io/v1?timeout=32s
time="2024-02-29T18:28:30Z" level=error msg="cannot list VolumeSnapshotClass no matches for kind \"VolumeSnapshotClass\" in version \"snapshot.storage.k8s.io/v1\"" backup=velero/lokibu-9 error="no matches for kind \"VolumeSnapshotClass\" in version \"snapshot.storage.k8s.io/v1\"" logSource="internal/volume/volumes_information.go:468"
time="2024-02-29T18:28:31Z" level=error msg="Error uploading log file" backup=lokibu-9 bucket=velero-clusterx error="rpc error: code = Unknown desc = error putting object backups/lokibu-9/lokibu-9-logs.gz: operation error S3: PutObject, https response error StatusCode: 403, RequestID: *****, HostID: *****=, api error AccessDenied: Access Denied" error.file="/go/src/velero-plugin-for-aws/velero-plugin-for-aws/object_store.go:246" error.function="main.(*ObjectStore).PutObject" logSource="pkg/persistence/object_store.go:252" prefix=
time="2024-02-29T18:28:31Z" level=info msg="Initial backup processing complete, moving to Finalizing" backup=velero/lokibu-9 logSource="pkg/controller/backup_controller.go:743"
time="2024-02-29T18:28:31Z" level=error msg="backup failed" backuprequest=velero/lokibu-9 controller=backup error="rpc error: code = Unknown desc = error putting object backups/lokibu-9/velero-backup.json: operation error S3: PutObject, https response error StatusCode: 403, RequestID: *****, HostID: *****=, api error AccessDenied: Access Denied" logSource="pkg/controller/backup_controller.go:288"
time="2024-02-29T18:28:31Z" level=info msg="Updating backup's final status" backuprequest=velero/lokibu-9 controller=backup logSource="pkg/controller/backup_controller.go:307"
time="2024-02-29T18:28:54Z" level=info msg="Validating BackupStorageLocation" backup-storage-location=velero/default controller=backup-storage-location logSource="pkg/controller/backup_storage_location_controller.go:141"
time="2024-02-29T18:28:54Z" level=info msg="BackupStorageLocations is valid, marking as available" backup-storage-location=velero/default controller=backup-storage-location logSource="pkg/controller/backup_storage_location_controller.go:126"
Main concern seems to be operation error S3: PutObject, https response error StatusCode: 403. Which is confusing as the same bucket with same policy is accessible from other cluster. Also, i tried manually doing aws s3 cp loki9-logs.gz s3://velero-clusterx/ and it works fine.
So, I am not sure whats the issue here?
