0

I have followed step by step the instructions to deploy the MySQL operator on Kubernetes. Following the instructions from the official page. Once deployed without errors, the operator is not in ready state. I have tried it via manifests and Helm but getting the same response.

https://dev.mysql.com/doc/mysql-operator/en/mysql-operator-installation-kubectl.html

After deployed it, operator is not ready

$ kubectl get all -n mysql-operator
NAME                                  READY   STATUS    RESTARTS   AGE
pod/mysql-operator-84ffdf9dbc-m2n4c   0/1     Running   0          31s

NAME                             READY   UP-TO-DATE   AVAILABLE   AGE
deployment.apps/mysql-operator   0/1     1            0           31s

NAME                                        DESIRED   CURRENT   READY   AGE
replicaset.apps/mysql-operator-84ffdf9dbc   1         1         0       31s

I have tested it on Kubernetes versions 1.24 and 1.27, obtaining the same results.

It seems Operator tries to get a xxx-privsecrets but cannot find it

Has someone encountered the same issue and found a solution?

You're welcome!

logs

2024-05-29 06:46:52: Info: mysqlsh   Ver 8.4.0 for Linux on x86_64 - for MySQL 8.4.0 (MySQL Community Server (GPL)) - build 15032060 - commit_id 4de2d1695fdc7089e81c99dcb042953620e444fb - product_id el8-x86-64bit rpm
2024-05-29 06:46:52: Info: Using credential store helper: /usr/bin/mysql-secret-store-login-path
2024-05-29 06:46:52: Info: Loading startup files...
2024-05-29 06:46:52: Info: Loading plugins...
[2024-05-29 06:46:55,639] root                 [INFO    ] Auto-detected cluster domain: cluster.local
[2024-05-29 06:46:55,644] kopf.activities.star [INFO    ] MySQL Operator/operator.py=2.1.3 timestamp=2024-04-29T07:16:37 kopf=1.35.4 uid=2
[2024-05-29 06:46:55,659] kopf.activities.star [INFO    ] KUBERNETES_VERSION =1.24
[2024-05-29 06:46:55,659] kopf.activities.star [INFO    ] OPERATOR_VERSION   =2.1.3
[2024-05-29 06:46:55,659] kopf.activities.star [INFO    ] OPERATOR_EDITION   =community
[2024-05-29 06:46:55,659] kopf.activities.star [INFO    ] OPERATOR_EDITIONS  =['community', 'enterprise']
[2024-05-29 06:46:55,660] kopf.activities.star [INFO    ] SHELL_VERSION      =8.4.0
[2024-05-29 06:46:55,660] kopf.activities.star [INFO    ] DEFAULT_VERSION_TAG=8.4.0
[2024-05-29 06:46:55,660] kopf.activities.star [INFO    ] SIDECAR_VERSION_TAG=8.4.0-2.1.3
[2024-05-29 06:46:55,660] kopf.activities.star [INFO    ] DEFAULT_IMAGE_REPOSITORY   =container-registry.oracle.com/mysql
[2024-05-29 06:46:55,660] kopf.activities.star [INFO    ] setuptools           = 50.3.2    
[2024-05-29 06:46:55,660] kopf.activities.star [INFO    ] pip                  = 20.2.4    
[2024-05-29 06:46:55,660] kopf.activities.star [INFO    ] yarl                 = 1.9.4     
[2024-05-29 06:46:55,660] kopf.activities.star [INFO    ] websocket-client     = 1.7.0     
[2024-05-29 06:46:55,660] kopf.activities.star [INFO    ] urllib3              = 2.2.0     
[2024-05-29 06:46:55,660] kopf.activities.star [INFO    ] typing-extensions    = 4.9.0     
[2024-05-29 06:46:55,661] kopf.activities.star [INFO    ] six                  = 1.16.0    
[2024-05-29 06:46:55,661] kopf.activities.star [INFO    ] rsa                  = 4.9       
[2024-05-29 06:46:55,661] kopf.activities.star [INFO    ] requests             = 2.31.0    
[2024-05-29 06:46:55,661] kopf.activities.star [INFO    ] requests-oauthlib    = 1.3.1     
[2024-05-29 06:46:55,661] kopf.activities.star [INFO    ] PyYAML               = 6.0.1     
[2024-05-29 06:46:55,661] kopf.activities.star [INFO    ] python-json-logger   = 2.0.7     
[2024-05-29 06:46:55,661] kopf.activities.star [INFO    ] python-dateutil      = 2.8.2     
[2024-05-29 06:46:55,661] kopf.activities.star [INFO    ] pyasn1               = 0.5.1     
[2024-05-29 06:46:55,661] kopf.activities.star [INFO    ] pyasn1-modules       = 0.3.0     
[2024-05-29 06:46:55,661] kopf.activities.star [INFO    ] oauthlib             = 3.2.2     
[2024-05-29 06:46:55,661] kopf.activities.star [INFO    ] multidict            = 6.0.5     
[2024-05-29 06:46:55,661] kopf.activities.star [INFO    ] kubernetes           = 23.6.0    
[2024-05-29 06:46:55,661] kopf.activities.star [INFO    ] kopf                 = 1.35.4    
[2024-05-29 06:46:55,662] kopf.activities.star [INFO    ] iso8601              = 2.1.0     
[2024-05-29 06:46:55,662] kopf.activities.star [INFO    ] idna                 = 3.6       
[2024-05-29 06:46:55,662] kopf.activities.star [INFO    ] google-auth          = 2.27.0    
[2024-05-29 06:46:55,662] kopf.activities.star [INFO    ] frozenlist           = 1.4.1     
[2024-05-29 06:46:55,662] kopf.activities.star [INFO    ] click                = 8.1.7     
[2024-05-29 06:46:55,662] kopf.activities.star [INFO    ] charset-normalizer   = 3.3.2     
[2024-05-29 06:46:55,662] kopf.activities.star [INFO    ] certifi              = 2024.2.2  
[2024-05-29 06:46:55,662] kopf.activities.star [INFO    ] cachetools           = 5.3.2     
[2024-05-29 06:46:55,662] kopf.activities.star [INFO    ] attrs                = 23.2.0    
[2024-05-29 06:46:55,662] kopf.activities.star [INFO    ] async-timeout        = 4.0.3     
[2024-05-29 06:46:55,662] kopf.activities.star [INFO    ] aiosignal            = 1.3.1     
[2024-05-29 06:46:55,662] kopf.activities.star [INFO    ] aiohttp              = 3.8.5     
[2024-05-29 06:46:55,663] kopf.activities.star [INFO    ] pytz                 = 2024.1    
[2024-05-29 06:46:55,663] kopf.activities.star [INFO    ] pyOpenSSL            = 23.3.0    
[2024-05-29 06:46:55,663] kopf.activities.star [INFO    ] PyNaCl               = 1.4.0     
[2024-05-29 06:46:55,663] kopf.activities.star [INFO    ] pycparser            = 2.21      
[2024-05-29 06:46:55,663] kopf.activities.star [INFO    ] paramiko             = 2.11.0    
[2024-05-29 06:46:55,663] kopf.activities.star [INFO    ] oci                  = 2.122.0   
[2024-05-29 06:46:55,663] kopf.activities.star [INFO    ] cryptography         = 41.0.7    
[2024-05-29 06:46:55,663] kopf.activities.star [INFO    ] circuitbreaker       = 1.4.0     
[2024-05-29 06:46:55,663] kopf.activities.star [INFO    ] cffi                 = 1.16.0    
[2024-05-29 06:46:55,663] kopf.activities.star [INFO    ] bcrypt               = 3.2.2     
[2024-05-29 06:46:55,664] kopf.activities.star [INFO    ] antlr4-python3-runtime = 4.13.1    
[2024-05-29 06:46:55,687] kopf.activities.star [ERROR   ] Activity 'on_startup' failed with an exception. Will retry.
Traceback (most recent call last):
  File "/usr/lib/mysqlsh/python-packages/kopf/_core/actions/execution.py", line 279, in execute_handler_once
    result = await invoke_handler(
  File "/usr/lib/mysqlsh/python-packages/kopf/_core/actions/execution.py", line 374, in invoke_handler
    result = await invocation.invoke(
  File "/usr/lib/mysqlsh/python-packages/kopf/_core/actions/invocation.py", line 139, in invoke
    await asyncio.shield(future)  # slightly expensive: creates tasks
  File "/usr/lib64/python3.9/concurrent/futures/thread.py", line 58, in run
    result = self.fn(*self.args, **self.kwargs)
  File "/usr/lib/mysqlsh/python-packages/mysqloperator/controller/operator.py", line 44, in on_startup
    operator_cluster.monitor_existing_clusters(clusters, logger)
  File "/usr/lib/mysqlsh/python-packages/mysqloperator/controller/innodbcluster/operator_cluster.py", line 45, in monitor_existing_clusters
    g_group_monitor.monitor_cluster(
  File "/usr/lib/mysqlsh/python-packages/mysqloperator/controller/group_monitor.py", line 196, in monitor_cluster
    account = RetryLoop(logger).call(cluster.get_admin_account)
  File "/usr/lib/mysqlsh/python-packages/mysqloperator/controller/shellutils.py", line 93, in call
    return f(*args)
  File "/usr/lib/mysqlsh/python-packages/mysqloperator/controller/innodbcluster/cluster_api.py", line 1920, in get_admin_account
    secrets = self.get_private_secrets()
  File "/usr/lib/mysqlsh/python-packages/mysqloperator/controller/innodbcluster/cluster_api.py", line 1822, in get_private_secrets
    api_core.read_namespaced_secret(f"{self.name}-privsecrets", self.namespace))
  File "/usr/lib/mysqlsh/python-packages/kubernetes/client/api/core_v1_api.py", line 24803, in read_namespaced_secret
    return self.read_namespaced_secret_with_http_info(name, namespace, **kwargs)  # noqa: E501
  File "/usr/lib/mysqlsh/python-packages/kubernetes/client/api/core_v1_api.py", line 24890, in read_namespaced_secret_with_http_info
    return self.api_client.call_api(
  File "/usr/lib/mysqlsh/python-packages/kubernetes/client/api_client.py", line 348, in call_api
    return self.__call_api(resource_path, method,
  File "/usr/lib/mysqlsh/python-packages/kubernetes/client/api_client.py", line 180, in __call_api
    response_data = self.request(
  File "/usr/lib/mysqlsh/python-packages/kubernetes/client/api_client.py", line 373, in request
    return self.rest_client.GET(url,
  File "/usr/lib/mysqlsh/python-packages/kubernetes/client/rest.py", line 240, in GET
    return self.request("GET", url,
  File "/usr/lib/mysqlsh/python-packages/kubernetes/client/rest.py", line 234, in request
    raise ApiException(http_resp=r)
kubernetes.client.exceptions.ApiException: (404)
Reason: Not Found
HTTP response headers: HTTPHeaderDict({'Audit-Id': '5c0f3f37-3732-4581-89e2-3bca443c832d', 'Cache-Control': 'no-cache, private', 'Content-Type': 'application/json', 'X-Kubernetes-Pf-Flowschema-Uid': 'b1d36932-a72f-4dcb-8aa3-d0a7c8ffd4d2', 'X-Kubernetes-Pf-Prioritylevel-Uid': 'b6ec6262-e0cf-415e-bf76-09433fc2daa3', 'Date': 'Wed, 29 May 2024 06:46:55 GMT', 'Content-Length': '224'})
HTTP response body: {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"secrets \"mysql-cluster-privsecrets\" not found","reason":"NotFound","details":{"name":"mysql-cluster-privsecrets","kind":"secrets"},"code":404}

1 Answer 1

0

The problem was that there was an orphaned deployment from a cluster, causing the operator to search for the secret. The solution is to delete that InnoDB cluster.

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .