I have the below flow jmeter concurrent api request -> magento-host-server -> magento-container
- The below error occurs only during the concurrent requests from apache jmeter of testing team are hit.
- not all API throws this error (only this API throws error api/account/userprofile)
- not all the time it thows error, lets say 50% of the times this error occurs, and the error rate is around 30% and the only error is SSL handshake.
- i cannot recreate this error on my local machine, occurs during only cocurrent requests from jmeter of testing team.
- during this error, the load on server is around 20-30% of CPU usage.
- I don't see any significant error log in container apache and application log
Below is the nginx reverse proxy server configuration of the host machine where the magento application container is running
upstream backend { # added for load testing server 127.0.0.1:8484; # added for load testing } # added for load testing server { listen 443 ssl; # managed by Certbot server_name magento.acc.abcd.eu; client_max_body_size 10M; location / { proxy_pass https://127.0.0.1:8484; proxy_ssl_name magento.acc.abcd.eu; # added for load testing proxy_ssl_server_name on; # added for load testing proxy_buffers 16 16k; proxy_buffer_size 32k; proxy_set_header Host $http_host; proxy_set_header X-Real-IP $remote_addr; proxy_set_header X-Forwarded-For https; proxy_redirect off; proxy_set_header X-Forwarded-Proto $scheme; proxy_cache off; proxy_request_buffering off; proxy_ssl_protocols TLSv1 TLSv1.1 TLSv1.2; send_timeout 3600s; # added for load testing proxy_connect_timeout 3600s; # added for load testing proxy_read_timeout 3600s; # added for load testing memcached_connect_timeout 3600s; # added for load testing memcached_read_timeout 3600s; # added for load testing memcached_send_timeout 3600s; # added for load testing } access_log /var/log/nginx/magento-access.log; error_log /var/log/nginx/magento-error.log; ssl_certificate /etc/nginx/ssl/star_acc_abcd_eu_full_chain.crt; # managed by Certbot ssl_certificate_key /etc/nginx/ssl/star_acc_abcd_eu.key; # managed by Certbot ssl_session_timeout 1d; ssl_session_cache shared:MozSSL:10m; ssl_session_tickets off; # intermediate configuration ssl_protocols TLSv1.2 TLSv1.3; ssl_ciphers ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES128-GCM-SHA256:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-CHACHA20-POLY1305:ECDHE-RSA-CHACHA20-POLY1305:DHE-RSA-AES128-GCM-SHA256:DHE-RSA-AES256-GCM-SHA384; ssl_prefer_server_ciphers on; ssl_dhparam /etc/nginx/certs/abcd-dhparam.pem; } server { listen [::]:80; listen 80; server_name magento.acc.abcd.eu; return 301 https://magento.acc.abcd.eu$request_uri; }
Below is the nginx log
2023/12/18 07:15:01 [error] 1639325#0: *159305 peer closed connection in SSL handshake while SSL handshaking to upstream, client: 12.12.12.12, server: magento.acc.abcd.eu, request: "GET /nl/rest/all/V1/customers/search?searchCriteria%5BfilterGroups%5D%5B0%5D%5Bfilters%5D%5B0%5D%5Bfield%5D=email&searchCriteria%5BfilterGroups%5D%5B0%5D%5Bfilters%5D%5B0%5D%5Bvalue%5D=saritaumale%40abcdtestmail.com&searchCriteria%5BfilterGroups%5D%5B0%5D%5Bfilters%5D%5B0%5D%5BconditionType%5D=eq&searchCriteria%5BsortOrders%5D%5B0%5D%5Bfield%5D=email&searchCriteria%5BsortOrders%5D%5B0%5D%5Bdirection%5D=asc HTTP/1.1", upstream: "https://127.0.0.1:8484/nl/rest/all/V1/customers/search?searchCriteria%5BfilterGroups%5D%5B0%5D%5Bfilters%5D%5B0%5D%5Bfield%5D=email&searchCriteria%5BfilterGroups%5D%5B0%5D%5Bfilters%5D%5B0%5D%5Bvalue%5D=saritaumale%40abcdtestmail.com&searchCriteria%5BfilterGroups%5D%5B0%5D%5Bfilters%5D%5B0%5D%5BconditionType%5D=eq&searchCriteria%5BsortOrders%5D%5B0%5D%5Bfield%5D=email&searchCriteria%5BsortOrders%5D%5B0%5D%5Bdirection%5D=asc", host: "magento.acc.abcd.eu"
Below is the magento container apache2 configuration
lessthan sign VirtualHost *:443 greterthansign ServerName magento-ssl-proxy.acc.abcd.eu ServerAlias magento.acc.abcd.eu fairs-m2.acc.abcd.eu clubid-m2.acc.abcd.eu appstore-m2.acc.abcd.eu whitelabel-m2.acc.abcd.eu brochure-m2.acc.abcd.eu acsifreelife-m2.acc.abcd.eu checkoutflow-m2.acc.abcd.eu subscription-m2.acc.abcd.eu forms-m2.acc.abcd.eu Header always set Strict-Transport-Security "max-age=31536000; includeSubDomains; preload" Header always set Referrer-Policy "strict-origin-when-cross-origin" Header always set Permissions-Policy "accelerometer=(self), ambient-light-sensor=(), autoplay=(), battery=(), camera=(), cross-origin-isolated=(self), display-capture=(), document-domain=(self), encrypted-media=(self), execution-while-not-rendered=(self), execution-while-out-of-viewport=(self), fullscreen=(self), geolocation=(self), gyroscope=(), keyboard-map=(self), magnetometer=(), microphone=(), midi=(), navigation-override=(self), payment=(self), picture-in-picture=(self), publickey-credentials-get=(self), screen-wake-lock=(), sync-xhr=(self), usb=(), web-share=(), xr-spatial-tracking=(), clipboard-read=(self), clipboard-write=(self), gamepad=(), speaker-selection=(), conversion-measurement=(), focus-without-user-activation=(), hid=(), idle-detection=(), interest-cohort=(), serial=(), sync-script=(), trust-token-redemption=(), window-placement=(), vertical-scroll=()" SSLEngine On SSLCertificateFile "/usr/local/share/ca-certificates/star_acc_acsi_eu.crt" SSLCertificateKeyFile "/usr/local/share/ca-certificates/star_acc_acsi_eu.key" SSLCACertificateFile "/usr/local/share/ca-certificates/star_acc_acsi_eu.ca-bundle" ProxyPreserveHost On ProxyPass / http://127.0.0.1:80/ connectiontimeout=60 timeout=1200 Keepalive=On # added for load testing Timeout 1200 ProxyTimeout 300 # added for load testing RequestHeader set X-Forwarded-Port "443" RequestHeader set X-Forwarded-Proto "https" ErrorLog "logs/ssl-proxy-error.log" CustomLog "logs/ssl-proxy-access.log" proxy CustomLog logs/ssl_request_log "%t %h %{SSL_PROTOCOL}x %{SSL_CIPHER}x \"%r\" %b" lessthan-sign/VirtualHost greaterthan-sign
below is the container apache2 log, this log generated few seconds after the ssl handshake error occurs on host machine nginx server lever
[Tue Dec 19 06:32:47.463827 2023] [proxy_fcgi:error] [pid 268875] (70007)The timeout specified has expired: [client 127.0.0.1:40270] AH01075: Error dispatching request to : (polling) [Tue Dec 19 06:32:48.239519 2023] [proxy_fcgi:error] [pid 268508] (70007)The timeout specified has expired: [client 127.0.0.1:40670] AH01075: Error dispatching request to : (polling)
What did i try?
I tried all the options that i tagged in the below configuration with # added for load testing