How to reduce the time it takes a request to pass from a ALB to the actual Fargate Server?

Question

I have a webhook endpoint where our service provider send a payload which I have to respond to within 2 seconds. I've been getting way too many timeout errors from the service provider, meaning I wasn't able to respond within 2 seconds.

I did some digging as to when the Fargate Server gets the payload vs when the ALB receives it. I went through some of the access logs from the ALB and found that it takes about a second or so to pass the payload from ALB to the fargate server.

Here's the timestamp at which the request arrived to the ALB - 15:19:20.01 and my server recieved it at - 15:19:21.69.

There's over a second of difference, I wanna know how to reduce it. One of the solution I thought of was that instead of registering my domain + the URI to the service provider to send webhook to, I set my IP + the URI so there's no need of forwarding done by ALB.

Let me know what you guys think.

EDIT - The solution I thought of was pretty stupid because fargate provides a new IP everytime a new task is deployed (as far as I know). Also the ALB forwards the request / payload to the ECS Target Group, just throwing this fact in as well.