Version 10 Webhook failures

Collapse
X
 
  • Time
  • Show
Clear All
new posts
  • Bizi Office
    Junior Member
    • Jul 2026
    • 5

    #1

    Version 10 Webhook failures

    We upgraded our instance to version 10 and our webhooks now fail to send. We logged into a 9.3.9 version and input the same api endpoint and it functions as expected. The new version fails and has the following in its output log. This endpoint has not changed and can be tested using curl inside the EspoCRM container and responds with 200. Has something changed? (yes the endpoint was replaced with **** in my post. Not actual url)

    [2026-07-06 05:12:03] ERROR: (500) Webhook Queue: Webhook '6a40575a0898553e4' sending failed.
    [object] (Espo\Core\Exceptions\Error(code: 500): Connect error. at /var/www/html/application/Espo/Core/Webhook/Sender.php:95)
    [previous exception] [object] (Espo\Core\HttpClient\Exceptions\ConnectException( code: 0): at /var/www/html/application/Espo/Core/HttpClient/Exceptions/ConnectException.php:44)
    [previous exception] [object] (GuzzleHttp\Exception\ConnectException(code: 0): cURL error 7: Failed to connect to api-************.bizioffice.com port 443 after 0 ms: Could not connect to server (see https://curl.se/libcurl/c/libcurl-errors.html) for https://api-**********.bizioffice.co...f-560e57538710 at /var/www/html/vendor/guzzlehttp/guzzle/src/Handler/CurlFactory.php:624)
    Last edited by Bizi Office; Yesterday, 06:01 AM.
  • yuri
    EspoCRM product developer
    • Mar 2014
    • 9927

    #2
    We switched from using CURL library directly to the Guzzle library.

    Comment


    • Bizi Office
      Bizi Office commented
      Editing a comment
      So is something needed from our part to correct? I just tried to rebuild webhook and get same failures
      Last edited by Bizi Office; Yesterday, 06:27 AM.

    • yuri
      yuri commented
      Editing a comment
      It's supposed to work. I don't know what's the reason yet. Any help with investigation appreciated.
  • yuri
    EspoCRM product developer
    • Mar 2014
    • 9927

    #3
    Just tested, it worked fine for me. I'm not yet sure what it could be.

    Comment


    • yuri
      yuri commented
      Editing a comment
      We did more testing on our cloud instance, tested integration with Zapier (it uses our webhooks). Worked fine.
  • Bizi Office
    Junior Member
    • Jul 2026
    • 5

    #4
    Update on debugging Webhook Queue cURL error 7 (0 ms) failures:

    I ran isolated tests inside the EspoCRM v10 Docker container (root@8b07bd21d0c9), and isolated the root cause to how Espo\Core\HttpClient handles dual-stack (IPv4/IPv6) resolution compared to raw Guzzle:

    1. Raw Guzzle works perfectly inside the container: Running a standalone script using \GuzzleHttp\Client()->request("POST", "https://api-********.bizioffice.com/...") returns Success: 200. Standard Guzzle connects without issue.

    2. The container networking does not support IPv6: Running curl -6 -Iv https://api-***********.bizioffice.com instantly fails with: Immediate connect fail for 2606:4700:3035::6815:5856: Network is unreachable curl: (7) Failed to connect... Could not connect to server (Standard curl without -6 succeeds immediately by falling back to IPv4).

    Conclusion / Bug Report: Because raw Guzzle succeeds but Espo v10's Webhook Queue fails instantly (0 ms), the updated v10 Espo\Core\HttpClient layer (likely the SSRF IP validation/pre-resolution logic) appears to be binding or resolving the endpoint to its Cloudflare IPv6 (AAAA) address and passing that fixed IP to cURL. When cURL is forced to connect to the IPv6 address inside an IPv4-only Docker network, it fails instantly with cURL error 7.

    Could you inspect Espo\Core\HttpClient to ensure that pre-resolved IPs support IPv4 fallback (CURL_IPRESOLVE_V4), or verify if the SSRF validation mechanism forces single-IP binding when dual-stack DNS records are present?

    Comment


    • yuri
      yuri commented
      Editing a comment
      I'm not sure that the solution with CURL_IPRESOLVE_V4 is good, as it won't allow to connect via IPv6 at all.
  • yuri
    EspoCRM product developer
    • Mar 2014
    • 9927

    #5
    For information, we had the same CURLOPT_RESOLVE with the same logic in v9.3.9.

    Comment


    • yuri
      yuri commented
      Editing a comment
      I see that we added IPv6 support for CURL_RESOLVE in v10.0.
  • yuri
    EspoCRM product developer
    • Mar 2014
    • 9927

    #6
    If you comment out this line, will it start to work? https://github.com/espocrm/espocrm/b...Check.php#L113

    Comment

  • Bizi Office
    Junior Member
    • Jul 2026
    • 5

    #7
    Final Update & Technical Summary: Webhook Queue cURL error 7 (0 ms) in v10.0

    After extensive testing across isolated environments, we have confirmed that enabling dual-stack (IPv6) routing at the container/network level completely resolves the webhook delivery failures.

    For the benefit of the community and future version planning, here is a breakdown of why this occurs in v10 and a feature suggestion for containerized deployments:

    The Root Cause in v10.0


    In EspoCRM v10.0, the HTTP client layer (Espo\Core\HttpClient\Client) introduces pre-flight DNS resolution that populates CURLOPT_RESOLVE with IPv6 (AAAA) records when interacting with dual-stack target domains (such as endpoints proxied through Cloudflare).

    When cURL receives an IPv6 mapping in CURLOPT_RESOLVE, it bypasses native DNS fallback and hard-binds exclusively to the IPv6 address (AF_INET6).

    Impact on Standard Docker Deployments


    Because standard Docker container bridge networks operate strictly on IPv4 by default (--ipv6=false), any containerized instance of EspoCRM attempting to send a webhook to a dual-stack endpoint will immediately fail with: cURL error 7: Failed to connect... Network is unreachable (after 0 ms)

    To make webhooks function in v10, Docker users currently have to manually upgrade their host daemon and custom bridge networks to assign and NAT IPv6 addresses.

    Constructive Feature Request for v10+


    Given that many self-hosted and Docker Compose environments run exclusively on IPv4 networks, forcing IPv6 socket binding creates a breaking change out-of-the-box for common webhook workflows.

    Would the development team consider adding a configurable deployment variable or config.php parameter?

    For example:
    • An environment variable or configuration toggle (e.g., 'httpPreferIpv4' => true or 'webhookResolveIpv6' => false) that tells UrlCheck to filter out AAAA records when constructing the CURLOPT_RESOLVE array.
    • Alternatively, allowing native libcurl dual-stack fallback (CURL_IPRESOLVE_V4) when pre-resolved IP arrays are passed.

    Having a built-in toggle would allow system administrators running standard IPv4 Docker environments to deploy v10 seamlessly without needing to re-architect their underlying container network stack.

    Thank you again to Yuri and the team for assisting in isolating this behavior!
    Last edited by Bizi Office; Yesterday, 11:28 PM.

    Comment

    • yuri
      EspoCRM product developer
      • Mar 2014
      • 9927

      #8
      Maybe we should filter out IPv6 only if the list includes IPv4.

      The question why it chooses IPv6 over IPv4 in your case.

      Comment


      • yuri
        yuri commented
        Editing a comment
        Should not CURL fallback to IPv4 if IPv6 failed.
    • yuri
      EspoCRM product developer
      • Mar 2014
      • 9927

      #9
      Could you try this fix? https://github.com/espocrm/espocrm/c...bce690456428f3

      Comment


      • Bizi Office
        Bizi Office commented
        Editing a comment
        Sorry, had to get a little rest. Since I deployed IPv6 through the stack I will have to try in another environment. Give me a little time to setup and I will test in an IPv4 environment

      • yuri
        yuri commented
        Editing a comment
        This patch is already in release. Thank you for assistance.
    Working...