Watson Workspace will no longer be available after February 28, 2019. Learn more.

Developers

Outbound webhooks down?

Comments

8 comments

  • Avatar
    JARED WALLACE

    Looks like the offsets are still an issue. My app just got a message replay from yesterday at 7:39pm CST, today at 7:39pm CST.

     

    So right at a 24 hour difference.



  • Avatar
    Herbert Dürr

    I checked the reported issue and can confirm that the notification (created on 2017-01-10 01:39:22 UTC) was delivered and confirmed only 24h later (2017-01-11 01:39:27 UTC).

     

    The root cause seems to be that the callback failed to reply / timed out until it was finally delivered on 2017-01-10 20:26:39 UTC. Retries are initially done every 30secs, then increasing e.g. 3h, 6h, 12h, 24h. The instance that waited for the successful delivery of the "blocking" notification was in a status at "retry after 24h" and didn't get that the blocking notification was delivered in the meantime. That handling is being reevaluated.

  • Avatar
    Mikhail Voytenko

    When retry logic retries to send a message to a webhook, the following messages are waiting in a kind of queue. When the message is delivered the outbound webhook tries to resend the following message immediately.

    In this case it looks like the service was restarted before it could handle the whole queue of following messages. As result the first remaining message in the queue was handed after service restart without information that previous message has just succeeded and thus the retry logic was waiting for the specified delay ( since the message was older that 12h the next time to try was 24h).

    That is going to be changed soon to let the service retry to send the first message in the queue after service restart.

  • Avatar
    Herbert Dürr

    Jared, you used the wording 'got a message replay from yesterday'. Are you sure you meant to write replay or did you mean delay? Because there is no indication in the system log that the message in question was already delivered once.

  • Avatar
    Herbert Dürr

    Thanks for checking. So it seems it was delivered exactly once (but with delay) instead of twice. This fits the "that message was blocked by the non-acknowledged older notification" analysis in my comment above.

  • Avatar
    Mikhail Voytenko

    If you want to workaround the problem, please try to change the webhook ( for example to enable and disable it ), that should reset the resend interval sequence and let the messages be resent.

  • Avatar
    JARED WALLACE

    The workaround succeeded, and my app is processing the queue now. Thanks :)

Please sign in to leave a comment.