Overview

Webhooks is a feature that allows the communication between Liquid Content and Evoq whenever a specific event has been made. This article explains the functionality and configuration of the webhooks feature.

Prerequisites

• Access to a SuperUser account.

Introduction

There has always been the need of communicating Evoq with SC, mostly due to maintaining the content consistency between both elements. In Evoq 9.0 this communication was based on UI events thrown in the SC (Structured Content) administration pages (Content Items, Content Types and Visualizers). These events were sent to a module's endpoint, which was in charge of clearing some cache elements. This way the modified content in the microservice was kept consistent in the Evoq site.

However, this doesn't cover all the possible scenarios:

• FB-SC Connector, this is a new microservice that uses the API of SC to create new content. But all these operations didn't throw any interaction with the Evoq sites, so possible inconsistencies may appear once direct publish be enabled.
• In the near future, any third party client that uses SC may potentially create or modify the existing content without using the current administration UI. So all these modifications would not be notified to the Evoq sites that make uses of that content.

To handle these scenarios, a webhook mechanism has been implemented. It allows us to notify the sites in any kind of scenario even when the user is not using the administration UI.

Communication Scenarios: Overview

When we are talking about webhooks there two scenarios involved:

• Subscription. This happens when a new client notifies that he wants to be notified when certain events happen.
• Notification. When one event associated with a webhook occurs the system notifies to all the subscribers of that event.

In our implementation, the subscriber will be any Evoq sites that want to be up to date of any change in their SC items. And the emission of that change will be the SC microservice.

Figures 1 and 2 show a simplified view of this process. In later sections, all the elements involved in these scenarios will be explained in deep.

Figure 1. Subscription overview.

Figure 2. Notification overview.

Webhook Events

As we mentioned a webhook is associated with one or more events. Currently, the events available in the SC microservices are:

• For content items:
  • ContentItemPublished,
  • ContentItemCreated,
  • ContentItemUpdated,
  • ContentItemDeleted,
• For content types:
  • ContentTypeCreated,
  • ContentTypeUpdated,
  • ContentTypeDeleted,
• For visualizers:
  
  • VisualizerCreated,
  • VisualizerUpdated,
  • VisualizerDeleted

It is important to mention here that currently all the webhooks created by Evoq are subscribed to all these events. However, the mechanism is prepared for other more advanced scenarios in which the subscribers are interested only in part of these events.

Description

Webhook subscription

When This Subscription Is Done?

Evoq has to create a webhook subscription in two main scenarios:

• When SC is enabled through the Opt-in process.
• When a new child site is created.

Both events require to create a new webhook that will be notified when changes are done in the content associated with a tenant.

The subscription process is done in two decoupled steps:

1. Add a new pending subscription task.
2. Create the webhook subscription.

The second step is done asynchronously, this avoids the user to wait until the webhook is created in the microservice, which can take a lot of time (in particular in the opt-in scenario).

Adding a New Pending Subscription Task

As mentioned, the second part of the subscription process is done asynchronously. To handle this scenario, when a new subscription is requested, a new row is added to the table "StructuredContent_Library_PendingHookSubscriptions". After adding this request to the table, a task for handling new requests is queued into a task queue service that we will see in the next section.

The next figure summarises step 1 the Opt-in scenario. A key thing here is that multiple requests will be added to the table, one per site. The handling task will have to take care of all of them since this task is done asynchronously, the user that enables the opt-in will not be affected by this heavy process.

The following figure summarizes the step 1 when a child site is added. In this case, only one row will be added to the table.

As a result of any of these scenarios, new rows of pending subscriptions are added to the system and a new task is queued to the task queue service to handle all the pending subscriptions.

Handling New Pending Subscriptions

Once a new handling task is queued into the task queue service, which ensures that only one task will be executed at the same time. However, in a web farm environment, multiple servers may want to execute a handling process for the same requests. To avoid this a lock mechanism has been implemented at row level. This way one row will be only handled by one process at the same time.

Once the queued task is executed, the handler:

• It creates a secret that will be used to sign future notifications.
• Sets the callback URL, the endpoint that will be used to send the notification.
• Adds a list of event names, this list contains the subscribed events.
• Finally, creates a new webhook for each one of them, as the next figure shows.

As mentioned, all this process uses a lock mechanism to avoid that a request is handled multiple times. If something goes wrong during the creation process, this row will keep locked what would be a problem. To avoid this a retry mechanism has been implemented to cover this scenario. Please review the section "Retry Policies" for more details.

Webhooks Notification

The second part of the process occurs when any event occurs in the SC microservice. Since the process of sending notifications can take a lot of time, this process has been also decoupled. The user that makes a change doesn't have to wait until the whole notification process finishes.

To make this the implementation makes uses of Azure Service Bus. Each time that an event happens, a new message is put in a topic (a special kind of queue that allows multiple subscribers). Then a subscriber of this topic will read the message and will notify all the webhooks. Currently, the only subscriber is the one in charge of notifying the webhooks and has been implemented as an Azure web job.

Retry Strategies

All the processes that involve communication between different applications need to have a retry policy. This way we ensure that the process is done even when the target application is not available temporally.

In the implement mechanism, retry policies have been created when:

1. In Evoq when the task for handling a new pending subscription is executed. This process will lock the request, but a retry policy will be executed over locked requests based on these two hidden host settings:
   • StructuredContent_Webhook_MaxTimeOfLockingInMinutes. This setting sets how many times a request can be locked before it will be handled by other tasks. The default value is 30 minutes.
   • StructuredContent_Webhook_MaxNumberOfAttemptsAllowed. Based on the previous settings a pending request can be handled multiple times in case of error. This setting sets the maximum number of times that is can be reprocessed. The default value is 3.
   However, this mechanism is not applied if the error is due to a wrong request for creating the hook, in this case, the request is marked as erroneous in the database (with information about the error) and will not be included in the handling process anymore.
2. When Evoq invokes the SC API to create a new hook.
3. When the notification is sent to a target webhook.

Are webhook configuration copied in Copy Tenant?

No, copy tenant does not copy webhook configuration as those will point to production site causing side effects.

How do I activate the webhooks if it is an upgraded website or for some other reason webhooks are not enabled for my website?

In this case, you will need to re-enable your Form Builder/Structured Content feature again. You can following steps:

1. Run the following queries in SQL Console

   1. Form Builder: 

      1. DELETE FROM {databaseOwner}[{objectQualifier}PortalSettings] WHERE SettingName IN ('FormBuilderClientCredentials','FormBuilderEnabled')

      2. DELETE FROM {databaseOwner}[{objectQualifier}HostSettings] WHERE SettingName IN ('FormBuilderEnabled')

   2. Structured Content

      1. DELETE FROM {databaseOwner}[{objectQualifier}PortalSettings] WHERE SettingName IN ('StructuredContentClientCredentials','StructuredContentEnabled')

      2. DELETE FROM {databaseOwner}[{objectQualifier}HostSettings] WHERE SettingName IN ('StructuredContentEnabled')

2. Restart the application and open the Form Builder and Structured Content and enable them.

Your website is registered with Webhooks now.