Failover Behavior and Recommendations

Miscellaneous

What is Allowlisting and Denylisting?

When a Slicer is first detected (either for the first time or after being fixed after a failover event), the Slicer Failover system denylists that Slicer. When denylisted, a Slicer is not used as an option for Slicer Failover. This prevents the system from thrashing in case the Slicer is not truly healthy.

Once the Slicer Failover system considers the new Slicer healthy, it is allowlisted. When allowlisted, a Slicer is used as an option for Slicer Failover. The transition from a denylisted to an allowlisted state can take several minutes.

What Exactly is a Warm State Slicer Doing?

In the Warm state, backup Slicers are still operating normally. A healthy Warm state Slicer indicates that it is "Slicing" (or "Ads") since it is operating normally.

After the Warm state Slicer sends its content to the cloud, it is not encoded, which provides the reduced cost. In a Warm state, the Slicer is essentially in blackout mode, meaning that its video is being discarded.

In the Event Log, a Warm state Slicer has a state of 1 from the Broker, which is the same state as ad break. The Broker is the component that receives the video from the Slicer. A state of 1 means the Broker is discarding the video from the Slicer. A state of 0 means that the Broker is not discarding the Slicer content.

The amount of slate displayed in Hot-Warm failover can range from a few seconds to over 30 seconds and is dependent on a number of factors such as the thresholds used, system loading, etc.

Auto Failback

Auto Failback after a failover condition occurs under the following conditions:

• Auto Failback is ENABLED.
• The formerly failed Slicer has been considered "healthy" for 30 seconds.
• After being healthy, the system takes 1-2 minutes for the Slicer to become allowlisted.
• The priority of the formerly failed Slicer is higher than the current one.

The system is designed to failover quickly and failback slowly to avoid toggling.

Event Log Info

State for xxxxx changed from 0 to 1:

• Means that Slicer xxxxx is going from Slicing state (0) to ad break or a Warm state (1).
• If going into a Warm state, there is usually an additional log message near this one.

A denylisted Slicer is unhealthy while an allowlisted Slicer is considered healthy.

Things to Watch Out For

When changing the contents of the Slicer tab, make sure to select Save so that the changes take effect.

When you enable a Slicer from the Slicer tab, it takes about 1-2 minutes for the Slicer to truly become active in the Failover Group. This ensures that the enabled Slicer is truly healthy before being considered for failover.

If you manually kill a Slicer, the Slicer status on the Slicer tab and the Content tab still says it is Slicing. For the Slicer tab, will try and use Slicer Health instead to provide more accurate information. Slicer Monitoring shows the status of the Slicer accurately and Slicer Failover itself will work.

If the backup Slicers have unstable or bad input, this could prevent a "failed" Slicer from doing failover since the backup Slicers would not be considered healthy. For instance, if two Slicers in a Failover Group have the same input feed and that feed stops, failover does not occur since both Slicers are considered unhealthy.

When a Slicer is in the Warm state, its status is still "Slicing" (or "Ads") since it is operating properly. In the Warm state, the cloud encoding is disabled, not the Slicer itself.

If you get an error reading Failover Groups in the CMS, you may still have a permissions issue.

If after adding a Failover Group to a channel there is no thumbnail for the channel displayed in the CMS, restart the Slicers in the Failover Group.

Make sure the Slicer config has enable_remote_config: 1 in it to enable the Slicer to get configurations from a centralized database. enable_remote_config: 1 has to be after the failover_id for Slicer versions prior to Slicer Release 2306600.

If using any previous versions of Slicer Failover, they must be disabled before this Slicer Failover can be used.

With Cloud Slicer Live, having enable_remote_config: 1 in the config file may prevent Slicer Failover from working.

Removing a Slicer Permanently from Slicer Failover

To remove a Slicer from Slicer Failover entirely:

• Remove or comment out the failover_id in the configuration file, if present.
• Remove or comment out the enable_remote_config: 1 in the configuration file.
• Remove the Slicer from the Failover Group in the CMS.
• Reboot the Slicer.

Recommendations

Slicer Failover reports events using SNS so it is recommended to use this functionality.

If the network feeding the Slicer has stability issues, it is recommended to set the Input Loss set threshold to higher than 0 to reduce the sensitivity.

When using Flexible Hot-Warm, it is recommended to use Prioritized or Custom as the Priority Mode because Flat Mode may pick a Warm Slicer instead of a Hot one since the backup Slicer is chosen randomly.