Automate via Slicebot

Use Slicebot to monitor one or more directories for new files. Once a file is detected, it is automatically sliced and uploaded to the cloud.

<Info>Slicebot is included with the Slicer installation.</Info>

<Info>Slicebot is not officially supported on Windows.</Info>

Quick Start

Setting up Slicebot consists of the following steps:

1. Create a configuration file that defines the directories to be monitored and how content will be processed.
2. Start Slicebot by running the following command: ./slicebot /ConfigurationFilePath

Basic Information

This section explains:

• The available configuration methods.
• How to set up the main Slicebot configuration file.
• How to set up a file-specific configuration file.
• How to define configuration settings within a file name.
• How to test out a configuration.
• How to end a Slicebot session.

Configuration

Configuration settings may be defined through the following mechanisms:

• Main Configuration File

  • Defines a default configuration.
  • Identifies the set of directories to be monitored.
  • Defines directory-specific configurations.

• File-specific Configuration File: Defines how a specific file will be processed.

• File Name: Defines how a specific file will be processed.

Settings will be applied in the following order:

1. [Global] section of the main configuration file.
2. Directory-specific sections in the main configuration file.
3. File-specific configuration file.
4. File name.

As a result, settings defined within a file name will take precedence over any conflicting settings defined elsewhere.

Main Configuration File

Slicebot requires a configuration file that contains settings defining:

• How and when source media will be processed.
• The directories that will be monitored for source media.

View configuration settings.

Key Information:

• Upon starting Slicebot, it must be informed where this configuration file is located.
• Create this configuration file using a text editor.
  • The file extension for this file should be "cfg" in lower-case (e.g., slicebot.cfg).
  • This configuration file may be saved in any directory that is accessible by Slicebot.

Categorize settings into the following sections:

• Global: Defines a default configuration for all monitored directories.

• Directory-specific Settings: A section must be created for each desired monitored directory.

  • The header for this type of section should be set to the full path to the desired directory.

    Example:

    ini

    [~/sales/videos/2016/]

• The settings defined in this type of section:

  • Only apply to that directory.
  • Override any conflicting settings defined in the Global section.

• A section may be left blank. A blank section indicates that the specified directory should be monitored and processed according to the configuration defined in:

  • The Global section of the main configuration file.
  • A file-specific configuration file.
  • A file-name.

Define settings using the following format: Name = Value

Sample Configuration

[global]
enc_wait = 1
mail_host = smtp.gmail.com:587
done_emails = [email protected], [email protected]
fail_emails = [email protected]
[~/t/slice/a/]
[~/t/slice/b/]
enc_wait = 0
username = [email protected]
apikey = abcDEFghiJKLmnoPQRtuvWXYz123ABCdefGHIJKL

File-Specific Configuration File

Instructions on when and how a specific file is processed may be defined by creating a configuration file that contains the desired configuration settings.

<Info>Although most configuration settings may be defined within a file-specific configuration, there are exceptions (e.g., halt_on_error and ignore). The description associated with each setting indicates any limitations on where it may be implemented.</Info>

Key information

• File-specific settings may be defined in a text file that meets the following conditions:

  • It must be stored in the same directory as the desired source media.

  • It must be named after the desired source media file.

  • The file extension for this file should be "cfg" in lower-case (e.g., myvideo.mp4.cfg).

    <Info>Use the helper_noext setting to exclude the source media's file extension (e.g., mp4).</Info>

• This type of configuration file should not contain section headers (e.g., [global] or [~/folder/]).

• File-specific settings override conflicting settings defined in the main configuration file.

• The recommended best practice is to ensure that a file-specific configuration file is present in a monitored directory before the corresponding media file is copied into it. This ensures that the media file will be processed according to the configuration defined in the file-specific configuration file.

• Define the require_config setting within the main configuration file to require a file-specific configuration before processing a file.

A sample configuration file is provided below.

enc_wait = 0
done_emails = [email protected], [email protected]
fail_emails = [email protected]

File Name Configuration

Instructions on when and how a specific file is processed may be defined by including the desired settings within the source media's file name.

<Info>Slicebot will only honor settings defined within a file name when the allow_inline setting has been enabled via either the main or a file-specific configuration file.</Info>

<Info>Although most configuration settings may be defined within a file name, there are exceptions (e.g., halt_on_error and ignore). The description associated with each setting indicates any limitations on where it may be implemented.</Info>

**Syntax**: FileName^Setting1=Value1^Setting2=Value2^SettingN=ValueN.ext

Example

Original file name: marketing.mp4

New file name: marketing^max_slices=3^external_id=1FA1272b.mp4

Follow these guidelines when defining a setting within a file name:

• Each setting should be appended to the file name.

• Each setting should be prepended with a caret (^) symbol.

• A file name should never contain spaces.

  • Convert space characters within a setting's value to the plus (+) symbol.

    Example: my_movie^description=Standard+Trailer.mp4

    In the above example, the asset's name will be set "Standard Trailer."

• A matching file name must still be assigned to any supporting files (e.g., closed captioning or file-specific configuration file).

  Closed Captioning Example: my_movie^max_slices=3^external_id=1FA1272b.mp4.ttml

  <Info></Info>The helper_noext setting determines whether source media's file extension (e.g., mp4) needs to be included when naming supporting files.

• Settings defined within a file name override conflicting settings defined in the Slicebot or a file-specific configuration file.

• Avoid creating file names longer than those supported by the OS.

<Tip>Use file-specific configuration files to avoid OS limitations.</Tip>

Testing

Use --dry-run option to test your configuration without processing files.

./slicebot --dry-run slicebot.cfg

End a Slicebot Session

Slicebot runs indefinitely until one of the following conditions is true:

• It is manually interrupted. The recommended method of interrupting Slicebot is through the SIGINT signal. This signal allows Slicebot to finish the current slicing job before shutting down. Pass this signal to Slicebot by running the following command: kill -SIGINT SlicebotPID
• Slicebot encounters a fatal error causing it to exit with a non-zero code. This may occur when there is an issue with the configuration file.

Configuration Settings

This section describes the supported configuration settings that can be used to control how slicebot processes files.

Key information

• The config file format considers lines starting with # to be comments, so they are ignored.

• Although most settings can be used anywhere (global section, directory sections, or per-file configs), you will find that some settings are more naturally used in some places more than others. For example, the global section will typically hold things like username, API key, and email settings, although they are legal in other places as well.

• Setting values support basic value substitution:

  [FILENAME] - replaced with the current source filename, minus the path. Example, with a source file of /mnt/files/ad.mp4:
  
  external_id=foo[FILENAME]bar
  would become
  external_id=fooad.mp4bar

  [FILENAME_SHORT] - replaced with the current source filename, minus the path and file extension. Example, with a source file of /mnt/files/ad.mp4:
  external_id=foo[FILENAME_SHORT]bar
  would become
  external_id=fooadbar

The following configuration settings may be leveraged when defining:

• The main configuration file.
• A file-specific configuration file.
• Configuration settings by file name.

Closed Captions

You can provide slicebot with closed caption information by placing closed caption data in a TTML file in the same directory as the media file. TTML files are simple text files that can be created with third party tools or with any text editor:

<?xml version="1.0" encoding="utf-8"?>
<tt xml:lang="en" xmlns="http://www.w3.org/2006/04/ttaf1" xmlns:tts="http://www.w3.org/2006/04/ttaf1#styling">
  <head>
    <styling>
      <style id="defaultSpeaker" tts:fontSize="24" tts:fontFamily="Arial" tts:fontWeight="normal" tts:fontStyle="normal" tts:textDecoration="none" tts:color="white" tts:backgroundColor="black" tts:textAlign="center" />
      <style id="defaultCaption" tts:fontSize="24" tts:fontFamily="Arial" tts:fontWeight="normal" tts:fontStyle="normal" tts:textDecoration="none" tts:color="white" tts:backgroundColor="black" tts:textAlign="center" />
    </styling>
  </head>
  <body style="defaultCaption" id="thebody">
    <div xml:lang="en">
      <p begin="00:00:01.420" end="00:00:02.620"><metadata ccrow="14" cccol="3" /><span tts:fontStyle="italic">THIS IS SOME TEXT...</span></p>
      <p begin="00:00:06.620" end="00:00:09.790"><metadata ccrow="13" cccol="1" />(Doorbell rings)</p>
      <p begin="00:00:11.100" end="00:00:13.000"><metadata ccrow="14" cccol="8" />♪ LA LA LA SINGING ♪</p>
      <p begin="00:00:20.460" end="00:00:22.170"><metadata ccrow="14" cccol="0" />THIS IS MORE TEXT</p>
    </div>
  </body>
</tt>

<Info>The require_captions config setting can be used to ensure that slicebot will skip a file if it does not have a corresponding TTML file. As with other settings, it can be used in the global, directory, or per-file context.</Info>

The TTML files use a naming convention similar to config files: take the name of the associated media file and add a ".ttml" extension to the filename. For example, if slicebot is processing a file named /tmp/foo/film.mp4, it will look to see if there is a file named /tmp/foo/film.mp4.ttml and, if so, use it for the source of closed caption data. If no corresponding TTML file exists, the video will not have any closed caption data. Note that the helper_noext setting can alter the naming convention for TTML files.

<Info>Only hh:mm:ss.sss time formats are currently supported in the TTML file.</Info>

Metadata Files

You can assign metadata key-value pairs to an asset using individual settings; for larger amounts of metadata, however, this can be cumbersome, so it may be more convenient to make use of a metadata file. This file is a JSON text file that has the metadata you want associated with the media file. As with caption and config files, the name of the metadata file is the media filename with an additional extension: .meta.json. For example, if a media file is named video123.mp4, then an associated metadata file for it would be named video123.mp4.meta.json.

The contents of the metadata file must be a valid JSON object, or processing the file will produce an error. Structured metadata is supported, although presently it is not fully displayed or editable in the CMS UI.

{
    "genre" : "comedy",
    "network" : "ZNN",
    "season" : 1,
    "episode" : 13
}

In general, a given media file will use individual metadata settings or a metadata file, but not both. However, in the event that both are specified for a particular file, metadata from a file is applied first, followed by any metadata settings.

HTTP Notifications

HTTP notifications provide an additional way to integrate a system with your content production workflow. This type of notification allows Slicebot to submit a HTTP POST request whenever one of the following events takes place:

• Slicing Starts (start_url)
• Slicing Completes (done_url)
• Slicing Error (fail_url)

<Tip>Enable HTTP notifications by setting one or more of the above settings to the URL that will be requested.</Tip>

Email notifications may also be sent whenever slicing completes (done_emails) or when an error occurs (fail_emails).

HTTP POST Request

Slicebot will submit a HTTP POST request using the default Internet media type (i.e., application/x-www-form-urlencoded). The body of this request will consist of a single line with the following parameters:

In addition to the above parameters, any custom query string parameters defined in the configuration setting (i.e., start_url, done_url, and fail_url) will be automatically stripped from the request and inserted into the request body. This allows custom parameters to be passed to the remote server as needed.

View a sample scenario.

<Info>Custom query string parameters whose names match one of the above system-defined parameters (e.g., filename or description) will be ignored.</Info>

<Info>To allow for future expansion, the server process that receives the HTTP POST should silently ignore unrecognized parameters.</Info>

Notification Issues

Slicebot will log an error under the following conditions:

• It has been configured to request an invalid URL.
• A valid URL has been provided, but the server is unreachable.
• A valid URL has been provided, but it generates a non-200 response.

After which, it will reattempt the HTTP notification several times as determined by the notify_retries setting. After each attempt, it will pause for a few seconds as determined by the notify_retry_wait setting. If it is unable to successfully submit the HTTP POST request, then it will move on to the next file.

Sample HTTP Notification Scenario

This sample HTTP notification scenario assumes that the done_url setting is configured to:

http://www.example.com/slicing/handler?custom=1234&other=5678

The following sample HTTP POST request will be submitted once a file called "video.m4v" has been successfully sliced.

POST /slicing/handler HTTP/1.1
Host: www.example.com
Accept-Encoding: identity
Content-Length: 161
Content-Type: application/x-www-form-urlencoded
Connection: close
User-Agent: upLynk-slicebot/1.0
status=ok&other=5678&description=The%20file&filename=%2Fopt%2Ft%2Fmyvideos%2Fvideo.m4v&external_id=dummy_id&custom=1234&asset_id=a9119d7afc5242319d028794ae283ea7

Notice that the custom query string parameters (i.e., custom and other) defined in the done_url setting were automatically stripped from the request and inserted into the request body.<Info