Engines preferences

Background

A DataMapper engine extracts data from a data file. A Merge engine merges the template and the data to create Email and Web output, or to create an intermediary file for Printed output. The intermediary file is in turn used by a Weaver engine to prepare the Print output. A Merge engine merges the template and the data to create an intermediary file for Printed output. The intermediary file is in turn used by a Weaver engine to prepare the Print output.

This configuration page allows control over those Connect engine settings.

This page appears in a both the Designer Preferences and the Connect Server Configuration tool preferences, but with some different options between each.
There are also differences between OL Connect Master/Standalone and Client installations (OL Connect Enterprise edition, only).


For further background on Engine Configuration, see this page: Engine configuration.

Settings

Configure engine options

  • Maximum Data Converters selection. This setting only appears in the OL Connect Server Configuration tool (Master/Standalone) Preferences, but not in the OL Connect Server Configuration tool (Client) Preferences or the Designer Preferences.

    AFP, PCL and PostScript input have to be converted to PDF before data mapping can be performed. For this, a DataMapper engine will request a conversion engine.

    Note: AFP and IPDS are only available in OL Connect Enterprise edition.


    The selection here manages the maximum number of AFP/PCL/PostScript conversion requests that can be handled in parallel.

    When printing from the Designer, the Designer offloads AFP/PCL conversions to the Server but locally handles PostScript conversions.

    Note: The option to use AFP input is included in the Enterprise edition of OL Connect.

    AFP input is dependent upon a third party library (CDP), which only allows OL Connect to run up to 4 AFP input processes on a given machine at a given time.
    In a Server Clustering setup these processes can only currently be run on the Main Server.

  • Tasks handled by:
    This setting only appears in the OL Connect Server Configuration tool (Master \ Standalone) configuration page, but not in the Server Configuration tool (Client) configuration page.
    Select how Connect jobs are to be processed. The choices are between:
    • Engines (separate processes): Select to have a number of separate engines (for Data Mapping, Merging and Weaver, as appropriate) to process jobs.

      Note: This is the best choice for production or clustered environments. This is the best choice for production environments.

    • Connect Designer (uses less memory): /
      Connect Server (not scalable, use for low memory non-production situations): 
      This setting differs between the Connect Server Configuration and the Connect Designer preferences.
      If selected in the Connect Designer preferences, this runs the engines as part of the Designer.
      If selected in the Connect Server Configuration preferences, this makes the engines run inside the Server.
      These options effectively mean the same thing: Do not run the selected engines as separate executables.

      Note: This option is not scalable and should only really ever be used for low memory, non-production (testing) environments.

Number of engines

This group is only available if Engines were selected as the processing architecture , except in the OL Connect Server Configuration tool (Client) configuration page, where it is always available.

  • DataMapper Engines selection: This selection is only available in the Connect Server Configuration utility. It is not available in the Connect Designer Preferences.

    A DataMapper engine extracts data from a data file. A Merge engine merges the template and the data to create Email and Web output, or to create an intermediary file for Printed output. The intermediary file is in turn used by a Weaver engine to prepare the Print output. A Merge engine merges the template and the data to create an intermediary file for Printed output. The intermediary file is in turn used by a Weaver engine to prepare the Print output.

    Select the desired amount of DataMapper Engines.

    • Adding DataMapper engines might be useful in the following circumstances:

    • When large data mapping operations have to be run simultaneously for many jobs.
    • When frequently using PDF or XML based input. Particularly in the case of XML input with large individual records.
    • When the All In One plugin is used often in Workflow configurations and there are more than two Merge engines running.

    The OL Connect MariaDB database needs a fast storage system (SSD or other fast devices) to be able to keep up with two or more DataMapper engines.
    When the database is installed on a system with a slow hard drive, adding a DataMapper engine may not increase the overall performance.

  • Merge Engines selection: Select the desired amount of Merge Engines.

    A DataMapper engine extracts data from a data file. A Merge engine merges the template and the data to create Email and Web output, or to create an intermediary file for Printed output. The intermediary file is in turn used by a Weaver engine to prepare the Print output. A Merge engine merges the template and the data to create an intermediary file for Printed output. The intermediary file is in turn used by a Weaver engine to prepare the Print output.

    Generally, launching a relatively high number of Merge engines results in better performance, as Merge engines are involved in the creation of output of all kinds (Print, Email and Web) and because content creation is relatively time-consuming.

    Note: The OL Connect installer automatically sets the number of Merge Engines based upon the machine CPUs/Processors (the number of cores) and available Memory (RAM). The default setting will likely be enough for most users.

    However, virtual machines can provide an exception to this. In some virtual environments the OL Connect installer cannot accurately determine the number of logical cores actually available to the virtual machine, and will thus use a lesser number than that which could actually be supported.


    You can confirm how many Merge engines are running on your machine by running Task Manager and locating the mergeengine.exe entries in your process list.
    Note that there will be an extra Merge and Weaver engine in your task manager if you are running Designer.

    The Merge engine is responsible for the plugins Create Print Content, Create Email Content and Create Web Content.

    License restrictions only apply to the Merge engine when creating Email or Web content. They do not apply for Create Print Content. This means you are allowed to start an infinite amount of Merge engines on any given Connect Sever to run print jobs, but you will be restricted to a set number of Merge engines for your Email and HTML jobs.
  • Weaver Engines selection: This option is read-only in OL Connect Designer Preferences.
    Select the desired amount of Weaver (Output) Engines.

    A DataMapper engine extracts data from a data file. A Merge engine merges the template and the data to create Email and Web output, or to create an intermediary file for Printed output. The intermediary file is in turn used by a Weaver engine to prepare the Print output. A Merge engine merges the template and the data to create an intermediary file for Printed output. The intermediary file is in turn used by a Weaver engine to prepare the Print output.

    Adding extra Weaver (Output) engine(s) might be useful when large Print jobs are to be run simultaneously with smaller Print jobs. However, too many running engines will waste precious RAM and CPU cycles to idle processes, whilst too few could create a bottleneck.

Memory

This group is only available if Engines were selected as the processing architecture, except in the OL Connect Server Configuration tool (Client) configuration page, where it is always available.
Specify the maximum amount of random access memory (RAM) in megabytes that will be used per engine in order to make optimal use of the machine's memory.

The selections made here will apply to all Engines of that type.

Note: These settings only control the maximum size of the Java heap memory that an engine can use; the total amount of memory that will used by an engine is actually a bit higher.

Also keep in mind that the Connect Server and the operating system itself will need memory to keep running.

By default, each engine is set to use up to a predetermined amount of RAM. To make optimum use of a machine's capabilities it might be useful to increase the amount of memory that the various engines can use.

Here are some simple scenarios where you might want to increase an Engine's memory usage:

  • DataMapper engines may perform better with greater memory when running jobs containing a lot of data.

  • For complex templates with a lot of pages per document, there is a chance that Merge engines will run better with more memory.

  • The maximum memory usage of a Weaver engine can be relevant for jobs with heavy graphics; or for jobs that use Cut & Stack impositioning; or for jobs using particular variables that entail page buffering (see Content variables).

The options available in the Memory group are:

  • DataMapper Engine (MB) selection: This selection is only available in the Connect Server Configuration utility. It is not available in the Connect Designer Preferences.
    Select the desired amount of DataMapper Engine memory.

    DataMapper engines may perform better with greater memory when running jobs containing a lot of data.

  • Merge Engines (MB) selection: Select the desired amount of Merge Engine memory.

    For complex templates with a lot of pages per document, there is a chance that Merge engines will run better with more memory.

  • Weaver Engine (MB) selection:
    Select the desired amount of Weaver (Output) Engine memory.

    When running really large jobs, it often pays to increase Weaver memory allocation, even if only for the duration of the production job(s).

    The maximum memory usage of a Weaver engine can be relevant for jobs with heavy graphics; or for jobs that use Cut & Stack impositioning; or for jobs using particular variables that entail page buffering (see Content variables).

Multi-threading settings (Enterprise edition only)

This group allows you to set Multi-threading rasterization options for form ripping when generating AFP or IPDS output.

Note: AFP and IPDS are only available in OL Connect Enterprise edition.

  • Enable multi-threaded reading check box: This setting controls how the engine reads its input, with multi-threading reading being the default.

    Note: This setting is primarily for trouble shooting. We recommended always leaving multi-threading on, unless otherwise advised by Support.

  • Number of multi-threaded form rips selection: Set the amount of individual threads to assign to form ripping.

    This setting is useful for when graphics content triggers rasterization when writing AFP or IPDS output. Typical examples of content triggering rasterization are: transparency; fill patterns; gradient fills; or text rotated at a non-right angle.

    Increasing the number of threads should speed up Output Creation under such circumstances.

    Note: How effective this setting is at speeding up the output will depend upon the number of cores available on the processing machine and how many other tasks the machine is running simultaneously.

    If only one Output Creation task is running, setting this value as equal to the number of (logical) cores will likely give the best performance.


Configure merge engine options

  • On the Merge engine tab you can set the Template reuse limit. Each merge engine keeps the resources of the most recently used templates open in memory, for performance reasons. The template reuse limit determines how many times those resources get reused before they are recreated. The default value is 100. A low value is more appropriate for large jobs, while a high value is more appropriate for small jobs.

Buttons

The Engines preferences also provides you with buttons to :

  • Restore Defaults. This option restores the preferences to Defaults. This applies to the current Preferences page only, but not other Preferences.

  • Apply: This option applies the settings made within the current Preferences page, but does not close the Preferences dialog.