Engines preferences

Background

A DataMapper engine extracts data from a data file. A Merge engine merges the template and the data to create Email and Web output, or to create an intermediary file for Printed output. The intermediary file is in turn used by a Weaver engine to prepare the Print output.

This preferences page allows control over those Connect engine settings.

This page appears in a both the Designer Preferences and the Connect Server Configuration tool preferences, but with some different options between each.

This preference page was remodelled in 2019.2 to improve ease of control over output production settings.

For further background on Engine Configuration, see this page: Engine configuration.

Settings

Configure engine options

  • Maximum Data Converters selection. This setting only appears in the PlanetPress Connect Server Configuration tool Preferences, but not in the Designer Preferences.
    AFP, PCL and PostScript input have to be converted to PDF before data mapping can be performed. For this, a DataMapper engine will request a conversion engine.

    The selection here manages the maximum number of AFP/PCL/PostScript conversion requests that can be handled in parallel.

    When printing from the Designer, the Designer offloads AFP/PCL conversions to the Server but locally handles PostScript conversions.

    Note: AFP input is dependent upon a third party library (CDP), which only allows PlanetPress Connect to run up to 4 AFP input processes on a given machine at a given time.

    In a Server Clustering setup these processes can only currently be run on the Main Server.

  • Tasks handled by:
    Select how Connect jobs are to be processed. The choices are between:
    • Engines (separate processes): Select to have a number of separate engines (for Data Mapping, Merging and Weaver, as appropriate) to process jobs.

      Note: This is the best choice for production environments.

    • Connect Designer (uses less memory): /
      Connect Server (not scalable, use for low memory non-production situations): 
      This setting differs between the Connect Server Configuration and the Connect Designer preferences.
      If selected in the Connect Designer preferences, this runs the engines as part of the Designer.
      If selected in the Connect Server Configuration preferences, this makes the engines run inside the Server.
      These options effectively mean the same thing: Do not run the selected engines as separate executables.

      Note: This option is not scalable and should only really ever be used for low memory, non-production (testing) environments.

Number of engines

This group is only available if Engines were selected as the processing architecture .

  • DataMapper Engines selection: This selection is only available in the Connect Server Configuration utility. It is not available in the Connect Designer Preferences.

    A DataMapper engine extracts data from a data file. A Merge engine merges the template and the data to create Email and Web output, or to create an intermediary file for Printed output. The intermediary file is in turn used by a Weaver engine to prepare the Print output.

    Select the desired amount of DataMapper Engines.

    • Adding DataMapper engines might be useful in the following circumstances:

    • When large data mapping operations have to be run simultaneously for many jobs.
    • When frequently using PDF or XML based input. Particularly in the case of XML input with large individual records.
    • When the All In One plugin is used often in Workflow configurations and there are more than two Merge engines running.

    The Connect MariaDB database needs a fast storage system (SSD or other fast devices) to be able to keep up with two or more DataMapper engines.
    When the database is installed on a system with a slow hard drive, adding a DataMapper engine may not increase the overall performance.

  • Merge Engines selection: Select the desired amount of Merge Engines.

    A DataMapper engine extracts data from a data file. A Merge engine merges the template and the data to create Email and Web output, or to create an intermediary file for Printed output. The intermediary file is in turn used by a Weaver engine to prepare the Print output.

    Generally, launching a relatively high number of Merge engines results in better performance, as Merge engines are involved in the creation of output of all kinds (Print, Email and Web) and because content creation is relatively time-consuming.

    Note: The Connect installer automatically sets the number of Merge Engines based upon the machine CPUs/Processors (the number of cores) and available Memory (RAM). The default setting will likely be enough for most users.

    However, virtual machines can provide an exception to this. In some virtual environments the Connect installer cannot accurately determine the number of logical cores actually available to the virtual machine, and will thus use a lesser number than that which could actually be supported.


    You can confirm how many Merge engines are running on your machine by running Task Manager and locating the mergeengine.exe entries in your process list.
    Note that there will be an extra Merge and Weaver engine in your task manager if you are running Designer.

    The Merge engine is responsible for the plugins Create Print Content, Create Email Content and Create Web Content.

    License restrictions only apply to the Merge engine when creating Email or Web content. They do not apply for Create Print Content. This means you are allowed to start an infinite amount of Merge engines on any given Connect Sever to run print jobs, but you will be restricted to a set number of Merge engines for your Email and HTML jobs.
  • Weaver Engines selection: This option is read-only in PlanetPress Connect Designer Preferences.
    Select the desired amount of Weaver (Output) Engines.

    A DataMapper engine extracts data from a data file. A Merge engine merges the template and the data to create Email and Web output, or to create an intermediary file for Printed output. The intermediary file is in turn used by a Weaver engine to prepare the Print output.

    Adding extra Weaver (Output) engine(s) might be useful when large Print jobs are to be run simultaneously with smaller Print jobs. However, too many running engines will waste precious RAM and CPU cycles to idle processes, whilst too few could create a bottleneck.

Memory

This group is only available if Engines were selected as the processing architecture.
Specify the maximum amount of random access memory (RAM) in megabytes that will be used per engine in order to make optimal use of the machine's memory.

The selections made here will apply to all Engines of that type.

Note: These settings only control the maximum size of the Java heap memory that an engine can use; the total amount of memory that will used by an engine is actually a bit higher.

Also keep in mind that the Connect Server and the operating system itself will need memory to keep running.

By default, each engine is set to use up to a predetermined amount of RAM. To make optimum use of a machine's capabilities it might be useful to increase the amount of memory that the various engines can use.

Here are some simple scenarios where you might want to increase an Engine's memory usage:

  • DataMapper engines may perform better with greater memory when running jobs containing a lot of data.

  • For complex templates with a lot of pages per document, there is a chance that Merge engines will run better with more memory.

  • The maximum memory usage of a Weaver engine can be relevant for jobs with heavy graphics; or for jobs that use Cut & Stack impositioning; or for jobs using particular variables that entail page buffering (see Content variables).

The options available in the Memory group are:

  • DataMapper Engine (MB) selection: This selection is only available in the Connect Server Configuration utility. It is not available in the Connect Designer Preferences.
    Select the desired amount of DataMapper Engine memory.

    DataMapper engines may perform better with greater memory when running jobs containing a lot of data.

  • Merge Engines (MB) selection: Select the desired amount of Merge Engine memory.

    For complex templates with a lot of pages per document, there is a chance that Merge engines will run better with more memory.

  • Weaver Engine (MB) selection:
    Select the desired amount of Weaver (Output) Engine memory.

    When running really large jobs, it often pays to increase Weaver memory allocation, even if only for the duration of the production job(s).

    The maximum memory usage of a Weaver engine can be relevant for jobs with heavy graphics; or for jobs that use Cut & Stack impositioning; or for jobs using particular variables that entail page buffering (see Content variables).

Buttons

The Engines preferences also provides you with buttons to :

  • Restore Defaults. This option restores the preferences to Defaults. This applies to the current Preferences page only, but not other Preferences.

  • Apply: This option applies the settings made within the current Preferences page, but does not close the Preferences dialog.