Preprocessing AFP input

Note: Option is only available in the Enterprise edition of OL Connect.

In some cases it is useful or even necessary to pre-process AFP input files before OL Connect extracts data from them. This topic explains why and how to preprocess AFP input.

Embedding external resources

AFP data streams often use external files: overlays, page segments, presentation pages, form definitions, page definitions, coded fonts, character sets, code pages, TrueType fonts, and more.

If resources are missing from AFP input files, it can make it impossible to extract data from them. The content will not look as intended and will therefore no longer be fit for further processing. In case of a missing font, for example, text may be converted into another font and become too long for a table cell or overlap with other text.

Example of malformed AFP output

Assuming you have the corresponding source files, there are two solutions: you can put the source files in a folder where OL Connect will find them, or you can embed them in the AFP.

Note: External resources are picked up immediately after they have been made available. There is no need to restart the Designer or OL Connect Server. However, if a previously imported or processed AFP input file is not formatted correctly after you add or embed source files, this is likely because the results of AFP input processing are cached. Both OL Connect Server and Designer (DataMapper) cache the results of AFP input processing. In that case, make sure to restart both.

Providing resources via a folder

The OL Connect Server will automatically use external AFP resources if it can find the files in the following folder:

C:\ProgramData\Objectif Lune\OL Connect\Resources\AFP

It will also search the following subfolders of that folder for resource files: FormDef,
PageDef, Overlay, PageSeg, PresentPage, CodedFont, Charset, Codepage, TrueType.

Although these folder names suggest a certain resource type, it actually doesn’t matter in which folder a particular resource type is stored.

The extension of a file name is ignored, so there is no need to change extensions of source files.

Note: This will also work in Designer (DataMapper) since all AFP processing is actually done by OL Connect Server.

Embedding resources with AFPMerge

AFPMerge is a command-line tool that can embed external resources in one or more AFP files, merge multiple AFP files into one, and indicate what form definition to include in the output AFP.

The tool is installed with OL Connect Enterprise when either the OL Connect Designer or Server component is installed.

AFPMerge can be found in the OL Connect installation folder:
C:\Program Files\Objectif Lune\OL Connect\AFPMerge\bin

To run AFPMerge with a single AFP file, open the Command Prompt and use the following command:

"C:\Program Files\Objectif Lune\OL Connect\AFPMerge\bin\AFPMerge.exe" -i <inputfile> -r <resource folder> -o myoutputfile.afp

  • -i specifies an input file.

    AFPMerge can also be used to embed resources in a collection of AFP files which will then be merged into one file. How to do this is explained below, in Merging AFP files.

  • -r specifies the folder in which the files to embed are stored.

    Files to be embedded do not have to have a file extension. AFPMerge looks up files by name, first without an extension, and then with file extensions that are accepted with the respective resource type.

  • -o specifies the name of the output file.

Example:
"C:\Program Files\Objectif Lune\OL Connect\AFPMerge\bin\AFPMerge.exe" -i c:\input\myinputfile.afp -r c:\myresourcefolder -o c:\output\myoutputfile.afp

Tip: The command "C:\Program Files\Objectif Lune\OL Connect\AFPMerge\bin\AFPMerge.exe" -h (or --help) displays a list of all of AFPMerge's command line options and what they do.

Merging AFP files

Extracting data from many small AFP files that have the same structure will be less efficient than when those files are combined in a single AFP file.

AFPMerge, a command-line tool that is designed to embed external resources into AFP files, can merge AFP files at the same time.

To run AFPMerge on a collection of AFP input files:

  1. Create a manifest file. This is a text file that contains the list of the AFP files that must be merged and, optionally, the grouping key of each (tab-delimited). See Manifest file format.

  2. Open the Command Prompt and use the following command:

    "C:\Program Files\Objectif Lune\OL Connect\AFPMerge\bin\AFPMerge.exe" -l <manifest file> -r <resource folder> -o myoutputfile.afp

    • -l (lowercase L) specifies the manifest file.

    • -r specifies the folder in which the files to embed are stored. This is required; however, if the given resource folder does not contain any resource files, the tool will proceed and only merge the files listed in the manifest file.

    • -o specifies the name of the file into which the listed files are merged.

    Example:
    "C:\Program Files\Objectif Lune\OL Connect\AFPMerge\bin\AFPMerge.exe" -l c:\input\mymanifest.txt -r c:\myresourcefolder -o c:\output\myoutputfile.afp

    Tip: In the Command Prompt, a . (dot) at the start of (or instead of) a folder name indicates the current folder.

Manifest file format

The manifest file (used with the -l option of AFPMerge) must be a UTF-8 encoded file that lists one file per line.

Example:
print75X-1.AFP
print75X-12.AFP
print75X-3.AFP
print75X-34.AFP
print75X-5.AFP
print75X-56.AFP
print75X-7.AFP
print75X-78.AFP

AFPMerge will wrap each file in a page group, and that page group gets a Tagged Logical Element (TLE) named GROUP, with its value set to the input filename.
Because DataMapper can set boundaries using AFP page groups, the combined AFP files can be easily processed into one record per input file.

Optionally, the file can have two columns separated by a tab.

Example:
print75X-1.AFP doc1
print75X-12.AFP doc1
print75X-3.AFP doc1
print75X-34.AFP doc2
print75X-5.AFP doc2
print75X-56.AFP doc3
print75X-7.AFP doc3
print75X-78.AFP doc3

When a second column is present, AFPMerge uses it as a grouping key and puts files with the same grouping key in the same page group. Each page group gets a TLE named GROUP that holds the value of the grouping key.

The above example will create a combined AFP file with 3 page groups that will map to 3 records in DataMapper.

Tip: TLEs can be mapped to fields in DataMapper, so it is easy to pick up the grouping key and use that in subsequent processing.

Specifying a form definition

AFP files can have multiple form definitions of which only one can be included in the output. AFPMerge allows to select the form definition that is to be included in the output with the option -f or --formdef.

For example:

"C:\Program Files\Objectif Lune\OL Connect\AFPMerge\bin\AFPMerge.exe" -l c:\input\mymanifest.txt -r c:\myresourcefolder -o c:\output\myoutputfile.afp -f F1MYFDEF

Preprocessing AFP files in a Workflow process

To preprocess AFP files in a Workflow process, AFPMerge can be run with the External Program action task.

  • In the task properties, select to run the program minimized.

  • Enter the command line options in the Parameters field. Enclose file and folder names in double quotes.

  • If you want log messages to be included in OL Connect log files, check the Log the program output option.

    Note: AFPMerge has several log levels. For information on setting the log level, see AFPMerge's log level options.

The output of AFPMerge can be loaded using the Load External File action task, which can pass the file to the OL Connect Execute Data Mapping task.

External Program task screenshot

AFPMerge's log level options

When AFPMerge is invoked without a log level option, error and warning messages are logged, as well as info messages that occur once per run. Info messages that are repeated for each input file or even more often, are not shown.

The following command line options change the log level.

  • -q, --quiet puts logging in quiet mode. When invoked with the -q option, AFPmerge shows no log messages.

  • -v, --verbose turns on verbose logging. When invoked with the -v option, AFPmerge logs error, warning, and info messages.

  • -d, --debug turns on detailed logging. When invoked with the -d option, AFPmerge logs error, warning, info, debug, and trace messages.