Data Processing Functions

Data Quality section. Data administrator interface

Contents

General Info

Data processing functions are used in quality rules and are intended to perform specified actions on data. The function requires input data, which is implemented by incoming ports. The results of work (processed data or checked values) are sent to the outgoing ports. There are different kinds of functions:

  • Simple. Available by default. They represent simple actions, such as removing extra spaces.
  • Composite. They are created in a user interface with a special editor. Composite function can use one or more simple functions, as well as special blocks, such as a condition.
  • Custom. They are implemented in Java and loaded into the system as JAR-files.

The following actions are available in the “Functions” tab:

Test Run

To test a function:

  1. Find the required function in the list of functions on the left side of the screen.
  2. Highlight the function.
  3. This will display a description of the function.
  4. Specify a test value for the function's incoming ports.
  5. Click the “Test” button.
  6. As a result of the action, the result of the function will be displayed on the outgoing ports. For example, if you specify numbers in the incoming ports of the sum function, the outgoing port will display the sum of the numbers.

Testing the function

Figure 1. Testing the function

Composite Functions

To create composite function:

  1. Make sure draft mode is enabled and draft is selected.
  2. Click the “Create” button located at the bottom of the functions list.
  3. The action will open a list of creating options above the button.
  4. Select the “Composite” option.
  5. As a result of the action, the window for setting the new function will open.
  6. Click the “Edit” button located in the upper right corner of the feature setup window.
  7. Specify basic settings of the function:
    • Name.
    • Displayed name.
    • Set the required function processing modes: global and/or local. For more information about the modes, see function_processing_modes below.
  8. Configure the composite function in the visual editor:
    • Add incoming ports by clicking the “Add” button on the corresponding block.
    • Add outgoing ports by clicking the “Add” button on the corresponding block.
    • If necessary, drag and drop the function blocks located in the list on the left.
    • Make connections between the blocks.
  9. Save the changes. Click the “Save” button located in the upper right corner of the function configuration window.
  10. After all the data has been entered and saved, publish the draft.

Function Processing Modes

GLOBAL. Processing mode, when the function processes the port once, regardless of the size of the attributes collection it gets into.

LOCAL. Processing mode in which the function processes the port as many times as there are attributes in the collection.

Features of the execution context:

  1. In GLOBAL mode, the function processes a port once, regardless of the size of the collection;
  2. GLOBAL is the default mode;
  3. LOCAL mode allows you to exclude from processing those attributes that might cause errors in the function's execution during attribute collection packing.
  4. In LOCAL mode, the function processes the port as many times as the number of attributes were packed into the collection;
  5. In LOCAL mode, ports can be selected either higher in structure or 1 level below the specified complex attribute;
  6. Attributes selected above the context are direct ancestors of attributes (parents and grandparents) specified in the context;
  7. Each standard data processing function supports one of the modes. A choice between modes is available for individual standard functions. Information about the processing mode is specified in the “Functions” section when selecting a standard function from the list;
  8. When creating a composite function a choice of possible processing modes is available;
  9. When implementing a third-party function, you also need to specify the value of the parameter that specifies the available processing modes;
  10. Upath-expression, specified in the “Execution context” field must match the expressions of the incoming and outgoing ports.

Adding Custom Function

To add a custom function:

  1. Make sure draft option is enabled and draft mode is selected.
  2. Make sure that the custom function has been added to the system as a library.
  3. Click the “Create” button located at the bottom of the functions list.
  4. The action will open a list of creation options above the button.
  5. Select the “Java” option.
  6. The action will open the add functions wizard.
  7. In the first step, select the required library.
  8. In the second step, add one or more functions contained in the library.
  9. In the third step, specify a group, a logical and display name for each function. If necessary, add a description.
  10. Click “Add”.
  11. After all the data has been entered and saved, publish the draft.

Function Group Setup

To set up a group:

  1. Make sure draft mode is enabled and draft is selected.
  2. Click the “Set groups” button located at the bottom of the function list.
  3. This action will open the “Set up groups” window.
  4. Rename the existing groups, if necessary.
  5. Add a new group if necessary. To do this:
    • Highlight the required nesting level of the groups. Level 1 group is added when you highlight the “Root” group.
    • Click the “Add” button.
    • Enter a name for the group.
  6. Save the changes: press the “Save” button.