2014-02-11

Common Table for SchemaMapper and Dynamic Schema

Sometimes I have to read CSV table which doesn't have field names row and write the records into other format table after defining appropriate field names.

When there is not field names row, the CSV reader generates default attribute names. i.e. "col0, col1, col2 ...".
If there were few columns, it would be easy to rename them with the AttributeRenamer and to set User Attributes of the writer feature type manually. But if there were very many columns, it would be so troublesome. In fact I have encountered a case where there were 200+ columns. Naturally, I would consider using the SchemaMapper and Dynamic Schema option.

In such a case, I create a common table which can be used as Attributes / Feature Type Mapping table (for the SchemaMapper) and also Schema Source table (for Dynamic Schema of the writer feature type). Call it "Common Schema Table" here.

1. Common Schema Table Definition

The common schema table looks like this. Although only 3 attributes are shown here, imagine that there are much more attributes in fact.
DestFeatureSrcAttrDestAttrDataTypeOrderGeomTypeSrcFeature
outputcol0IDfme_decimal(4,0)1fme_no_geomdata
outputcol1Namefme_varchar(16)2
outputcol2Addressfme_varchar(16)3
...............

"DestFeature" (destination feature type name) and "DestAttr" (destination attribute names) are defined for both SchemaMapper and Dynamic Schema.
"SrcFeature" (source feature type name) and "SrcAttr" (source attribute names) are only for SchemaMapper.
Others are for Dynamic Schema. "Order" is optional.
Format of the table is arbitrary. I prefer to use CSV format.

2. SchemaMapper Settings

Insert a SchemaMapper between reader and writer on the workspace, specify the table to its Dataset and set Actions like this.
-----
Map Attributes: SrcAttr -> DestAttr
Map Feature Types: SrcFeature -> DestFeature
-----
The SchemaMapper will rename SrcAttr (col0, col1, col2 ...) to DestAttr (ID, Name, Address ...), and also change the feature type name "data" to "output".

3. Dynamic Schema Settings

Add a "Schema (From Table)" reader to read the common schema table as a Workspace Resource (Menu: Readers > Add Reader as Resource). Parameter settings are:
-----
Feature type: DestFeature
Attribute name: DestAttr
Attribute data type: DataType
Geometry type: GeomType
Attribute sequence: Order
-----
And then, specify the table to "Schema Sources" of the writer feature type.

To create the table takes time and effort, but the maintenance will be easy after once creating.
The method is an application of Example 3 and Example 4 in this article.
> FMEpedia: Dynamic Schema Examples
=====
2014-12-12: The link has been updated. See here instead.
> Dynamic Workflow Tutorial: Destination Schema is Derived from a Lookup Table
====
2014-12-15: Related article:
> Read Schemas from Database with Schema (Any Format) Reader
=====

No comments:

Post a Comment