2013-08-24

Japanese character problem

=====
2103-12-19: I confirmed that almost all the problems  related to Japanese characters has been solved in FME 2014 Beta Build 14220.
> Japanese Character Problems in FME 2014 Beta
=====

The Japanese Windows usually uses the code page 932 to process character strings. In this environment, a standard Japanese character is represented with two byte code.
There are some characters whose second byte is equal to 0x5c. Since 0x5c indicates a back slash (\) in ascii character code set, those characters could cause problem in string processing. So, some people often call them as だめ文字 (damemoji: dumb characters).

Almost all the damemoji are seldom used, but one of them - 表 (hyou) exceptionally very often occurs in FME use cases, because it means "table".
What is worse, FME currently doesn't work if a Reader / Writer Feature Type Name ends with a damemoji. In fact, there are many database tables, Excel worksheets, csv files named "****表" (foobar_table). We have to replace "表" with a non-damemoji character or remove it in such a case. This is the only workaround in the meantime.
It's a serious problem for the Japanese FME users.
Hope strongly that the problem is solved as soon as possible.

=====
2013-11-29: On Writers, it seems that this problem has been solved in FME 2013 SP2+.
Many thanks, Safe!

=====
2013-11-29: A friend asked me. "There should be many countries other than Japan using 2-byte character code set. Are there Damemoji in those countries?"
Probably, yes. According to "Shift_JIS" (Wikipedia Japanese version), GBK (Chinese, cp936) and Big5 (Traditional Chinese, cp950) also have some Damemoji - i.e. characters whose second byte is 0x5C.
But I have not heard that the problem similar to Shift JIS (Japanese, cp932) occurs in FME use cases. If there is an opportunity, I would like to discuss with Chinese FME users about this problem.

Next: Japanese Character Problems in FME 2014 Beta

Find the shortest path visiting all the given points

If the order of visiting points is defined, these transformers can be used to do that.
PointConnector: create a line connecting the points in the order of visiting.
Chopper: split the line into individual line segments.
ShortestPathFinder: find shortest paths for all the segments (FROM-TO lines).
* FME 2013+

-----
2013-09-18
It's not necessary to chop the line into individual line segments.
ShortestPathFinder help:
"It can contain intermediate stops before the final destination. For example, a FROM-TO line may be used to find the path from A to B to C to D. This can also be read as “the path from A to D that also passes through B and C.”
-----

It's not so difficult, I recently created a practical workspace based on this approach for an actual project.
But when the the order of visiting is unknown - in other words, when you need to find the optimal route which visits all the given points, it will be the famous "Travelling salesman problem". Several approximation algorithms have been invented to solve the problem, however, FME currently doesn't support them.
> Community: Shortest path along a network between multiple points

An Example of Copying Files

For example, there are these files in my computer disk system:
C:\tmp\a\001.txt
C:\tmp\a\002.txt
C:\tmp\b\001.txt
C:\tmp\b\002.txt

and I want to copy them to another directory with renaming like this:
C:\tmp\folder\a_001.txt
C:\tmp\folder\a_002.txt
C:\tmp\folder\b_001.txt
C:\tmp\folder\b_002.txt

Directory and File Pathnames Reader and File Copy Writer are very useful to do such a operation.

Directory and File Pathnames Reader retrieves properties of every file in the specified directories, creates non-spatial features having attributes named "path_****". Each attribute stores a file property - file full path, directory path, file name, extension etc.
Source Directory and File Pathnames Directory: C:\tmp\a,C:\tmp\b

FilenamePartExtractor extracts parts of the file path.
Source File Name: path_windows
Full Filename Attribute: _filename
Directory Name Attribute: _dirname

AttributeCreator renames the attribute storing source file path, and creates the destination filename as a new attribute. "filecopy_****" is a format attribute of the File Copy Writer.
Attribute Name <- Value
filecopy_source_dataset <- path_windows
filecopy_dest_filename <- @Value(_dirname)_@Value(_filename)

File Copy Writer copies "filecopy_source_dataset" to "filecopy_dest_filename" in the specified directory.
Destination File Copy Directory: C:\tmp\folder

This is likely to be applicable in various scenes.

2013-08-23

Can ArcGIS Data Interoperability run existing FME workspaces?

I finally found a clear answer to this question.

> FME & Esri ArcGIS Data Interoperability FAQ
Can ArcGIS Data Interoperability run my existing FME workspaces?
No. ArcGIS Data Interoperability cannot run existing FME workspaces. If an FME user has workspaces, they will have to recreate the functionality of their existing workspaces within ArcGIS. This may or may not be successful, as the ArcGIS Data Interoperability extension doesn’t contain full FME functionality (see next question).
Note: It is possible to copy and paste transformers and data models between workspaces made with FME into custom tools made with Workbench inside of ArcGIS.

So, when I need to transplant an existing workspace to ArcGIS Data Interoperability, I have to re-create that in the ArcGIS Interoperability environment or copy & paste all the items of the workspace. However it may not be successfull because there are formats or transformers unsupported by ArcGIS Interoperability.

Associated links:
http://www.safe.com/solutions/for-applications/esri-arcgis/formats/
http://www.safe.com/solutions/for-applications/esri-arcgis/transformers/
http://www.safe.com/solutions/for-applications/esri-arcgis/data-interop-comparison/