Hello Shulian:

Visual COBOL was not tested under AIX 5.3.  It was tested and certified only under AIX 6.1 and AIX 7.1.  It may work anyway, but the chances are not good.

Here is a Knowledgebase article describing the file $COBDIR/docs/env.txt, and the general subject of supported versus unsupported OS versions:

community.microfocus.com/.../21405.what-unixlinux-version-is-supported-by-this-release-of-micro-focus-cobol-will-the-product-run-on-unsupported-unixlinux-versions.aspx

It is worth noting here that IBM themselves no longer support AIX 5.3.  IBM has withdrawn AIX 5.3 from mainstream support.