A COBOL program is making two calls to a C program and passing a PIC X(20) string as a parameter.  The C program moves a value to this parameter beffore returning to COBOL.

I found that when I perform two calls to the C program, and the second string returned by C is smaller than the first one, COBOL does not recognize the 0x0 character and shows the two strings overlapped.to a C subprogram.

For example:

The first call returns 'file11111.txt', and the second 'file2.txt'.
What COBOL sees is 'file2.txt txt'.

Why is this?

This is to be expected as a NULL terminated string is a C construct and not a COBOL one.  In COBOL a PIC X data item can contain any character including a X"00" (NULL).

In order to get the expected results in COBOL you should either initialize the field to spaces before calling the C program or extract the data from the field after the call using something like:

    unstring p_szFile delimited by x"00"
       into p_szFile

This has the effect of removing the NULL byte and anything following it.