Open-source Languages & Tools for z/OS

View Only

vim - untagged files forced to ISO8859-1

Andy Styles posted 03-29-2022 04:35

Hi all,

We have a version of vim from several years ago, which was obtained from the download site as a .tgz and installed in the pre-conda days. We've been happily using this on a daily basis for some time, but recently there has been some more interest in some of the other tools such as Python.

I've now installed all the free packages from the anaconda site via conda, which obviously includes the latest vim.

I've noticed a difference in behaviour of vim between the two packages. The original vim was compiled in 2017:

VIM - Vi IMproved 8.0 (2016 Sep 12, compiled Jun 7 2017 07:22:47)
Included patches: 1-22
Compiled by PDKUZM@RS12.rocketsoftware.com
Big version without GUI. Features included (+) or not (-):
+acl +cmdline_hist +ebcdic -gettext +listcmds +mouse_sgr +persistent_undo +smartindent +textobjects +wildmenu
-arabic +cmdline_info +emacs_tags -hangul_input +localmap -mouse_sysmouse +postscript +startuptime +timers +windows
+autocmd +comments +eval -iconv -lua +mouse_urxvt +printer +statusline +title +writebackup
-balloon_eval +conceal +ex_extra +insert_expand +menu +mouse_xterm -profile -sun_workshop -toolbar -X11
-browse +cryptv +extra_search +job +mksession -multi_byte -python +syntax +user_commands -xfontset
++builtin_terms +cscope -farsi +jumplist +modify_fname +multi_lang -python3 -tag_binary +vertsplit -xim
+byte_offset +cursorbind +file_in_path +keymap +mouse -mzscheme +quickfix +tag_old_static +virtualedit -xpm
+channel +cursorshape +find_in_path +lambda -mouseshape +netbeans_intg +reltime -tag_any_white +visual -xsmp
+cindent +dialog_con +float +langmap +mouse_dec +num64 -rightleft -tcl +visualextra -xterm_clipboard
-clientserver +diff +folding +libcall -mouse_gpm +packages -ruby +termguicolors +viminfo -xterm_save
-clipboard +digraphs -footer +linebreak -mouse_jsbterm +path_extra +scrollbind +terminfo +vreplace
+cmdline_compl -dnd +fork() +lispindent +mouse_netterm -perl +signs +termresponse +wildignore
system vimrc file: "$VIM/vimrc"
user vimrc file: "$HOME/.vimrc"
2nd user vimrc file: "~/.vim/vimrc"
user exrc file: "$HOME/.exrc"
defaults file: "$VIMRUNTIME/defaults.vim"
fall-back for $VIM: "/rsusr/ported/share/vim"
Compilation: c99 -c -I. -Iproto -DHAVE_CONFIG_H -D_ALL_SOURCE -D_EXT -D_UNIX03_SOURCE -D_XOPEN_SOURCE_EXTENDED=1 -D_ISOC99_SOURCE -D_OE_SOCKETS -D_OPEN_MSGQ_EXT -D_OPEN_SY
S -D_OPEN_THREADS -D_POSIX_SOURCE -D_SHARE_EXT_VARS -D_UNIX03_SOURCE -D_UNIX03_WITHDRAWN -D_XOPEN_SOURCE=600 -D_OPEN_SYS_FILE_EXT=1 -O -qlanglvl=extended:extc89:extc99 -qxp
link -qdll -qfloat=ieee -qlongname -qenum=int -qhaltonmsg=3296:4108
Linking: c99 -O -qxplink -qdll -o vim -lm -lcurses

The one just installed:

VIM - Vi IMproved 8.0 (2016 Sep 12, compiled Mar 18 2020 12:17:08)
Included patches: 1-22
Compiled by TSPORT@RS12.rocketsoftware.com
Big version without GUI. Features included (+) or not (-):
+acl +cmdline_hist -ebcdic -gettext +listcmds +mouse_sgr +persistent_undo +smartindent +textobjects +wildmenu
+arabic +cmdline_info +emacs_tags -hangul_input +localmap -mouse_sysmouse +postscript +startuptime +timers +windows
+autocmd +comments +eval +iconv -lua +mouse_urxvt +printer +statusline +title +writebackup
-balloon_eval +conceal +ex_extra +insert_expand +menu +mouse_xterm -profile -sun_workshop -toolbar -X11
-browse +cryptv +extra_search -job +mksession +multi_byte -python +syntax +user_commands -xfontset
++builtin_terms +cscope +farsi +jumplist +modify_fname +multi_lang -python3 +tag_binary +vertsplit -xim
+byte_offset +cursorbind +file_in_path +keymap +mouse -mzscheme +quickfix +tag_old_static +virtualedit -xpm
-channel +cursorshape +find_in_path +lambda -mouseshape -netbeans_intg +reltime -tag_any_white +visual -xsmp
+cindent +dialog_con +float +langmap +mouse_dec +num64 +rightleft -tcl +visualextra -xterm_clipboard
-clientserver +diff +folding +libcall -mouse_gpm +packages -ruby +termguicolors +viminfo -xterm_save
-clipboard +digraphs -footer +linebreak -mouse_jsbterm +path_extra +scrollbind +terminfo +vreplace
+cmdline_compl -dnd +fork() +lispindent +mouse_netterm -perl +signs +termresponse +wildignore
system vimrc file: "$VIM/vimrc"
user vimrc file: "$HOME/.vimrc"
2nd user vimrc file: "~/.vim/vimrc"
user exrc file: "$HOME/.exrc"
defaults file: "$VIMRUNTIME/defaults.vim"
fall-back for $VIM: "/SERVICE/rocket/envs/lbg-dev/share/vim"
Compilation: xlc_echocmd -c -I. -Iproto -DHAVE_CONFIG_H -D_XOPEN_SOURCE=600 -D_OPEN_SYS_SOCK_IPV6=1 -D_ENHANCED_ASCII_EXT=0xFFFFFFFF -D_OPEN_SYS_FILE_EXT=1 -D_TR1_C99 -I/S
ERVICE/rocket/envs/lbg-dev/include -qdll -qexportall -qascii -q64 -qnocse -qgonum -qasm -qbitfield=signed -qenum=int -qhalton=3296:3304:3950 -qtarget=zosv2r1 -qarch=10 -qtu
ne=12 -O2 -qstrict -qfloat=ieee:nomaf -qlanglvl=extc1x
Linking: xlc_echocmd -q64 -L/SERVICE/rocket/envs/lbg-dev/lib -o vim -lm -ltag -lncurses

so it would appear the codebase for vim hasn't changed, though the compile/link options have.

Anyway onto the difference. What I'm seeing is that if I use the old vim on untagged files, when saved, they remain in their original format. With the newer vim, untagged files are forced into ISO8859-1; this 'corrupts' anything untagged - for example, if I edit my .profile which is untagged and thus in IBM-1047 and save it with the new vim, it's force converted to ISO8859-1 and then I get errors at login.

Example:

$ touch oldvim.txt
$ touch newvim.txt
$ ls -laT *vim.txt
- untagged T=off -rw-r--r-- 1 XXXXXXX XXXXXXXX 0 Mar 29 09:17 newvim.txt
- untagged T=off -rw-r--r-- 1 XXXXXXX XXXXXXXX 0 Mar 29 09:17 oldvim.txt

$ vim --version
VIM - Vi IMproved 8.0 (2016 Sep 12, compiled Jun 7 2017 07:22:47) [etc etc etc]

[vim edit session, adding one line of text and :wq]
"oldvim.txt" 1L, 15C written

$ ls -laT *vim.txt
- untagged T=off -rw-r--r-- 1 XXXXXXX XXXXXXXX  0 Mar 29 09:17 newvim.txt
- untagged T=off -rw-r--r-- 1 XXXXXXX XXXXXXXX 15 Mar 29 09:20 oldvim.txt

[switch to newer vim]

$ vim --version
VIM - Vi IMproved 8.0 (2016 Sep 12, compiled Mar 18 2020 12:17:08) [etc etc etc]

[vim edit session, adding one line of text and :wq]
"newvim.txt" 1L, 15C written

$ ls -laT *vim.txt
t ISO8859-1 T=on -rw-r--r-- 1 XXXXXXX XXXXXXXX 15 Mar 29 09:23 newvim.txt
- untagged T=off -rw-r--r-- 1 XXXXXXX XXXXXXXX 15 Mar 29 09:20 oldvim.txt

I have the following environmental variables set both times:

_BPXK_AUTOCVT=ON
_CEE_RUNOPTS='FILETAG(AUTOCVT,AUTOTAG) POSIX(ON)'

Can anyone offer insight as this is not the behaviour I would expect or want?

Many thanks,

Andy.

Tom Longfellow posted 03-30-2022 09:20

Not an answer actually - just a line of thought based upon my prior experience with the anaconda delivered packages.

I got burned by the version of sed that was installed by one of the new packages. It had AUTOCVT type issues that made previously working tasks fail.

I access the rocket code via symbolic links from /usr/bin to the Rocket/bin directory. To correct the problem, I removed the sed link from my /usr/bin.

In this new 'code once - run anywhere' world. Development platform arrogance can burn you in the 'tagging' environment. Target ported environments may have special quirks not handled in the development environment. I guess special steps have to be taken in the code to be aware of what an untagged file really means, but I am just a systems person, not a developer.

ROCKETEER Vladimir Kudriakov posted 03-30-2022 14:07

Hello Andy,

Source code of vim was changed a little bit between those two builds. It was changed in the way how vim works with tagged/untagged files in z/OS. And the recent build of vim (compiled Mar 18 2020), installed via the Conda, always saves modified content of an untagged file in ASCII encoding and sets file tag to ISO8859-1.

Agree that it is not the expected behavior, and it seems we should change such behavior in the next ported version of vim.

I would like to add that the recent build of vim actually does NOT corrupt the content of an untagged file after updating and saving it. It just converts the content's encoding from EBCDIC to ASCII if that content was in EBCDIC encoding. So all z/OS utilities, as ported and native, should read such updated file correctly as it has correct tag.

The only way to avoid such unexpected behavior with the recent build of vim is to avoid editing of untagged files. It's worth to tag files as much as possible.

Regards,
Vladimir

Andrew Rowley posted 03-30-2022 18:45

"the recent build of vim actually does NOT corrupt the content of an untagged file after updating and saving it. It just converts the content's encoding from EBCDIC to ASCII if that content was in EBCDIC encoding"

There is no guaranteed round-trip translation between EBCDIC and ASCII so unintended translation should definitely be considered data corruption. (Even intended translation is risky.)

"So all z/OS utilities, as ported and native, should read such updated file correctly as it has correct tag."

I think there are many functions on z/OS that will not check for a tag when reading the file. I suspect the ones that do not check would be a majority. I'm not sure of the specifics of file tagging and automatic translation, but my understanding is that it requires writing in C and compilation using specific compiler options. Is that correct?

Andy Styles posted 03-31-2022 03:49

In response to Vladimir:

My intention was not to say that it corrupted the file - I used quotes around the word corrupt to indicate that it could be considered corruption if the program reading the file did not understand the converted content; that's certainly the case here with things like .profile - and I've had similar issues editing REXX programs that run under cron.

I've also noticed that if I display the hex values of the character under the cursor in vim (normal mode ga command), the values are reported always in ASCII (or ISO8859-1 if you prefer), regardless of the incoming file format; again, this seems wrong - and I would hope your indication that this might be changed to the previous behaviour would also fix that.

I know this is all unpaid support and there's no driver (other than doing the right thing) to correct it, but wondered if you have any timescales for an updated build? As it stands, the free conda install of vim is dangerous to use (I can't speak for the paid-for support version).

I also note that the github source for vim is now many patches ahead of the one we have on z/OS - are there any plans to bring the z/OS one up to date?

Many thanks,

Andy

Gary Grossi posted 03-31-2022 10:28

Hello,
I'm not sure if this will help. For vim, we are using this environment variable as a default to tag a new file or to allow vim to edit an untagged file.
thanks

export _ENCODE_FILE_NEW=IBM-1047

Andy Styles posted 04-01-2022 05:20

Thanks Gary; I did spot that environmental variable (and its cousins), but I couldn't find any official documentation around it - all the hits on Google seem to turn up discussion threads, rather than proper documentation.

I would like to have seen some release notes for these tools - in the share/vim/vim80/doc directory, there is an os_390.txt file, but it's from the era when IBM distributed the ported tools, and it's woefully out of date (as may be guessed from the filename!).

Mike Großmann posted 07-04-2022 02:07

Hello,

I have the same problem.
We are using NFS mounts, a lot.
Using VIM or SED on any file on a NFS mount
are basically destroying these files.
These files are untagged ASCII files (NFS do not support tagging).
VIM is converting these files from 1047 to ascii, on save.

Thats a very strange source change. SED has such issue, too.

Best, Mike

ROCKETEER Alexander Klochkov posted 07-11-2022 09:04

Hi Mike Großmann,

What versions of vim and sed are you using? I just checked vim-8.0.0022-0 and sed-4.2.2-12 from conda server, they are ok for your case.

Thanks,
Alexander

Open-source Languages & Tools for z/OS

vim - untagged files forced to ISO8859-1

Contact Us

Quick Links