Source control for UniVerse

I have used CVS, SVN, Git, and Team Foundation. The problem is that Universe/Unidata do not lend themselves to version control because of their hashed files. What you need is a tool that can convert U2 files to flat files so that version control can access them. That means you can write scripts for Windows or Unix systems that transmit the data flattened out or use a tool that does this automatically. An open-source tool I have been using for a number of years is EclipseIDE. It works great with a Diff editor and many more features built into EclipseIDE.

------------------------------
Doug Averch
Owner
U2 Logic
------------------------------

Hi,

SJ+ used to have a package for source code control, though I am not familiar with the current status. See https://sjplus.com/why-prc/

Regards

JJ

------------------------------
John Jenkins
Thame, Oxfordshire
------------------------------

Hi Doc,

Welcome to this little world we call MultiValue.  PRC is a complete, flexible, mature and well-supported source control /IT Governance tool written in and for the MultiValue platform, including Unidata, Universe, SB+ and more.

Email, call, schedule a call or web meet?

https://sjplus.com

Hi John,

PRC and I are still doing our thing over here. Thanks for the mention. Hope things are good for you.

Susan

Revelation software has a GIT interface which you can use to manage code in Universe via their MVBFS which attaches Universe tables.   Here is a screen shot showing a program that resides in Universe being edited with OpenInsight's full text editor.  You can also use it to create very nice Windows desktop applications.  In this particular example GIT is not enabled but you can see the menu options.....

Hi Doc!

  We use GitHub. Combining it with the Rocket MV Basic extension for Visual Studio Code, you obtain an excellent coding and source control environment with Visual Studio Code, that allows for code completion for BASIC and you can even compile and catalog from VS Code.

Kind Regards

------------------------------
Enrique Ignacio Murphy
Software Engineer
Aleator Software - RN40 Software
Argentina
------------------------------

EclipseIDE allows for source code control of:

1. Procs
2. Paragraphs
3. VOC items
4. Dictionaries
5. Data
6. Any other miscellaneous files you need.

Surprisingly, we use data all of the time for source code control. I cannot tell you how many times our clients have deleted source code files that are required for the system to run.

------------------------------
Doug Averch
Owner
U2 Logic
------------------------------

Hi and welcome Doc,

I'd recommend Git and VSCode as the floor for SCM. I've used a few different source control systems and Git is it. The VSCode IDE makes it easy to use at both a basic level and in more sophisticated scenarios using tagging, versioning and branching. Their ubiquity mean that training, support and utilities are plentiful.

In general, whichever SCM system you choose will influence your DevOps workflow and you need to consider how you want to approach that and what level of maturity you want to achieve.

If you haven't already, consider what artifacts are part of the source that needs to be managed. Starting with mvBasic code which is probably already in OS files (Dir, Type19, etc) should be easy to start managing. Other source will include dictionaries, VOC entries, paragraphs, procs, scripts, configs, Java, Python code, etc.  Some of these may need a process to manage moving records out and in between hashed files <-> text artifacts.  You should also consider what not to manage. Exclude things like logs and artifacts provided by other systems (depending on the maturity level you aim for, note how you will manage these dependencies if you need to). Avoid storing BLOBS like raw hashed files.

If you choose Git you will probably define a "remote" repository for distributed/shared SCM. This can be on a server of your choice, either with or without installing a workflow platform (GitLab, GitHub, etc), or you can use a commercial workflow platform like GitHub or a cloud vendor's (Azure, AWS, etc). Of those, I recommend having a look at the Azure DevOps platform as the remote repo and using the workflow tools that come with it.

There is no major issues with using Git or any other source control for MV.  Your two big items are Encoding and MV hashed files.  We have been working on a best practices document that is based on feedback from many customers already doing.  Here are a few items until that is available.

1. Basic code is the easiest since most MV systems can use directories.
2. Non Basic items, such as Proc's, dictionaries, MD/Voc items, your own items.  If these have to be in MV hash files then you need to move them out for source control.  One recommendation is to create your own scripts for a branch, kind of like makefiles and as you are working with items add them to the script.  The script should be able to copy items in and out of the MV files.  This is not a uncommon process, for example mysql based systems may have to move schema's in and out, create tables, populate tables, etc.  No different for MV.  Use this same script for DB activities also, such as creating files, indexes, etc.  You can then run the export script to push changes back into your Git directories and when other users are deploying, they can use the import script to push changes into your mv files and do db/other activities.  These scripts should also include compile catalog statements.  Come up with a standard naming concept for these "makefiles" and the normal Diff engines in Git will also be helpful on differences between branches.
3. CodePage.  If you are dealing with MV items that have Attribute/Value/Subvalue characters a easy way around this is to use one of the Windows-1252 code pages vs UTF-8.  Those characters will work just fine and git version controlling will work and based on the language you choose they will typically show up as umlat u and y characters
4. Don't forget about your application meta items.  For example if you are using a 4gl with meta data driving screens you should include those.
5. For very tough items to diff you may want to ETL them to something else like YAML to allow Git more easily diff them.
6. Be careful just pushing large numbers of DB items out to git.  For example, if your project was pushing a new file that needed to be populated it is better to make that file a cvs/pipe delimited import/export vs creating a directory in git and just copying the items out.  MV will do that but Git performance can fall off with large number of "items" in directories.  
7. Watch your new line endings.  Based on your MV host being Windows or Linux you will want to make sure Git is not messing this up.  There are configuration options for git to enforce what your server wants.

Creating MV helper scripts will help with assistance in these areas, but I recommend starting with doing this manually until you are comfortable and then start automating items with scripts and triggers.  

Examples

1. Creating a MV helper script to assist in copying items in and out. 
2. Import/Export functions if dealing with DB file items if you need CSV/PIPE files
3. Triggers to assist you on MV changes that need the running of the export functions to get the changes into the Git directories (exporting items)

Hi, 

Following up on what the others have said... I've used various source control systems over the years and in modern times helped a number of clients 'go GIT'. 

As Patrick pointed out, the main issue with GIT is that it is based on a merge model, and so your assets need to be in text format. That's the obvious stuff like programs, paragraphs etc. but also things like parameter/control items and your file layouts. The moment you do a SELECT statement inside your code, your files, dictionaries and indices all become part of your source code and you need a way to manage those sensibly using some form of DDL. (There are free tools on my site to help).

The other issue is around segregating commits. Unlike earlier source control systems, git uses a two stage commit where you lock out and commit individual code items, each GIT commit should be logically complete and consistent representing an item of work, especially if you are triggering devops operations from a push or a pull request. That's easy if you are a solo developer or everyone is working on different pieces of code that don't depend on one another, but one of the advantages of GIT is to enable parallel development so you don't have to lock out shared code. The flip side is how to manage that if all your developers share the same repository.

The gold standard is to have all developers working in their own refreshable accounts, each with a separate repository, and to pull and merge your work - that is what GIT is designed for (distributed version control). That makes for a more complex workload AND means you need to sometimes handle merge conflicts and deal with how GIT chooses to merge your changes... which means more testing to make sure it hasn't messed stuff up. That's where automated tests come to the fore to identify broken builds.

That said, GIT is incredibly powerful with many, many different workflows you can read up on the web, and just because it CAN do all those things, doesn't mean you should! You can start really simple with your repository separate from your code base and have whatever you currently use for delivery (I hope you do) update the repository with the selected items you are promoting, before looking at more complex scenarios.

Happy to talk these things through.