z/OS Tools & Language

Expand all | Collapse all

miniconda's git and diskspace

  • 1.  miniconda's git and diskspace

    Posted 21 days ago

    Hi all,
    I am a bit concerned seeing how much diskspace git 2.26.2-78 uses.

    When I have installed miniconda the `~/miniconda` directory uses approx. 508MB

    Now after having installed git: `conda install -c zoss-appdev git` which includes


    ```
    curl: 7.66.0-1 zoss-appdev
    expat: 2.2.9-4 zoss-appdev
    git: 2.26.2-pl524_78 zoss-appdev
    libtag: 1.4-1 zoss-appdev
    perl: 5.24.4-4 zoss-appdev
    ```

    the `~/miniconda` directory uses approx 1.6GB disk space.

    Looking at the Rocket Ported Tools I find

    curl   202MB
    perl 128MB
    git 164MB

    If I sum this up I get approx 490MB.

    As I cannot see that expat and libtag utilize much diskpace I am asking myself why git 2.26.2-78 uses so much diskspace.

    Any idea what's going on here?

    Thanks a lot,
    Manfred



    ------------------------------
    Manfred Lotz
    IBM
    ------------------------------


  • 2.  RE: miniconda's git and diskspace

    Posted 18 days ago
    Hello Manfred,

    What tool do you use to track the used disk space? Some tools might give inaccurate results.

    When you install a package in miniconda, first it gets downloaded to the directory miniconda/pkgs, and unpacked there. Then, miniconda installs it into the environment directory by creating hard links to the unpacked files. Each file seems to be present in both pkgs and envs directories, though it's actually a single file object and it takes as much space as a single file does. However, if you just calculate the size of the miniconda directory, this hard-linking doubles the numbers you get. Also, please keep in mind that package tarballs are not deleted from the pkgs directory, and this takes some disk space, too.

    Regards,
    Vladimir

    ------------------------------
    Vladimir Ein
    Rocket Software
    ------------------------------



  • 3.  RE: miniconda's git and diskspace

    Posted 18 days ago

    Hi Vladimir,
    I use the z/OS UNIX du command to check disk space which handles hardlinks well.

    So, I dare to say that the sizes I gave are ok. 

    I just checked for hardlinks. There are not many hardlinks for files > 10m. Here how I looked

    find . -type f -size +10M -print0 | xargs -0 ls -li | sort


    --
    Manfred



    ------------------------------
    Manfred Lotz
    IBM
    ------------------------------



  • 4.  RE: miniconda's git and diskspace

    Posted 18 days ago
    Edited by Manfred Lotz 18 days ago
    Well, I have to admit that the following is better

    $find . -type f -size +5M -links +1 -ls
    44200 2652 -rwxrwxr-x 2 DEMNT15 STCGRP 5419008 Apr 14 18:02 ./pkgs/perl-5.24.4-4/lib/perl5/5.24.4/os390/CORE/libperl.so
    43597 5880 -rw-rw-r-- 2 DEMNT15 STCGRP 12027794 Jan 14 2020 ./pkgs/openssl-1.0.2k-5/lib/libcrypto.a
    58594 21448 -rwxrwxr-x 2 DEMNT15 STCGRP 43892736 Apr 1 2020 ./pkgs/curl-7.66.0-1/bin/curl
    58594 21448 -rwxrwxr-x 2 DEMNT15 STCGRP 43892736 Apr 1 2020 ./bin/curl
    44200 2652 -rwxrwxr-x 2 DEMNT15 STCGRP 5419008 Apr 14 18:02 ./lib/perl5/5.24.4/os390/CORE/libperl.so
    43597 5880 -rw-rw-r-- 2 DEMNT15 STCGRP 12027794 Jan 14 2020 ./lib/libcrypto.a




    -- 
    Manfred

    ------------------------------
    Manfred Lotz
    IBM
    ------------------------------



  • 5.  RE: miniconda's git and diskspace

    Posted 18 days ago
    Hi Manfred,

    Somehow executable files in the recent builds of Git are 2-3 times larger than they were in the version shipped via the Customer Portal. This appears to be the reason why Git has become that large. Thanks for reporting it - we're going to look into this as time permits.

    Thanks again,
    Vladimir

    ------------------------------
    Vladimir Ein
    Rocket Software
    ------------------------------