Skip to main content

miniconda's git and diskspace

  • October 2, 2020
  • 4 replies
  • 0 views
A
  • Anonymous

Hi all,
I am a bit concerned seeing how much diskspace git 2.26.2-78 uses.

When I have installed miniconda the `~/miniconda` directory uses approx. 508MB

Now after having installed git: `conda install -c zoss-appdev git` which includes


```
curl: 7.66.0-1 zoss-appdev
expat: 2.2.9-4 zoss-appdev
git: 2.26.2-pl524_78 zoss-appdev
libtag: 1.4-1 zoss-appdev
perl: 5.24.4-4 zoss-appdev
```

the `~/miniconda` directory uses approx 1.6GB disk space.

Looking at the Rocket Ported Tools I find

curl   202MB
perl 128MB
git 164MB

If I sum this up I get approx 490MB.

As I cannot see that expat and libtag utilize much diskpace I am asking myself why git 2.26.2-78 uses so much diskspace.

Any idea what's going on here?

Thanks a lot,
Manfred



------------------------------
Manfred Lotz
IBM
------------------------------

    4 replies

    Vladimir Ein
    Forum|alt.badge.img+1
    • Vladimir EinRocketeer
    • Rocketeer
    • 110 replies
    • October 5, 2020

    Hi all,
    I am a bit concerned seeing how much diskspace git 2.26.2-78 uses.

    When I have installed miniconda the `~/miniconda` directory uses approx. 508MB

    Now after having installed git: `conda install -c zoss-appdev git` which includes


    ```
    curl: 7.66.0-1 zoss-appdev
    expat: 2.2.9-4 zoss-appdev
    git: 2.26.2-pl524_78 zoss-appdev
    libtag: 1.4-1 zoss-appdev
    perl: 5.24.4-4 zoss-appdev
    ```

    the `~/miniconda` directory uses approx 1.6GB disk space.

    Looking at the Rocket Ported Tools I find

    curl   202MB
    perl 128MB
    git 164MB

    If I sum this up I get approx 490MB.

    As I cannot see that expat and libtag utilize much diskpace I am asking myself why git 2.26.2-78 uses so much diskspace.

    Any idea what's going on here?

    Thanks a lot,
    Manfred



    ------------------------------
    Manfred Lotz
    IBM
    ------------------------------
    Hello Manfred,

    What tool do you use to track the used disk space? Some tools might give inaccurate results.

    When you install a package in miniconda, first it gets downloaded to the directory miniconda/pkgs, and unpacked there. Then, miniconda installs it into the environment directory by creating hard links to the unpacked files. Each file seems to be present in both pkgs and envs directories, though it's actually a single file object and it takes as much space as a single file does. However, if you just calculate the size of the miniconda directory, this hard-linking doubles the numbers you get. Also, please keep in mind that package tarballs are not deleted from the pkgs directory, and this takes some disk space, too.

    Regards,
    Vladimir

    ------------------------------
    Vladimir Ein
    Rocket Software
    ------------------------------
    A
    • Anonymous
    • 0 replies
    • October 5, 2020
    Hello Manfred,

    What tool do you use to track the used disk space? Some tools might give inaccurate results.

    When you install a package in miniconda, first it gets downloaded to the directory miniconda/pkgs, and unpacked there. Then, miniconda installs it into the environment directory by creating hard links to the unpacked files. Each file seems to be present in both pkgs and envs directories, though it's actually a single file object and it takes as much space as a single file does. However, if you just calculate the size of the miniconda directory, this hard-linking doubles the numbers you get. Also, please keep in mind that package tarballs are not deleted from the pkgs directory, and this takes some disk space, too.

    Regards,
    Vladimir

    ------------------------------
    Vladimir Ein
    Rocket Software
    ------------------------------

    Hi Vladimir,
    I use the z/OS UNIX du command to check disk space which handles hardlinks well.

    So, I dare to say that the sizes I gave are ok. 

    I just checked for hardlinks. There are not many hardlinks for files > 10m. Here how I looked

    find . -type f -size +10M -print0 | xargs -0 ls -li | sort


    --
    Manfred



    ------------------------------
    Manfred Lotz
    IBM
    ------------------------------
    A
    • Anonymous
    • 0 replies
    • October 5, 2020

    Hi Vladimir,
    I use the z/OS UNIX du command to check disk space which handles hardlinks well.

    So, I dare to say that the sizes I gave are ok. 

    I just checked for hardlinks. There are not many hardlinks for files > 10m. Here how I looked

    find . -type f -size +10M -print0 | xargs -0 ls -li | sort


    --
    Manfred



    ------------------------------
    Manfred Lotz
    IBM
    ------------------------------
    Well, I have to admit that the following is better

    $find . -type f -size +5M -links +1 -ls
    44200 2652 -rwxrwxr-x 2 DEMNT15 STCGRP 5419008 Apr 14 18:02 ./pkgs/perl-5.24.4-4/lib/perl5/5.24.4/os390/CORE/libperl.so
    43597 5880 -rw-rw-r-- 2 DEMNT15 STCGRP 12027794 Jan 14 2020 ./pkgs/openssl-1.0.2k-5/lib/libcrypto.a
    58594 21448 -rwxrwxr-x 2 DEMNT15 STCGRP 43892736 Apr 1 2020 ./pkgs/curl-7.66.0-1/bin/curl
    58594 21448 -rwxrwxr-x 2 DEMNT15 STCGRP 43892736 Apr 1 2020 ./bin/curl
    44200 2652 -rwxrwxr-x 2 DEMNT15 STCGRP 5419008 Apr 14 18:02 ./lib/perl5/5.24.4/os390/CORE/libperl.so
    43597 5880 -rw-rw-r-- 2 DEMNT15 STCGRP 12027794 Jan 14 2020 ./lib/libcrypto.a




    -- 
    Manfred

    ------------------------------
    Manfred Lotz
    IBM
    ------------------------------
    Vladimir Ein
    Forum|alt.badge.img+1
    • Vladimir EinRocketeer
    • Rocketeer
    • 110 replies
    • October 5, 2020
    Well, I have to admit that the following is better

    $find . -type f -size +5M -links +1 -ls
    44200 2652 -rwxrwxr-x 2 DEMNT15 STCGRP 5419008 Apr 14 18:02 ./pkgs/perl-5.24.4-4/lib/perl5/5.24.4/os390/CORE/libperl.so
    43597 5880 -rw-rw-r-- 2 DEMNT15 STCGRP 12027794 Jan 14 2020 ./pkgs/openssl-1.0.2k-5/lib/libcrypto.a
    58594 21448 -rwxrwxr-x 2 DEMNT15 STCGRP 43892736 Apr 1 2020 ./pkgs/curl-7.66.0-1/bin/curl
    58594 21448 -rwxrwxr-x 2 DEMNT15 STCGRP 43892736 Apr 1 2020 ./bin/curl
    44200 2652 -rwxrwxr-x 2 DEMNT15 STCGRP 5419008 Apr 14 18:02 ./lib/perl5/5.24.4/os390/CORE/libperl.so
    43597 5880 -rw-rw-r-- 2 DEMNT15 STCGRP 12027794 Jan 14 2020 ./lib/libcrypto.a




    -- 
    Manfred

    ------------------------------
    Manfred Lotz
    IBM
    ------------------------------
    Hi Manfred,

    Somehow executable files in the recent builds of Git are 2-3 times larger than they were in the version shipped via the Customer Portal. This appears to be the reason why Git has become that large. Thanks for reporting it - we're going to look into this as time permits.

    Thanks again,
    Vladimir

    ------------------------------
    Vladimir Ein
    Rocket Software
    ------------------------------


    Terms & ConditionsAccessibility statement