Technology Sharing

한어Русский языкEnglishFrançaisIndonesianSanskrit日本語DeutschPortuguêsΕλληνικάespañolItalianoSuomalainenLatina

Anaconda virtual environment packaging

The experiment was successfully verified on this machine and needs to be migrated to advanced network devices. For a single 4090D, it took two and a half hours to run a 128k multimodal large model for inference and evaluation on a data set. This is still too time-consuming, and the important task needs to be handed over to the 8A100 machine.

To migrate the anaconda environment from the source computer to the target computer, first install anaconda on both computers, and then on this basis.

1. Environment packaging of source computer

Using Anaconda Prompt

1. Install conda-pack tool

conda install conda-pack
1

2. Determine the environment

conda env list
1

Find the environment name you want to package. For example, my environment name is VLM

3. Packaging environment

conda pack -n VLM -o VLM.tar.gz
1

4. Copy the packaged environment to the USB drive

The Windows environment is packaged in the current directory, such as XX is the user name and the path is
C:/Users/XX/VLM.tar.gz
The same applies to Linux

2. Migrate the environment to the target computer

①Method 1
1. Use Anaconda Prompt to create a new environment, for example, the new environment is py310:

conda create -n py310 python=3.10
1

2. Copy the packaged environment in the USB drive to the username directory of the target computer:
C:/Users/XX/Py36.tar.gz
3. Use Anaconda Prompt to unzip the packaged environment to the new environment path:

tar -zxvf Py36.tar.gz -C E:anaconda3envspy36
1

4. Activate the new environment

conda activate py310
1

Note: If this fails, try method 2:
②Method 2
1. Find the environment folder of the target computer
The path of the environment is the envs folder under the directory where anaconda is located. For example, mine is: E:anaconda3envs. The following steps are all performed under this path.
2. Copy the packaged environment in the USB drive to the environment folder envs.
3. Use the cmd command line in the environment path to create a new folder and name it py36:

mkdir py36
1

Or manually create a new folder
4. Use the cmd command line to decompress the packaged environment into the new environment

tar -zxvf Py36.tar.gz -C ./py36
1

3. Exception handling

pip install -e. causes pack to fail → ignore

The error is as follows

Collecting packages...
CondaPackError: Cannot pack an environment with editable packages
installed (e.g. from `python setup.py develop` or
 `pip install -e`). Editable packages found:

- /home/xxx
1
2
3
4
5
6

Use the following method instead

conda pack -n VLM -o VLM.tar.gz --ignore-editable-packages
1

Decompress using

mkdir ~/ananconda/envs/pcdet-tmp
tar -xf pcdet.tar.gz -C ~/ananconda/envs/pcdet-tmp
cd ~/ananconda/envs/pcdet-tmp
source ./bin/activate
1
2
3
4

The managed file has been deleted or overwritten → compressed into tar

CondaPackError:  
	Files managed by conda were found to have been deleted/overwritten in the following packages: 
		 - charset-normalizer 2.0.4:
		 - xxxx
This is usually due to `pip` uninstalling or clobbering conda managed files,
resulting in an inconsistent environment. Please check your environment for
conda/pip conflicts using `conda list`, and fix the environment by ensuring
only one version of each package is installed (conda preferred).
1
2
3
4
5
6
7
8

The solution tried to take advantage of WSL to package the environment env in anaconda directly under Windows, but a huge number of dependency errors and naming errors were reported.

Finally, I stopped and tried to package and compress it under Linux system to see if it was feasible.
In Linux, you can use the tar command to compress files or folders into .tar.gz format.
tar is a command used to package files.
-c creates a new archive file.
-z means use gzip compression.
-v means to display detailed information while archiving.
-f is followed by the name of the archive file.

For example, if you want to compress a folder named VLM under env in anaconda, you can use the following command:

tar -czvf VLM.tar.gz VLM
1

Notice

在Linux里去看位置，anaconda是不显示完整文件目录的，会直接不显示有env文件夹，但是可以直接cd 进env去看
1

If you want to compress a single file, such as myfile.txt, you can use:

tar -czvf myfile.tar.gz myfile.txt
1

These commands will create a .tar.gz file in the current directory. You can change the file path or specify the output directory if needed.

Compression success

Reactivate the environment

3. Step 3: Activate the environment
For tar compression, explain the compression execution

tar -xzvf VLM.tar.gz
1

If the .zip file is copied to another system and unzipped to the directory /xxxx/anaconda/envs/, then

unzip -d /xxxx/anaconda/envs/  pyenv.zip
1

Note! The packaged virtual environment records the source environment. The anaconda installation path on the new system may be different from the source environment. For example, the source environment is /xxxx/anaconda2023/, and the new system is /xxxx/anaconda/. In this case, you need to create an empty directory on the new system with the same anaconda installation path as the source environment, such as the original path wsl.localhostUbuntuhometestanaconda3envsVLM
mkdir /xxxx/anaconda3/envs/
Then create a soft link
ln -s /xxxx/anaconda/envs/pyenv /xxxx/anaconda3/envs/
Point the empty directory to the actual anaconda path
Then activate the environment source activate /xxxx/anaconda/envs/VLM

Hope it can be completed smoothly!

Well, if it is helpful to you, please copy it happily. If it is not easy to organize and reprint, please indicate qwq!
If you have better suggestions or opinions, please feel free to add them!
I’m Qi Yunpeng (Qi Qí), and I strive to share the joy of algorithms with everyone!

One picture per bet (1/1)↓

Reference：
conda-pack migration virtual environment
Offline environment conda virtual environment backup migration-conda pack problem