The proper method for setting up your python virtual environments
Whatever is your operating system, default python installation, here is the good way to create virtual environments for python properly.
Prerequisites on your computer, system-wide
Your computer embeds a default python engine. On macOS Monterey, and Ubuntu 22.04 LTS, this is python 3.10. However, your projects will require various python versions. pyenv will help you to retrieve and install any of them.
Install pyenv following the repository instructions. It is critical to correctly append your shell environment file with appending the required information, especially the PATH modification: pyenv must take precedence over other paths when it is enabled.
Now, let’s start !
You just retrieved a new python project on github, which looks very promising. You want to clone the source and start modifying the code, or test the python program on your computer.
Retrieve the required python version for your project
This information is populated into your Pipfile, on the [requires] section
[requires]
python_version = "3.9"If you’re using another packaging or dependencies management system (like poetry), this information will be located in the root folder (pyproject.toml for poetry, …).
Use pyenv to install the latest python runtime on your computer.
$ pyenv install --list
$ pyenv install 3.9.13
-> <https://www.python.org/ftp/python/3.9.13/Python-3.9.13.tar.xz>
Installing Python-3.9.13...
python-build: use tcl-tk from homebrew
python-build: use readline from homebrew
python-build: use zlib from xcode sdk
Installed Python-3.9.13 to /Users/ben/.pyenv/versions/3.9.13
$This operation usually takes a few minutes to complete. It will not affect your computer environment yet: the python engine is retrieved from the official source, then built and placed on your home folder.
Enable the required python version for your project
The operating system points, by default, to the system python engine that we want to avoid making changes on. Tell pyenv to use the newly installed python engine for our project.
$ cd my_repo
my_repo $
my_repo $ python --version
Python 3.10.6 # This is the default macos system version for python
my_repo $ pyenv local 3.9.13
my_repo $ python --version
Python 3.9.13
my_repo $Two observations here :
1- If this not behaves as expected on your computer, your pyenv installation is not operational. Remember to restart a shell window once the installation procedure has been applied. If this still fails, your path is not defined correctly.
2- A new file, named .python-version has been created by pyenv. It contains the python version to use in your folder.
my_repo $ cat .python-version
3.9.13
my_repo $Congratulations, at this point, you've set your environment up to use a dedicated python version. A good starting point, but not enough. We'll first install some tooling that will be available python-version-wide.
Install the required tooling for your python version
It is critical to install the pipenv linked with the correct python version; otherwise, you'll have dependencies issues while invoking pipenv on your system.
This article will focus on pipenv commands. Other dependencies managers will be using their own tooling instead of pipenv, however, the command arguments are often the same, or very similar than pipenv.
Preliminary check for rogue pipenv versions
Let's check if you have already pipenv installed it on your computer. Open a new shell and check for pipenv. You should expect a pipenv not found message.
$ pipenv
pyenv: pipenv: command not foundThe `pipenv' command exists in these Python versions:
3.7.13Note: See 'pyenv help global' for tips on allowing both
python2 and python3 to be found.
$
While locating pipenv, you should have either "not found" or pointing to a shims folder. This is what we want here.
$ which pipenv
/Users/ben/.pyenv/shims/pipenvIf you have anything other than the pipenv in the .pyenv folder, you first need to remove the legacy pipenv.
Install pipenv
Go back to your project folder. Your shell will automatically provide the python version expected.
$ cd my_repo
my_repo $ python --version
Python 3.9.13
my_repo $Now let's install the pipenv virtual environment manager. We'll get the relevant pipenv version for Python 3.9.13 and all its linked dependencies.
my_repo $ pip install pipenv
Looking in indexes: <https://pypi.org/simple>, <https://pypi.org/simple>
Collecting pipenv
Using cached <https://pypi.org/simple/fb9a7c9b104ffa64d631f6c3ab2dcb9b7ec7ec52d32f82466e9b785bbac0/pipenv-2022.9.24-py2.py3-none-any.whl> (3.3 MB)
... snip ...
Installing collected packages: distlib, virtualenv-clone, platformdirs, filelock, certifi, virtualenv, pipenv
Successfully installed certifi-2022.9.24 distlib-0.3.6 filelock-3.8.0 pipenv-2022.9.24 platformdirs-2.5.2 virtualenv-20.16.5 virtualenv-clone-0.5.7
WARNING: You are using pip version 22.0.4; however, version 22.2.2 is available.
You should consider upgrading via the '/Users/ben/.pyenv/versions/3.9.13/bin/python3.9 -m pip install --upgrade pip' command.my_repo $
You've installed the pipenv tool for the relevant version. You gained some isolation. Any library/program installed will not affect your system (using a different python version).
However, we want an additional isolation per project. This is what the virtual environments will bring to us.
Create your virtual environment
At this stage, your shell is pointing to the correct version, and you've installed the required tooling to create additional, per-project isolation. A Python Virtual Environment is used to containerize all the python libraries together, avoiding dependencies mismatches between projects.
For now, there is no virtual environment loaded into the project.
my_repo $ printenv | grep -i VIRTUAL_ENV
my_repo $Let's create the virtual environment with pipenv shell. It's required to define the python version as a command option (pipenv shell --python 3.9.13), as we're using the pipenv linked to a given, fixed python version
my_repo $ pipenv shell --python 3.9.13
Creating a virtualenv for this project...
Pipfile: /Users/ben/my_repo/Pipfile
Using /Users/ben/.pyenv/versions/3.9.13/bin/python3.9 (3.9.13) to create virtualenv...
⠸ Creating virtual environment...created virtual environment CPython3.9.13.final.0-64 in 190ms
creator CPython3Posix(dest=/Users/ben/.local/share/virtualenvs/serenity-module-geo-ip-QcpgNzbv, clear=False, no_vcs_ignore=False, global=False)
seeder FromAppData(download=False, pip=bundle, setuptools=bundle, wheel=bundle, via=copy, app_data_dir=/Users/ben/Library/Application Support/virtualenv)
added seed packages: pip==22.2.2, setuptools==65.3.0, wheel==0.37.1
activators BashActivator,CShellActivator,FishActivator,NushellActivator,PowerShellActivator,PythonActivator✔ Successfully created virtual environment!
Virtualenv location: /Users/ben/.local/share/virtualenvs/my_repo-QcpgNzbv
Launching subshell in virtual environment...my_repo $
Now your current shell operates in your required python version, within a given python virtual environment.
my_repo $ printenv | grep -i VIRTUAL_ENV
VIRTUAL_ENV=/Users/ben/.local/share/virtualenvs/my_repo-QcpgNzbv
my_repo $That means that any additional library installed beyond this point will be containerised and not interfere with other python libraries. Let's now import the python dependencies for our project.
Installing python dependencies
The python dependencies are declared in a Pipfile.lock versioned tree, which is barely readable, we usually declare our dependencies on the Pipfile instead.
[dev-packages]
yq = "*"
ipdb = "*"
ptvsd = "*"
pytest = "*"
pytest-cov = "*"
tavern = "*"When using pipenv sync , dependencies are installed from the Pipfile.lock. We can regenerate an updated Pipfile.lock using the command pipenv lock -d.
my_repo $ pipenv lock -d
Locking [packages] dependencies...
Building requirements...
Resolving dependencies...
✔ Success!
Locking [dev-packages] dependencies...
Building requirements...
Resolving dependencies...
✔ Success!
Updated Pipfile.lock (be54f6)!
my_repo $Then proceed with the dependencies installations with pipenv sync -d.
Life with virtual environments
Entering an existing virtual env
While entering a project folder with Virtual Environments, you'll not, by default, enter in the virtual environment attached to it. It is required to issue the pipenv shell command.
my_repo $
my_repo $ printenv | grep -i VIRTUAL_ENV
my_repo $
my_repo $ pipenv shell
my_repo $ printenv | grep -i VIRTUAL_ENV
VIRTUAL_ENV=/Users/ben/.local/share/virtualenvs/my_repo-QcpgNzbvLeaving an enabled virtual env
When you enter a virtual environment, a sub-shell is spawned with the required environment variables. To exit a given environment variable, simply enter exit. However, it's cleaner to close and reopen a new window to ensure that no leftovers remained from your last session.
my_repo $ exitDelete an existing virtual env
To delete the existing virtual environment created with pipenv :
my_repo $ pipenv --rm
Removing virtualenv (/Users/ben/.local/share/virtualenvs/my_repo-uh2L8Xwz)...my_repo $
A good piece of advice is to reopen a new container window to ensure that the previous pipenv has been completely removed.
Declare your virtual environment to your IDE
You need to declare to your IDE the custom virtual environment you're working on. On VSCode, this happens on the right bottom corner, on the python version number
Then enter the name of your repository in the search box. Your virtual environment should be displayed here.
Declare your virtual environment to your shell
It's advised to set up your shell prompt to be aware when operating on a python virtual environment mainly because you can leave your current python project folder and go to another python project.
You can use Oh My Zsh with the P10K theme. Here is a configuration excerpt from my `~/.p10k.zsh` :
Conclusion
We have explored the right way to create python environment variables on your computer.
Here are a few key points to remember :
- never install any additional python dependencies on your system-level python ;
- never rely on your system-level python version but establish your own python version ;
- the python tooling is version dependent: always install the python tooling for each python version ;
- always, always operate on separated virtual environments (to ensure there are no dependencies conflicts with other python projects)
