pyenv_post

Why Do You Need Different Python Versions?

Why would you want to install another Python version than the one your system already ships with?

Well, the default Python version that is bundled with your system is usually way too old and might not have what you need. For instance, Python 3.11 is now 10~60% faster than 3.10 which is a huge improvement and can improve your applications and scripts runtime.

In this post, I will explain how to install Pyenv and then optionally, integrate it with my favorite shell, fish on an Ubuntu machine.

What is Pyenv?

pyenv is a tool that allows you to easily manage and switch between multiple versions of Python on a single machine. It allows you to specify the version of Python that you want to use for a particular project, and it will automatically modify your environment so that the desired version of Python is used when you run your scripts or use the Python interpreter.

Pyenv Installation

  1. Get the installer:
cd /tmp/
curl -s -S -L -o pyenvInstaller.sh https://raw.githubusercontent.com/pyenv/pyenv-installer/master/bin/pyenv-installer
  1. Install it:
bash pyenvInstaller.sh

Pyenv installer depends on Git. If you do not have it installed, you can do so by running “sudo apt install -y git”

Right after the installation is done, we are presented with this message:

# Load pyenv automatically by appending
# the following to
~/.bash_profile if it exists, otherwise ~/.profile (for login shells)
and ~/.bashrc (for interactive shells) :

export PYENV_ROOT="$HOME/.pyenv"
command -v pyenv >/dev/null || export PATH="$PYENV_ROOT/bin:$PATH"
eval "$(pyenv init -)"

# Restart your shell for the changes to take effect.

# Load pyenv-virtualenv automatically by adding
# the following to ~/.bashrc:

eval "$(pyenv virtualenv-init -)"
  1. so, let’s modify the .bashrc file and add the required commands.
cat <<EOF>> ~/.bashrc

export PYENV_ROOT="\$HOME/.pyenv"
command -v pyenv >/dev/null || export PATH="\$PYENV_ROOT/bin:\$PATH"
eval "\$(pyenv init -)"
eval "\$(pyenv virtualenv-init -)"

EOF
  1. All good! Next, source the .bashrc file for our changes to take effect:
source ~/.bashrc

Running pyenv --version should give you an output similar to the result below:

$ pyenv --version
pyenv 2.3.9

Exploring the Available Python Versions

It is time to explore the plethora of the available Python versions at our fingertips. Say, you want to install the CPython (default and mainstream Python) implementation of Python version 3.11.

The command below lists all the available Python versions and I just narrow down the list to what I actually look for:

pyenv install --list | grep "^  3.11"
  3.11.0
  3.11-dev
  3.11.1

Install a Custom Version of Python

Some dependencies are required for pyenv to successfully build and install your desired Python version. Chances are that you already have them. Although it is better to make sure they are installed by running:

sudo apt update
sudo apt install -y gcc build-essential libffi-dev python3-pip

With all that our of the way, let’s install Python 3.11.1:

pyenv install 3.11.1

pyenv puts the Python binaries inside ~/.pyenv/versions directory.

After the installation, run the interpreter to make sure everything is fine:

~/.pyenv/versions/3.11.1/bin/python3

Using the New Python Version

There are a few ways to utilize the new Python version:

  1. Invoke the executable directly on your apps, ~/.pyenv/versions/3.11.1/bin/python3 app.py
  2. Add ~/.pyenv/versions/3.11.1/bin/ to your $PATH
  3. Leverage pyenv local 3.11.1
  4. Use virtualenvs

The first two options are just not convenient and spiral out of control once you have a few different versions installed.

The third option is quite clever; it generates a .python-version file in the current directory and runs python 3.11.1 when you use the Python command inside that directory and its sub-directories.

The fourth option, virtual environment is my favorite way of managing different Python versions.

Let’s see how to use them.

Python Virtual Environments

A Python virtual environment or venv is a tool used to isolate specific Python environments on a single machine, allowing you to work on multiple projects with different software requirements concurrently and in a safe manner.

To give a simplified example, it enables you to have:

  • Project A with Python 3.8 and requests library version 1
  • Project B with Python 3.11.1 and requests library version 2

While for instance your system’s default Python version is 3.6.

Virtual environments will keep your Python versions and your project’s dependencies in a safe and fluffy place, ensuring that you don’t accidentally break your system (in case of Linux/Mac OS) and that there are no interference between different projects.

Creating a venv is pretty straightforward:

python3 -m venv /tmp/test

Now, we need to activate the virtual environment:

source /tmp/test/bin/activate

This will slightly change your Bash prompt, indicating that we are in a venv:

(test) pouriya@dev:/tmp$ python -V
Python 3.6

Notice the (test); name of the venv we just created in the Bash prompt and the Python version which is my system’s default.

Exit from the venv by issuing:

deactivate

This will again change your Bash prompt by removing (test), indicating that we are no longer in a venv.

At this point, you might wonder what happened to version 3.11.1 and why do we see version 3.6 instead?

It all lies withing the fact that we just ran python3 -m venv /tmp/test which uses my default system’s Python installation. In order to make a specific venv version, we need to explicitly specify it:

 ~/.pyenv/versions/3.11.1/bin/python3 -m venv /tmp/3.11

Activate it and check the version:

$ source /tmp/3.11/bin/activate
(3.11) pouriya@3dc918f6bff8:/tmp$ python -V
Python 3.11.1

Better Virtual Environment Management with virtualenvwrapper

virtualenvwrapper is a set of shell scripts that provide additional functionality for managing Python virtual environments which works on top of virtualenv. It provides several features that can make it easier to work with virtual environments, such as:

  • A set of commands for creating, deleting, and managing virtual environments.
  • A centralized location for storing all of your virtual environments, making it easier to organize and manage them.
  • The ability to specify a default Python interpreter for each virtual environment.
  • The ability to set environment variables for each virtual environment.

Install virtualenvwrapper

  1. Let’s start off by installing the virtualenvwrapper:
pip3 install virtualenvwrapper

Take a note of where virtualenvwrapper.sh will be placed according to the output. Mine was placed in ~/.local/bin/

Most modern text editors and IDEs can automatically detect and activate a venv if:

  • The venv is in the same directory as your project
  • There’s a directory called .virtualenvs in your $HOME
  1. I prefer to have all of my venvs in one location, so I’ll create a new directory under the $HOME directory:
mkdir -p ~/.virtualenvs
  1. Modify the ~.bashrc to specify the location of your venvs and to initialize virtualenvwrapper whenever you open a new shell:
cat <<EOF>> ~/.bashrc

export VIRTUALENVWRAPPER_PYTHON=/usr/bin/python3
export WORKON_HOME=~/.virtualenvs
source ~/.local/bin/virtualenvwrapper.sh
EOF
  1. source the file for the changes to take effect:
source ~/.bashrc

Now, we will have some new commands such as:

  • mkvirtualenv: Creates a new virtual environment
  • rmvirtualenv: Deletes a virtual environment
  • workon: Activates a virtual environment

Create Python venvs using virtualenvwrapper

Let’s create a new venv called p3.11 using Python 3.11.1:

mkvirtualenv p3.11 -p ~/.pyenv/versions/3.11.1/bin/python3

The venv also gets activated by the command above.

Take note of the argument -p For this environment, it instructs the virtualenvwrapper to use Python 3.11.1. Otherwise, the system’s built-in Python would be utilized.

If you have multiple venvs, you can easily switch between them using the workon command:

workon p3.11
workon p3.10
workon dev
workon test

Exiting from a venv is similar to what I previously showed, just type deactivate and done.

Deleting a venv is achieved by running:

rmvirtualenv p3.11

Make sure you deactivate the venv before deleting it.

In case you faced errors complaining about command not found or PATH, just add the $HOME directory to the system PATH:

export PATH=~/.local/bin:$PATH

Optionally, make it consistent by adding it to the ~/.bashrc file.

export PATH=~/.local/bin:$PATH

Fish Shell Integration

I assume you already have fish installed on your system. Otherwise you can follow the steps mentioned here.

  1. Install the VirtualFish which is a Python virtual environment manager for the Fish shell. I am going to shorten the steps mentioned in the official documentation.
python3 -m pip install --user virtualfish

This will place the files under ~/.local/bin/ directory. Make sure it is included in the $PATH as depicted in the previous section.

  1. Install the VirtualFish loader by running:
vf install
  1. Install the plugins:
vf install compat_aliases projects environment
  1. Add the plugins:
vf addplugins compat_aliases projects environment
  1. Customize your fish_prompt by editing ~/.config/fish/functions/fish_prompt.fish and adding the following lines somewhere in the middle of the file:
if set -q VIRTUAL_ENV
    echo -n -s (set_color -b blue white) "(" (basename "$VIRTUAL_ENV") ")" (set_color normal) " "
end
  1. Apply the changes:
exec fish

Now, you should be able to use the virtualenvwrapper with the fish shell.

Summary

In this post I demonstrated how to utilize pyenv and virtualenvwrapper to manage and maintain multiple Python versions on your system in an organized way and gradually, we worked our way towards integrating it all with the fish shell.

I hope this post helps you and improve your development experience.