Automating Convention: Linting and Formatting Python Code

Introduction

“Readability counts.”

– The Zen of Python

I’ve found that most programmers favor a consistent code style and format. However, I’ve also found that consistently applying that style and format on every commit requires a lot of discipline. Instead of manually formatting code and applying code style, we can automate and have the computer apply our format and style.

Keeping our code style and format consistent makes our code base more readable and keeps code reviews focused on the substance of the implementation. Personally, using these tools reduces the time I spend reviewing code and allows me to forget about manually formatting code. Previously, I spent a lot of time manually formatting long function signatures and nested dictionaries, but adding tools like black to my git commit process saved me time.

Project Set Up

I’ll take you through an example project with configuration that I use in my production applications.

1. Clone the github repo
   • git clone https://github.com/laactech/pre-commit-config.git
2. Create a virtual environment
   • python3 -m venv pcc_venv
3. Activate the virtual environment
   • source ./pcc_venv/bin/activate
4. Install the requirements
   • pip install -r requirements.txt
5. Install the git hooks using pre-commit
   • pre-commit install

Tools for Automation

pre-commit installs and manages the environments for the code linting and formatting tools:

• flake8 a linter to enforce coding style
  • .flake8 stores the configuration
• bandit a linter to check for security vulnerabilities
  • The bandit section of the .pre-commit-config.yaml stores the configuration
• black an automatic code formatter
  • The tool.black section of the pyproject.toml stores the configuration
• isort an automatic import formatter
  • .isort.cfg stores the configuration
• seed-isort-config a tool to statically populate the known_third_party part of the .isort.cfg

This collection of tools automates the formatting of our Python code and imports, style guide linting, and security linting. With pre-commit, all of these tools run on the code staged for every commit. This ensures the quality of the code being committed to our repository. Additionally for extra insurance, pre-commit can run our tools across the entire code base as part of a continuous integration process.

Walk Through

Now let’s run pre-commit run -a on the example project to walk through how the linting and formatting works and how we would remedy any problems.

Black outputs the first failure:

It tells us that it reformatted both of our Python files. If you look at the example.py file, you see the reformatted code.

Before formatting:

After formatting:

When black fails on the git hook, black reformatted our code. The only action required from us is to git add the changed code.

Flake8 outputs the next failure:

Flake8 gives us the file name:line number:character number:error code for each style issue. Let’s fix our flake8 issues. First we need to remove the unused imports for pendulum and Dict in example.py and requests in another_example.py. We can safely delete those import lines. Next let’s remove the -u on line 41 to fix the undefined name issues. After that on line 49, let’s replace the ellipse with a f.read() to fix our local variable never used. Finally, remove the # fmt: off and # fmt: on around the custom_formatting list so that black can reformat it.

Bandit outputs one failure:

If you read the message output, bandit says that input is safe on Python 3. Due to this, let’s add a # nosec next to our output on line 16 to silence this issue.

Seed isort output:

The seed isort output tells us that the .isort.cfg changed. If we open the file, we see that the known_third_party line updated with our third party packages pendulum and requests. No action is required on our part aside from git adding the changes.

Finally, the isort output:

The isort output tells us which files changed. If we look at both our Python files, we see that our imports were sorted. Once again, no action is required from us besides git adding the changes.

Now we’re ready to commit the changes. As a reminder, we need to add the changes to the .isort.cfg, example.py, and another_example.py. Let’s attempt to commit our changes using git commit.

Black reformatted our custom_formatting list and helped fix our flake8 issues. Seed isort removed pendulum from the known_third_party line in our .isort.cfg since we removed that import from our code. Finally, let’s try another git commit after adding the changes to example.py and .isort.cfg.

Our code committed with a full pass of our linters and formatters. These examples contained a lot of changes, but generally in my experience, the number of actions required per commit is low. The failing hooks tend to be black and isort which require no action. Occasionally, flake8 will fail, but in my opinion, most flake8 failures are easy to solve, and it helps keeps your code clean.

Using the Tools in Your Project

Hopefully, the walk through convinced you that adding these tools to your Python project is easy. Now let’s go through how to add these to your new or existing Python project.

1. Copy the following files to the root of your Python project’s git repository:
   • .pre-commit-config.yaml
   • .flake8
   • .isort.cfg
   • pyproject.toml
2. git add the previous files to your git repository
3. Run pip install pre-commit
4. Add pre-commit to your project’s requirements
5. Run pre-commit install
6. git commit the new configuration files
7. Run pre-commit run -a to lint and format your entire project
8. git add and git commit the formatting and linting changes once you’ve resolved any issues

Now on every commit, pre-commit will use a git hook to run the tools.

Final Thoughts

Automating the linting and formatting of your code gives you consistent code without the manual time investment. I’ve used this configuration in numerous production applications and with multiple development teams with great success. I hope that you’ll find success with them as well.