Migrating from black and flake8 to ruff

Posted by __Aaliyan__@reddit | Python | View on Reddit | 42 comments

as the title says, so i'm currently working on a relatively huge python/django codebase, built over the course of 6 years, which has been using black and flake8 for formatting and linting in pre-commit hook, both have their versions unupdated for about 3 years, now i have a somewhat difficult task on hand.

the formatting and linting engine is to be moved to ruff but in such a way that the formatting and linting changes reflected in codebase due to ruff are minimal, i can't seem to figure out a way of exporting either configs from black and flake8 in their current state so i can somehow replicate them in ruff to control the changes due to formatting. if anyone has been in a similar situation or know any potential way i can approach this, that woyld greatly help. cheers!

pre-commit-config.yaml (in its current state, as you can see versions are a bit older)

repos:
-   repo: https://github.com/psf/black
    rev: 19.10b0
    hooks:
    - id: black
      additional_dependencies: ['click==8.0.4']
-   repo: https://github.com/pre-commit/pre-commit-hooks
    rev: v1.2.3
    hooks:
    - id: flake8
      args: [--max-line-length=120]
    - id: check-yaml

[-]

Aaliyan@reddit (OP)

agreed with you on just accepting whatever formatting changes are coming with the formatter & linter, but can't seem to get the approval unless i can keep changes to minimal.

do you have any suggestions on identifying what the exact version of ruff that matches the current config of my black and flake8?

[-]

JimDabell@reddit

do you have any suggestions on identifying what the exact version of tuff that matches the current config of my black and flake8?

Why are you getting stuck here? Just install the latest version of Ruff and take a look at what it flags. It takes thirty seconds to do. You seem to be waiting for people here to spell out in exhaustive detail every single aspect of what you need to do. It’s your job, man.

[-]

justheretolurk332@reddit

You don’t need to look for a specific version of ruff, I’m guessing that commenter hasn’t used it before. Just look up the ruff documentation and configure the rules that you want to enforce. The rules are organized by the tool that they originally come from, so it should be easy to find the flake 8 section. The ruff docs are honestly fantastic in my opinion.

As for formatting, ruff formatting is designed to be a drop-in replacement for black so you probably shouldn’t need to do anything there, but again all known differences are well-covered in the docs.

[-]

Erelde@reddit

I commented one level up about that. You can customize git blame so that big formatting commits aren't a problem.

https://www.reddit.com/r/Python/s/kC9rAR7sqj

[-]

schfourteen-teen@reddit

But "minimal" is quite vague, and you are potentially in a position of controlling the flow of information. Minimal could mean "these are the minimal changes necessary to get on a current version of a reliable linting and formatting tool", even if there is some other solution that would technically make fewer changes.

[-]

violentlymickey@reddit

Not sure. Would maybe ask ChatGPT or look at the ruff changelog.

[-]

pacific_plywood@reddit

Asking ChatGPT for a specific version of something is… not likely to help you

fail_fast: true repos: - repo: local hooks: - id: ruff-check name: Ruff check entry: poetry run ruff check args: [--fix] language: system types: [file, python] - id: ruff-format name: Ruff format entry: poetry run ruff format language: system types: [file, python] - id: pyright name: Pyright type check entry: poetry run pyright language: system types: [file, python]

repos: - repo: https://github.com/astral-sh/ruff-pre-commit rev: "v0.7.4" hooks: - id: ruff args: ["--fix", --max-line-length=120] - id: ruff-format - repo: https://github.com/pre-commit/pre-commit-hooks rev: v5.0.0 hooks: - id: check-yaml

The problem with pre-commit hooks like this is that they don’t sync with your project dependencies and can go out of date easily. What I’ve done at my company is run bash invocations of pyinvoke tasks that call whatever formatting or linting libraries we have defined in our dev dependencies.

As for ruff, I would try to pin the version to whatever is closest to the behavior of black and flake8 that was used on your repo. If it were me though I would just push for updating to the latest version and accepting whatever formatting and linting changes arise from that.

would push for updating to the latest version and accepting whatever formatting and linting changes arise from that.

Git blame can take a file of revs to be ignored, it's intended in part for that use case

https://git-scm.com/docs/git-blame

What I do is put a .git-blame-ignore-revs file in the repo, put each revs on a line preceded by a comment explaining why that rev should be ignored.

Acceptable_Durian868@reddit

Oh that's awesome, thanks.

Delicious_Gap_2350@reddit

This is not helpful for the discussion , but holy smokes i read the question and didn't understand anything and then i read this reply and i still don't understand anything. Im on week 2 of CS50 but damn it looks like there is a long long long way for me to go gif

SentinelReborn@reddit

The post is about automated code formatting, a topic that beginners don't need to touch until starting to make bigger projects (maybe after 6+ months), so still a bit of a gap for you, a delicious gap you might say. Enjoy learning programming fundamentals first.

Haha thanks man !

ducdetronquito@reddit

Good advice !

At work we use poetry to manage a project dependencies with a "dev" group for development dependencies like ruff.

Typically you would do something like poetry add ruff --group dev.

With that in place, our pre-commit hooks looks like this:

If you don't use poetry, you should just have to change the "entry" accordingly.

VovaViliReddit@reddit

What I’ve done at my company is run bash invocations in the precommit hooks of pyinvoke tasks that call whatever formatting or linting libraries we have defined in our dev dependencies.

Can you please share how you did that?

scykei@reddit

In my project, we add a lint check in the CI so it fails if the pre commit hooks were not run or if they're running the wrong versions of the library or anything.

mvaliente2001@reddit

One strategy that has worked very well for me:

Install and configure all development tools using pyproject/poetry.
use make (or even better, just) as the entry point for all dev tools: init venv, linting, type checking, docstring checking, etc.

Then, it's easy to guarantee you'll get the exact same result when running dev tools in the CLI, by git hooks, in the CI pipeline.

wyldstallionesquire@reddit

Don’t do pre commit, make lint part of CI so bad formatting isn’t merged, and developers will quickly get in the habit of setting up format on save.

mistabuda@reddit

Wouldn't you want to know about the formatting issue before pushing to remote?

I find the ability to commit non-compliant code locally very useful, for example for alternating work between branches.

That's why we have the --no-verify flag tho is it not?

infiniteAggression-@reddit

What I’ve done in my team is that the CI runs the ruff lint stage on every merge request and blocks the merge if it finds out warnings/errors according to our own config. You can then just push those fixes to the same merge request and squash on merge. We're a very small team so it works for us.

Yea I work somewhere that does that. And tbh it's kinda annoying waiting for a runner to pick up the job then waiting for the job to finish since it scans the whole codebase instead of the changed files exclusively like precommit

It’s just part of our test script so if I miss something it’s caught before I push

You sure you don't mean "before merge"? Because if it's happening before you push wouldn't that mean it's executing locally?

It’s both. Test lets me know if I missed something locally, then CI makes sure a merge can’t happen with errors

Ok_Raspberry5383@reddit

Why not do both?

wineblood@reddit

Where I work we use precommit and it's called as part of the CI. Easy win.

I always find precommit more trouble than it’s worth. I already run test suite before I push anything anyway. Keeps the git config simpler. Ezpz

JerMenKoO@reddit

Reformat codebase over the weekend, merge to master, let others rebase on top of the reformatted version on Monday.

sonobanana33@reddit

I see you're not in the habit of supporting more than 1 branch

Good point, most of my experience is in monorepo world (and large means tens of millions of line of code)

The 2 aren't mutually exclusive at all.

cheese_is_available@reddit

Replace everything with

Treat that PR like you had upgraded black to 2023's style (should have done that 23 months ago, frankly) and be done with it.

LargeSale8354@reddit

We made sure that both our CICD pipelines and pre-commit hooks use the same version of Ruff. We migrated from Flake8 and Black and the pyproject.toml file allowed us to get the settings more or less the same. We took the view that linting/formatting is to achieve and enforce consistency. Yesterday we used older lint/format tech and our code style was consitent. Today we use Ruff and our code style is consistent. We don't care about consistency with a tool (Flake8 etc)we no longer use. The post implementation commit was surprisingly small.

EternityForest@reddit

I would just do a format pass and change the style to match what Ruff does, just standardize the whole project on the current de facto standard.

The exception is the line lengths, I use 120 in the pre-commit for legacy projects, but only in pre-commit.

I hate long lines, and I tell the formatter and IDE linter not to allow them, but if there's a file full of hundreds people sometimes need to make a quick change without fixing them all at once, so I allow it in pre-commit and fix them gradually over time.

jdgtrplyr@reddit

Ruff is great.

DoubtConnect6689@reddit

Almost unrelated question. In my pre-commit hooks I use darker instead of black because our team (sigh) doesn't have an indentation policy. Is there anything that runs ruff only on your diff (so that I don't mess with unchanged code of modified files) and fixes it automatically?