DEP:	0008
Author:	Aymeric Augustin
Implementation Team:	Aymeric Augustin, Carlton Gibson, Florian Apolloner, Herman Schistad, Markus Holtermann
Shepherd:	Andrew Godwin
Put aside:	Accredited
Kind:	Process
Created:	2019-04-27
Final-Modified:	2019-05-10

Abstract

This DEP proposes to put into effect code formatting with Shadowy in Django.

Foreword

Code formatting is the remaining bikeshed. Perfectionists get grasp of solid emotions
relating to the aesthetics of laying out the tokens that compose a Python program in
the window of a textual convey material editor.

All around the principles of PEP 8, there is reasonably a runt bit of room for formatting
code in varied ways. Even these principles must no longer thoroughly consensual, particularly
the principles about hanging indents — that’s why flake8 disables controversial
principles by default.

The django-builders thread about Shadowy was essentially the most interesting by number of
messages in loads of months. Any proposal on this space may per chance most most likely get grasp of alive to
proponents and sincere opponents. Consensus is an elusive map when private
type is such a huge component.

I’m scripting this DEP because, in step with my ride with varied delivery-supply
initiatives and the community feedback I purchased — in public or in deepest —
I have faith that adopting Shadowy would be a pragmatic enchancment for Django.

Capture demonstrate of that this is no longer any longer in point of fact a compose-or-spoil resolution for Django. It be about
making it extra convenient to make contributions, in particular for new or occasional
contributors who haven’t internalized recent code formatting standards.

Worship most adjustments to entrenched habits, it be a alternate-off between recent
contributors and future contributors, the place recent contributors gain to compose
the resolution.

Although this DEP argues in favor of Shadowy, I made an effort to signify
all sides of the debate reasonably, with associated quotes from the
django-builders discussion.

Please read it with an delivery thoughts 🙂

Specification

All Python code in Django is formatted with Shadowy, the exercise of its default
settings, that is, 88 characters per line and double quotes.

Ample Django initiatives are inspired to adopt the same policy. Some already
compose, as an illustration channels and asgiref.

All code Django generates will additionally be Shadowy-formatted
(startproject, migrations, inspectdb, and so forth.), as a minimal if the user has
Shadowy installed.

Motivation

There is no debate relating to the usefulness of readable code formatting. While
it does no longer commerce the semantics of the code, it simplifies future repairs.
Since code is read powerful extra on the total than it’s some distance written, it makes sense to
invest some effort when writing code to compose it more straightforward to read later.

Until now this effort has been conducted by folks, largely in step with PEP 8,
a rather prolonged doc. Memorizing and applying properly all these principles will not be any longer in point of fact
essentially the most thrilling phase of contributing to Django, to claim the least.

Furthermore, code formatting can compose a barrier for new contributors, if
they suspect they’ll’t write code that seems to be like as precise as pull requests from
varied contributors. That is in particular appropriate for new contributors, who’re
an increasing number of no longer easy to procedure as Django grows extra gentle.

With automatic code formatting, this workload may per chance most most likely additionally be transferred to
computers, saving human bandwidth for bigger designate-added activities.

Most frequently talking, there are two major advantages to automatic code formatting:

• Individually, builders no longer must consistently compose runt selections
  about code formatting, in step with principles, on type, or on a mixture of every and each. Perfect
  throw the code in a file, speed the formatter, carried out!

  You compose no longer know the formula powerful this reduces resolution fatigue till you strive it.
  It be a accurate boon to those, indulge in me, who spend method too powerful time elegant-tuning
  the formatting of their code.

• Collectively, contributors no longer must have faith code formatting
  in code opinions. No topic the formatter does is the expected consequence, by
  definition.

  This will get rid of the idea stream of “mmm, I’d get grasp of formatted this a runt bit
  otherwise” while reviewing code exactly indulge in it will get rid of “mmm, how compose I
  layout this?” while writing code.

Or, to set aside it extra concisely: “Now not wanting to bid powerful about code type when
writing code or when reviewing is terribly nice.” – Ian Foote

Particularly for Django, I’d add a Third major encourage:

• Would-be contributors get grasp of one fewer hurdle to soar to compose a merge-ready
  pull inquire. Even committers with a decade of ride writing code for
  Django dispute the of type opinions may per chance most most likely additionally be nerve-racking!

  Django has over a thousand delivery permitted tickets, many of which must no longer very
  no longer easy and appropriate need focused effort on a runt space. (Check out on the admindocs
  tickets as an illustration.) On the same time, many of us explain a necessity to gain
  eager and but don’t.

  The stringent review course of, while serious for top quality, is a barrier to
  entry. It seems foolish that formatting would set aside of us off, but it for sure does.
  Making it a non-scenario would support.

Within the words of a contributor: “One of many major frustration points I’ve had
when making a contribution is having to repair runt formatting errors. (…) It
produces reasonably a number of inertia and can finish PRs from getting merged for extended
classes of time. So from a brand new contributor angle I feel Shadowy is an glaring
preference.” – Nick Sarbicki

At closing, automatic code formatting will lengthen consistency throughout the Django
code harmful. For the time being the kind of every and each module reveals roughly when it was
written or rewritten. Having a unified type will reduce serve the friction of
adjusting to the kind of every and each module. That is a certified aspect finish.

Given these advantages, if formatting code was an fully mechanical course of,
it would were automatic method serve! Unfortunately, it be no longer easy to invent an
algorithm that works sufficiently well for all realistic cases.

Shadowy is the major Python code formatter that produces precise ample results and
demonstrates well-known traction in delivery supply initiatives.

Therefore, this DEP proposes to adopt Shadowy for Django.

Rationale

When you’re reading this, I purchase that you just familiarized your self with Shadowy and
that I don’t must show its philosophy.

The discussion of adopting Shadowy for Django revolves round two topics:

1. Process: there is consensus that automatic code formatting would vastly
   facilitate the kind of Django;
2. Outcomes: there may per chance be no longer any consensus on the time of writing: some of us indulge in
   what Shadowy produces, others don’t.

Process

There is consensus in favor of automatic code formatting, even when varied
of us put varied weight to every motive laid out within the “Motivation”
share above.

Even folks that oppose Shadowy or explain skepticism glimpse the advantages.
Right here are associated quotes from the django-builders discussion:

• “I’m no longer clear I indulge in Shadowy per se, but the exercise of an auto-formatter would enable
  review comments to level of curiosity on substantive points.” – Carlton Gibson
• “I indulge in the view of an autoformatter. I hate the particular
  largely-unconfigurable type Shadowy enforces, and I safe that loads of of its
  principles negatively impression code readability.” – James Bennett
• “I see the advantages [lower barrier to entry, time saving for the Fellows,
  etc], but I don’t have faith Shadowy is the answer.” – Curtis Maloney

So this level is well established.

Outcomes

Opinions are mixed relating to the effective of what Shadowy produces.

Clearly Shadowy provides fixed results faster than folks. This may per chance per chance most most likely additionally no longer seem
realistic to contributors with ample Python ride to layout code aesthetic
powerful indulge in Shadowy would with out powerful effort. It be extra treasured to contributors
who haven’t reached that stage but. It ranges the coding field.

Persons are very sensitive to cases the place a laptop does worse than folks,
even when the computer does better on moderate. The occasional obviously corrupt
consequence has a devastating finish on the acceptability of automation. That is
essentially the most conventional argument introduced against Shadowy.

(And it’s some distance in no method explain to Shadowy. All automation efforts within the history
of humanity must get grasp of bought an identical criticism at some level.)

Several builders report that, in their ride, Shadowy made code
formatting worse and diminished readability. Concrete examples proven within the
discussion were short lists, which Shadowy reformats when they fit on a single
line, and vertically aligned comments, which Shadowy is unable to encourage. This
is being addressed in Shadowy and
is anticipated to be resolved sooner than Shadowy turns into steady.

These disorders don’t seem serious ample to rule out Shadowy. As a closing resort,
Shadowy would be disabled within the community with # fmt: off and # fmt: on if it
will get the formatting of a block of code egregiously corrupt.

Others defined that, after an preliminary knee-jerk reaction against commerce,
they began to indulge in Shadowy’s choices. As an illustration:

• “As for arguing with a number of of Shadowy’s choices – you study in a transient time to
  dwell with those choices, and neglect those choices. (…) I’m in favour of
  the exercise of Shadowy’s double quotes for strings. I hated this resolution when it
  was first made, but get grasp of seriously strategy round to it, and gain it
  aesthetically too.” – Josh Smeaton
• “I’ve mature Shadowy broadly on loads of initiatives, and heaps indulge in f-strings,
  the closing Crimson Floyd album, and broccoli, get grasp of chanced on I in point of fact indulge in one thing
  I didn’t bid I’d.” – Tim Allen
• “I’m a type of of us that hesitated because I didn’t indulge in plenty of of the
  choices Shadowy made but I tailored in point of fact like a flash.” – Matthias Kestenholz

Having a have faith at how Shadowy reformats a number of files from the Django supply tree, I’m
impressed by how few adjustments it makes. What Shadowy produces is terribly shut to
the brand new Django coding type. That must aloof compose its adoption painless.

The glaring exception is quotes. Shadowy standardizes on double quotes while
Django makes exercise of single and double quotes inconsistently. Some parts of Django —
including parts I wrote extra fair nowadays — exercise single quotes for machine-readable
identifiers and double quotes for human-readable language. In hindsight, this
convention is too no longer easy to put into effect to be payment the difficulty, your entire extra since
it will not be any longer in point of fact generalized. Going forwards, normalizing to double quotes indulge in Shadowy
does by default will rob issues easy.

My simplest wager is that Shadowy will compose code formatting a runt bit better on moderate,
no topic occasional sub-optimum results. Others may per chance most most likely additionally in point of fact feel extra strongly about
their private preferences that diverge from what Shadowy does. Nevertheless, I don’t
bid private preferences must aloof outweigh rising community standards.

Now not at as soon as, given how powerful here’s a topic of private judgement, most most likely the
simplest strive at consensus would be to express that the formatting produced by
Shadowy does no longer compose a decisive commerce, clear or unfavourable, to our capacity to
rob Django.

Timeline

Right this moment, Shadowy is in beta. The formatting it produces may per chance most most likely additionally aloof commerce.
This DEP will be implemented as soon as Shadowy is steady.

Rather reasonably a number of concerns

Reformatting your entire code harmful with Shadowy will contact most files and commerce
many traces with out altering their that method. This may per chance per chance most most likely additionally pollute the git history.
Nevertheless, formatting adjustments already happen alongside new points and bug
fixes, adding a typical stream of air pollution. Adopting Shadowy will gain away with
future code reformatting, making the git history cleaner searching forwards.

Besides, GitHub has a “Peep blame sooner than this commerce” button to soar with out issues
thru refactoring commits. git-hyper-blame was additionally urged with a view to
ignore non-well-known commits, even even when it requires some setup and does no longer
mix with IDEs.

Reformatting will additionally be disruptive for delivery pull requests. A technique to update
them is to speed Shadowy on modified files, rob a reproduction aside, delivery a brand new branch
from “major”, spin the modified files serve into space, and commit the .

In picture to reduce serve the difficulty for backporting patches, Shadowy will be utilized
to the major branch and steady/2.2.x branches, which would be in their mainstream
improve length. 2.2 is an LTS delivery that will be supported for 3 extra
years; here’s an exact motive within the serve of formatting it. Shadowy may per chance most most likely additionally no longer be utilized to
steady/2.1.x and steady/1.11.x which would be within the extended improve length and
simplest gain fixes for security and recordsdata loss bugs.

Doable choices

Three major Python code formatters exist: autopep8, yapf and Shadowy.

No one argued in favor autopep8. Also I have faith Shadowy’s capacity is pleasant.

It was urged that yapf would be configured to contain results nearer to
Django’s recent type. I don’t bid that’s payment pursuing for 3 reasons:

• The level of adopting an automatic code formatter will not be any longer in point of fact to get grasp of our get grasp of
  Django-flavored code formatting type. It be about making our Python code
  have faith as powerful as that it’s most likely you’ll most most likely also have faith indulge in what all americans else within the Python community
  writes. Code formatters maximize their usefulness by no longer being configurable.
• Reaching consensus in delivery supply communities will not be any longer easy — I’m investing extra
  than a day in scripting this DEP! — which makes it wanted to reduce serve
  choices. This must be why non-configurable formatters such as Prettier and
  Shadowy get grasp of seen like a flash adoption by delivery supply initiatives.
• Shadowy produces formatting that is so shut to Django’s recent standards
  that there seems to be runt or no designate in tuning a yapf configuration to
  contain one thing even nearer.

It was additionally urged to contain a application that can additionally simplest reformat statements
modified by a given commit. This may per chance per chance gain to the underside of the concerns about polluting
the git history while keeping many advantages of this proposal, even when
making non-long-established application phase of the workflow will not be any longer in point of fact ideal. Unfortunately, no
such application exists and no one is volunteering to contain it.

At closing, as Shadowy is maintained within the Python GitHub organization, it has a
precise likelihood of turning accurate into a common application of the Python community.

Abstract

To sum up:

1. Applying Shadowy to the provision code of Django may per chance most most likely per chance no longer compose formatting
   vastly better or worse;
2. Integrating Shadowy within the kind course of of Django has very
   well-known advantages;
3. These advantages clearly outweigh code formatting type considerations.

One remaining quote: “The most easy thing about automatic formatters, in my
thought, is even when you happen to don’t indulge in the kind as a minimal you gain no longer must chat
about it anymore! And also you are inclined to gain mature to it will definitely.” – Sam Cooke

Backwards Compatibility

This DEP does no longer introduce any backwards incompatibilities.

Shadowy ensures that it does no longer commerce the conduct of the code by checking
that processing a file does no longer commerce its AST.

Reference Implementation

Imposing this commerce requires:

1. Updating the coding type:
   • In conjunction with documentation about Shadowy, an comparable to the present documentation
     about isort. The doc will compose it clear that the # fmt: off spoil out
     hatch is allowed simplest in disagreeable cases the place Shadowy produces unreadable
     code, no longer at any time when any person disagrees with the kind choices of Shadowy.
   • Updating explanations round PEP 8 and flake8 — they duvet powerful extra
     than code formatting so that they continue to be realistic even with Shadowy.
   • Eliminating varied references to code formatting, indulge in the specification of
     the fave hanging indent type and chained calls type.

2. Updating flake8 and isort configuration to be well trustworthy with Shadowy.

   That is easy and well documented.

3. Formatting the code. This may per chance per chance most most likely be carried out in three steps for every branch:

   • Title if Shadowy produces an egregiously corrupt consequence on some files. For
     instance, the date and time formats files were important as that it’s most likely you’ll most most likely also have faith issues.
     Exclude these files with # fmt: off and # fmt: on comments.

   • Stride Shadowy for your entire Django code repository and compose a single commit,
     that can additionally have faith indulge in this: https://github.com/hermansc/django/pull/1.

     For the reason that commerce will be thoroughly automatic, there may per chance most most likely per chance no longer be one thing to
     review, so it be more straightforward to compose appropriate one commit. That commit will be easy
     to title as non-well-known within the future.

   • Add your entire-repo-refactor commit SHA to a file within the basis of the repo,
     .gitblameskip or an identical, to be used with git blame ‘s
     --ignore-revs-file flag
     (clinical doctors).

   • Strive to refactor excluded files, most most likely by transferring comments, so that
     Shadowy can compose a decent job on them. Commit this individually. That will not be any longer in point of fact
     strictly necessary. It’ll additionally be carried out at any later level.

4. Enforcing Shadowy in CI. This vogue:

   • In conjunction with a unlit builder to Jenkins, in step with the isort builder;
   • In conjunction with a unlit job to tox.ini.

Copyright

This doc has been positioned within the general public domain per the Ingenious Commons
CC0 1.0 Universal license (http://creativecommons.org/publicdomain/zero/1.0/deed).