Python 3.11 Changes

Python 3.11 Changes#

New language features#

Fine-grained error locations in tracebacks#

  • In Python 3.11, decorative annotations are added to the tracebacks.

  • The symbols ^ and ~ are embedded within the traceback. They’re used to guide the attention to the part of the code that’s causing the error, instead of just the line.

def inverse(number):
    return 1 / number

print(inverse(0))

Error output in Python 3.10:

Traceback (most recent call last):
  File "/Users/surya.kumar/workspace/file.py", line 4, in <module>
    print(inverse(0))
  File "/Users/surya.kumar/workspace/file.py", line 2, in inverse
    return 1 / number
ZeroDivisionError: division by zero

Error output in Python 3.11 (use of symbols ^ and ~):

Traceback (most recent call last):
  File "/Users/surya.kumar/workspace/file.py", line 4, in <module>
    print(inverse(0))
          ^^^^^^^^^^
  File "/Users/surya.kumar/workspace/file.py", line 2, in inverse
    return 1 / number
           ~~^~~~~~~~
ZeroDivisionError: division by zero

Note:

  • This feature may result in a small increase in interpreter memory usage and disk usage for compiled Python files.

  • To avoid storing the extra information and deactivate printing the extra traceback information, use the -X no_debug_ranges command line option or the PYTHONNODEBUGRANGES environment variable.

  • See the Python 3.11 release notes for more information.

Exception groups#

  • Exception Groups enable a program to raise and handle multiple unrelated exceptions simultaneously. They’re like regular exceptions wrapping several other regular exceptions.

  • You create an exception group by giving it a description and listing the exceptions that it wraps:

eg = ExceptionGroup("eg", [TypeError("int"), ValueError(0)])

  • When an exception group is raised it displays a nice traceback that illustrates the grouping and nesting of errors:

  + Exception Group Traceback (most recent call last):
  |   File "<stdin>", line 1, in <module>
  | ExceptionGroup: eg (2 sub-exceptions)
  +-+---------------- 1 ----------------
    | TypeError: int
    +---------------- 2 ----------------
    | ValueError: 0
    +------------------------------------

  • The new except* syntax generalizes except to match subgroups of exception groups to for working effectively with them.

try:
    raise eg
except* ValueError as ve:
    print(f"ValueError handled: {ve.exceptions}")
except* TypeError as te:
    print(f"TypeError handled: {te.exceptions}")

# ValueError handled: (ValueError(0),)
# TypeError handled: (TypeError('int'),)

See Python 3.11: Exceptions for more information.

Exception notes#

  • The add_note() method is added to BaseException. It can be used to enrich exceptions with context information that is not available at the time when the exception is raised.

  • A note can be added to an exception, and existing notes can be viewed by inspecting the .__notes__ attribute in the following way:

err = ValueError(0)
err.add_note("0 is not allowed")
err.add_note("Python 3.11")

print(err.__notes__)
# ['0 is not allowed', 'Python 3.11']

raise err
# ValueError: 0
# 0 is not allowed
# Python 3.11

New typing features#

See Python 3.11: New typing features for more information on the changes in this section.

Variadic generics with TypeVarTuple#

Python 3.11 introduces a specialized type variable, TypeVarTuple, enabling variadic generics.

Using type variable tuples, you can create type hints that specify generic data structures of arbitrary shape and type:

from typing import TypeVarTuple, Generic

TValues = TypeVarTuple("TValues")

class Record(Generic[*TValues]):
    key: int
    values: Tuple[*TValues]

    def __init__(self, key: int, *args: *TValues):
        self.key = key
        self.values = args

    def update(self, *args: *TValues):
        self.values = args

record1 = Record(1001, "Hello", 12.3, 1234)
record1. update("new word", 123.2, 32343)

Like regular tuples, type variable tuples can be of arbitrary length:

record1.update("more", "words", -0.342, True, {"hi": "I am a dict"})
record1.update()

See Python 3.11: Variadic generics for more information.

Self type#

  • The new Self annotation provides a simple and intuitive way to annotate methods that return an instance of their class.

  • Self can also be used to annotate method parameters or attributes of the same type as their enclosing class.

  • In the below example, the annotation -> Self indicates that __enter__(self) will return an instance of the current class.

from typing import Self

class MyLock:
    def __enter__(self) -> Self:
        self.lock()
        return self

Required and NotRequired with TypedDict#

  • Python 3.11 introduces two new type qualifiers, typing.Required and typing.NotRequired, which allow defining a single TypedDict with a mix of both required and potentially-missing keys.

  • All the fields are still required by default.

class Movie(TypedDict):
   title: str
   year: NotRequired[int]

m1: Movie = {"title": "Black Panther", "year": 2018}  # Type Check Passed
m2: Movie = {"title": "Star Wars"}  # Type Check Passed (year is not required)
m3: Movie = {"year": 2022}  # # Type Check Failed (missing required field title)

LiteralString#

  • A new type is added to the typing module: LiteralString is a special kind of string type that’s defined literally in your code. I.e LiteralString only accepts string values that are known to be made of literals. (A string literal is where you specify the contents of a string in a program.)

  • Type checkers that can enforce sensitive functions (with LiteralString type), such as those that execute SQL statements or shell commands, are called only with static arguments. This provides protection against injection attacks.

from typing import LiteralString

def execute_sql(query: LiteralString):
    ...

execute_sql("SELECT * FROM users")            # Type check Pass
table = "users"
execute_sql(f"SELECT * FROM {table}")         # Type check Pass

user_input = input()
execute_sql("SELECT * FROM " + user_input)    # Type check Fail

  • In the last example, even though the value of user_input happens to be the same as the value of table from earlier, the type checker will raise an error here.

  • Users control the value of user_input and can potentially change it to something that’s unsafe for your application.

  • See PEP 675 for more details.

Data class transforms#

  • Python 3.11 introduces a new decorator function in the typing module named dataclass_transform.

  • This decorator can be applied to either a function that is itself a decorator, a class, or a metaclass.

  • The presence of @dataclass_transform() tells a static type checker that the decorated object performs runtime “magic” that transforms a class, giving it dataclass-like behaviors.

import typing

@typing.dataclass_transform()
def create_model(cls: Type[T]) -> Type[T]:
    cls.__init__ = ...
    cls.__eq__ = ...
    cls.__ne__ = ...
    return cls

# The create_model decorator can now be used to create new model classes:
@create_model
class CustomerModel:
    id: int
    name: str

See PEP 681: Data class transforms for more details.

Other language changes#

Starred unpacking expressions in for statements#

Starred unpacking expressions can now be used in for statements.

This functionality was present, but not documented, in Python 3.9 and 3.10 stable releases.

See bpo-46725 for more details.

a = [1, 2, 3]
b = [11, 22, 33]

for x in *a, *b:
    print(x, end=' ')

# 1, 2, 3, 11, 22, 33

Asynchronous comprehensions bug fix#

Asynchronous comprehensions are now allowed inside comprehensions in asynchronous functions. Outer comprehensions implicitly become asynchronous in this case. Introduced in bpo-33346 change.

Error handling for objects not supporting context managers#

A TypeError is raised instead of an AttributeError in with statements and contextlib.ExitStack.enter_context() for objects that don’t support the context manager protocol.

Similarly, in async with statements and contextlib.AsyncExitStack.enter_async_context(), a TypeError is raised for objects that don’t support the asynchronous context manager protocol.

See the Python 3.11 release notes for more details.

New method provides default implementation for __getstate__#

A new object.__getstate__() method has been added, providing a default implementation for __getstate__(). Now copying and pickling instances of subclasses of certain built-in types (like bytearray, set, frozenset, and others) now copies and pickles instance attributes implemented as slots.

This change has an unintended side effect: It trips up a small minority of existing Python projects not expecting object.__getstate__() to exist. See the discussion for more details.

New command line option -P and environment variable PYTHONSAFEPATH#

A new -P command line option and PYTHONSAFEPATH environment variable have been added to disable the automatic prepending to sys.path of the script’s directory when running a script, or the current directory when using -c and -m.

This change ensures that only standard library and installed modules are imported, preventing local (user-writable) directories from unintentionally or maliciously shadowing standard modules.

New string format specifier for signed zeros#

In Python 3.11, you can add a literal z to the format string. This will force any zeros to be normalized to positive zero before formatting. (PEP 682 introduces a small extension to the format mini-language used by f-strings and str.format().)

small = -0.00311
f"A small number: {small:z.2f}"

# Python 3.11
# 'A small number: 0.00'

# Python 3.10
# ValueError: Invalid format specifier

See PEP 682 for more details.

Bytes no longer accepted on sys.path#

Bytes are no longer accepted on sys.path; see the Python 3.11 release notes for more information.

Optimizations#

String formatting#

The compiler now optimizes simple printf-style % formatting on string literals containing only the format codes %s, %r and %a and makes it as fast as a corresponding f-string expression.

python -m timeit -s 'x = "%s has been building autonomous AI agents" % "mystring"'

Difference in M3 Pro, 11 core

  • Python 3.10 -> 6.51 nsec per loop

  • Python 3.11 -> 4.4 nsec per loop

Integer division#

Integer division (//) is better tuned for optimization by compilers. It is now around 20% faster on x86-64 when dividing an int by a value smaller than 2**30.

sum function#

sum is now nearly 30% faster for integers smaller than 2**30.

Difference in M3 Pro, 11 core:

  • Python 3.10: 6.51 nsec per loop

  • Python 3.11: 4.13 nsec per loop

List resizing#

Resizing lists is streamlined for the common case, speeding up list.append() by ≈15% and simple list comprehensions by up to 20-30%.

Dictionary hashing#

Dictionaries no longer store hash values when all keys are Unicode objects, decreasing dict size.

For example, sys.getsizeof(dict.fromkeys("abcdefg")) is reduced from 352 bytes to 272 bytes (23% smaller) on 64-bit platforms.

Large file transfers with asyncio.DatagramProtocol#

Using asyncio.DatagramProtocol is now orders of magnitude faster when transferring large files over UDP, with speeds over 100 times higher for a ≈60 MiB file. Find out more at gh-91487.

math.comb and math.perm with large arguments#

math functions comb() and perm are now ≈10 times faster for large arguments (with a larger speedup for larger k). Find out more at bpo-37295.

statistics.mean, statistics.variance, and statistics.stdev#

The statistics functions mean, variance and stdev now consume iterators in one pass rather than converting them to a list first. This is twice as fast and can save substantial memory.

Faster unicode normalization of ASCII strings#

unicodedata.normalize() now normalizes pure-ASCII strings in constant time. Check more at bpo-44987.

Improved modules#

datetime#

datetime.UTC is now added in Python 3.11, a convenience alias for datetime.timezone.utc.

In Python 3.11, the following functions have been enhanced to parse almost all ISO 8601 formats:

  • datetime.date.fromisoformat

  • datetime.time.fromisoformat

  • datetime.datetime.fromisoformat

The only exceptions are those formats that include fractional hours and minutes.

inspect#

getmembers_static() is added. It returns all the members of an object in a list of (name, value) pairs sorted by name without triggering dynamic lookup via the descriptor protocol, __getattr__ or __getattribute__. Optionally, only return members that satisfy a given predicate.

See Python bug 30533 for more details.

math#

  • Added math.exp2(x): return 2 raised to the power of x.

  • Added math.cbrt(x): return the cube root of x.

  • The behaviour of two math.pow() corner cases was changed, for consistency.

    • The operations math.pow(0.0, -math.inf) and math.pow(-0.0, -math.inf) now return inf.

    • Previously they raised ValueError.

  • The math.nan value is now always available. (Prior to 3.11, In some implementations or platforms, this value might have been unavailable due to platform-specific constraints or variations in the underlying C library.)

pathlib#

  • glob() and rglob() return only directories if pattern ends with a pathname components separator: sep or altsep.

from pathlib import Path

list_of_all = [x for x in Path('.').glob('*/')]

# Python 3.10 -> list_of_all contains all directories & files
# Python 3.11 -> list_of_all contains only directories (as glob pattern is '*/')

traceback#

typing#

  • typing.reveal_type() is added. This is useful for asking a type checker what type it has inferred for a given expression. At runtime it prints the type of the received value.

    from typing import reveal_type

a = 15
b = 7.2

c = reveal_type(a) + b
reveal_type(c)
print(c)

    Output in python file.py:

    Runtime type is 'int'
Runtime type is 'float'
22.2

    Output in mypy file.py:

    file.py:6: note: Revealed type is "builtins.int"
file.py:7: note: Revealed type is "builtins.float"

  • Any can now be used as a base class. This can be useful for avoiding type checker errors with classes that can duck type anywhere or are highly dynamic, such as mocks.

    This means that the following code returns True in 3.11, whereas in 3.10 it returns False

    from typing import Any
isinstance(Any, type)

  • The representation of empty tuple types (Tuple[()]) is simplified. This affects introspection, e.g. get_args(Tuple[()]) now evaluates to () instead of ((),).

Deprecations#

Chaining of classmethod descriptors#

Chaining classmethod descriptors is now deprecated.

  • It can no longer be used to wrap other descriptors such as property.

  • It is due to design flaw of the feature and number of downstream problems.

  • To “pass-through” a classmethod, consider using the __wrapped__ attribute

  • Check more at gh-89519

Invalid octal escape sequences#

Octal escapes in string and bytes literals with values larger than 0o377 (255 in decimal) now produce a DeprecationWarning. In a future Python version, they will raise a SyntaxWarning and eventually a SyntaxError.

Delegation of int to __trunc__#

The delegation of int() to __trunc__() is now deprecated. Calling int(a) when type(a) implements __trunc__() but not __int__() or __index__() now raises a DeprecationWarning. Check bpo-44977 for more.

Deprecated modules#

The following modules are deprecated in Python 3.11, and slated for removal in Python 3.13:

Updated deprecation warnings in configparser#

In configparser, warnings have been updated for the following deprecated objects:

Older importlib.resources functions#

The older set of importlib.resources functions were deprecated in favor of the replacements added in Python 3.9.

locale.getdefaultlocale function#

The locale.getdefaultlocale() function is deprecated and will be removed in Python 3.15. Use locale.setlocale(), locale.getpreferredencoding(False) and locale.getlocale() functions instead.

locale.resetlocale#

The locale.resetlocale() function is deprecated and will be removed in Python 3.13. Use locale.setlocale(locale.LC_ALL, "") instead.

typing.Text#

typing.Text, which exists solely to provide compatibility support between Python 2 and Python 3 code, is now deprecated. It is encouraged to use str instead.

Keyword argument syntax for constructing typing.TypedDict#

The keyword argument syntax for constructing typing.TypedDict types is now deprecated.

Test case return values#

The behavior of returning a value from a TestCase and IsolatedAsyncioTestCase test methods (other than the default None value) is now deprecated.

unittest functions#

  • Deprecated the following not-formally-documented unittest functions, scheduled for removal in Python 3.13:

    • unittest.findTestCases

    • unittest.makeSuite

    • unittest.getTestCaseNames

    See the Python 3.11 release notes for alternatives.

  • unittest.TestProgram.usageExit is deprecated, to be removed in 3.13.

re.template, re.TEMPLATE, and re.T#

The re.template function and the corresponding re.TEMPLATE and re.T flags are deprecated.

webbrowser.MacOSX#

webbrowser.MacOSX is deprecated and will be removed in Python 3.13.

Removals#

  • @asyncio.coroutine() decorator deprecated since Python 3.8 is removed now; enabling legacy generator-based coroutines to be compatible with async / await code. Use async def instead.

  • Removed asyncio.coroutines.CoroWrapper used for wrapping legacy generator-based coroutine objects in the debug mode.

  • Due to significant security concerns, the _reuse_address_ parameter of asyncio.loop.create_datagram_endpoint(), disabled in Python 3.9, is now entirely removed.

  • Removed the deprecated binhex module. Also removed the related, similarly-deprecated these binasciifunctions.

  • Removed the deprecated distutils bdist_msi command. Use bdist_wheel (wheel packages) instead.

  • Removed the deprecated __getitem__() methods of xml.dom.pulldom.DOMEventStream, wsgiref.util.FileWrapper and fileinput.FileInput,

  • Removed the deprecated gettext functions lgettext(), ldgettext(), lngettext() and ldngettext(). Also removed the bind_textdomain_codeset() function, the NullTranslations.output_charset() and NullTranslations.set_output_charset() methods, and the _codeset_ parameter of translation() and install(), since they are only used for the l*gettext() functions.

  • The deprecated getargspec(), formatargspec() function and Signature.from_builtin() and Signature.from_function() methods from inspect are removed. Use these instead.

  • Removed the __class_getitem__() method from pathlib.PurePath

  • Removed the MailmanProxy class in the smtpd module, as it is unusable without the external mailman package.

  • Removed the deprecated split() method of _tkinter.TkappType.

  • Removed namespace package support from unittest discovery.

  • Removed the undocumented private float.__set_format__() method (float.__setformat__() previously).

  • The --experimental-isolated-subinterpreters configure flag (and corresponding EXPERIMENTAL_ISOLATED_SUBINTERPRETERS macro) have been removed.

  • Pynche has been moved out and is being developed independently from the Python source tree.