Module paroxython.preprocess_source

Bring together everything relative to the pre-processing of a given source code.

Cleaning

A useful, albeit not essential, step is to cleanse the code of all its algorithmically irrelevant features (blank lines, comments, docstrings, etc.). Lots of examples are provided in the Cleanup class documentation below.

Manual hints: preservation, centrifugation and scheduling

Certain comments intentionally survive the cleaning step, the so-called “manual hints”, introduced here.

• If they appear at the end of a nonempty line of code, they are preserved as they are.
• If they extend over an entire line (excluding the initial blanks), they are considered to be featured by the entire program. They are replaced by two hints, one put at the beginning and the other at the end of the source code, thus delimiting the totality of the program. This operation is called “centrifugation”.

In either case, these hints were manually placed by the user to modify the behavior of Paroxython, by asking it either to add the taxa that it missed, or to delete those that it unduly produced. This is at this step that they are scheduled for addition or deletion (respectively). For the moment, the information is just stored: ProgramParser will process it when required.

from collections import defaultdict
from token import DEDENT, INDENT, NEWLINE, STRING
from tokenize import COMMENT, NL, generate_tokens
from typing import Callable, Dict, List, Tuple, Set

import regex  # type: ignore

from .goodies import print_fail
from .user_types import LabelName, LabelsSpans, Source, Span

HINT_COMMENT = "# paroxython:"


class Cleanup:
    def __init__(self, cleanup_strategy: str):
        self.run = lambda source: source
        if cleanup_strategy == "full":
            self.run = Cleanup.full_cleaning

    @staticmethod
    def full_cleaning(source: Source) -> Source:
        r'''Remove as much noise (comments, etc.) as possible in the given source code.

        Description:
            1. Suppress initial comments.
            2. Suppress `if __name__ == '__main__'` part.
            3. Suppress `sys.path` injections (frequently used to simplify `import` behavior).
            4. Replace tabulations with spaces.
            5. Normalize Paroxython hint comments.
            6. Replace docstrings with `pass` statements (most of which being suppressed at the end
               of the process).
            7. Suppress empty or blank lines.
            8. Suppress useless `pass` statements.

        Examples:
            - Empty or blank lines are suppressed.
            ```python
            foo = bar


            bar = buzz

            ```
            ```python
            foo = bar
            bar = buzz
            ```
            - Normal inline comments are suppressed.
            ```python
            foo = bar # lorem ipsum
            fizz = [
                "foo", # lorem
                "bar", # ipsum
            ]
            ```
            ```python
            foo = bar
            fizz = [
                "foo",
                "bar",
            ]
            ```
            - Commented lines are suppressed.
            ```python
            # lorem ipsum
            # dolor amet
            def foo(bar):
                # lorem
                # ipsum
                if foo == bar:
                    # lorem
                    return []
                # lorem
            # ipsum
            ```
            ```python
            def foo(bar):
                if foo == bar:
                    return []
            ```
            - Shebang is suppressed.
            ```python
            #!/usr/bin/env python
            foobar()
            ```
            ```python
            foobar()
            ```
            - Encoding declaration is suppressed.
            ```python
            # coding=utf8
            foobar()
            ```
            ```python
            foobar()
            ```
            - Docstrings of classes and functions are suppressed.
            ```python
            class Foo:
                def foo(self, bar):
                    return 1
            def bar():
                return 2
            ```
            ```python
            class Foo:
                def foo(self, bar):
                    return 1
            def bar():
                return 2
            ```
            - Docstrings of modules are suppressed.
            ```python
            a = """consectetur
            adipiscing
            elit.
            ```
            - A `pass` statement is added to empty classes if needed.
            ```python
            class Foo:
            ```
            ```python
            class Foo:
                pass
            ```
            - Useless `pass` statements are suppressed.
            ```python
            foo()
            pass
            if bar():
                pass
            else:
                pass
                foobar()
                pass
            ```
            ```python
            foo()
            if bar():
                pass
            else:
                foobar()
                pass
            ```
            - Mixes of comments and docstrings are suppressed.
            ```python
            # Lorem
            # Ipsum
            foobar()
            ```
            ```python
            foobar()
            ```
            - Paroxython's label hints are **preserved** and **normalized**.
            ```python
            foo = bar #   Paroxython   :   hint_1   hint_2
            #paroxython: hint_3
            fizz = [
                "foo", # lorem
                "bar", # paroxython: hint_4
            ]
            ```
            ```python
            foo = bar # paroxython: hint_1   hint_2
            # paroxython: hint_3
            fizz = [
                "foo",
                "bar", # paroxython: hint_4
            ]
            ```
            - `if __name__ == "__main__":` part is suppressed.
            ```python
            foo = bar
            if __name__ == "__main__":
                bar = foo
            ```
            ```python
            foo = bar
            ```
            - `sys.path` injections are suppressed.
            ```python
            __import__("sys").path[0:0] = ["programs"]
            (28433 * 2**7830457 + 1) % 10**10
            ```
            ```python
            (28433 * 2**7830457 + 1) % 10**10
            ```

            All examples above automatically extracted from
            [test_cleanup_source.py](https://github.com/laowantong/paroxython/blob/master/tests/test_cleanup_source.py).

        .. warning::
            Useless pass statements are preserved when they are not followed by a line with a same
            indentation level.
        '''
        result = []
        previous_token = INDENT
        (previous_end_row, previous_end_col) = (-1, 0)
        text = str(source)
        text = Cleanup.suppress_first_comments(text)
        text = Cleanup.suppress_main_guard(text)
        text = Cleanup.suppress_sys_path_injection(text)
        text = text.replace("\t", "    ")
        lines = iter(text.split("\n"))
        for token_info in generate_tokens(lambda: next(lines) + "\n"):
            (token, string, (start_row, start_col), (end_row, end_col), _) = token_info
            if start_row > previous_end_row:
                previous_end_col = 0
            result.append(" " * max(0, start_col - previous_end_col))
            if token == COMMENT:
                (string, n) = Cleanup.normalize_paroxython_comments(string)
                if n == 0:
                    continue
                result.append(string)
            elif token == STRING and previous_token in (INDENT, DEDENT, NEWLINE):
                result.append("pass\n")  # replace the docstring by a pass statement
            else:
                result.append(string)
            if (previous_token, token) == (NEWLINE, NL):
                previous_token = NEWLINE
            else:
                previous_token = token
            (previous_end_row, previous_end_col) = (end_row, end_col)
        text = "".join(result).strip()
        text = Cleanup.suppress_blank_lines(text)
        text = Cleanup.suppress_useless_pass_statements(text)
        return Source(text)

    @staticmethod
    def suppress_first_comments(
        source: str, sub: Callable = regex.compile(r"\A(#.*\n)*").sub
    ) -> str:
        return sub("", source)

    @staticmethod
    def suppress_main_guard(
        source: str, sub: Callable = regex.compile(r"(?ms)^if +__name__ *== *.__main__. *:.+").sub
    ) -> str:
        return sub("", source)

    @staticmethod
    def suppress_sys_path_injection(
        source: str,
        sub: Callable = regex.compile(r'(?m)^__import__\("sys"\)\.path\[0:0\] = .+\n').sub,
    ) -> str:
        return sub("", source)

    @staticmethod
    def normalize_paroxython_comments(
        source: str,
        subn: Callable = regex.compile(
            r"\s*".join(regex.split(r"(\w+)", HINT_COMMENT.replace(" ", ""))) + r"\s*",
            regex.IGNORECASE,
        ).subn,
    ) -> Tuple[str, int]:
        return subn(f"{HINT_COMMENT} ", source)

    @staticmethod
    def suppress_blank_lines(source: str, sub: Callable = regex.compile(r"\s*\n").sub) -> str:
        return sub("\n", source)

    @staticmethod
    def suppress_useless_pass_statements(
        source: str,
        sub: Callable = regex.compile(r"(?m)^( *)pass\n\1(?!\s)").sub,
    ) -> str:
        return sub(r"\1", source)


def centrifugate_hints(
    source: Source, match_isolated_hints: Callable = regex.compile(fr"\s*{HINT_COMMENT} (.+)").match
) -> Source:
    lines = []
    hints: Set[str] = set()
    for line in source.split("\n"):
        m = match_isolated_hints(line)
        if m:
            hints.update(m[1].split())
        else:
            lines.append(line)
    if not hints:
        return source
    for i in (0, -1):
        if f" {HINT_COMMENT}" not in lines[i]:
            lines[i] += f" {HINT_COMMENT}"
    for hint in sorted(hints):
        lines[0] += f" {hint}..."
        lines[-1] += f" ...{hint}"
    return Source("\n".join(lines))


class HintBuffer:
    def __init__(self, description: str) -> None:
        self.result: LabelsSpans = defaultdict(list)
        self.stack: Dict[LabelName, List[int]] = defaultdict(list)
        self.description = description

    def append_hint(self, label_name: LabelName, line_number: int) -> None:
        self.result[label_name].append(Span(line_number, line_number))

    def open_hint(self, label_name: LabelName, line_number: int) -> None:
        self.stack[label_name].append(line_number)

    def get_champion(self, label_name: LabelName) -> List[Tuple[int, "HintBuffer"]]:
        return [(self.stack[label_name][-1], self)] if self.stack[label_name] else []

    def close_hint(self, label_name: LabelName, line_number: int) -> None:
        self.result[label_name].append(Span(self.stack[label_name].pop(), line_number))

    def ensure_stack_is_empty(self) -> None:
        if any(self.stack.values()):
            print_fail(f"Unmatched opening hints for {self.description}: {self.stack}.")

    def get_result(self) -> LabelsSpans:
        return {k: sorted(v) for (k, v) in self.result.items()}


def collect_hints(
    source: Source,
    match_label: Callable = regex.compile(r"^((?:-|\+|\.\.\.|…)?)(\w.*?)((?:\.\.\.|…)?)$").match,
) -> Tuple[LabelsSpans, LabelsSpans]:
    addition = HintBuffer("addition")
    deletion = HintBuffer("deletion")
    for (i, line) in enumerate(source.split("\n"), 1):
        (_, separator, hints) = line.partition(f"{HINT_COMMENT} ")
        if not separator:
            continue
        for hint in hints.split():
            m = match_label(hint)
            if m is None:
                print_fail(f"Malformed hint '{hint}' on line {i}.")
            (before, label, after) = m.groups()
            if before in ("", "+", "-"):
                buffer = deletion if before == "-" else addition
                if after:
                    buffer.open_hint(label, i)
                else:
                    buffer.append_hint(label, i)
            else:  # `before` is in ("...", "…")
                if after:
                    print_fail(f"Illegal last part for hint {m.groups()} on line {i}.")
                champions = addition.get_champion(label) + deletion.get_champion(label)
                if not champions:
                    print_fail(f"Unmatched closing hint {m.groups()} on line {i}.")
                max(champions)[1].close_hint(label, i)
    addition.ensure_stack_is_empty()
    deletion.ensure_stack_is_empty()
    return (addition.get_result(), deletion.get_result())


def remove_hints(
    source: Source,
    sub_hints: Callable = regex.compile(fr"\s*{HINT_COMMENT} .*").sub,
) -> Source:
    return Source(sub_hints("", source).strip())

Functions

def centrifugate_hints(source: Source,
match_isolated_hints: Callable = <built-in method match of regex>
) ‑> Source

Transform the isolated hints into all-encompassing hints.

Description

When a hint is isolated on its own line, it is considered to extend to the entire program. These hints are moved to both ends of the source (centrifugated), and made open on the first line and close on the last line.

Examples

• Some isolated hints (hint_3 and hint_5 are centrifugated).

foo = bar # paroxython: hint_1 hint_2
# paroxython: hint_3
fizz = [
    "foo",
    "bar", # paroxython: hint_4
    # paroxython: hint_5
]

foo = bar # paroxython: hint_1 hint_2 hint_3... hint_5...
fizz = [
    "foo",
    "bar", # paroxython: hint_4
] # paroxython: ...hint_3 ...hint_5

• Some trailing isolated hints (hint_5 and hint_6 are centrifugated).

foo = bar # paroxython: hint_1 hint_2
fizz = [
    "foo",
    "bar", # paroxython: hint_3
] # paroxython: hint_4
# paroxython: hint_5
# paroxython: hint_6

foo = bar # paroxython: hint_1 hint_2 hint_5... hint_6...
fizz = [
    "foo",
    "bar", # paroxython: hint_3
] # paroxython: hint_4 ...hint_5 ...hint_6

• No isolated hints (no centrifugation of the existing hints).

foo = bar # paroxython: hint_1 hint_2
fizz = [
    "foo",
    "bar", # paroxython: hint_4
]

foo = bar # paroxython: hint_1 hint_2
fizz = [
    "foo",
    "bar", # paroxython: hint_4
]

All examples above automatically extracted from test_centrifugate_hints.py.

Argument match_isolated_hints not to be explicitly provided.

Expand source code Browse GitHub

def centrifugate_hints(
    source: Source, match_isolated_hints: Callable = regex.compile(fr"\s*{HINT_COMMENT} (.+)").match
) -> Source:
    lines = []
    hints: Set[str] = set()
    for line in source.split("\n"):
        m = match_isolated_hints(line)
        if m:
            hints.update(m[1].split())
        else:
            lines.append(line)
    if not hints:
        return source
    for i in (0, -1):
        if f" {HINT_COMMENT}" not in lines[i]:
            lines[i] += f" {HINT_COMMENT}"
    for hint in sorted(hints):
        lines[0] += f" {hint}..."
        lines[-1] += f" ...{hint}"
    return Source("\n".join(lines))

def collect_hints(source: Source,
match_label: Callable = <built-in method match of regex>
) ‑> Tuple[Dict[LabelName, List[Span]], Dict[LabelName, List[Span]]]

Schedule for addition or deletion the hints appearing in the comments.

Description

On this stage, the source code is scanned for manual hints. This function set them aside in two separate containers (depending on whether the user intends to add or remove them). They will later be used to tweak the labelling results obtained by static analysis.

Args

source : Source

A source code.

match_label : Callable, optional

A function matching a string:

• optionally starting with "+", "-", "..." or "…";
• continuing with one alphanumeric character ("[_0-9a-zA-Z]") and optionally a sequence of characters other than a newline (together they constitute the label);
• optionally ending with "..." or "…".

Not to be explicitly provided.

Raises

ValueError

Raised in various cases of malformed input.

Returns

Tuple[LabelsSpans, LabelsSpans]

A couple of dictionaries associating labels with lists of spans. The first [resp. second] member collects the labels that the source code author has manually added [resp. suppressed].

Example

>>> source = '''
...     1   foo() # paroxython: +hint_1...
...     2   bar() # paroxython: hint_2 hint_3 -hint_4
...     3   biz() # paroxython: ...hint_1
...     4   buz() # paroxython: -hint_2
... '''
>>> collect_hints(source)
(
    {
        "hint_1": [Span(start=1, end=3)],
        "hint_2": [Span(start=2, end=2)],
        "hint_3": [Span(start=2, end=2)],
    },
    {
        "hint_2": [Span(start=4, end=4)],
        "hint_4": [Span(start=2, end=2)],
    },
)

```

Expand source code Browse GitHub

def collect_hints(
    source: Source,
    match_label: Callable = regex.compile(r"^((?:-|\+|\.\.\.|…)?)(\w.*?)((?:\.\.\.|…)?)$").match,
) -> Tuple[LabelsSpans, LabelsSpans]:
    addition = HintBuffer("addition")
    deletion = HintBuffer("deletion")
    for (i, line) in enumerate(source.split("\n"), 1):
        (_, separator, hints) = line.partition(f"{HINT_COMMENT} ")
        if not separator:
            continue
        for hint in hints.split():
            m = match_label(hint)
            if m is None:
                print_fail(f"Malformed hint '{hint}' on line {i}.")
            (before, label, after) = m.groups()
            if before in ("", "+", "-"):
                buffer = deletion if before == "-" else addition
                if after:
                    buffer.open_hint(label, i)
                else:
                    buffer.append_hint(label, i)
            else:  # `before` is in ("...", "…")
                if after:
                    print_fail(f"Illegal last part for hint {m.groups()} on line {i}.")
                champions = addition.get_champion(label) + deletion.get_champion(label)
                if not champions:
                    print_fail(f"Unmatched closing hint {m.groups()} on line {i}.")
                max(champions)[1].close_hint(label, i)
    addition.ensure_stack_is_empty()
    deletion.ensure_stack_is_empty()
    return (addition.get_result(), deletion.get_result())

def remove_hints(source: Source,
sub_hints: Callable = <built-in method sub of regex>
) ‑> Source

Once they are collected, remove all Paroxython hints from the given source.

Argument sub_hints not to be explicitly provided.

Expand source code Browse GitHub

def remove_hints(
    source: Source,
    sub_hints: Callable = regex.compile(fr"\s*{HINT_COMMENT} .*").sub,
) -> Source:
    return Source(sub_hints("", source).strip())

Classes

class Cleanup (cleanup_strategy: str)

Collection of cleaning static methods to be applied to a source code.

Args

cleanup_strategy : str

The name of the transformation. Currently, any other name than "full" (corresponding to full_cleaning() comes down to the identity function).

Returns

Callable[[Source], Source]

A function that will clean up a given source code.

Expand source code Browse GitHub

class Cleanup:
    def __init__(self, cleanup_strategy: str):
        self.run = lambda source: source
        if cleanup_strategy == "full":
            self.run = Cleanup.full_cleaning

    @staticmethod
    def full_cleaning(source: Source) -> Source:
        r'''Remove as much noise (comments, etc.) as possible in the given source code.

        Description:
            1. Suppress initial comments.
            2. Suppress `if __name__ == '__main__'` part.
            3. Suppress `sys.path` injections (frequently used to simplify `import` behavior).
            4. Replace tabulations with spaces.
            5. Normalize Paroxython hint comments.
            6. Replace docstrings with `pass` statements (most of which being suppressed at the end
               of the process).
            7. Suppress empty or blank lines.
            8. Suppress useless `pass` statements.

        Examples:
            - Empty or blank lines are suppressed.
            ```python
            foo = bar


            bar = buzz

            ```
            ```python
            foo = bar
            bar = buzz
            ```
            - Normal inline comments are suppressed.
            ```python
            foo = bar # lorem ipsum
            fizz = [
                "foo", # lorem
                "bar", # ipsum
            ]
            ```
            ```python
            foo = bar
            fizz = [
                "foo",
                "bar",
            ]
            ```
            - Commented lines are suppressed.
            ```python
            # lorem ipsum
            # dolor amet
            def foo(bar):
                # lorem
                # ipsum
                if foo == bar:
                    # lorem
                    return []
                # lorem
            # ipsum
            ```
            ```python
            def foo(bar):
                if foo == bar:
                    return []
            ```
            - Shebang is suppressed.
            ```python
            #!/usr/bin/env python
            foobar()
            ```
            ```python
            foobar()
            ```
            - Encoding declaration is suppressed.
            ```python
            # coding=utf8
            foobar()
            ```
            ```python
            foobar()
            ```
            - Docstrings of classes and functions are suppressed.
            ```python
            class Foo:
                def foo(self, bar):
                    return 1
            def bar():
                return 2
            ```
            ```python
            class Foo:
                def foo(self, bar):
                    return 1
            def bar():
                return 2
            ```
            - Docstrings of modules are suppressed.
            ```python
            a = """consectetur
            adipiscing
            elit.
            ```
            - A `pass` statement is added to empty classes if needed.
            ```python
            class Foo:
            ```
            ```python
            class Foo:
                pass
            ```
            - Useless `pass` statements are suppressed.
            ```python
            foo()
            pass
            if bar():
                pass
            else:
                pass
                foobar()
                pass
            ```
            ```python
            foo()
            if bar():
                pass
            else:
                foobar()
                pass
            ```
            - Mixes of comments and docstrings are suppressed.
            ```python
            # Lorem
            # Ipsum
            foobar()
            ```
            ```python
            foobar()
            ```
            - Paroxython's label hints are **preserved** and **normalized**.
            ```python
            foo = bar #   Paroxython   :   hint_1   hint_2
            #paroxython: hint_3
            fizz = [
                "foo", # lorem
                "bar", # paroxython: hint_4
            ]
            ```
            ```python
            foo = bar # paroxython: hint_1   hint_2
            # paroxython: hint_3
            fizz = [
                "foo",
                "bar", # paroxython: hint_4
            ]
            ```
            - `if __name__ == "__main__":` part is suppressed.
            ```python
            foo = bar
            if __name__ == "__main__":
                bar = foo
            ```
            ```python
            foo = bar
            ```
            - `sys.path` injections are suppressed.
            ```python
            __import__("sys").path[0:0] = ["programs"]
            (28433 * 2**7830457 + 1) % 10**10
            ```
            ```python
            (28433 * 2**7830457 + 1) % 10**10
            ```

            All examples above automatically extracted from
            [test_cleanup_source.py](https://github.com/laowantong/paroxython/blob/master/tests/test_cleanup_source.py).

        .. warning::
            Useless pass statements are preserved when they are not followed by a line with a same
            indentation level.
        '''
        result = []
        previous_token = INDENT
        (previous_end_row, previous_end_col) = (-1, 0)
        text = str(source)
        text = Cleanup.suppress_first_comments(text)
        text = Cleanup.suppress_main_guard(text)
        text = Cleanup.suppress_sys_path_injection(text)
        text = text.replace("\t", "    ")
        lines = iter(text.split("\n"))
        for token_info in generate_tokens(lambda: next(lines) + "\n"):
            (token, string, (start_row, start_col), (end_row, end_col), _) = token_info
            if start_row > previous_end_row:
                previous_end_col = 0
            result.append(" " * max(0, start_col - previous_end_col))
            if token == COMMENT:
                (string, n) = Cleanup.normalize_paroxython_comments(string)
                if n == 0:
                    continue
                result.append(string)
            elif token == STRING and previous_token in (INDENT, DEDENT, NEWLINE):
                result.append("pass\n")  # replace the docstring by a pass statement
            else:
                result.append(string)
            if (previous_token, token) == (NEWLINE, NL):
                previous_token = NEWLINE
            else:
                previous_token = token
            (previous_end_row, previous_end_col) = (end_row, end_col)
        text = "".join(result).strip()
        text = Cleanup.suppress_blank_lines(text)
        text = Cleanup.suppress_useless_pass_statements(text)
        return Source(text)

    @staticmethod
    def suppress_first_comments(
        source: str, sub: Callable = regex.compile(r"\A(#.*\n)*").sub
    ) -> str:
        return sub("", source)

    @staticmethod
    def suppress_main_guard(
        source: str, sub: Callable = regex.compile(r"(?ms)^if +__name__ *== *.__main__. *:.+").sub
    ) -> str:
        return sub("", source)

    @staticmethod
    def suppress_sys_path_injection(
        source: str,
        sub: Callable = regex.compile(r'(?m)^__import__\("sys"\)\.path\[0:0\] = .+\n').sub,
    ) -> str:
        return sub("", source)

    @staticmethod
    def normalize_paroxython_comments(
        source: str,
        subn: Callable = regex.compile(
            r"\s*".join(regex.split(r"(\w+)", HINT_COMMENT.replace(" ", ""))) + r"\s*",
            regex.IGNORECASE,
        ).subn,
    ) -> Tuple[str, int]:
        return subn(f"{HINT_COMMENT} ", source)

    @staticmethod
    def suppress_blank_lines(source: str, sub: Callable = regex.compile(r"\s*\n").sub) -> str:
        return sub("\n", source)

    @staticmethod
    def suppress_useless_pass_statements(
        source: str,
        sub: Callable = regex.compile(r"(?m)^( *)pass\n\1(?!\s)").sub,
    ) -> str:
        return sub(r"\1", source)

Static methods

def full_cleaning(source: Source) ‑> Source

Remove as much noise (comments, etc.) as possible in the given source code.

Description

1. Suppress initial comments.
2. Suppress if __name__ == '__main__' part.
3. Suppress sys.path injections (frequently used to simplify import behavior).
4. Replace tabulations with spaces.
5. Normalize Paroxython hint comments.
6. Replace docstrings with pass statements (most of which being suppressed at the end of the process).
7. Suppress empty or blank lines.
8. Suppress useless pass statements.

Examples

• Empty or blank lines are suppressed.

foo = bar


bar = buzz

foo = bar
bar = buzz

• Normal inline comments are suppressed.

foo = bar # lorem ipsum
fizz = [
    "foo", # lorem
    "bar", # ipsum
]

foo = bar
fizz = [
    "foo",
    "bar",
]

• Commented lines are suppressed.

# lorem ipsum
# dolor amet
def foo(bar):
    # lorem
    # ipsum
    if foo == bar:
        # lorem
        return []
    # lorem
# ipsum

def foo(bar):
    if foo == bar:
        return []

• Shebang is suppressed.

#!/usr/bin/env python
foobar()

foobar()

• Encoding declaration is suppressed.

# coding=utf8
foobar()

foobar()

• Docstrings of classes and functions are suppressed.

class Foo:
    """Lorem.

    Ipsum dolor.
    """
    def foo(self, bar):
        """Lorem ipsum."""
        return 1
def bar():
    """Ipsum dolor"""
    return 2

class Foo:
    def foo(self, bar):
        return 1
def bar():
    return 2

• Docstrings of modules are suppressed.

"""Lorem.

Ipsum Dolor."""

"""Lorem ipsum dolor sic amet."""

a = """consectetur
adipiscing
elit.
"""

a = """consectetur
adipiscing
elit.
"""

• A pass statement is added to empty classes if needed.

class Foo:
    """Lorem.

    Ipsum dolor.
    """

class Foo:
    pass

• Useless pass statements are suppressed.

foo()
pass
if bar():
    pass
else:
    pass
    foobar()
    pass

foo()
if bar():
    pass
else:
    foobar()
    pass

• Mixes of comments and docstrings are suppressed.

# Lorem
# Ipsum
"""Dolor."""
foobar()

foobar()

• Paroxython's label hints are preserved and normalized.

foo = bar #   Paroxython   :   hint_1   hint_2
#paroxython: hint_3
fizz = [
    "foo", # lorem
    "bar", # paroxython: hint_4
]

foo = bar # paroxython: hint_1   hint_2
# paroxython: hint_3
fizz = [
    "foo",
    "bar", # paroxython: hint_4
]

• if __name__ == "__main__": part is suppressed.

foo = bar
if __name__ == "__main__":
    bar = foo

foo = bar

• sys.path injections are suppressed.

__import__("sys").path[0:0] = ["programs"]
(28433 * 2**7830457 + 1) % 10**10

(28433 * 2**7830457 + 1) % 10**10

All examples above automatically extracted from test_cleanup_source.py.

Warning

Useless pass statements are preserved when they are not followed by a line with a same indentation level.

Expand source code Browse GitHub

@staticmethod
def full_cleaning(source: Source) -> Source:
    r'''Remove as much noise (comments, etc.) as possible in the given source code.

    Description:
        1. Suppress initial comments.
        2. Suppress `if __name__ == '__main__'` part.
        3. Suppress `sys.path` injections (frequently used to simplify `import` behavior).
        4. Replace tabulations with spaces.
        5. Normalize Paroxython hint comments.
        6. Replace docstrings with `pass` statements (most of which being suppressed at the end
           of the process).
        7. Suppress empty or blank lines.
        8. Suppress useless `pass` statements.

    Examples:
        - Empty or blank lines are suppressed.
        ```python
        foo = bar


        bar = buzz

        ```
        ```python
        foo = bar
        bar = buzz
        ```
        - Normal inline comments are suppressed.
        ```python
        foo = bar # lorem ipsum
        fizz = [
            "foo", # lorem
            "bar", # ipsum
        ]
        ```
        ```python
        foo = bar
        fizz = [
            "foo",
            "bar",
        ]
        ```
        - Commented lines are suppressed.
        ```python
        # lorem ipsum
        # dolor amet
        def foo(bar):
            # lorem
            # ipsum
            if foo == bar:
                # lorem
                return []
            # lorem
        # ipsum
        ```
        ```python
        def foo(bar):
            if foo == bar:
                return []
        ```
        - Shebang is suppressed.
        ```python
        #!/usr/bin/env python
        foobar()
        ```
        ```python
        foobar()
        ```
        - Encoding declaration is suppressed.
        ```python
        # coding=utf8
        foobar()
        ```
        ```python
        foobar()
        ```
        - Docstrings of classes and functions are suppressed.
        ```python
        class Foo:
            def foo(self, bar):
                return 1
        def bar():
            return 2
        ```
        ```python
        class Foo:
            def foo(self, bar):
                return 1
        def bar():
            return 2
        ```
        - Docstrings of modules are suppressed.
        ```python
        a = """consectetur
        adipiscing
        elit.
        ```
        - A `pass` statement is added to empty classes if needed.
        ```python
        class Foo:
        ```
        ```python
        class Foo:
            pass
        ```
        - Useless `pass` statements are suppressed.
        ```python
        foo()
        pass
        if bar():
            pass
        else:
            pass
            foobar()
            pass
        ```
        ```python
        foo()
        if bar():
            pass
        else:
            foobar()
            pass
        ```
        - Mixes of comments and docstrings are suppressed.
        ```python
        # Lorem
        # Ipsum
        foobar()
        ```
        ```python
        foobar()
        ```
        - Paroxython's label hints are **preserved** and **normalized**.
        ```python
        foo = bar #   Paroxython   :   hint_1   hint_2
        #paroxython: hint_3
        fizz = [
            "foo", # lorem
            "bar", # paroxython: hint_4
        ]
        ```
        ```python
        foo = bar # paroxython: hint_1   hint_2
        # paroxython: hint_3
        fizz = [
            "foo",
            "bar", # paroxython: hint_4
        ]
        ```
        - `if __name__ == "__main__":` part is suppressed.
        ```python
        foo = bar
        if __name__ == "__main__":
            bar = foo
        ```
        ```python
        foo = bar
        ```
        - `sys.path` injections are suppressed.
        ```python
        __import__("sys").path[0:0] = ["programs"]
        (28433 * 2**7830457 + 1) % 10**10
        ```
        ```python
        (28433 * 2**7830457 + 1) % 10**10
        ```

        All examples above automatically extracted from
        [test_cleanup_source.py](https://github.com/laowantong/paroxython/blob/master/tests/test_cleanup_source.py).

    .. warning::
        Useless pass statements are preserved when they are not followed by a line with a same
        indentation level.
    '''
    result = []
    previous_token = INDENT
    (previous_end_row, previous_end_col) = (-1, 0)
    text = str(source)
    text = Cleanup.suppress_first_comments(text)
    text = Cleanup.suppress_main_guard(text)
    text = Cleanup.suppress_sys_path_injection(text)
    text = text.replace("\t", "    ")
    lines = iter(text.split("\n"))
    for token_info in generate_tokens(lambda: next(lines) + "\n"):
        (token, string, (start_row, start_col), (end_row, end_col), _) = token_info
        if start_row > previous_end_row:
            previous_end_col = 0
        result.append(" " * max(0, start_col - previous_end_col))
        if token == COMMENT:
            (string, n) = Cleanup.normalize_paroxython_comments(string)
            if n == 0:
                continue
            result.append(string)
        elif token == STRING and previous_token in (INDENT, DEDENT, NEWLINE):
            result.append("pass\n")  # replace the docstring by a pass statement
        else:
            result.append(string)
        if (previous_token, token) == (NEWLINE, NL):
            previous_token = NEWLINE
        else:
            previous_token = token
        (previous_end_row, previous_end_col) = (end_row, end_col)
    text = "".join(result).strip()
    text = Cleanup.suppress_blank_lines(text)
    text = Cleanup.suppress_useless_pass_statements(text)
    return Source(text)

def suppress_first_comments(source: str,
sub: Callable = <built-in method sub of regex>
) ‑> str

Replace all comments placed on the first lines of the source code.

Argument sub not to be explicitly provided.

Expand source code Browse GitHub

@staticmethod
def suppress_first_comments(
    source: str, sub: Callable = regex.compile(r"\A(#.*\n)*").sub
) -> str:
    return sub("", source)

def suppress_main_guard(source: str,
sub: Callable = <built-in method sub of regex>
) ‑> str

Suppress if __name__ == '__main__' part.

Argument sub not to be explicitly provided.

Expand source code Browse GitHub

@staticmethod
def suppress_main_guard(
    source: str, sub: Callable = regex.compile(r"(?ms)^if +__name__ *== *.__main__. *:.+").sub
) -> str:
    return sub("", source)

def suppress_sys_path_injection(source: str,
sub: Callable = <built-in method sub of regex>
) ‑> str

Suppress lines starting with __import__("sys").path[0:0] =.

Argument sub not to be explicitly provided.

Expand source code Browse GitHub

@staticmethod
def suppress_sys_path_injection(
    source: str,
    sub: Callable = regex.compile(r'(?m)^__import__\("sys"\)\.path\[0:0\] = .+\n').sub,
) -> str:
    return sub("", source)

def normalize_paroxython_comments(source: str,
subn: Callable = <built-in method subn of regex>
) ‑> Tuple[str, int]

Replace and count all the hint comment strings, made space- and case-insensitive.

For instance, if HINT_COMMENT is "# paroxython:" (its default value), the actual search pattern will be r"(?i)#\s*paroxython\s*:\s*".

Argument subn not to be explicitly provided.

Expand source code Browse GitHub

@staticmethod
def normalize_paroxython_comments(
    source: str,
    subn: Callable = regex.compile(
        r"\s*".join(regex.split(r"(\w+)", HINT_COMMENT.replace(" ", ""))) + r"\s*",
        regex.IGNORECASE,
    ).subn,
) -> Tuple[str, int]:
    return subn(f"{HINT_COMMENT} ", source)

def suppress_blank_lines(source: str,
sub: Callable = <built-in method sub of regex>
) ‑> str

Suppress all empty or blank lines in the given source.

Argument sub not to be explicitly provided.

Expand source code Browse GitHub

@staticmethod
def suppress_blank_lines(source: str, sub: Callable = regex.compile(r"\s*\n").sub) -> str:
    return sub("\n", source)

def suppress_useless_pass_statements(source: str,
sub: Callable = <built-in method sub of regex>
) ‑> str

Suppress all pass statements followed by a line with a same level of indentation.

Argument sub not to be explicitly provided.

Expand source code Browse GitHub

@staticmethod
def suppress_useless_pass_statements(
    source: str,
    sub: Callable = regex.compile(r"(?m)^( *)pass\n\1(?!\s)").sub,
) -> str:
    return sub(r"\1", source)

class HintBuffer (description: str)

Layer on the operations of appending, opening and closing a hint, used by collect_hints().

Expand source code Browse GitHub

class HintBuffer:
    def __init__(self, description: str) -> None:
        self.result: LabelsSpans = defaultdict(list)
        self.stack: Dict[LabelName, List[int]] = defaultdict(list)
        self.description = description

    def append_hint(self, label_name: LabelName, line_number: int) -> None:
        self.result[label_name].append(Span(line_number, line_number))

    def open_hint(self, label_name: LabelName, line_number: int) -> None:
        self.stack[label_name].append(line_number)

    def get_champion(self, label_name: LabelName) -> List[Tuple[int, "HintBuffer"]]:
        return [(self.stack[label_name][-1], self)] if self.stack[label_name] else []

    def close_hint(self, label_name: LabelName, line_number: int) -> None:
        self.result[label_name].append(Span(self.stack[label_name].pop(), line_number))

    def ensure_stack_is_empty(self) -> None:
        if any(self.stack.values()):
            print_fail(f"Unmatched opening hints for {self.description}: {self.stack}.")

    def get_result(self) -> LabelsSpans:
        return {k: sorted(v) for (k, v) in self.result.items()}

Methods

def append_hint(self,
label_name: LabelName,
line_number: int
) ‑> NoneType

Label a span consisting in the given line number.

Expand source code Browse GitHub

def append_hint(self, label_name: LabelName, line_number: int) -> None:
    self.result[label_name].append(Span(line_number, line_number))

def open_hint(self,
label_name: LabelName,
line_number: int
) ‑> NoneType

Store the line number associated with a given opening hint.

Expand source code Browse GitHub

def open_hint(self, label_name: LabelName, line_number: int) -> None:
    self.stack[label_name].append(line_number)

def get_champion(self,
label_name: LabelName
) ‑> List[Tuple[int, HintBuffer]]

Return the last line number (if any) of the opening label, plus itself.

Expand source code Browse GitHub

def get_champion(self, label_name: LabelName) -> List[Tuple[int, "HintBuffer"]]:
    return [(self.stack[label_name][-1], self)] if self.stack[label_name] else []

def close_hint(self,
label_name: LabelName,
line_number: int
) ‑> NoneType

Label a span opened above and closed on the given line number.

Expand source code Browse GitHub

def close_hint(self, label_name: LabelName, line_number: int) -> None:
    self.result[label_name].append(Span(self.stack[label_name].pop(), line_number))

def ensure_stack_is_empty(self) ‑> NoneType

Raise an error iff there remains at least one open label.

Expand source code Browse GitHub

def ensure_stack_is_empty(self) -> None:
    if any(self.stack.values()):
        print_fail(f"Unmatched opening hints for {self.description}: {self.stack}.")

def get_result(self) ‑> Dict[LabelName, List[Span]]

Return the result with sorted spans.

Expand source code Browse GitHub

def get_result(self) -> LabelsSpans:
    return {k: sorted(v) for (k, v) in self.result.items()}