Coverage for C:\leo.repo\leo-editor\leo\core\leoAst.py: 100%
Shortcuts on this page
r m x toggle line displays
j k next/prev highlighted chunk
0 (zero) top of page
1 (one) first highlighted chunk
Shortcuts on this page
r m x toggle line displays
j k next/prev highlighted chunk
0 (zero) top of page
1 (one) first highlighted chunk
1# -*- coding: utf-8 -*-
2#@+leo-ver=5-thin
3#@+node:ekr.20141012064706.18389: * @file leoAst.py
4#@@first
5# This file is part of Leo: https://leoeditor.com
6# Leo's copyright notice is based on the MIT license: http://leoeditor.com/license.html
7#@+<< docstring >>
8#@+node:ekr.20200113081838.1: ** << docstring >> (leoAst.py)
9"""
10leoAst.py: This file does not depend on Leo in any way.
12The classes in this file unify python's token-based and ast-based worlds by
13creating two-way links between tokens in the token list and ast nodes in
14the parse tree. For more details, see the "Overview" section below.
17**Stand-alone operation**
19usage:
20 leoAst.py --help
21 leoAst.py [--fstringify | --fstringify-diff | --orange | --orange-diff] PATHS
22 leoAst.py --py-cov [ARGS]
23 leoAst.py --pytest [ARGS]
24 leoAst.py --unittest [ARGS]
26examples:
27 --py-cov "-f TestOrange"
28 --pytest "-f TestOrange"
29 --unittest TestOrange
31positional arguments:
32 PATHS directory or list of files
34optional arguments:
35 -h, --help show this help message and exit
36 --fstringify leonine fstringify
37 --fstringify-diff show fstringify diff
38 --orange leonine Black
39 --orange-diff show orange diff
40 --py-cov run pytest --cov on leoAst.py
41 --pytest run pytest on leoAst.py
42 --unittest run unittest on leoAst.py
45**Overview**
47leoAst.py unifies python's token-oriented and ast-oriented worlds.
49leoAst.py defines classes that create two-way links between tokens
50created by python's tokenize module and parse tree nodes created by
51python's ast module:
53The Token Order Generator (TOG) class quickly creates the following
54links:
56- An *ordered* children array from each ast node to its children.
58- A parent link from each ast.node to its parent.
60- Two-way links between tokens in the token list, a list of Token
61 objects, and the ast nodes in the parse tree:
63 - For each token, token.node contains the ast.node "responsible" for
64 the token.
66 - For each ast node, node.first_i and node.last_i are indices into
67 the token list. These indices give the range of tokens that can be
68 said to be "generated" by the ast node.
70Once the TOG class has inserted parent/child links, the Token Order
71Traverser (TOT) class traverses trees annotated with parent/child
72links extremely quickly.
75**Applicability and importance**
77Many python developers will find asttokens meets all their needs.
78asttokens is well documented and easy to use. Nevertheless, two-way
79links are significant additions to python's tokenize and ast modules:
81- Links from tokens to nodes are assigned to the nearest possible ast
82 node, not the nearest statement, as in asttokens. Links can easily
83 be reassigned, if desired.
85- The TOG and TOT classes are intended to be the foundation of tools
86 such as fstringify and black.
88- The TOG class solves real problems, such as:
89 https://stackoverflow.com/questions/16748029/
91**Known bug**
93This file has no known bugs *except* for Python version 3.8.
95For Python 3.8, syncing tokens will fail for function call such as:
97 f(1, x=2, *[3, 4], y=5)
99that is, for calls where keywords appear before non-keyword args.
101There are no plans to fix this bug. The workaround is to use Python version
1023.9 or above.
105**Figures of merit**
107Simplicity: The code consists primarily of a set of generators, one
108for every kind of ast node.
110Speed: The TOG creates two-way links between tokens and ast nodes in
111roughly the time taken by python's tokenize.tokenize and ast.parse
112library methods. This is substantially faster than the asttokens,
113black or fstringify tools. The TOT class traverses trees annotated
114with parent/child links even more quickly.
116Memory: The TOG class makes no significant demands on python's
117resources. Generators add nothing to python's call stack.
118TOG.node_stack is the only variable-length data. This stack resides in
119python's heap, so its length is unimportant. In the worst case, it
120might contain a few thousand entries. The TOT class uses no
121variable-length data at all.
123**Links**
125Leo...
126Ask for help: https://groups.google.com/forum/#!forum/leo-editor
127Report a bug: https://github.com/leo-editor/leo-editor/issues
128leoAst.py docs: http://leoeditor.com/appendices.html#leoast-py
130Other tools...
131asttokens: https://pypi.org/project/asttokens
132black: https://pypi.org/project/black/
133fstringify: https://pypi.org/project/fstringify/
135Python modules...
136tokenize.py: https://docs.python.org/3/library/tokenize.html
137ast.py https://docs.python.org/3/library/ast.html
139**Studying this file**
141I strongly recommend that you use Leo when studying this code so that you
142will see the file's intended outline structure.
144Without Leo, you will see only special **sentinel comments** that create
145Leo's outline structure. These comments have the form::
147 `#@<comment-kind>:<user-id>.<timestamp>.<number>: <outline-level> <headline>`
148"""
149#@-<< docstring >>
150#@+<< imports >>
151#@+node:ekr.20200105054219.1: ** << imports >> (leoAst.py)
152import argparse
153import ast
154import codecs
155import difflib
156import glob
157import io
158import os
159import re
160import sys
161import textwrap
162import tokenize
163import traceback
164from typing import Any, Callable, Dict, Generator, List, Optional, Tuple, Union
165#@-<< imports >>
166Node = ast.AST
167ActionList = List[Tuple[Callable, Any]]
168v1, v2, junk1, junk2, junk3 = sys.version_info
169py_version = (v1, v2)
171# Async tokens exist only in Python 3.5 and 3.6.
172# https://docs.python.org/3/library/token.html
173has_async_tokens = (3, 5) <= py_version <= (3, 6)
175# has_position_only_params = (v1, v2) >= (3, 8)
176#@+others
177#@+node:ekr.20191226175251.1: ** class LeoGlobals
178#@@nosearch
181class LeoGlobals: # pragma: no cover
182 """
183 Simplified version of functions in leoGlobals.py.
184 """
186 total_time = 0.0 # For unit testing.
188 #@+others
189 #@+node:ekr.20191226175903.1: *3* LeoGlobals.callerName
190 def callerName(self, n: int) -> str:
191 """Get the function name from the call stack."""
192 try:
193 f1 = sys._getframe(n)
194 code1 = f1.f_code
195 return code1.co_name
196 except Exception:
197 return ''
198 #@+node:ekr.20191226175426.1: *3* LeoGlobals.callers
199 def callers(self, n: int=4) -> str:
200 """
201 Return a string containing a comma-separated list of the callers
202 of the function that called g.callerList.
203 """
204 i, result = 2, []
205 while True:
206 s = self.callerName(n=i)
207 if s:
208 result.append(s)
209 if not s or len(result) >= n:
210 break
211 i += 1
212 return ','.join(reversed(result))
213 #@+node:ekr.20191226190709.1: *3* leoGlobals.es_exception & helper
214 def es_exception(self, full: bool=True) -> Tuple[str, int]:
215 typ, val, tb = sys.exc_info()
216 for line in traceback.format_exception(typ, val, tb):
217 print(line)
218 fileName, n = self.getLastTracebackFileAndLineNumber()
219 return fileName, n
220 #@+node:ekr.20191226192030.1: *4* LeoGlobals.getLastTracebackFileAndLineNumber
221 def getLastTracebackFileAndLineNumber(self) -> Tuple[str, int]:
222 typ, val, tb = sys.exc_info()
223 if typ == SyntaxError:
224 # IndentationError is a subclass of SyntaxError.
225 # SyntaxError *does* have 'filename' and 'lineno' attributes.
226 return val.filename, val.lineno
227 #
228 # Data is a list of tuples, one per stack entry.
229 # The tuples have the form (filename, lineNumber, functionName, text).
230 data = traceback.extract_tb(tb)
231 item = data[-1] # Get the item at the top of the stack.
232 filename, n, functionName, text = item
233 return filename, n
234 #@+node:ekr.20200220065737.1: *3* LeoGlobals.objToString
235 def objToString(self, obj: Any, tag: str=None) -> str:
236 """Simplified version of g.printObj."""
237 result = []
238 if tag:
239 result.append(f"{tag}...")
240 if isinstance(obj, str):
241 obj = g.splitLines(obj)
242 if isinstance(obj, list):
243 result.append('[')
244 for z in obj:
245 result.append(f" {z!r}")
246 result.append(']')
247 elif isinstance(obj, tuple):
248 result.append('(')
249 for z in obj:
250 result.append(f" {z!r}")
251 result.append(')')
252 else:
253 result.append(repr(obj))
254 result.append('')
255 return '\n'.join(result)
256 #@+node:ekr.20220327132500.1: *3* LeoGlobals.pdb
257 def pdb(self) -> None:
258 import pdb as _pdb
259 # pylint: disable=forgotten-debug-statement
260 _pdb.set_trace()
261 #@+node:ekr.20191226190425.1: *3* LeoGlobals.plural
262 def plural(self, obj: Any) -> str:
263 """Return "s" or "" depending on n."""
264 if isinstance(obj, (list, tuple, str)):
265 n = len(obj)
266 else:
267 n = obj
268 return '' if n == 1 else 's'
269 #@+node:ekr.20191226175441.1: *3* LeoGlobals.printObj
270 def printObj(self, obj: Any, tag: str=None) -> None:
271 """Simplified version of g.printObj."""
272 print(self.objToString(obj, tag))
273 #@+node:ekr.20220327120618.1: *3* LeoGlobals.shortFileName
274 def shortFileName(self, fileName: str) -> str:
275 """Return the base name of a path."""
276 return os.path.basename(fileName) if fileName else ''
277 #@+node:ekr.20191226190131.1: *3* LeoGlobals.splitLines
278 def splitLines(self, s: str) -> List[str]:
279 """Split s into lines, preserving the number of lines and
280 the endings of all lines, including the last line."""
281 # g.stat()
282 if s:
283 return s.splitlines(True)
284 # This is a Python string function!
285 return []
286 #@+node:ekr.20191226190844.1: *3* LeoGlobals.toEncodedString
287 def toEncodedString(self, s: Any, encoding: str='utf-8') -> bytes:
288 """Convert unicode string to an encoded string."""
289 if not isinstance(s, str):
290 return s
291 try:
292 s = s.encode(encoding, "strict")
293 except UnicodeError:
294 s = s.encode(encoding, "replace")
295 print(f"toEncodedString: Error converting {s!r} to {encoding}")
296 return s
297 #@+node:ekr.20191226190006.1: *3* LeoGlobals.toUnicode
298 def toUnicode(self, s: Any, encoding: str='utf-8') -> str:
299 """Convert bytes to unicode if necessary."""
300 tag = 'g.toUnicode'
301 if isinstance(s, str):
302 return s
303 if not isinstance(s, bytes):
304 print(f"{tag}: bad s: {s!r}")
305 return ''
306 b: bytes = s
307 try:
308 s2 = b.decode(encoding, 'strict')
309 except(UnicodeDecodeError, UnicodeError):
310 s2 = b.decode(encoding, 'replace')
311 print(f"{tag}: unicode error. encoding: {encoding!r}, s2:\n{s2!r}")
312 g.trace(g.callers())
313 except Exception:
314 g.es_exception()
315 print(f"{tag}: unexpected error! encoding: {encoding!r}, s2:\n{s2!r}")
316 g.trace(g.callers())
317 return s2
318 #@+node:ekr.20191226175436.1: *3* LeoGlobals.trace
319 def trace(self, *args: Any) -> None:
320 """Print a tracing message."""
321 # Compute the caller name.
322 try:
323 f1 = sys._getframe(1)
324 code1 = f1.f_code
325 name = code1.co_name
326 except Exception:
327 name = ''
328 print(f"{name}: {' '.join(str(z) for z in args)}")
329 #@+node:ekr.20191226190241.1: *3* LeoGlobals.truncate
330 def truncate(self, s: str, n: int) -> str:
331 """Return s truncated to n characters."""
332 if len(s) <= n:
333 return s
334 s2 = s[: n - 3] + f"...({len(s)})"
335 return s2 + '\n' if s.endswith('\n') else s2
336 #@-others
337#@+node:ekr.20200702114522.1: ** leoAst.py: top-level commands
338#@+node:ekr.20200702114557.1: *3* command: fstringify_command
339def fstringify_command(files: List[str]) -> None:
340 """
341 Entry point for --fstringify.
343 Fstringify the given file, overwriting the file.
344 """
345 for filename in files: # pragma: no cover
346 if os.path.exists(filename):
347 print(f"fstringify {filename}")
348 Fstringify().fstringify_file_silent(filename)
349 else:
350 print(f"file not found: {filename}")
351#@+node:ekr.20200702121222.1: *3* command: fstringify_diff_command
352def fstringify_diff_command(files: List[str]) -> None:
353 """
354 Entry point for --fstringify-diff.
356 Print the diff that would be produced by fstringify.
357 """
358 for filename in files: # pragma: no cover
359 if os.path.exists(filename):
360 print(f"fstringify-diff {filename}")
361 Fstringify().fstringify_file_diff(filename)
362 else:
363 print(f"file not found: {filename}")
364#@+node:ekr.20200702115002.1: *3* command: orange_command
365def orange_command(files: List[str]) -> None:
367 for filename in files: # pragma: no cover
368 if os.path.exists(filename):
369 print(f"orange {filename}")
370 Orange().beautify_file(filename)
371 else:
372 print(f"file not found: {filename}")
373#@+node:ekr.20200702121315.1: *3* command: orange_diff_command
374def orange_diff_command(files: List[str]) -> None:
376 for filename in files: # pragma: no cover
377 if os.path.exists(filename):
378 print(f"orange-diff {filename}")
379 Orange().beautify_file_diff(filename)
380 else:
381 print(f"file not found: {filename}")
382#@+node:ekr.20160521104628.1: ** leoAst.py: top-level utils
383if 1: # pragma: no cover
384 #@+others
385 #@+node:ekr.20200702102239.1: *3* function: main (leoAst.py)
386 def main() -> None:
387 """Run commands specified by sys.argv."""
388 description = textwrap.dedent("""\
389 leo-editor/leo/unittests/core/test_leoAst.py contains unit tests (100% coverage).
390 """)
391 parser = argparse.ArgumentParser(description=description, formatter_class=argparse.RawTextHelpFormatter)
392 parser.add_argument('PATHS', nargs='*', help='directory or list of files')
393 group = parser.add_mutually_exclusive_group(required=False) # Don't require any args.
394 add = group.add_argument
395 add('--fstringify', dest='f', action='store_true', help='leonine fstringify')
396 add('--fstringify-diff', dest='fd', action='store_true', help='show fstringify diff')
397 add('--orange', dest='o', action='store_true', help='leonine Black')
398 add('--orange-diff', dest='od', action='store_true', help='show orange diff')
399 args = parser.parse_args()
400 files = args.PATHS
401 if len(files) == 1 and os.path.isdir(files[0]):
402 files = glob.glob(f"{files[0]}{os.sep}*.py")
403 if args.f:
404 fstringify_command(files)
405 if args.fd:
406 fstringify_diff_command(files)
407 if args.o:
408 orange_command(files)
409 if args.od:
410 orange_diff_command(files)
411 #@+node:ekr.20200107114409.1: *3* functions: reading & writing files
412 #@+node:ekr.20200218071822.1: *4* function: regularize_nls
413 def regularize_nls(s: str) -> str:
414 """Regularize newlines within s."""
415 return s.replace('\r\n', '\n').replace('\r', '\n')
416 #@+node:ekr.20200106171502.1: *4* function: get_encoding_directive
417 # This is the pattern in PEP 263.
418 encoding_pattern = re.compile(r'^[ \t\f]*#.*?coding[:=][ \t]*([-_.a-zA-Z0-9]+)')
420 def get_encoding_directive(bb: bytes) -> str:
421 """
422 Get the encoding from the encoding directive at the start of a file.
424 bb: The bytes of the file.
426 Returns the codec name, or 'UTF-8'.
428 Adapted from pyzo. Copyright 2008 to 2020 by Almar Klein.
429 """
430 for line in bb.split(b'\n', 2)[:2]:
431 # Try to make line a string
432 try:
433 line2 = line.decode('ASCII').strip()
434 except Exception:
435 continue
436 # Does the line match the PEP 263 pattern?
437 m = encoding_pattern.match(line2)
438 if not m:
439 continue
440 # Is it a known encoding? Correct the name if it is.
441 try:
442 c = codecs.lookup(m.group(1))
443 return c.name
444 except Exception:
445 pass
446 return 'UTF-8'
447 #@+node:ekr.20200103113417.1: *4* function: read_file
448 def read_file(filename: str, encoding: str='utf-8') -> Optional[str]:
449 """
450 Return the contents of the file with the given name.
451 Print an error message and return None on error.
452 """
453 tag = 'read_file'
454 try:
455 # Translate all newlines to '\n'.
456 with open(filename, 'r', encoding=encoding) as f:
457 s = f.read()
458 return regularize_nls(s)
459 except Exception:
460 print(f"{tag}: can not read {filename}")
461 return None
462 #@+node:ekr.20200106173430.1: *4* function: read_file_with_encoding
463 def read_file_with_encoding(filename: str) -> Tuple[str, str]:
464 """
465 Read the file with the given name, returning (e, s), where:
467 s is the string, converted to unicode, or '' if there was an error.
469 e is the encoding of s, computed in the following order:
471 - The BOM encoding if the file starts with a BOM mark.
472 - The encoding given in the # -*- coding: utf-8 -*- line.
473 - The encoding given by the 'encoding' keyword arg.
474 - 'utf-8'.
475 """
476 # First, read the file.
477 tag = 'read_with_encoding'
478 try:
479 with open(filename, 'rb') as f:
480 bb = f.read()
481 except Exception:
482 print(f"{tag}: can not read {filename}")
483 if not bb:
484 return 'UTF-8', ''
485 # Look for the BOM.
486 e, bb = strip_BOM(bb)
487 if not e:
488 # Python's encoding comments override everything else.
489 e = get_encoding_directive(bb)
490 s = g.toUnicode(bb, encoding=e)
491 s = regularize_nls(s)
492 return e, s
493 #@+node:ekr.20200106174158.1: *4* function: strip_BOM
494 def strip_BOM(bb: bytes) -> Tuple[Optional[str], bytes]:
495 """
496 bb must be the bytes contents of a file.
498 If bb starts with a BOM (Byte Order Mark), return (e, bb2), where:
500 - e is the encoding implied by the BOM.
501 - bb2 is bb, stripped of the BOM.
503 If there is no BOM, return (None, bb)
504 """
505 assert isinstance(bb, bytes), bb.__class__.__name__
506 table = (
507 # Test longer bom's first.
508 (4, 'utf-32', codecs.BOM_UTF32_BE),
509 (4, 'utf-32', codecs.BOM_UTF32_LE),
510 (3, 'utf-8', codecs.BOM_UTF8),
511 (2, 'utf-16', codecs.BOM_UTF16_BE),
512 (2, 'utf-16', codecs.BOM_UTF16_LE),
513 )
514 for n, e, bom in table:
515 assert len(bom) == n
516 if bom == bb[: len(bom)]:
517 return e, bb[len(bom) :]
518 return None, bb
519 #@+node:ekr.20200103163100.1: *4* function: write_file
520 def write_file(filename: str, s: str, encoding: str='utf-8') -> None:
521 """
522 Write the string s to the file whose name is given.
524 Handle all exeptions.
526 Before calling this function, the caller should ensure
527 that the file actually has been changed.
528 """
529 try:
530 # Write the file with platform-dependent newlines.
531 with open(filename, 'w', encoding=encoding) as f:
532 f.write(s)
533 except Exception as e:
534 g.trace(f"Error writing {filename}\n{e}")
535 #@+node:ekr.20200113154120.1: *3* functions: tokens
536 #@+node:ekr.20191223093539.1: *4* function: find_anchor_token
537 def find_anchor_token(node: Node, global_token_list: List["Token"]) -> Optional["Token"]:
538 """
539 Return the anchor_token for node, a token such that token.node == node.
541 The search starts at node, and then all the usual child nodes.
542 """
544 node1 = node
546 def anchor_token(node: Node) -> Optional["Token"]:
547 """Return the anchor token in node.token_list"""
548 # Careful: some tokens in the token list may have been killed.
549 for token in get_node_token_list(node, global_token_list):
550 if is_ancestor(node1, token):
551 return token
552 return None
554 # This table only has to cover fields for ast.Nodes that
555 # won't have any associated token.
557 fields = (
558 # Common...
559 'elt', 'elts', 'body', 'value',
560 # Less common...
561 'dims', 'ifs', 'names', 's',
562 'test', 'values', 'targets',
563 )
564 while node:
565 # First, try the node itself.
566 token = anchor_token(node)
567 if token:
568 return token
569 # Second, try the most common nodes w/o token_lists:
570 if isinstance(node, ast.Call):
571 node = node.func
572 elif isinstance(node, ast.Tuple):
573 node = node.elts # type:ignore
574 # Finally, try all other nodes.
575 else:
576 # This will be used rarely.
577 for field in fields:
578 node = getattr(node, field, None)
579 if node:
580 token = anchor_token(node)
581 if token:
582 return token
583 else:
584 break
585 return None
586 #@+node:ekr.20191231160225.1: *4* function: find_paren_token (changed signature)
587 def find_paren_token(i: int, global_token_list: List["Token"]) -> int:
588 """Return i of the next paren token, starting at tokens[i]."""
589 while i < len(global_token_list):
590 token = global_token_list[i]
591 if token.kind == 'op' and token.value in '()':
592 return i
593 if is_significant_token(token):
594 break
595 i += 1
596 return None
597 #@+node:ekr.20200113110505.4: *4* function: get_node_tokens_list
598 def get_node_token_list(node: Node, global_tokens_list: List["Token"]) -> List["Token"]:
599 """
600 tokens_list must be the global tokens list.
601 Return the tokens assigned to the node, or [].
602 """
603 i = getattr(node, 'first_i', None)
604 j = getattr(node, 'last_i', None)
605 return [] if i is None else global_tokens_list[i : j + 1]
606 #@+node:ekr.20191124123830.1: *4* function: is_significant & is_significant_token
607 def is_significant(kind: str, value: str) -> bool:
608 """
609 Return True if (kind, value) represent a token that can be used for
610 syncing generated tokens with the token list.
611 """
612 # Making 'endmarker' significant ensures that all tokens are synced.
613 return (
614 kind in ('async', 'await', 'endmarker', 'name', 'number', 'string') or
615 kind == 'op' and value not in ',;()')
617 def is_significant_token(token: "Token") -> bool:
618 """Return True if the given token is a syncronizing token"""
619 return is_significant(token.kind, token.value)
620 #@+node:ekr.20191224093336.1: *4* function: match_parens
621 def match_parens(filename: str, i: int, j: int, tokens: List["Token"]) -> int:
622 """Match parens in tokens[i:j]. Return the new j."""
623 if j >= len(tokens):
624 return len(tokens)
625 # Calculate paren level...
626 level = 0
627 for n in range(i, j + 1):
628 token = tokens[n]
629 if token.kind == 'op' and token.value == '(':
630 level += 1
631 if token.kind == 'op' and token.value == ')':
632 if level == 0:
633 break
634 level -= 1
635 # Find matching ')' tokens *after* j.
636 if level > 0:
637 while level > 0 and j + 1 < len(tokens):
638 token = tokens[j + 1]
639 if token.kind == 'op' and token.value == ')':
640 level -= 1
641 elif token.kind == 'op' and token.value == '(':
642 level += 1
643 elif is_significant_token(token):
644 break
645 j += 1
646 if level != 0: # pragma: no cover.
647 line_n = tokens[i].line_number
648 raise AssignLinksError(
649 f"\n"
650 f"Unmatched parens: level={level}\n"
651 f" file: {filename}\n"
652 f" line: {line_n}\n")
653 return j
654 #@+node:ekr.20191223053324.1: *4* function: tokens_for_node
655 def tokens_for_node(filename: str, node: Node, global_token_list: List["Token"]) -> List["Token"]:
656 """Return the list of all tokens descending from node."""
657 # Find any token descending from node.
658 token = find_anchor_token(node, global_token_list)
659 if not token:
660 if 0: # A good trace for debugging.
661 print('')
662 g.trace('===== no tokens', node.__class__.__name__)
663 return []
664 assert is_ancestor(node, token)
665 # Scan backward.
666 i = first_i = token.index
667 while i >= 0:
668 token2 = global_token_list[i - 1]
669 if getattr(token2, 'node', None):
670 if is_ancestor(node, token2):
671 first_i = i - 1
672 else:
673 break
674 i -= 1
675 # Scan forward.
676 j = last_j = token.index
677 while j + 1 < len(global_token_list):
678 token2 = global_token_list[j + 1]
679 if getattr(token2, 'node', None):
680 if is_ancestor(node, token2):
681 last_j = j + 1
682 else:
683 break
684 j += 1
685 last_j = match_parens(filename, first_i, last_j, global_token_list)
686 results = global_token_list[first_i : last_j + 1]
687 return results
688 #@+node:ekr.20200101030236.1: *4* function: tokens_to_string
689 def tokens_to_string(tokens: List[Any]) -> str:
690 """Return the string represented by the list of tokens."""
691 if tokens is None:
692 # This indicates an internal error.
693 print('')
694 g.trace('===== token list is None ===== ')
695 print('')
696 return ''
697 return ''.join([z.to_string() for z in tokens])
698 #@+node:ekr.20191223095408.1: *3* node/token nodes...
699 # Functions that associate tokens with nodes.
700 #@+node:ekr.20200120082031.1: *4* function: find_statement_node
701 def find_statement_node(node: Node) -> Optional[Node]:
702 """
703 Return the nearest statement node.
704 Return None if node has only Module for a parent.
705 """
706 if isinstance(node, ast.Module):
707 return None
708 parent = node
709 while parent:
710 if is_statement_node(parent):
711 return parent
712 parent = parent.parent
713 return None
714 #@+node:ekr.20191223054300.1: *4* function: is_ancestor
715 def is_ancestor(node: Node, token: "Token") -> bool:
716 """Return True if node is an ancestor of token."""
717 t_node = token.node
718 if not t_node:
719 assert token.kind == 'killed', repr(token)
720 return False
721 while t_node:
722 if t_node == node:
723 return True
724 t_node = t_node.parent
725 return False
726 #@+node:ekr.20200120082300.1: *4* function: is_long_statement
727 def is_long_statement(node: Node) -> bool:
728 """
729 Return True if node is an instance of a node that might be split into
730 shorter lines.
731 """
732 return isinstance(node, (
733 ast.Assign, ast.AnnAssign, ast.AsyncFor, ast.AsyncWith, ast.AugAssign,
734 ast.Call, ast.Delete, ast.ExceptHandler, ast.For, ast.Global,
735 ast.If, ast.Import, ast.ImportFrom,
736 ast.Nonlocal, ast.Return, ast.While, ast.With, ast.Yield, ast.YieldFrom))
737 #@+node:ekr.20200120110005.1: *4* function: is_statement_node
738 def is_statement_node(node: Node) -> bool:
739 """Return True if node is a top-level statement."""
740 return is_long_statement(node) or isinstance(node, (
741 ast.Break, ast.Continue, ast.Pass, ast.Try))
742 #@+node:ekr.20191231082137.1: *4* function: nearest_common_ancestor
743 def nearest_common_ancestor(node1: Node, node2: Node) -> Optional[Node]:
744 """
745 Return the nearest common ancestor node for the given nodes.
747 The nodes must have parent links.
748 """
750 def parents(node: Node) -> List[Node]:
751 aList = []
752 while node:
753 aList.append(node)
754 node = node.parent
755 return list(reversed(aList))
757 result = None
758 parents1 = parents(node1)
759 parents2 = parents(node2)
760 while parents1 and parents2:
761 parent1 = parents1.pop(0)
762 parent2 = parents2.pop(0)
763 if parent1 == parent2:
764 result = parent1
765 else:
766 break
767 return result
768 #@+node:ekr.20191231072039.1: *3* functions: utils...
769 # General utility functions on tokens and nodes.
770 #@+node:ekr.20191119085222.1: *4* function: obj_id
771 def obj_id(obj: Any) -> str:
772 """Return the last four digits of id(obj), for dumps & traces."""
773 return str(id(obj))[-4:]
774 #@+node:ekr.20191231060700.1: *4* function: op_name
775 #@@nobeautify
777 # https://docs.python.org/3/library/ast.html
779 _op_names = {
780 # Binary operators.
781 'Add': '+',
782 'BitAnd': '&',
783 'BitOr': '|',
784 'BitXor': '^',
785 'Div': '/',
786 'FloorDiv': '//',
787 'LShift': '<<',
788 'MatMult': '@', # Python 3.5.
789 'Mod': '%',
790 'Mult': '*',
791 'Pow': '**',
792 'RShift': '>>',
793 'Sub': '-',
794 # Boolean operators.
795 'And': ' and ',
796 'Or': ' or ',
797 # Comparison operators
798 'Eq': '==',
799 'Gt': '>',
800 'GtE': '>=',
801 'In': ' in ',
802 'Is': ' is ',
803 'IsNot': ' is not ',
804 'Lt': '<',
805 'LtE': '<=',
806 'NotEq': '!=',
807 'NotIn': ' not in ',
808 # Context operators.
809 'AugLoad': '<AugLoad>',
810 'AugStore': '<AugStore>',
811 'Del': '<Del>',
812 'Load': '<Load>',
813 'Param': '<Param>',
814 'Store': '<Store>',
815 # Unary operators.
816 'Invert': '~',
817 'Not': ' not ',
818 'UAdd': '+',
819 'USub': '-',
820 }
822 def op_name(node: Node) -> str:
823 """Return the print name of an operator node."""
824 class_name = node.__class__.__name__
825 assert class_name in _op_names, repr(class_name)
826 return _op_names[class_name].strip()
827 #@+node:ekr.20200107114452.1: *3* node/token creators...
828 #@+node:ekr.20200103082049.1: *4* function: make_tokens
829 def make_tokens(contents: str) -> List["Token"]:
830 """
831 Return a list (not a generator) of Token objects corresponding to the
832 list of 5-tuples generated by tokenize.tokenize.
834 Perform consistency checks and handle all exeptions.
835 """
837 def check(contents: str, tokens: List["Token"]) -> bool:
838 result = tokens_to_string(tokens)
839 ok = result == contents
840 if not ok:
841 print('\nRound-trip check FAILS')
842 print('Contents...\n')
843 g.printObj(contents)
844 print('\nResult...\n')
845 g.printObj(result)
846 return ok
848 try:
849 five_tuples = tokenize.tokenize(
850 io.BytesIO(contents.encode('utf-8')).readline)
851 except Exception:
852 print('make_tokens: exception in tokenize.tokenize')
853 g.es_exception()
854 return None
855 tokens = Tokenizer().create_input_tokens(contents, five_tuples)
856 assert check(contents, tokens)
857 return tokens
858 #@+node:ekr.20191027075648.1: *4* function: parse_ast
859 def parse_ast(s: str) -> Optional[Node]:
860 """
861 Parse string s, catching & reporting all exceptions.
862 Return the ast node, or None.
863 """
865 def oops(message: str) -> None:
866 print('')
867 print(f"parse_ast: {message}")
868 g.printObj(s)
869 print('')
871 try:
872 s1 = g.toEncodedString(s)
873 tree = ast.parse(s1, filename='before', mode='exec')
874 return tree
875 except IndentationError:
876 oops('Indentation Error')
877 except SyntaxError:
878 oops('Syntax Error')
879 except Exception:
880 oops('Unexpected Exception')
881 g.es_exception()
882 return None
883 #@+node:ekr.20191231110051.1: *3* node/token dumpers...
884 #@+node:ekr.20191027074436.1: *4* function: dump_ast
885 def dump_ast(ast: Node, tag: str='dump_ast') -> None:
886 """Utility to dump an ast tree."""
887 g.printObj(AstDumper().dump_ast(ast), tag=tag)
888 #@+node:ekr.20191228095945.4: *4* function: dump_contents
889 def dump_contents(contents: str, tag: str='Contents') -> None:
890 print('')
891 print(f"{tag}...\n")
892 for i, z in enumerate(g.splitLines(contents)):
893 print(f"{i+1:<3} ", z.rstrip())
894 print('')
895 #@+node:ekr.20191228095945.5: *4* function: dump_lines
896 def dump_lines(tokens: List["Token"], tag: str='Token lines') -> None:
897 print('')
898 print(f"{tag}...\n")
899 for z in tokens:
900 if z.line.strip():
901 print(z.line.rstrip())
902 else:
903 print(repr(z.line))
904 print('')
905 #@+node:ekr.20191228095945.7: *4* function: dump_results
906 def dump_results(tokens: List["Token"], tag: str='Results') -> None:
907 print('')
908 print(f"{tag}...\n")
909 print(tokens_to_string(tokens))
910 print('')
911 #@+node:ekr.20191228095945.8: *4* function: dump_tokens
912 def dump_tokens(tokens: List["Token"], tag: str='Tokens') -> None:
913 print('')
914 print(f"{tag}...\n")
915 if not tokens:
916 return
917 print("Note: values shown are repr(value) *except* for 'string' tokens.")
918 tokens[0].dump_header()
919 for i, z in enumerate(tokens):
920 # Confusing.
921 # if (i % 20) == 0: z.dump_header()
922 print(z.dump())
923 print('')
924 #@+node:ekr.20191228095945.9: *4* function: dump_tree
925 def dump_tree(tokens: List["Token"], tree: Node, tag: str='Tree') -> None:
926 print('')
927 print(f"{tag}...\n")
928 print(AstDumper().dump_tree(tokens, tree))
929 #@+node:ekr.20200107040729.1: *4* function: show_diffs
930 def show_diffs(s1: str, s2: str, filename: str='') -> None:
931 """Print diffs between strings s1 and s2."""
932 lines = list(difflib.unified_diff(
933 g.splitLines(s1),
934 g.splitLines(s2),
935 fromfile=f"Old {filename}",
936 tofile=f"New {filename}",
937 ))
938 print('')
939 tag = f"Diffs for {filename}" if filename else 'Diffs'
940 g.printObj(lines, tag=tag)
941 #@+node:ekr.20191225061516.1: *3* node/token replacers...
942 # Functions that replace tokens or nodes.
943 #@+node:ekr.20191231162249.1: *4* function: add_token_to_token_list
944 def add_token_to_token_list(token: "Token", node: Node) -> None:
945 """Insert token in the proper location of node.token_list."""
946 if getattr(node, 'first_i', None) is None:
947 node.first_i = node.last_i = token.index
948 else:
949 node.first_i = min(node.first_i, token.index)
950 node.last_i = max(node.last_i, token.index)
951 #@+node:ekr.20191225055616.1: *4* function: replace_node
952 def replace_node(new_node: Node, old_node: Node) -> None:
953 """Replace new_node by old_node in the parse tree."""
954 parent = old_node.parent
955 new_node.parent = parent
956 new_node.node_index = old_node.node_index
957 children = parent.children
958 i = children.index(old_node)
959 children[i] = new_node
960 fields = getattr(old_node, '_fields', None)
961 if fields:
962 for field in fields:
963 field = getattr(old_node, field)
964 if field == old_node:
965 setattr(old_node, field, new_node)
966 break
967 #@+node:ekr.20191225055626.1: *4* function: replace_token
968 def replace_token(token: "Token", kind: str, value: str) -> None:
969 """Replace kind and value of the given token."""
970 if token.kind in ('endmarker', 'killed'):
971 return
972 token.kind = kind
973 token.value = value
974 token.node = None # Should be filled later.
975 #@-others
976#@+node:ekr.20191027072910.1: ** Exception classes
977class AssignLinksError(Exception):
978 """Assigning links to ast nodes failed."""
981class AstNotEqual(Exception):
982 """The two given AST's are not equivalent."""
985class FailFast(Exception):
986 """Abort tests in TestRunner class."""
987#@+node:ekr.20220402062255.1: ** Classes
988#@+node:ekr.20141012064706.18390: *3* class AstDumper
989class AstDumper: # pragma: no cover
990 """A class supporting various kinds of dumps of ast nodes."""
991 #@+others
992 #@+node:ekr.20191112033445.1: *4* dumper.dump_tree & helper
993 def dump_tree(self, tokens: List["Token"], tree: Node) -> str:
994 """Briefly show a tree, properly indented."""
995 self.tokens = tokens
996 result = [self.show_header()]
997 self.dump_tree_and_links_helper(tree, 0, result)
998 return ''.join(result)
999 #@+node:ekr.20191125035321.1: *5* dumper.dump_tree_and_links_helper
1000 def dump_tree_and_links_helper(self, node: Node, level: int, result: List[str]) -> None:
1001 """Return the list of lines in result."""
1002 if node is None:
1003 return
1004 # Let block.
1005 indent = ' ' * 2 * level
1006 children: List[ast.AST] = getattr(node, 'children', [])
1007 node_s = self.compute_node_string(node, level)
1008 # Dump...
1009 if isinstance(node, (list, tuple)):
1010 for z in node:
1011 self.dump_tree_and_links_helper(z, level, result)
1012 elif isinstance(node, str):
1013 result.append(f"{indent}{node.__class__.__name__:>8}:{node}\n")
1014 elif isinstance(node, ast.AST):
1015 # Node and parent.
1016 result.append(node_s)
1017 # Children.
1018 for z in children:
1019 self.dump_tree_and_links_helper(z, level + 1, result)
1020 else:
1021 result.append(node_s)
1022 #@+node:ekr.20191125035600.1: *4* dumper.compute_node_string & helpers
1023 def compute_node_string(self, node: Node, level: int) -> str:
1024 """Return a string summarizing the node."""
1025 indent = ' ' * 2 * level
1026 parent = getattr(node, 'parent', None)
1027 node_id = getattr(node, 'node_index', '??')
1028 parent_id = getattr(parent, 'node_index', '??')
1029 parent_s = f"{parent_id:>3}.{parent.__class__.__name__} " if parent else ''
1030 class_name = node.__class__.__name__
1031 descriptor_s = f"{node_id}.{class_name}: " + self.show_fields(
1032 class_name, node, 30)
1033 tokens_s = self.show_tokens(node, 70, 100)
1034 lines = self.show_line_range(node)
1035 full_s1 = f"{parent_s:<16} {lines:<10} {indent}{descriptor_s} "
1036 node_s = f"{full_s1:<62} {tokens_s}\n"
1037 return node_s
1038 #@+node:ekr.20191113223424.1: *5* dumper.show_fields
1039 def show_fields(self, class_name: str, node: Node, truncate_n: int) -> str:
1040 """Return a string showing interesting fields of the node."""
1041 val = ''
1042 if class_name == 'JoinedStr':
1043 values = node.values
1044 assert isinstance(values, list)
1045 # Str tokens may represent *concatenated* strings.
1046 results = []
1047 fstrings, strings = 0, 0
1048 for z in values:
1049 assert isinstance(z, (ast.FormattedValue, ast.Str))
1050 if isinstance(z, ast.Str):
1051 results.append(z.s)
1052 strings += 1
1053 else:
1054 results.append(z.__class__.__name__)
1055 fstrings += 1
1056 val = f"{strings} str, {fstrings} f-str"
1057 elif class_name == 'keyword':
1058 if isinstance(node.value, ast.Str):
1059 val = f"arg={node.arg}..Str.value.s={node.value.s}"
1060 elif isinstance(node.value, ast.Name):
1061 val = f"arg={node.arg}..Name.value.id={node.value.id}"
1062 else:
1063 val = f"arg={node.arg}..value={node.value.__class__.__name__}"
1064 elif class_name == 'Name':
1065 val = f"id={node.id!r}"
1066 elif class_name == 'NameConstant':
1067 val = f"value={node.value!r}"
1068 elif class_name == 'Num':
1069 val = f"n={node.n}"
1070 elif class_name == 'Starred':
1071 if isinstance(node.value, ast.Str):
1072 val = f"s={node.value.s}"
1073 elif isinstance(node.value, ast.Name):
1074 val = f"id={node.value.id}"
1075 else:
1076 val = f"s={node.value.__class__.__name__}"
1077 elif class_name == 'Str':
1078 val = f"s={node.s!r}"
1079 elif class_name in ('AugAssign', 'BinOp', 'BoolOp', 'UnaryOp'): # IfExp
1080 name = node.op.__class__.__name__
1081 val = f"op={_op_names.get(name, name)}"
1082 elif class_name == 'Compare':
1083 ops = ','.join([op_name(z) for z in node.ops])
1084 val = f"ops='{ops}'"
1085 else:
1086 val = ''
1087 return g.truncate(val, truncate_n)
1088 #@+node:ekr.20191114054726.1: *5* dumper.show_line_range
1089 def show_line_range(self, node: Node) -> str:
1091 token_list = get_node_token_list(node, self.tokens)
1092 if not token_list:
1093 return ''
1094 min_ = min([z.line_number for z in token_list])
1095 max_ = max([z.line_number for z in token_list])
1096 return f"{min_}" if min_ == max_ else f"{min_}..{max_}"
1097 #@+node:ekr.20191113223425.1: *5* dumper.show_tokens
1098 def show_tokens(self, node: Node, n: int, m: int, show_cruft: bool=False) -> str:
1099 """
1100 Return a string showing node.token_list.
1102 Split the result if n + len(result) > m
1103 """
1104 token_list = get_node_token_list(node, self.tokens)
1105 result = []
1106 for z in token_list:
1107 val = None
1108 if z.kind == 'comment':
1109 if show_cruft:
1110 val = g.truncate(z.value, 10) # Short is good.
1111 result.append(f"{z.kind}.{z.index}({val})")
1112 elif z.kind == 'name':
1113 val = g.truncate(z.value, 20)
1114 result.append(f"{z.kind}.{z.index}({val})")
1115 elif z.kind == 'newline':
1116 # result.append(f"{z.kind}.{z.index}({z.line_number}:{len(z.line)})")
1117 result.append(f"{z.kind}.{z.index}")
1118 elif z.kind == 'number':
1119 result.append(f"{z.kind}.{z.index}({z.value})")
1120 elif z.kind == 'op':
1121 if z.value not in ',()' or show_cruft:
1122 result.append(f"{z.kind}.{z.index}({z.value})")
1123 elif z.kind == 'string':
1124 val = g.truncate(z.value, 30)
1125 result.append(f"{z.kind}.{z.index}({val})")
1126 elif z.kind == 'ws':
1127 if show_cruft:
1128 result.append(f"{z.kind}.{z.index}({len(z.value)})")
1129 else:
1130 # Indent, dedent, encoding, etc.
1131 # Don't put a blank.
1132 continue
1133 if result and result[-1] != ' ':
1134 result.append(' ')
1135 #
1136 # split the line if it is too long.
1137 # g.printObj(result, tag='show_tokens')
1138 if 1:
1139 return ''.join(result)
1140 line, lines = [], []
1141 for r in result:
1142 line.append(r)
1143 if n + len(''.join(line)) >= m:
1144 lines.append(''.join(line))
1145 line = []
1146 lines.append(''.join(line))
1147 pad = '\n' + ' ' * n
1148 return pad.join(lines)
1149 #@+node:ekr.20191110165235.5: *4* dumper.show_header
1150 def show_header(self) -> str:
1151 """Return a header string, but only the fist time."""
1152 return (
1153 f"{'parent':<16} {'lines':<10} {'node':<34} {'tokens'}\n"
1154 f"{'======':<16} {'=====':<10} {'====':<34} {'======'}\n")
1155 #@+node:ekr.20141012064706.18392: *4* dumper.dump_ast & helper
1156 annotate_fields = False
1157 include_attributes = False
1158 indent_ws = ' '
1160 def dump_ast(self, node: Node, level: int=0) -> str:
1161 """
1162 Dump an ast tree. Adapted from ast.dump.
1163 """
1164 sep1 = '\n%s' % (self.indent_ws * (level + 1))
1165 if isinstance(node, ast.AST):
1166 fields = [(a, self.dump_ast(b, level + 1)) for a, b in self.get_fields(node)]
1167 if self.include_attributes and node._attributes:
1168 fields.extend([(a, self.dump_ast(getattr(node, a), level + 1))
1169 for a in node._attributes])
1170 if self.annotate_fields:
1171 aList = ['%s=%s' % (a, b) for a, b in fields]
1172 else:
1173 aList = [b for a, b in fields]
1174 name = node.__class__.__name__
1175 sep = '' if len(aList) <= 1 else sep1
1176 return '%s(%s%s)' % (name, sep, sep1.join(aList))
1177 if isinstance(node, list):
1178 sep = sep1
1179 return 'LIST[%s]' % ''.join(
1180 ['%s%s' % (sep, self.dump_ast(z, level + 1)) for z in node])
1181 return repr(node)
1182 #@+node:ekr.20141012064706.18393: *5* dumper.get_fields
1183 def get_fields(self, node: Node) -> Generator:
1185 return (
1186 (a, b) for a, b in ast.iter_fields(node)
1187 if a not in ['ctx',] and b not in (None, [])
1188 )
1189 #@-others
1190#@+node:ekr.20191222083453.1: *3* class Fstringify
1191class Fstringify:
1192 """A class to fstringify files."""
1194 silent = True # for pytest. Defined in all entries.
1195 line_number = 0
1196 line = ''
1198 #@+others
1199 #@+node:ekr.20191222083947.1: *4* fs.fstringify
1200 def fstringify(self, contents: str, filename: str, tokens: List["Token"], tree: Node) -> str:
1201 """
1202 Fstringify.fstringify:
1204 f-stringify the sources given by (tokens, tree).
1206 Return the resulting string.
1207 """
1208 self.filename = filename
1209 self.tokens = tokens
1210 self.tree = tree
1211 # Prepass: reassign tokens.
1212 ReassignTokens().reassign(filename, tokens, tree)
1213 # Main pass.
1214 for node in ast.walk(tree):
1215 if (
1216 isinstance(node, ast.BinOp)
1217 and op_name(node.op) == '%'
1218 and isinstance(node.left, ast.Str)
1219 ):
1220 self.make_fstring(node)
1221 results = tokens_to_string(self.tokens)
1222 return results
1223 #@+node:ekr.20200103054101.1: *4* fs.fstringify_file (entry)
1224 def fstringify_file(self, filename: str) -> bool: # pragma: no cover
1225 """
1226 Fstringify.fstringify_file.
1228 The entry point for the fstringify-file command.
1230 f-stringify the given external file with the Fstrinfify class.
1232 Return True if the file was changed.
1233 """
1234 tag = 'fstringify-file'
1235 self.filename = filename
1236 self.silent = False
1237 tog = TokenOrderGenerator()
1238 try:
1239 contents, encoding, tokens, tree = tog.init_from_file(filename)
1240 if not contents or not tokens or not tree:
1241 print(f"{tag}: Can not fstringify: {filename}")
1242 return False
1243 results = self.fstringify(contents, filename, tokens, tree)
1244 except Exception as e:
1245 print(e)
1246 return False
1247 # Something besides newlines must change.
1248 changed = regularize_nls(contents) != regularize_nls(results)
1249 status = 'Wrote' if changed else 'Unchanged'
1250 print(f"{tag}: {status:>9}: {filename}")
1251 if changed:
1252 write_file(filename, results, encoding=encoding)
1253 return changed
1254 #@+node:ekr.20200103065728.1: *4* fs.fstringify_file_diff (entry)
1255 def fstringify_file_diff(self, filename: str) -> bool: # pragma: no cover
1256 """
1257 Fstringify.fstringify_file_diff.
1259 The entry point for the diff-fstringify-file command.
1261 Print the diffs that would resulf from the fstringify-file command.
1263 Return True if the file would be changed.
1264 """
1265 tag = 'diff-fstringify-file'
1266 self.filename = filename
1267 self.silent = False
1268 tog = TokenOrderGenerator()
1269 try:
1270 contents, encoding, tokens, tree = tog.init_from_file(filename)
1271 if not contents or not tokens or not tree:
1272 return False
1273 results = self.fstringify(contents, filename, tokens, tree)
1274 except Exception as e:
1275 print(e)
1276 return False
1277 # Something besides newlines must change.
1278 changed = regularize_nls(contents) != regularize_nls(results)
1279 if changed:
1280 show_diffs(contents, results, filename=filename)
1281 else:
1282 print(f"{tag}: Unchanged: {filename}")
1283 return changed
1284 #@+node:ekr.20200112060218.1: *4* fs.fstringify_file_silent (entry)
1285 def fstringify_file_silent(self, filename: str) -> bool: # pragma: no cover
1286 """
1287 Fstringify.fstringify_file_silent.
1289 The entry point for the silent-fstringify-file command.
1291 fstringify the given file, suppressing all but serious error messages.
1293 Return True if the file would be changed.
1294 """
1295 self.filename = filename
1296 self.silent = True
1297 tog = TokenOrderGenerator()
1298 try:
1299 contents, encoding, tokens, tree = tog.init_from_file(filename)
1300 if not contents or not tokens or not tree:
1301 return False
1302 results = self.fstringify(contents, filename, tokens, tree)
1303 except Exception as e:
1304 print(e)
1305 return False
1306 # Something besides newlines must change.
1307 changed = regularize_nls(contents) != regularize_nls(results)
1308 status = 'Wrote' if changed else 'Unchanged'
1309 # Write the results.
1310 print(f"{status:>9}: {filename}")
1311 if changed:
1312 write_file(filename, results, encoding=encoding)
1313 return changed
1314 #@+node:ekr.20191222095754.1: *4* fs.make_fstring & helpers
1315 def make_fstring(self, node: Node) -> None:
1316 """
1317 node is BinOp node representing an '%' operator.
1318 node.left is an ast.Str node.
1319 node.right reprsents the RHS of the '%' operator.
1321 Convert this tree to an f-string, if possible.
1322 Replace the node's entire tree with a new ast.Str node.
1323 Replace all the relevant tokens with a single new 'string' token.
1324 """
1325 trace = False
1326 assert isinstance(node.left, ast.Str), (repr(node.left), g.callers())
1327 # Careful: use the tokens, not Str.s. This preserves spelling.
1328 lt_token_list = get_node_token_list(node.left, self.tokens)
1329 if not lt_token_list: # pragma: no cover
1330 print('')
1331 g.trace('Error: no token list in Str')
1332 dump_tree(self.tokens, node)
1333 print('')
1334 return
1335 lt_s = tokens_to_string(lt_token_list)
1336 if trace:
1337 g.trace('lt_s:', lt_s) # pragma: no cover
1338 # Get the RHS values, a list of token lists.
1339 values = self.scan_rhs(node.right)
1340 if trace: # pragma: no cover
1341 for i, z in enumerate(values):
1342 dump_tokens(z, tag=f"RHS value {i}")
1343 # Compute rt_s, self.line and self.line_number for later messages.
1344 token0 = lt_token_list[0]
1345 self.line_number = token0.line_number
1346 self.line = token0.line.strip()
1347 rt_s = ''.join(tokens_to_string(z) for z in values)
1348 # Get the % specs in the LHS string.
1349 specs = self.scan_format_string(lt_s)
1350 if len(values) != len(specs): # pragma: no cover
1351 self.message(
1352 f"can't create f-fstring: {lt_s!r}\n"
1353 f":f-string mismatch: "
1354 f"{len(values)} value{g.plural(len(values))}, "
1355 f"{len(specs)} spec{g.plural(len(specs))}")
1356 return
1357 # Replace specs with values.
1358 results = self.substitute_values(lt_s, specs, values)
1359 result = self.compute_result(lt_s, results)
1360 if not result:
1361 return
1362 # Remove whitespace before ! and :.
1363 result = self.clean_ws(result)
1364 # Show the results
1365 if trace: # pragma: no cover
1366 before = (lt_s + ' % ' + rt_s).replace('\n', '<NL>')
1367 after = result.replace('\n', '<NL>')
1368 self.message(
1369 f"trace:\n"
1370 f":from: {before!s}\n"
1371 f": to: {after!s}")
1372 # Adjust the tree and the token list.
1373 self.replace(node, result, values)
1374 #@+node:ekr.20191222102831.3: *5* fs.clean_ws
1375 ws_pat = re.compile(r'(\s+)([:!][0-9]\})')
1377 def clean_ws(self, s: str) -> str:
1378 """Carefully remove whitespace before ! and : specifiers."""
1379 s = re.sub(self.ws_pat, r'\2', s)
1380 return s
1381 #@+node:ekr.20191222102831.4: *5* fs.compute_result & helpers
1382 def compute_result(self, lt_s: str, tokens: List["Token"]) -> str:
1383 """
1384 Create the final result, with various kinds of munges.
1386 Return the result string, or None if there are errors.
1387 """
1388 # Fail if there is a backslash within { and }.
1389 if not self.check_back_slashes(lt_s, tokens):
1390 return None # pragma: no cover
1391 # Ensure consistent quotes.
1392 if not self.change_quotes(lt_s, tokens):
1393 return None # pragma: no cover
1394 return tokens_to_string(tokens)
1395 #@+node:ekr.20200215074309.1: *6* fs.check_back_slashes
1396 def check_back_slashes(self, lt_s: str, tokens: List["Token"]) -> bool:
1397 """
1398 Return False if any backslash appears with an {} expression.
1400 Tokens is a list of lokens on the RHS.
1401 """
1402 count = 0
1403 for z in tokens:
1404 if z.kind == 'op':
1405 if z.value == '{':
1406 count += 1
1407 elif z.value == '}':
1408 count -= 1
1409 if (count % 2) == 1 and '\\' in z.value:
1410 if not self.silent:
1411 self.message( # pragma: no cover (silent during unit tests)
1412 f"can't create f-fstring: {lt_s!r}\n"
1413 f":backslash in {{expr}}:")
1414 return False
1415 return True
1416 #@+node:ekr.20191222102831.7: *6* fs.change_quotes
1417 def change_quotes(self, lt_s: str, aList: List[Any]) -> bool:
1418 """
1419 Carefully check quotes in all "inner" tokens as necessary.
1421 Return False if the f-string would contain backslashes.
1423 We expect the following "outer" tokens.
1425 aList[0]: ('string', 'f')
1426 aList[1]: ('string', a single or double quote.
1427 aList[-1]: ('string', a single or double quote matching aList[1])
1428 """
1429 # Sanity checks.
1430 if len(aList) < 4:
1431 return True # pragma: no cover (defensive)
1432 if not lt_s: # pragma: no cover (defensive)
1433 self.message("can't create f-fstring: no lt_s!")
1434 return False
1435 delim = lt_s[0]
1436 # Check tokens 0, 1 and -1.
1437 token0 = aList[0]
1438 token1 = aList[1]
1439 token_last = aList[-1]
1440 for token in token0, token1, token_last:
1441 # These are the only kinds of tokens we expect to generate.
1442 ok = (
1443 token.kind == 'string' or
1444 token.kind == 'op' and token.value in '{}')
1445 if not ok: # pragma: no cover (defensive)
1446 self.message(
1447 f"unexpected token: {token.kind} {token.value}\n"
1448 f": lt_s: {lt_s!r}")
1449 return False
1450 # These checks are important...
1451 if token0.value != 'f':
1452 return False # pragma: no cover (defensive)
1453 val1 = token1.value
1454 if delim != val1:
1455 return False # pragma: no cover (defensive)
1456 val_last = token_last.value
1457 if delim != val_last:
1458 return False # pragma: no cover (defensive)
1459 #
1460 # Check for conflicting delims, preferring f"..." to f'...'.
1461 for delim in ('"', "'"):
1462 aList[1] = aList[-1] = Token('string', delim)
1463 for z in aList[2:-1]:
1464 if delim in z.value:
1465 break
1466 else:
1467 return True
1468 if not self.silent: # pragma: no cover (silent unit test)
1469 self.message(
1470 f"can't create f-fstring: {lt_s!r}\n"
1471 f": conflicting delims:")
1472 return False
1473 #@+node:ekr.20191222102831.6: *5* fs.munge_spec
1474 def munge_spec(self, spec: str) -> Tuple[str, str]:
1475 """
1476 Return (head, tail).
1478 The format is spec !head:tail or :tail
1480 Example specs: s2, r3
1481 """
1482 # To do: handle more specs.
1483 head, tail = [], []
1484 if spec.startswith('+'):
1485 pass # Leave it alone!
1486 elif spec.startswith('-'):
1487 tail.append('>')
1488 spec = spec[1:]
1489 if spec.endswith('s'):
1490 spec = spec[:-1]
1491 if spec.endswith('r'):
1492 head.append('r')
1493 spec = spec[:-1]
1494 tail_s = ''.join(tail) + spec
1495 head_s = ''.join(head)
1496 return head_s, tail_s
1497 #@+node:ekr.20191222102831.9: *5* fs.scan_format_string
1498 # format_spec ::= [[fill]align][sign][#][0][width][,][.precision][type]
1499 # fill ::= <any character>
1500 # align ::= "<" | ">" | "=" | "^"
1501 # sign ::= "+" | "-" | " "
1502 # width ::= integer
1503 # precision ::= integer
1504 # type ::= "b" | "c" | "d" | "e" | "E" | "f" | "F" | "g" | "G" | "n" | "o" | "s" | "x" | "X" | "%"
1506 format_pat = re.compile(r'%(([+-]?[0-9]*(\.)?[0.9]*)*[bcdeEfFgGnoxrsX]?)')
1508 def scan_format_string(self, s: str) -> List[re.Match]:
1509 """Scan the format string s, returning a list match objects."""
1510 result = list(re.finditer(self.format_pat, s))
1511 return result
1512 #@+node:ekr.20191222104224.1: *5* fs.scan_rhs
1513 def scan_rhs(self, node: Node) -> List[Any]:
1514 """
1515 Scan the right-hand side of a potential f-string.
1517 Return a list of the token lists for each element.
1518 """
1519 trace = False
1520 # First, Try the most common cases.
1521 if isinstance(node, ast.Str):
1522 token_list = get_node_token_list(node, self.tokens)
1523 return [token_list]
1524 if isinstance(node, (list, tuple, ast.Tuple)):
1525 result = []
1526 elts = node.elts if isinstance(node, ast.Tuple) else node
1527 for i, elt in enumerate(elts):
1528 tokens = tokens_for_node(self.filename, elt, self.tokens)
1529 result.append(tokens)
1530 if trace: # pragma: no cover
1531 g.trace(f"item: {i}: {elt.__class__.__name__}")
1532 g.printObj(tokens, tag=f"Tokens for item {i}")
1533 return result
1534 # Now we expect only one result.
1535 tokens = tokens_for_node(self.filename, node, self.tokens)
1536 return [tokens]
1537 #@+node:ekr.20191226155316.1: *5* fs.substitute_values
1538 def substitute_values(self, lt_s: str, specs: List[re.Match], values: List) -> List["Token"]:
1539 """
1540 Replace specifiers with values in lt_s string.
1542 Double { and } as needed.
1543 """
1544 i, results = 0, [Token('string', 'f')]
1545 for spec_i, m in enumerate(specs):
1546 value = tokens_to_string(values[spec_i])
1547 start, end, spec = m.start(0), m.end(0), m.group(1)
1548 if start > i:
1549 val = lt_s[i:start].replace('{', '{{').replace('}', '}}')
1550 results.append(Token('string', val[0]))
1551 results.append(Token('string', val[1:]))
1552 head, tail = self.munge_spec(spec)
1553 results.append(Token('op', '{'))
1554 results.append(Token('string', value))
1555 if head:
1556 results.append(Token('string', '!'))
1557 results.append(Token('string', head))
1558 if tail:
1559 results.append(Token('string', ':'))
1560 results.append(Token('string', tail))
1561 results.append(Token('op', '}'))
1562 i = end
1563 # Add the tail.
1564 tail = lt_s[i:]
1565 if tail:
1566 tail = tail.replace('{', '{{').replace('}', '}}')
1567 results.append(Token('string', tail[:-1]))
1568 results.append(Token('string', tail[-1]))
1569 return results
1570 #@+node:ekr.20200214142019.1: *4* fs.message
1571 def message(self, message: str) -> None: # pragma: no cover.
1572 """
1573 Print one or more message lines aligned on the first colon of the message.
1574 """
1575 # Print a leading blank line.
1576 print('')
1577 # Calculate the padding.
1578 lines = g.splitLines(message)
1579 pad = max(lines[0].find(':'), 30)
1580 # Print the first line.
1581 z = lines[0]
1582 i = z.find(':')
1583 if i == -1:
1584 print(z.rstrip())
1585 else:
1586 print(f"{z[:i+2].strip():>{pad+1}} {z[i+2:].strip()}")
1587 # Print the remaining message lines.
1588 for z in lines[1:]:
1589 if z.startswith('<'):
1590 # Print left aligned.
1591 print(z[1:].strip())
1592 elif z.startswith(':') and -1 < z[1:].find(':') <= pad:
1593 # Align with the first line.
1594 i = z[1:].find(':')
1595 print(f"{z[1:i+2].strip():>{pad+1}} {z[i+2:].strip()}")
1596 elif z.startswith('>'):
1597 # Align after the aligning colon.
1598 print(f"{' ':>{pad+2}}{z[1:].strip()}")
1599 else:
1600 # Default: Put the entire line after the aligning colon.
1601 print(f"{' ':>{pad+2}}{z.strip()}")
1602 # Print the standard message lines.
1603 file_s = f"{'file':>{pad}}"
1604 ln_n_s = f"{'line number':>{pad}}"
1605 line_s = f"{'line':>{pad}}"
1606 print(
1607 f"{file_s}: {self.filename}\n"
1608 f"{ln_n_s}: {self.line_number}\n"
1609 f"{line_s}: {self.line!r}")
1610 #@+node:ekr.20191225054848.1: *4* fs.replace
1611 def replace(self, node: Node, s: str, values: List["Token"]) -> None:
1612 """
1613 Replace node with an ast.Str node for s.
1614 Replace all tokens in the range of values with a single 'string' node.
1615 """
1616 # Replace the tokens...
1617 tokens = tokens_for_node(self.filename, node, self.tokens)
1618 i1 = i = tokens[0].index
1619 replace_token(self.tokens[i], 'string', s)
1620 j = 1
1621 while j < len(tokens):
1622 replace_token(self.tokens[i1 + j], 'killed', '')
1623 j += 1
1624 # Replace the node.
1625 new_node = ast.Str()
1626 new_node.s = s
1627 replace_node(new_node, node)
1628 # Update the token.
1629 token = self.tokens[i1]
1630 token.node = new_node
1631 # Update the token list.
1632 add_token_to_token_list(token, new_node)
1633 #@-others
1634#@+node:ekr.20220330191947.1: *3* class IterativeTokenGenerator
1635class IterativeTokenGenerator:
1636 """
1637 Self-contained iterative token syncing class.
1638 """
1640 begin_end_stack: List[str] = [] # A stack of node names.
1641 n_nodes = 0 # The number of nodes that have been visited.
1642 node = None # The current node.
1643 node_index = 0 # The index into the node_stack.
1644 node_stack: List[ast.AST] = [] # The stack of parent nodes.
1646 #@+others
1647 #@+node:ekr.20220402095550.1: *4* iterative: Init...
1648 # Same as in the TokenOrderGenerator class.
1649 #@+node:ekr.20220402095550.2: *5* iterative.balance_tokens
1650 def balance_tokens(self, tokens: List["Token"]) -> int:
1651 """
1652 TOG.balance_tokens.
1654 Insert two-way links between matching paren tokens.
1655 """
1656 count, stack = 0, []
1657 for token in tokens:
1658 if token.kind == 'op':
1659 if token.value == '(':
1660 count += 1
1661 stack.append(token.index)
1662 if token.value == ')':
1663 if stack:
1664 index = stack.pop()
1665 tokens[index].matching_paren = token.index
1666 tokens[token.index].matching_paren = index
1667 else: # pragma: no cover
1668 g.trace(f"unmatched ')' at index {token.index}")
1669 if stack: # pragma: no cover
1670 g.trace("unmatched '(' at {','.join(stack)}")
1671 return count
1672 #@+node:ekr.20220402095550.3: *5* iterative.create_links (changed)
1673 def create_links(self, tokens: List["Token"], tree: Node, file_name: str='') -> List:
1674 """
1675 A generator creates two-way links between the given tokens and ast-tree.
1677 Callers should call this generator with list(tog.create_links(...))
1679 The sync_tokens method creates the links and verifies that the resulting
1680 tree traversal generates exactly the given tokens in exact order.
1682 tokens: the list of Token instances for the input.
1683 Created by make_tokens().
1684 tree: the ast tree for the input.
1685 Created by parse_ast().
1686 """
1687 # Init all ivars.
1688 self.file_name = file_name # For tests.
1689 self.node = None # The node being visited.
1690 self.tokens = tokens # The immutable list of input tokens.
1691 self.tree = tree # The tree of ast.AST nodes.
1692 # Traverse the tree.
1693 self.main_loop(tree)
1694 # Ensure that all tokens are patched.
1695 self.node = tree
1696 self.token(('endmarker', ''))
1697 # Return [] for compatibility with legacy code: list(tog.create_links).
1698 return []
1699 #@+node:ekr.20220402095550.4: *5* iterative.init_from_file
1700 def init_from_file(self, filename: str) -> Tuple[str, str, List["Token"], Node]: # pragma: no cover
1701 """
1702 Create the tokens and ast tree for the given file.
1703 Create links between tokens and the parse tree.
1704 Return (contents, encoding, tokens, tree).
1705 """
1706 self.filename = filename
1707 encoding, contents = read_file_with_encoding(filename)
1708 if not contents:
1709 return None, None, None, None
1710 self.tokens = tokens = make_tokens(contents)
1711 self.tree = tree = parse_ast(contents)
1712 self.create_links(tokens, tree)
1713 return contents, encoding, tokens, tree
1714 #@+node:ekr.20220402095550.5: *5* iterative.init_from_string
1715 def init_from_string(self, contents: str, filename: str) -> Tuple[List["Token"], Node]: # pragma: no cover
1716 """
1717 Tokenize, parse and create links in the contents string.
1719 Return (tokens, tree).
1720 """
1721 self.filename = filename
1722 self.tokens = tokens = make_tokens(contents)
1723 self.tree = tree = parse_ast(contents)
1724 self.create_links(tokens, tree)
1725 return tokens, tree
1726 #@+node:ekr.20220402094825.1: *4* iterative: Syncronizers...
1727 # Same as in the TokenOrderGenerator class.
1729 # The synchronizer sync tokens to nodes.
1730 #@+node:ekr.20220402094825.2: *5* iterative.find_next_significant_token
1731 def find_next_significant_token(self) -> Optional["Token"]:
1732 """
1733 Scan from *after* self.tokens[px] looking for the next significant
1734 token.
1736 Return the token, or None. Never change self.px.
1737 """
1738 px = self.px + 1
1739 while px < len(self.tokens):
1740 token = self.tokens[px]
1741 px += 1
1742 if is_significant_token(token):
1743 return token
1744 # This will never happen, because endtoken is significant.
1745 return None # pragma: no cover
1746 #@+node:ekr.20220402094825.3: *5* iterative.set_links
1747 last_statement_node = None
1749 def set_links(self, node: Node, token: "Token") -> None:
1750 """Make two-way links between token and the given node."""
1751 # Don't bother assigning comment, comma, parens, ws and endtoken tokens.
1752 if token.kind == 'comment':
1753 # Append the comment to node.comment_list.
1754 comment_list: List["Token"] = getattr(node, 'comment_list', [])
1755 node.comment_list = comment_list + [token]
1756 return
1757 if token.kind in ('endmarker', 'ws'):
1758 return
1759 if token.kind == 'op' and token.value in ',()':
1760 return
1761 # *Always* remember the last statement.
1762 statement = find_statement_node(node)
1763 if statement:
1764 self.last_statement_node = statement
1765 assert not isinstance(self.last_statement_node, ast.Module)
1766 if token.node is not None: # pragma: no cover
1767 line_s = f"line {token.line_number}:"
1768 raise AssignLinksError(
1769 f" file: {self.filename}\n"
1770 f"{line_s:>12} {token.line.strip()}\n"
1771 f"token index: {self.px}\n"
1772 f"token.node is not None\n"
1773 f" token.node: {token.node.__class__.__name__}\n"
1774 f" callers: {g.callers()}")
1775 # Assign newlines to the previous statement node, if any.
1776 if token.kind in ('newline', 'nl'):
1777 # Set an *auxilliary* link for the split/join logic.
1778 # Do *not* set token.node!
1779 token.statement_node = self.last_statement_node
1780 return
1781 if is_significant_token(token):
1782 # Link the token to the ast node.
1783 token.node = node
1784 # Add the token to node's token_list.
1785 add_token_to_token_list(token, node)
1786 #@+node:ekr.20220402094825.4: *5* iterative.sync_name (aka name)
1787 def sync_name(self, val: str) -> None:
1788 aList = val.split('.')
1789 if len(aList) == 1:
1790 self.sync_token(('name', val))
1791 else:
1792 for i, part in enumerate(aList):
1793 self.sync_token(('name', part))
1794 if i < len(aList) - 1:
1795 self.sync_op('.')
1797 name = sync_name # for readability.
1798 #@+node:ekr.20220402094825.5: *5* iterative.sync_op (aka op)
1799 def sync_op(self, val: str) -> None:
1800 """
1801 Sync to the given operator.
1803 val may be '(' or ')' *only* if the parens *will* actually exist in the
1804 token list.
1805 """
1806 self.sync_token(('op', val))
1808 op = sync_op # For readability.
1809 #@+node:ekr.20220402094825.6: *5* iterative.sync_token (aka token)
1810 px = -1 # Index of the previously synced token.
1812 ### def sync_token(self, kind: str, val: str) -> None:
1813 def sync_token(self, data: Tuple[Any, Any]) -> None:
1814 """
1815 Sync to a token whose kind & value are given. The token need not be
1816 significant, but it must be guaranteed to exist in the token list.
1818 The checks in this method constitute a strong, ever-present, unit test.
1820 Scan the tokens *after* px, looking for a token T matching (kind, val).
1821 raise AssignLinksError if a significant token is found that doesn't match T.
1822 Otherwise:
1823 - Create two-way links between all assignable tokens between px and T.
1824 - Create two-way links between T and self.node.
1825 - Advance by updating self.px to point to T.
1826 """
1827 kind, val = data ### New
1828 node, tokens = self.node, self.tokens
1829 assert isinstance(node, ast.AST), repr(node)
1830 # g.trace(
1831 # f"px: {self.px:2} "
1832 # f"node: {node.__class__.__name__:<10} "
1833 # f"kind: {kind:>10}: val: {val!r}")
1834 #
1835 # Step one: Look for token T.
1836 old_px = px = self.px + 1
1837 while px < len(self.tokens):
1838 token = tokens[px]
1839 if (kind, val) == (token.kind, token.value):
1840 break # Success.
1841 if kind == token.kind == 'number':
1842 val = token.value
1843 break # Benign: use the token's value, a string, instead of a number.
1844 if is_significant_token(token): # pragma: no cover
1845 line_s = f"line {token.line_number}:"
1846 val = str(val) # for g.truncate.
1847 raise AssignLinksError(
1848 f" file: {self.filename}\n"
1849 f"{line_s:>12} {token.line.strip()}\n"
1850 f"Looking for: {kind}.{g.truncate(val, 40)!r}\n"
1851 f" found: {token.kind}.{token.value!r}\n"
1852 f"token.index: {token.index}\n")
1853 # Skip the insignificant token.
1854 px += 1
1855 else: # pragma: no cover
1856 val = str(val) # for g.truncate.
1857 raise AssignLinksError(
1858 f" file: {self.filename}\n"
1859 f"Looking for: {kind}.{g.truncate(val, 40)}\n"
1860 f" found: end of token list")
1861 #
1862 # Step two: Assign *secondary* links only for newline tokens.
1863 # Ignore all other non-significant tokens.
1864 while old_px < px:
1865 token = tokens[old_px]
1866 old_px += 1
1867 if token.kind in ('comment', 'newline', 'nl'):
1868 self.set_links(node, token)
1869 #
1870 # Step three: Set links in the found token.
1871 token = tokens[px]
1872 self.set_links(node, token)
1873 #
1874 # Step four: Advance.
1875 self.px = px
1877 token = sync_token # For readability.
1878 #@+node:ekr.20220330164313.1: *4* iterative: Traversal...
1879 #@+node:ekr.20220402094946.2: *5* iterative.enter_node
1880 def enter_node(self, node: Node) -> None:
1881 """Enter a node."""
1882 # Update the stats.
1883 self.n_nodes += 1
1884 # Create parent/child links first, *before* updating self.node.
1885 #
1886 # Don't even *think* about removing the parent/child links.
1887 # The nearest_common_ancestor function depends upon them.
1888 node.parent = self.node
1889 if self.node:
1890 children: List[Node] = getattr(self.node, 'children', [])
1891 children.append(node)
1892 self.node.children = children
1893 # Inject the node_index field.
1894 assert not hasattr(node, 'node_index'), g.callers()
1895 node.node_index = self.node_index
1896 self.node_index += 1
1897 # begin_visitor and end_visitor must be paired.
1898 self.begin_end_stack.append(node.__class__.__name__)
1899 # Push the previous node.
1900 self.node_stack.append(self.node)
1901 # Update self.node *last*.
1902 self.node = node
1903 #@+node:ekr.20220402094946.3: *5* iterative.leave_node
1904 def leave_node(self, node: Node) -> None:
1905 """Leave a visitor."""
1906 # Make *sure* that begin_visitor and end_visitor are paired.
1907 entry_name = self.begin_end_stack.pop()
1908 assert entry_name == node.__class__.__name__, f"{entry_name!r} {node.__class__.__name__}"
1909 assert self.node == node, (repr(self.node), repr(node))
1910 # Restore self.node.
1911 self.node = self.node_stack.pop()
1912 #@+node:ekr.20220330120220.1: *5* iterative.main_loop
1913 def main_loop(self, node: Node) -> None:
1915 func = getattr(self, 'do_' + node.__class__.__name__, None)
1916 if not func: # pragma: no cover (defensive code)
1917 print('main_loop: invalid ast node:', repr(node))
1918 return
1919 exec_list: ActionList = [(func, node)]
1920 while exec_list:
1921 func, arg = exec_list.pop(0)
1922 result = func(arg)
1923 if result:
1924 # Prepend the result, a list of tuples.
1925 assert isinstance(result, list), repr(result)
1926 exec_list[:0] = result
1928 # For debugging...
1929 # try:
1930 # func, arg = data
1931 # if 0:
1932 # func_name = g.truncate(func.__name__, 15)
1933 # print(
1934 # f"{self.node.__class__.__name__:>10}:"
1935 # f"{func_name:>20} "
1936 # f"{arg.__class__.__name__}")
1937 # except ValueError:
1938 # g.trace('BAD DATA', self.node.__class__.__name__)
1939 # if isinstance(data, (list, tuple)):
1940 # for z in data:
1941 # print(data)
1942 # else:
1943 # print(repr(data))
1944 # raise
1945 #@+node:ekr.20220330155314.1: *5* iterative.visit
1946 def visit(self, node: Node) -> ActionList:
1947 """'Visit' an ast node by return a new list of tuples."""
1948 # Keep this trace.
1949 if False: # pragma: no cover
1950 cn = node.__class__.__name__ if node else ' '
1951 caller1, caller2 = g.callers(2).split(',')
1952 g.trace(f"{caller1:>15} {caller2:<14} {cn}")
1953 if node is None:
1954 return []
1955 # More general, more convenient.
1956 if isinstance(node, (list, tuple)):
1957 result = []
1958 for z in node:
1959 if isinstance(z, ast.AST):
1960 result.append((self.visit, z))
1961 else: # pragma: no cover (This might never happen).
1962 # All other fields should contain ints or strings.
1963 assert isinstance(z, (int, str)), z.__class__.__name__
1964 return result
1965 # We *do* want to crash if the visitor doesn't exist.
1966 assert isinstance(node, ast.AST), repr(node)
1967 method = getattr(self, 'do_' + node.__class__.__name__)
1968 # Don't call *anything* here. Just return a new list of tuples.
1969 return [
1970 (self.enter_node, node),
1971 (method, node),
1972 (self.leave_node, node),
1973 ]
1974 #@+node:ekr.20220330133336.1: *4* iterative: Visitors
1975 #@+node:ekr.20220330133336.2: *5* iterative.keyword: not called!
1976 # keyword arguments supplied to call (NULL identifier for **kwargs)
1978 # keyword = (identifier? arg, expr value)
1980 def do_keyword(self, node: Node) -> List: # pragma: no cover
1981 """A keyword arg in an ast.Call."""
1982 # This should never be called.
1983 # iterative.hande_call_arguments calls self.visit(kwarg_arg.value) instead.
1984 filename = getattr(self, 'filename', '<no file>')
1985 raise AssignLinksError(
1986 f"file: {filename}\n"
1987 f"do_keyword should never be called\n"
1988 f"{g.callers(8)}")
1989 #@+node:ekr.20220330133336.3: *5* iterative: Contexts
1990 #@+node:ekr.20220330133336.4: *6* iterative.arg
1991 # arg = (identifier arg, expr? annotation)
1993 def do_arg(self, node: Node) -> ActionList:
1994 """This is one argument of a list of ast.Function or ast.Lambda arguments."""
1996 annotation = getattr(node, 'annotation', None)
1997 result: List = [
1998 (self.name, node.arg),
1999 ]
2000 if annotation:
2001 result.extend([
2002 (self.op, ':'),
2003 (self.visit, annotation),
2004 ])
2005 return result
2007 #@+node:ekr.20220330133336.5: *6* iterative.arguments
2008 # arguments = (
2009 # arg* posonlyargs, arg* args, arg? vararg, arg* kwonlyargs,
2010 # expr* kw_defaults, arg? kwarg, expr* defaults
2011 # )
2013 def do_arguments(self, node: Node) -> ActionList:
2014 """Arguments to ast.Function or ast.Lambda, **not** ast.Call."""
2015 #
2016 # No need to generate commas anywhere below.
2017 #
2018 # Let block. Some fields may not exist pre Python 3.8.
2019 n_plain = len(node.args) - len(node.defaults)
2020 posonlyargs = getattr(node, 'posonlyargs', [])
2021 vararg = getattr(node, 'vararg', None)
2022 kwonlyargs = getattr(node, 'kwonlyargs', [])
2023 kw_defaults = getattr(node, 'kw_defaults', [])
2024 kwarg = getattr(node, 'kwarg', None)
2025 result: ActionList = []
2026 # 1. Sync the position-only args.
2027 if posonlyargs:
2028 for n, z in enumerate(posonlyargs):
2029 # self.visit(z)
2030 result.append((self.visit, z))
2031 # self.op('/')
2032 result.append((self.op, '/'))
2033 # 2. Sync all args.
2034 for i, z in enumerate(node.args):
2035 # self.visit(z)
2036 result.append((self.visit, z))
2037 if i >= n_plain:
2038 # self.op('=')
2039 # self.visit(node.defaults[i - n_plain])
2040 result.extend([
2041 (self.op, '='),
2042 (self.visit, node.defaults[i - n_plain]),
2043 ])
2044 # 3. Sync the vararg.
2045 if vararg:
2046 # self.op('*')
2047 # self.visit(vararg)
2048 result.extend([
2049 (self.op, '*'),
2050 (self.visit, vararg),
2051 ])
2052 # 4. Sync the keyword-only args.
2053 if kwonlyargs:
2054 if not vararg:
2055 # self.op('*')
2056 result.append((self.op, '*'))
2057 for n, z in enumerate(kwonlyargs):
2058 # self.visit(z)
2059 result.append((self.visit, z))
2060 val = kw_defaults[n]
2061 if val is not None:
2062 # self.op('=')
2063 # self.visit(val)
2064 result.extend([
2065 (self.op, '='),
2066 (self.visit, val),
2067 ])
2068 # 5. Sync the kwarg.
2069 if kwarg:
2070 # self.op('**')
2071 # self.visit(kwarg)
2072 result.extend([
2073 (self.op, '**'),
2074 (self.visit, kwarg),
2075 ])
2076 return result
2080 #@+node:ekr.20220330133336.6: *6* iterative.AsyncFunctionDef
2081 # AsyncFunctionDef(identifier name, arguments args, stmt* body, expr* decorator_list,
2082 # expr? returns)
2084 def do_AsyncFunctionDef(self, node: Node) -> ActionList:
2086 returns = getattr(node, 'returns', None)
2087 result: ActionList = []
2088 # Decorators...
2089 # @{z}\n
2090 for z in node.decorator_list or []:
2091 result.extend([
2092 (self.op, '@'),
2093 (self.visit, z)
2094 ])
2095 # Signature...
2096 # def name(args): -> returns\n
2097 # def name(args):\n
2098 result.extend([
2099 (self.name, 'async'),
2100 (self.name, 'def'),
2101 (self.name, node.name), # A string.
2102 (self.op, '('),
2103 (self.visit, node.args),
2104 (self.op, ')'),
2105 ])
2106 if returns is not None:
2107 result.extend([
2108 (self.op, '->'),
2109 (self.visit, node.returns),
2110 ])
2111 # Body...
2112 result.extend([
2113 (self.op, ':'),
2114 (self.visit, node.body),
2115 ])
2116 return result
2117 #@+node:ekr.20220330133336.7: *6* iterative.ClassDef
2118 def do_ClassDef(self, node: Node) -> ActionList:
2120 result: ActionList = []
2121 for z in node.decorator_list or []:
2122 # @{z}\n
2123 result.extend([
2124 (self.op, '@'),
2125 (self.visit, z),
2126 ])
2127 # class name(bases):\n
2128 result.extend([
2129 (self.name, 'class'),
2130 (self.name, node.name), # A string.
2131 ])
2132 if node.bases:
2133 result.extend([
2134 (self.op, '('),
2135 (self.visit, node.bases),
2136 (self.op, ')'),
2137 ])
2138 result.extend([
2139 (self.op, ':'),
2140 (self.visit, node.body),
2141 ])
2142 return result
2143 #@+node:ekr.20220330133336.8: *6* iterative.FunctionDef
2144 # FunctionDef(
2145 # identifier name, arguments args,
2146 # stmt* body,
2147 # expr* decorator_list,
2148 # expr? returns,
2149 # string? type_comment)
2151 def do_FunctionDef(self, node: Node) -> ActionList:
2153 returns = getattr(node, 'returns', None)
2154 result: ActionList = []
2155 # Decorators...
2156 # @{z}\n
2157 for z in node.decorator_list or []:
2158 result.extend([
2159 (self.op, '@'),
2160 (self.visit, z)
2161 ])
2162 # Signature...
2163 # def name(args): -> returns\n
2164 # def name(args):\n
2165 result.extend([
2166 (self.name, 'def'),
2167 (self.name, node.name), # A string.
2168 (self.op, '('),
2169 (self.visit, node.args),
2170 (self.op, ')'),
2171 ])
2172 if returns is not None:
2173 result.extend([
2174 (self.op, '->'),
2175 (self.visit, node.returns),
2176 ])
2177 # Body...
2178 result.extend([
2179 (self.op, ':'),
2180 (self.visit, node.body),
2181 ])
2182 return result
2183 #@+node:ekr.20220330133336.9: *6* iterative.Interactive
2184 def do_Interactive(self, node: Node) -> ActionList: # pragma: no cover
2186 return [
2187 (self.visit, node.body),
2188 ]
2189 #@+node:ekr.20220330133336.10: *6* iterative.Lambda
2190 def do_Lambda(self, node: Node) -> ActionList:
2192 return [
2193 (self.name, 'lambda'),
2194 (self.visit, node.args),
2195 (self.op, ':'),
2196 (self.visit, node.body),
2197 ]
2199 #@+node:ekr.20220330133336.11: *6* iterative.Module
2200 def do_Module(self, node: Node) -> ActionList:
2202 # Encoding is a non-syncing statement.
2203 return [
2204 (self.visit, node.body),
2205 ]
2206 #@+node:ekr.20220330133336.12: *5* iterative: Expressions
2207 #@+node:ekr.20220330133336.13: *6* iterative.Expr
2208 def do_Expr(self, node: Node) -> ActionList:
2209 """An outer expression."""
2210 # No need to put parentheses.
2211 return [
2212 (self.visit, node.value),
2213 ]
2214 #@+node:ekr.20220330133336.14: *6* iterative.Expression
2215 def do_Expression(self, node: Node) -> ActionList: # pragma: no cover
2216 """An inner expression."""
2217 # No need to put parentheses.
2218 return [
2219 (self.visit, node.body),
2220 ]
2221 #@+node:ekr.20220330133336.15: *6* iterative.GeneratorExp
2222 def do_GeneratorExp(self, node: Node) -> ActionList:
2223 # '<gen %s for %s>' % (elt, ','.join(gens))
2224 # No need to put parentheses or commas.
2225 return [
2226 (self.visit, node.elt),
2227 (self.visit, node.generators),
2228 ]
2229 #@+node:ekr.20220330133336.16: *6* iterative.NamedExpr
2230 # NamedExpr(expr target, expr value)
2232 def do_NamedExpr(self, node: Node) -> ActionList: # Python 3.8+
2234 return [
2235 (self.visit, node.target),
2236 (self.op, ':='),
2237 (self.visit, node.value),
2238 ]
2239 #@+node:ekr.20220402160128.1: *5* iterative: Operands
2240 #@+node:ekr.20220402160128.2: *6* iterative.Attribute
2241 # Attribute(expr value, identifier attr, expr_context ctx)
2243 def do_Attribute(self, node: Node) -> ActionList:
2245 return [
2246 (self.visit, node.value),
2247 (self.op, '.'),
2248 (self.name, node.attr), # A string.
2249 ]
2250 #@+node:ekr.20220402160128.3: *6* iterative.Bytes
2251 def do_Bytes(self, node: Node) -> ActionList:
2253 """
2254 It's invalid to mix bytes and non-bytes literals, so just
2255 advancing to the next 'string' token suffices.
2256 """
2257 token = self.find_next_significant_token()
2258 return [
2259 (self.token, ('string', token.value)),
2260 ]
2261 #@+node:ekr.20220402160128.4: *6* iterative.comprehension
2262 # comprehension = (expr target, expr iter, expr* ifs, int is_async)
2264 def do_comprehension(self, node: Node) -> ActionList:
2266 # No need to put parentheses.
2267 result: ActionList = [
2268 (self.name, 'for'),
2269 (self.visit, node.target), # A name
2270 (self.name, 'in'),
2271 (self.visit, node.iter),
2272 ]
2273 for z in node.ifs or []:
2274 result.extend([
2275 (self.name, 'if'),
2276 (self.visit, z),
2277 ])
2278 return result
2279 #@+node:ekr.20220402160128.5: *6* iterative.Constant
2280 def do_Constant(self, node: Node) -> ActionList: # pragma: no cover
2281 """
2283 https://greentreesnakes.readthedocs.io/en/latest/nodes.html
2285 A constant. The value attribute holds the Python object it represents.
2286 This can be simple types such as a number, string or None, but also
2287 immutable container types (tuples and frozensets) if all of their
2288 elements are constant.
2289 """
2290 # Support Python 3.8.
2291 if node.value is None or isinstance(node.value, bool):
2292 # Weird: return a name!
2293 return [
2294 (self.token, ('name', repr(node.value))),
2295 ]
2296 if node.value == Ellipsis:
2297 return [
2298 (self.op, '...'),
2299 ]
2300 if isinstance(node.value, str):
2301 return self.do_Str(node)
2302 if isinstance(node.value, (int, float)):
2303 return [
2304 (self.token, ('number', repr(node.value))),
2305 ]
2306 if isinstance(node.value, bytes):
2307 return self.do_Bytes(node)
2308 if isinstance(node.value, tuple):
2309 return self.do_Tuple(node)
2310 if isinstance(node.value, frozenset):
2311 return self.do_Set(node)
2312 g.trace('----- Oops -----', repr(node.value), g.callers())
2313 return []
2315 #@+node:ekr.20220402160128.6: *6* iterative.Dict
2316 # Dict(expr* keys, expr* values)
2318 def do_Dict(self, node: Node) -> ActionList:
2320 assert len(node.keys) == len(node.values)
2321 result: List = [
2322 (self.op, '{'),
2323 ]
2324 # No need to put commas.
2325 for i, key in enumerate(node.keys):
2326 key, value = node.keys[i], node.values[i]
2327 result.extend([
2328 (self.visit, key), # a Str node.
2329 (self.op, ':'),
2330 ])
2331 if value is not None:
2332 result.append((self.visit, value))
2333 result.append((self.op, '}'))
2334 return result
2335 #@+node:ekr.20220402160128.7: *6* iterative.DictComp
2336 # DictComp(expr key, expr value, comprehension* generators)
2338 # d2 = {val: key for key, val in d}
2340 def do_DictComp(self, node: Node) -> ActionList:
2342 result: ActionList = [
2343 (self.token, ('op', '{')),
2344 (self.visit, node.key),
2345 (self.op, ':'),
2346 (self.visit, node.value),
2347 ]
2348 for z in node.generators or []:
2349 result.extend([
2350 (self.visit, z),
2351 (self.token, ('op', '}')),
2352 ])
2353 return result
2355 #@+node:ekr.20220402160128.8: *6* iterative.Ellipsis
2356 def do_Ellipsis(self, node: Node) -> ActionList: # pragma: no cover (Does not exist for python 3.8+)
2358 return [
2359 (self.op, '...'),
2360 ]
2361 #@+node:ekr.20220402160128.9: *6* iterative.ExtSlice
2362 # https://docs.python.org/3/reference/expressions.html#slicings
2364 # ExtSlice(slice* dims)
2366 def do_ExtSlice(self, node: Node) -> ActionList: # pragma: no cover (deprecated)
2368 result: ActionList = []
2369 for i, z in enumerate(node.dims):
2370 # self.visit(z)
2371 result.append((self.visit, z))
2372 if i < len(node.dims) - 1:
2373 # self.op(',')
2374 result.append((self.op, ','))
2375 return result
2376 #@+node:ekr.20220402160128.10: *6* iterative.Index
2377 def do_Index(self, node: Node) -> ActionList: # pragma: no cover (deprecated)
2379 return [
2380 (self.visit, node.value),
2381 ]
2382 #@+node:ekr.20220402160128.11: *6* iterative.FormattedValue: not called!
2383 # FormattedValue(expr value, int? conversion, expr? format_spec)
2385 def do_FormattedValue(self, node: Node) -> ActionList: # pragma: no cover
2386 """
2387 This node represents the *components* of a *single* f-string.
2389 Happily, JoinedStr nodes *also* represent *all* f-strings,
2390 so the TOG should *never visit this node!
2391 """
2392 filename = getattr(self, 'filename', '<no file>')
2393 raise AssignLinksError(
2394 f"file: {filename}\n"
2395 f"do_FormattedValue should never be called")
2397 # This code has no chance of being useful...
2398 # conv = node.conversion
2399 # spec = node.format_spec
2400 # self.visit(node.value)
2401 # if conv is not None:
2402 # self.token('number', conv)
2403 # if spec is not None:
2404 # self.visit(node.format_spec)
2405 #@+node:ekr.20220402160128.12: *6* iterative.JoinedStr & helpers
2406 # JoinedStr(expr* values)
2408 def do_JoinedStr(self, node: Node) -> ActionList:
2409 """
2410 JoinedStr nodes represent at least one f-string and all other strings
2411 concatentated to it.
2413 Analyzing JoinedStr.values would be extremely tricky, for reasons that
2414 need not be explained here.
2416 Instead, we get the tokens *from the token list itself*!
2417 """
2418 # for z in self.get_concatenated_string_tokens():
2419 # self.gen_token(z.kind, z.value)
2420 return [
2421 (self.token, (z.kind, z.value))
2422 for z in self.get_concatenated_string_tokens()
2423 ]
2424 #@+node:ekr.20220402160128.13: *6* iterative.List
2425 def do_List(self, node: Node) -> ActionList:
2427 # No need to put commas.
2428 return [
2429 (self.op, '['),
2430 (self.visit, node.elts),
2431 (self.op, ']'),
2432 ]
2433 #@+node:ekr.20220402160128.14: *6* iterative.ListComp
2434 # ListComp(expr elt, comprehension* generators)
2436 def do_ListComp(self, node: Node) -> ActionList:
2438 result: List = [
2439 (self.op, '['),
2440 (self.visit, node.elt),
2441 ]
2442 for z in node.generators:
2443 result.append((self.visit, z))
2444 result.append((self.op, ']'))
2445 return result
2446 #@+node:ekr.20220402160128.15: *6* iterative.Name & NameConstant
2447 def do_Name(self, node: Node) -> ActionList:
2449 return [
2450 (self.name, node.id),
2451 ]
2453 def do_NameConstant(self, node: Node) -> ActionList: # pragma: no cover (Does not exist in Python 3.8+)
2455 return [
2456 (self.name, repr(node.value)),
2457 ]
2458 #@+node:ekr.20220402160128.16: *6* iterative.Num
2459 def do_Num(self, node: Node) -> ActionList: # pragma: no cover (Does not exist in Python 3.8+)
2461 return [
2462 (self.token, ('number', node.n)),
2463 ]
2464 #@+node:ekr.20220402160128.17: *6* iterative.Set
2465 # Set(expr* elts)
2467 def do_Set(self, node: Node) -> ActionList:
2469 return [
2470 (self.op, '{'),
2471 (self.visit, node.elts),
2472 (self.op, '}'),
2473 ]
2474 #@+node:ekr.20220402160128.18: *6* iterative.SetComp
2475 # SetComp(expr elt, comprehension* generators)
2477 def do_SetComp(self, node: Node) -> ActionList:
2479 result: List = [
2480 (self.op, '{'),
2481 (self.visit, node.elt),
2482 ]
2483 for z in node.generators or []:
2484 result.append((self.visit, z))
2485 result.append((self.op, '}'))
2486 return result
2487 #@+node:ekr.20220402160128.19: *6* iterative.Slice
2488 # slice = Slice(expr? lower, expr? upper, expr? step)
2490 def do_Slice(self, node: Node) -> ActionList:
2492 lower = getattr(node, 'lower', None)
2493 upper = getattr(node, 'upper', None)
2494 step = getattr(node, 'step', None)
2495 result: ActionList = []
2496 if lower is not None:
2497 result.append((self.visit, lower))
2498 # Always put the colon between upper and lower.
2499 result.append((self.op, ':'))
2500 if upper is not None:
2501 result.append((self.visit, upper))
2502 # Put the second colon if it exists in the token list.
2503 if step is None:
2504 result.append((self.slice_helper, node))
2505 else:
2506 result.extend([
2507 (self.op, ':'),
2508 (self.visit, step),
2509 ])
2510 return result
2512 def slice_helper(self, node: Node) -> ActionList:
2513 """Delayed evaluation!"""
2514 token = self.find_next_significant_token()
2515 if token and token.value == ':':
2516 return [
2517 (self.op, ':'),
2518 ]
2519 return []
2520 #@+node:ekr.20220402160128.20: *6* iterative.Str & helper
2521 def do_Str(self, node: Node) -> ActionList:
2522 """This node represents a string constant."""
2523 # This loop is necessary to handle string concatenation.
2525 # for z in self.get_concatenated_string_tokens():
2526 # self.gen_token(z.kind, z.value)
2528 return [
2529 (self.token, (z.kind, z.value))
2530 for z in self.get_concatenated_string_tokens()
2531 ]
2533 #@+node:ekr.20220402160128.21: *7* iterative.get_concatenated_tokens
2534 def get_concatenated_string_tokens(self) -> List:
2535 """
2536 Return the next 'string' token and all 'string' tokens concatenated to
2537 it. *Never* update self.px here.
2538 """
2539 trace = False
2540 tag = 'iterative.get_concatenated_string_tokens'
2541 i = self.px
2542 # First, find the next significant token. It should be a string.
2543 i, token = i + 1, None
2544 while i < len(self.tokens):
2545 token = self.tokens[i]
2546 i += 1
2547 if token.kind == 'string':
2548 # Rescan the string.
2549 i -= 1
2550 break
2551 # An error.
2552 if is_significant_token(token): # pragma: no cover
2553 break
2554 # Raise an error if we didn't find the expected 'string' token.
2555 if not token or token.kind != 'string': # pragma: no cover
2556 if not token:
2557 token = self.tokens[-1]
2558 filename = getattr(self, 'filename', '<no filename>')
2559 raise AssignLinksError(
2560 f"\n"
2561 f"{tag}...\n"
2562 f"file: {filename}\n"
2563 f"line: {token.line_number}\n"
2564 f" i: {i}\n"
2565 f"expected 'string' token, got {token!s}")
2566 # Accumulate string tokens.
2567 assert self.tokens[i].kind == 'string'
2568 results = []
2569 while i < len(self.tokens):
2570 token = self.tokens[i]
2571 i += 1
2572 if token.kind == 'string':
2573 results.append(token)
2574 elif token.kind == 'op' or is_significant_token(token):
2575 # Any significant token *or* any op will halt string concatenation.
2576 break
2577 # 'ws', 'nl', 'newline', 'comment', 'indent', 'dedent', etc.
2578 # The (significant) 'endmarker' token ensures we will have result.
2579 assert results
2580 if trace: # pragma: no cover
2581 g.printObj(results, tag=f"{tag}: Results")
2582 return results
2583 #@+node:ekr.20220402160128.22: *6* iterative.Subscript
2584 # Subscript(expr value, slice slice, expr_context ctx)
2586 def do_Subscript(self, node: Node) -> ActionList:
2588 return [
2589 (self.visit, node.value),
2590 (self.op, '['),
2591 (self.visit, node.slice),
2592 (self.op, ']'),
2593 ]
2594 #@+node:ekr.20220402160128.23: *6* iterative.Tuple
2595 # Tuple(expr* elts, expr_context ctx)
2597 def do_Tuple(self, node: Node) -> ActionList:
2599 # Do not call gen_op for parens or commas here.
2600 # They do not necessarily exist in the token list!
2602 return [
2603 (self.visit, node.elts),
2604 ]
2605 #@+node:ekr.20220330133336.40: *5* iterative: Operators
2606 #@+node:ekr.20220330133336.41: *6* iterative.BinOp
2607 def do_BinOp(self, node: Node) -> ActionList:
2609 return [
2610 (self.visit, node.left),
2611 (self.op, op_name(node.op)),
2612 (self.visit, node.right),
2613 ]
2615 #@+node:ekr.20220330133336.42: *6* iterative.BoolOp
2616 # BoolOp(boolop op, expr* values)
2618 def do_BoolOp(self, node: Node) -> ActionList:
2620 result: ActionList = []
2621 op_name_ = op_name(node.op)
2622 for i, z in enumerate(node.values):
2623 result.append((self.visit, z))
2624 if i < len(node.values) - 1:
2625 result.append((self.name, op_name_))
2626 return result
2627 #@+node:ekr.20220330133336.43: *6* iterative.Compare
2628 # Compare(expr left, cmpop* ops, expr* comparators)
2630 def do_Compare(self, node: Node) -> ActionList:
2632 assert len(node.ops) == len(node.comparators)
2633 result: List = [(self.visit, node.left)]
2634 for i, z in enumerate(node.ops):
2635 op_name_ = op_name(node.ops[i])
2636 if op_name_ in ('not in', 'is not'):
2637 for z in op_name_.split(' '):
2638 # self.name(z)
2639 result.append((self.name, z))
2640 elif op_name_.isalpha():
2641 # self.name(op_name_)
2642 result.append((self.name, op_name_))
2643 else:
2644 # self.op(op_name_)
2645 result.append((self.op, op_name_))
2646 # self.visit(node.comparators[i])
2647 result.append((self.visit, node.comparators[i]))
2648 return result
2649 #@+node:ekr.20220330133336.44: *6* iterative.UnaryOp
2650 def do_UnaryOp(self, node: Node) -> ActionList:
2652 op_name_ = op_name(node.op)
2653 result: List = []
2654 if op_name_.isalpha():
2655 # self.name(op_name_)
2656 result.append((self.name, op_name_))
2657 else:
2658 # self.op(op_name_)
2659 result.append((self.op, op_name_))
2660 # self.visit(node.operand)
2661 result.append((self.visit, node.operand))
2662 return result
2663 #@+node:ekr.20220330133336.45: *6* iterative.IfExp (ternary operator)
2664 # IfExp(expr test, expr body, expr orelse)
2666 def do_IfExp(self, node: Node) -> ActionList:
2668 #'%s if %s else %s'
2669 return [
2670 (self.visit, node.body),
2671 (self.name, 'if'),
2672 (self.visit, node.test),
2673 (self.name, 'else'),
2674 (self.visit, node.orelse),
2675 ]
2677 #@+node:ekr.20220330133336.46: *5* iterative: Statements
2678 #@+node:ekr.20220330133336.47: *6* iterative.Starred
2679 # Starred(expr value, expr_context ctx)
2681 def do_Starred(self, node: Node) -> ActionList:
2682 """A starred argument to an ast.Call"""
2683 return [
2684 (self.op, '*'),
2685 (self.visit, node.value),
2686 ]
2687 #@+node:ekr.20220330133336.48: *6* iterative.AnnAssign
2688 # AnnAssign(expr target, expr annotation, expr? value, int simple)
2690 def do_AnnAssign(self, node: Node) -> ActionList:
2692 # {node.target}:{node.annotation}={node.value}\n'
2693 result: ActionList = [
2694 (self.visit, node.target),
2695 (self.op, ':'),
2696 (self.visit, node.annotation),
2697 ]
2698 if node.value is not None: # #1851
2699 result.extend([
2700 (self.op, '='),
2701 (self.visit, node.value),
2702 ])
2703 return result
2704 #@+node:ekr.20220330133336.49: *6* iterative.Assert
2705 # Assert(expr test, expr? msg)
2707 def do_Assert(self, node: Node) -> ActionList:
2709 # No need to put parentheses or commas.
2710 msg = getattr(node, 'msg', None)
2711 result: List = [
2712 (self.name, 'assert'),
2713 (self.visit, node.test),
2714 ]
2715 if msg is not None:
2716 result.append((self.visit, node.msg))
2717 return result
2718 #@+node:ekr.20220330133336.50: *6* iterative.Assign
2719 def do_Assign(self, node: Node) -> ActionList:
2721 result: ActionList = []
2722 for z in node.targets:
2723 result.extend([
2724 (self.visit, z),
2725 (self.op, '=')
2726 ])
2727 result.append((self.visit, node.value))
2728 return result
2729 #@+node:ekr.20220330133336.51: *6* iterative.AsyncFor
2730 def do_AsyncFor(self, node: Node) -> ActionList:
2732 # The def line...
2733 # Py 3.8 changes the kind of token.
2734 async_token_type = 'async' if has_async_tokens else 'name'
2735 result: List = [
2736 (self.token, (async_token_type, 'async')),
2737 (self.name, 'for'),
2738 (self.visit, node.target),
2739 (self.name, 'in'),
2740 (self.visit, node.iter),
2741 (self.op, ':'),
2742 # Body...
2743 (self.visit, node.body),
2744 ]
2745 # Else clause...
2746 if node.orelse:
2747 result.extend([
2748 (self.name, 'else'),
2749 (self.op, ':'),
2750 (self.visit, node.orelse),
2751 ])
2752 return result
2753 #@+node:ekr.20220330133336.52: *6* iterative.AsyncWith
2754 def do_AsyncWith(self, node: Node) -> ActionList:
2756 async_token_type = 'async' if has_async_tokens else 'name'
2757 return [
2758 (self.token, (async_token_type, 'async')),
2759 (self.do_With, node),
2760 ]
2761 #@+node:ekr.20220330133336.53: *6* iterative.AugAssign
2762 # AugAssign(expr target, operator op, expr value)
2764 def do_AugAssign(self, node: Node) -> ActionList:
2766 # %s%s=%s\n'
2767 return [
2768 (self.visit, node.target),
2769 (self.op, op_name(node.op) + '='),
2770 (self.visit, node.value),
2771 ]
2772 #@+node:ekr.20220330133336.54: *6* iterative.Await
2773 # Await(expr value)
2775 def do_Await(self, node: Node) -> ActionList:
2777 #'await %s\n'
2778 async_token_type = 'await' if has_async_tokens else 'name'
2779 return [
2780 (self.token, (async_token_type, 'await')),
2781 (self.visit, node.value),
2782 ]
2783 #@+node:ekr.20220330133336.55: *6* iterative.Break
2784 def do_Break(self, node: Node) -> ActionList:
2786 return [
2787 (self.name, 'break'),
2788 ]
2789 #@+node:ekr.20220330133336.56: *6* iterative.Call & helpers
2790 # Call(expr func, expr* args, keyword* keywords)
2792 # Python 3 ast.Call nodes do not have 'starargs' or 'kwargs' fields.
2794 def do_Call(self, node: Node) -> ActionList:
2796 # The calls to op(')') and op('(') do nothing by default.
2797 # No need to generate any commas.
2798 # Subclasses might handle them in an overridden iterative.set_links.
2799 return [
2800 (self.visit, node.func),
2801 (self.op, '('),
2802 (self.handle_call_arguments, node),
2803 (self.op, ')'),
2804 ]
2805 #@+node:ekr.20220330133336.57: *7* iterative.arg_helper
2806 def arg_helper(self, node: Node) -> ActionList:
2807 """
2808 Yield the node, with a special case for strings.
2809 """
2810 result: List = []
2811 if isinstance(node, str):
2812 result.append((self.token, ('name', node)))
2813 else:
2814 result.append((self.visit, node))
2815 return result
2816 #@+node:ekr.20220330133336.58: *7* iterative.handle_call_arguments
2817 def handle_call_arguments(self, node: Node) -> ActionList:
2818 """
2819 Generate arguments in the correct order.
2821 Call(expr func, expr* args, keyword* keywords)
2823 https://docs.python.org/3/reference/expressions.html#calls
2825 Warning: This code will fail on Python 3.8 only for calls
2826 containing kwargs in unexpected places.
2827 """
2828 # *args: in node.args[]: Starred(value=Name(id='args'))
2829 # *[a, 3]: in node.args[]: Starred(value=List(elts=[Name(id='a'), Num(n=3)])
2830 # **kwargs: in node.keywords[]: keyword(arg=None, value=Name(id='kwargs'))
2831 #
2832 # Scan args for *name or *List
2833 args = node.args or []
2834 keywords = node.keywords or []
2836 def get_pos(obj: Any) -> Tuple:
2837 line1 = getattr(obj, 'lineno', None)
2838 col1 = getattr(obj, 'col_offset', None)
2839 return line1, col1, obj
2841 def sort_key(aTuple: Tuple) -> int:
2842 line, col, obj = aTuple
2843 return line * 1000 + col
2845 assert py_version >= (3, 9)
2847 places = [get_pos(z) for z in args + keywords]
2848 places.sort(key=sort_key)
2849 ordered_args = [z[2] for z in places]
2850 result: ActionList = []
2851 for z in ordered_args:
2852 if isinstance(z, ast.Starred):
2853 result.extend([
2854 (self.op, '*'),
2855 (self.visit, z.value),
2856 ])
2857 elif isinstance(z, ast.keyword):
2858 if getattr(z, 'arg', None) is None:
2859 result.extend([
2860 (self.op, '**'),
2861 (self.arg_helper, z.value),
2862 ])
2863 else:
2864 result.extend([
2865 (self.arg_helper, z.arg),
2866 (self.op, '='),
2867 (self.arg_helper, z.value),
2868 ])
2869 else:
2870 result.append((self.arg_helper, z))
2871 return result
2872 #@+node:ekr.20220330133336.59: *6* iterative.Continue
2873 def do_Continue(self, node: Node) -> ActionList:
2875 return [
2876 (self.name, 'continue'),
2877 ]
2878 #@+node:ekr.20220330133336.60: *6* iterative.Delete
2879 def do_Delete(self, node: Node) -> ActionList:
2881 # No need to put commas.
2882 return [
2883 (self.name, 'del'),
2884 (self.visit, node.targets),
2885 ]
2886 #@+node:ekr.20220330133336.61: *6* iterative.ExceptHandler
2887 def do_ExceptHandler(self, node: Node) -> ActionList:
2889 # Except line...
2890 result: List = [
2891 (self.name, 'except'),
2892 ]
2893 if getattr(node, 'type', None):
2894 result.append((self.visit, node.type))
2895 if getattr(node, 'name', None):
2896 result.extend([
2897 (self.name, 'as'),
2898 (self.name, node.name),
2899 ])
2900 result.extend([
2901 (self.op, ':'),
2902 # Body...
2903 (self.visit, node.body),
2904 ])
2905 return result
2906 #@+node:ekr.20220330133336.62: *6* iterative.For
2907 def do_For(self, node: Node) -> ActionList:
2909 result: ActionList = [
2910 # The def line...
2911 (self.name, 'for'),
2912 (self.visit, node.target),
2913 (self.name, 'in'),
2914 (self.visit, node.iter),
2915 (self.op, ':'),
2916 # Body...
2917 (self.visit, node.body),
2918 ]
2919 # Else clause...
2920 if node.orelse:
2921 result.extend([
2922 (self.name, 'else'),
2923 (self.op, ':'),
2924 (self.visit, node.orelse),
2925 ])
2926 return result
2927 #@+node:ekr.20220330133336.63: *6* iterative.Global
2928 # Global(identifier* names)
2930 def do_Global(self, node: Node) -> ActionList:
2932 result = [
2933 (self.name, 'global'),
2934 ]
2935 for z in node.names:
2936 result.append((self.name, z))
2937 return result
2938 #@+node:ekr.20220330133336.64: *6* iterative.If & helpers
2939 # If(expr test, stmt* body, stmt* orelse)
2941 def do_If(self, node: Node) -> ActionList:
2942 #@+<< do_If docstring >>
2943 #@+node:ekr.20220330133336.65: *7* << do_If docstring >>
2944 """
2945 The parse trees for the following are identical!
2947 if 1: if 1:
2948 pass pass
2949 else: elif 2:
2950 if 2: pass
2951 pass
2953 So there is *no* way for the 'if' visitor to disambiguate the above two
2954 cases from the parse tree alone.
2956 Instead, we scan the tokens list for the next 'if', 'else' or 'elif' token.
2957 """
2958 #@-<< do_If docstring >>
2959 # Use the next significant token to distinguish between 'if' and 'elif'.
2960 token = self.find_next_significant_token()
2961 result: List = [
2962 (self.name, token.value),
2963 (self.visit, node.test),
2964 (self.op, ':'),
2965 # Body...
2966 (self.visit, node.body),
2967 ]
2968 # Else and elif clauses...
2969 if node.orelse:
2970 # We *must* delay the evaluation of the else clause.
2971 result.append((self.if_else_helper, node))
2972 return result
2974 def if_else_helper(self, node: Node) -> ActionList:
2975 """Delayed evaluation!"""
2976 token = self.find_next_significant_token()
2977 if token.value == 'else':
2978 return [
2979 (self.name, 'else'),
2980 (self.op, ':'),
2981 (self.visit, node.orelse),
2982 ]
2983 return [
2984 (self.visit, node.orelse),
2985 ]
2986 #@+node:ekr.20220330133336.66: *6* iterative.Import & helper
2987 def do_Import(self, node: Node) -> ActionList:
2989 result: List = [
2990 (self.name, 'import'),
2991 ]
2992 for alias in node.names:
2993 result.append((self.name, alias.name))
2994 if alias.asname:
2995 result.extend([
2996 (self.name, 'as'),
2997 (self.name, alias.asname),
2998 ])
2999 return result
3000 #@+node:ekr.20220330133336.67: *6* iterative.ImportFrom
3001 # ImportFrom(identifier? module, alias* names, int? level)
3003 def do_ImportFrom(self, node: Node) -> ActionList:
3005 result: List = [
3006 (self.name, 'from'),
3007 ]
3008 for i in range(node.level):
3009 result.append((self.op, '.'))
3010 if node.module:
3011 result.append((self.name, node.module))
3012 result.append((self.name, 'import'))
3013 # No need to put commas.
3014 for alias in node.names:
3015 if alias.name == '*': # #1851.
3016 result.append((self.op, '*'))
3017 else:
3018 result.append((self.name, alias.name))
3019 if alias.asname:
3020 result.extend([
3021 (self.name, 'as'),
3022 (self.name, alias.asname),
3023 ])
3024 return result
3025 #@+node:ekr.20220402124844.1: *6* iterative.Match* (Python 3.10+)
3026 # Match(expr subject, match_case* cases)
3028 # match_case = (pattern pattern, expr? guard, stmt* body)
3030 # Full syntax diagram: # https://peps.python.org/pep-0634/#appendix-a
3032 def do_Match(self, node: Node) -> ActionList:
3034 cases = getattr(node, 'cases', [])
3035 result: List = [
3036 (self.name, 'match'),
3037 (self.visit, node.subject),
3038 (self.op, ':'),
3039 ]
3040 for case in cases:
3041 result.append((self.visit, case))
3042 return result
3043 #@+node:ekr.20220402124844.2: *7* iterative.match_case
3044 # match_case = (pattern pattern, expr? guard, stmt* body)
3046 def do_match_case(self, node: Node) -> ActionList:
3048 guard = getattr(node, 'guard', None)
3049 body = getattr(node, 'body', [])
3050 result: List = [
3051 (self.name, 'case'),
3052 (self.visit, node.pattern),
3053 ]
3054 if guard:
3055 result.extend([
3056 (self.name, 'if'),
3057 (self.visit, guard),
3058 ])
3059 result.append((self.op, ':'))
3060 for statement in body:
3061 result.append((self.visit, statement))
3062 return result
3063 #@+node:ekr.20220402124844.3: *7* iterative.MatchAs
3064 # MatchAs(pattern? pattern, identifier? name)
3066 def do_MatchAs(self, node: Node) -> ActionList:
3067 pattern = getattr(node, 'pattern', None)
3068 name = getattr(node, 'name', None)
3069 result: ActionList = []
3070 if pattern and name:
3071 result.extend([
3072 (self.visit, pattern),
3073 (self.name, 'as'),
3074 (self.name, name),
3075 ])
3076 elif pattern:
3077 result.append((self.visit, pattern)) # pragma: no cover
3078 else:
3079 result.append((self.name, name or '_'))
3080 return result
3081 #@+node:ekr.20220402124844.4: *7* iterative.MatchClass
3082 # MatchClass(expr cls, pattern* patterns, identifier* kwd_attrs, pattern* kwd_patterns)
3084 def do_MatchClass(self, node: Node) -> ActionList:
3086 cls = node.cls
3087 patterns = getattr(node, 'patterns', [])
3088 kwd_attrs = getattr(node, 'kwd_attrs', [])
3089 kwd_patterns = getattr(node, 'kwd_patterns', [])
3090 result: List = [
3091 (self.visit, node.cls),
3092 (self.op, '('),
3093 ]
3094 for pattern in patterns:
3095 result.append((self.visit, pattern))
3096 for i, kwd_attr in enumerate(kwd_attrs):
3097 result.extend([
3098 (self.name, kwd_attr), # a String.
3099 (self.op, '='),
3100 (self.visit, kwd_patterns[i]),
3101 ])
3102 result.append((self.op, ')'))
3103 return result
3104 #@+node:ekr.20220402124844.5: *7* iterative.MatchMapping
3105 # MatchMapping(expr* keys, pattern* patterns, identifier? rest)
3107 def do_MatchMapping(self, node: Node) -> ActionList:
3108 keys = getattr(node, 'keys', [])
3109 patterns = getattr(node, 'patterns', [])
3110 rest = getattr(node, 'rest', None)
3111 result: List = [
3112 (self.op, '{'),
3113 ]
3114 for i, key in enumerate(keys):
3115 result.extend([
3116 (self.visit, key),
3117 (self.op, ':'),
3118 (self.visit, patterns[i]),
3119 ])
3120 if rest:
3121 result.extend([
3122 (self.op, '**'),
3123 (self.name, rest), # A string.
3124 ])
3125 result.append((self.op, '}'))
3126 return result
3127 #@+node:ekr.20220402124844.6: *7* iterative.MatchOr
3128 # MatchOr(pattern* patterns)
3130 def do_MatchOr(self, node: Node) -> ActionList:
3132 patterns = getattr(node, 'patterns', [])
3133 result: List = []
3134 for i, pattern in enumerate(patterns):
3135 if i > 0:
3136 result.append((self.op, '|'))
3137 result.append((self.visit, pattern))
3138 return result
3139 #@+node:ekr.20220402124844.7: *7* iterative.MatchSequence
3140 # MatchSequence(pattern* patterns)
3142 def do_MatchSequence(self, node: Node) -> ActionList:
3143 patterns = getattr(node, 'patterns', [])
3144 result: List = []
3145 # Scan for the next '(' or '[' token, skipping the 'case' token.
3146 token = None
3147 for token in self.tokens[self.px + 1 :]:
3148 if token.kind == 'op' and token.value in '([':
3149 break
3150 if is_significant_token(token):
3151 # An implicit tuple: there is no '(' or '[' token.
3152 token = None
3153 break
3154 else:
3155 raise AssignLinksError('Ill-formed tuple') # pragma: no cover
3156 if token:
3157 result.append((self.op, token.value))
3158 for i, pattern in enumerate(patterns):
3159 result.append((self.visit, pattern))
3160 if token:
3161 val = ']' if token.value == '[' else ')'
3162 result.append((self.op, val))
3163 return result
3164 #@+node:ekr.20220402124844.8: *7* iterative.MatchSingleton
3165 # MatchSingleton(constant value)
3167 def do_MatchSingleton(self, node: Node) -> ActionList:
3168 """Match True, False or None."""
3169 return [
3170 (self.token, ('name', repr(node.value))),
3171 ]
3172 #@+node:ekr.20220402124844.9: *7* iterative.MatchStar
3173 # MatchStar(identifier? name)
3175 def do_MatchStar(self, node: Node) -> ActionList:
3177 name = getattr(node, 'name', None)
3178 result: List = [
3179 (self.op, '*'),
3180 ]
3181 if name:
3182 result.append((self.name, name))
3183 return result
3184 #@+node:ekr.20220402124844.10: *7* iterative.MatchValue
3185 # MatchValue(expr value)
3187 def do_MatchValue(self, node: Node) -> ActionList:
3189 return [
3190 (self.visit, node.value),
3191 ]
3192 #@+node:ekr.20220330133336.78: *6* iterative.Nonlocal
3193 # Nonlocal(identifier* names)
3195 def do_Nonlocal(self, node: Node) -> ActionList:
3197 # nonlocal %s\n' % ','.join(node.names))
3198 # No need to put commas.
3199 result: List = [
3200 (self.name, 'nonlocal'),
3201 ]
3202 for z in node.names:
3203 result.append((self.name, z))
3204 return result
3205 #@+node:ekr.20220330133336.79: *6* iterative.Pass
3206 def do_Pass(self, node: Node) -> ActionList:
3208 return ([
3209 (self.name, 'pass'),
3210 ])
3211 #@+node:ekr.20220330133336.80: *6* iterative.Raise
3212 # Raise(expr? exc, expr? cause)
3214 def do_Raise(self, node: Node) -> ActionList:
3216 # No need to put commas.
3217 exc = getattr(node, 'exc', None)
3218 cause = getattr(node, 'cause', None)
3219 tback = getattr(node, 'tback', None)
3220 result: List = [
3221 (self.name, 'raise'),
3222 (self.visit, exc),
3223 ]
3224 if cause:
3225 result.extend([
3226 (self.name, 'from'), # #2446.
3227 (self.visit, cause),
3228 ])
3229 result.append((self.visit, tback))
3230 return result
3232 #@+node:ekr.20220330133336.81: *6* iterative.Return
3233 def do_Return(self, node: Node) -> ActionList:
3235 return [
3236 (self.name, 'return'),
3237 (self.visit, node.value),
3238 ]
3239 #@+node:ekr.20220330133336.82: *6* iterative.Try
3240 # Try(stmt* body, excepthandler* handlers, stmt* orelse, stmt* finalbody)
3242 def do_Try(self, node: Node) -> ActionList:
3244 result: List = [
3245 # Try line...
3246 (self.name, 'try'),
3247 (self.op, ':'),
3248 # Body...
3249 (self.visit, node.body),
3250 (self.visit, node.handlers),
3251 ]
3252 # Else...
3253 if node.orelse:
3254 result.extend([
3255 (self.name, 'else'),
3256 (self.op, ':'),
3257 (self.visit, node.orelse),
3258 ])
3259 # Finally...
3260 if node.finalbody:
3261 result.extend([
3262 (self.name, 'finally'),
3263 (self.op, ':'),
3264 (self.visit, node.finalbody),
3265 ])
3266 return result
3267 #@+node:ekr.20220330133336.83: *6* iterative.While
3268 def do_While(self, node: Node) -> ActionList:
3270 # While line...
3271 # while %s:\n'
3272 result: List = [
3273 (self.name, 'while'),
3274 (self.visit, node.test),
3275 (self.op, ':'),
3276 # Body...
3277 (self.visit, node.body),
3278 ]
3279 # Else clause...
3280 if node.orelse:
3281 result.extend([
3282 (self.name, 'else'),
3283 (self.op, ':'),
3284 (self.visit, node.orelse),
3285 ])
3286 return result
3287 #@+node:ekr.20220330133336.84: *6* iterative.With
3288 # With(withitem* items, stmt* body)
3290 # withitem = (expr context_expr, expr? optional_vars)
3292 def do_With(self, node: Node) -> ActionList:
3294 expr: Optional[ast.AST] = getattr(node, 'context_expression', None)
3295 items: List[ast.AST] = getattr(node, 'items', [])
3296 result: List = [
3297 (self.name, 'with'),
3298 (self.visit, expr),
3299 ]
3300 # No need to put commas.
3301 for item in items:
3302 result.append((self.visit, item.context_expr))
3303 optional_vars = getattr(item, 'optional_vars', None)
3304 if optional_vars is not None:
3305 result.extend([
3306 (self.name, 'as'),
3307 (self.visit, item.optional_vars),
3308 ])
3309 result.extend([
3310 # End the line.
3311 (self.op, ':'),
3312 # Body...
3313 (self.visit, node.body),
3314 ])
3315 return result
3316 #@+node:ekr.20220330133336.85: *6* iterative.Yield
3317 def do_Yield(self, node: Node) -> ActionList:
3319 result: List = [
3320 (self.name, 'yield'),
3321 ]
3322 if hasattr(node, 'value'):
3323 result.extend([
3324 (self.visit, node.value),
3325 ])
3326 return result
3327 #@+node:ekr.20220330133336.86: *6* iterative.YieldFrom
3328 # YieldFrom(expr value)
3330 def do_YieldFrom(self, node: Node) -> ActionList:
3332 return ([
3333 (self.name, 'yield'),
3334 (self.name, 'from'),
3335 (self.visit, node.value),
3336 ])
3337 #@-others
3338#@+node:ekr.20200107165250.1: *3* class Orange
3339class Orange:
3340 """
3341 A flexible and powerful beautifier for Python.
3342 Orange is the new black.
3344 *Important*: This is a predominantly a *token*-based beautifier.
3345 However, orange.colon and orange.possible_unary_op use the parse
3346 tree to provide context that would otherwise be difficult to
3347 deduce.
3348 """
3349 # This switch is really a comment. It will always be false.
3350 # It marks the code that simulates the operation of the black tool.
3351 black_mode = False
3353 # Patterns...
3354 nobeautify_pat = re.compile(r'\s*#\s*pragma:\s*no\s*beautify\b|#\s*@@nobeautify')
3356 # Patterns from FastAtRead class, specialized for python delims.
3357 node_pat = re.compile(r'^(\s*)#@\+node:([^:]+): \*(\d+)?(\*?) (.*)$') # @node
3358 start_doc_pat = re.compile(r'^\s*#@\+(at|doc)?(\s.*?)?$') # @doc or @
3359 at_others_pat = re.compile(r'^(\s*)#@(\+|-)others\b(.*)$') # @others
3361 # Doc parts end with @c or a node sentinel. Specialized for python.
3362 end_doc_pat = re.compile(r"^\s*#@(@(c(ode)?)|([+]node\b.*))$")
3363 #@+others
3364 #@+node:ekr.20200107165250.2: *4* orange.ctor
3365 def __init__(self, settings: Optional[Dict]=None):
3366 """Ctor for Orange class."""
3367 if settings is None:
3368 settings = {}
3369 valid_keys = (
3370 'allow_joined_strings',
3371 'max_join_line_length',
3372 'max_split_line_length',
3373 'orange',
3374 'tab_width',
3375 )
3376 # For mypy...
3377 self.kind: str = ''
3378 # Default settings...
3379 self.allow_joined_strings = False # EKR's preference.
3380 self.max_join_line_length = 88
3381 self.max_split_line_length = 88
3382 self.tab_width = 4
3383 # Override from settings dict...
3384 for key in settings: # pragma: no cover
3385 value = settings.get(key)
3386 if key in valid_keys and value is not None:
3387 setattr(self, key, value)
3388 else:
3389 g.trace(f"Unexpected setting: {key} = {value!r}")
3390 #@+node:ekr.20200107165250.51: *4* orange.push_state
3391 def push_state(self, kind: str, value: Any=None) -> None:
3392 """Append a state to the state stack."""
3393 state = ParseState(kind, value)
3394 self.state_stack.append(state)
3395 #@+node:ekr.20200107165250.8: *4* orange: Entries
3396 #@+node:ekr.20200107173542.1: *5* orange.beautify (main token loop)
3397 def oops(self) -> None: # pragma: no cover
3398 g.trace(f"Unknown kind: {self.kind}")
3400 def beautify(self, contents: str, filename: str, tokens: List["Token"], tree: Node,
3402 max_join_line_length: Optional[int]=None, max_split_line_length: Optional[int]=None,
3403 ) -> str:
3404 """
3405 The main line. Create output tokens and return the result as a string.
3406 """
3407 # Config overrides
3408 if max_join_line_length is not None:
3409 self.max_join_line_length = max_join_line_length
3410 if max_split_line_length is not None:
3411 self.max_split_line_length = max_split_line_length
3412 # State vars...
3413 self.curly_brackets_level = 0 # Number of unmatched '{' tokens.
3414 self.decorator_seen = False # Set by do_name for do_op.
3415 self.in_arg_list = 0 # > 0 if in an arg list of a def.
3416 self.level = 0 # Set only by do_indent and do_dedent.
3417 self.lws = '' # Leading whitespace.
3418 self.paren_level = 0 # Number of unmatched '(' tokens.
3419 self.square_brackets_stack: List[bool] = [] # A stack of bools, for self.word().
3420 self.state_stack: List["ParseState"] = [] # Stack of ParseState objects.
3421 self.val = None # The input token's value (a string).
3422 self.verbatim = False # True: don't beautify.
3423 #
3424 # Init output list and state...
3425 self.code_list: List[Token] = [] # The list of output tokens.
3426 self.code_list_index = 0 # The token's index.
3427 self.tokens = tokens # The list of input tokens.
3428 self.tree = tree
3429 self.add_token('file-start', '')
3430 self.push_state('file-start')
3431 for i, token in enumerate(tokens):
3432 self.token = token
3433 self.kind, self.val, self.line = token.kind, token.value, token.line
3434 if self.verbatim:
3435 self.do_verbatim()
3436 else:
3437 func = getattr(self, f"do_{token.kind}", self.oops)
3438 func()
3439 # Any post pass would go here.
3440 return tokens_to_string(self.code_list)
3441 #@+node:ekr.20200107172450.1: *5* orange.beautify_file (entry)
3442 def beautify_file(self, filename: str) -> bool: # pragma: no cover
3443 """
3444 Orange: Beautify the the given external file.
3446 Return True if the file was changed.
3447 """
3448 self.filename = filename
3449 tog = TokenOrderGenerator()
3450 contents, encoding, tokens, tree = tog.init_from_file(filename)
3451 if not contents or not tokens or not tree:
3452 return False # #2529: Not an error.
3453 # Beautify.
3454 results = self.beautify(contents, filename, tokens, tree)
3455 # Something besides newlines must change.
3456 if regularize_nls(contents) == regularize_nls(results):
3457 return False
3458 if 0: # This obscures more import error messages.
3459 show_diffs(contents, results, filename=filename)
3460 # Write the results
3461 print(f"Beautified: {g.shortFileName(filename)}")
3462 write_file(filename, results, encoding=encoding)
3463 return True
3464 #@+node:ekr.20200107172512.1: *5* orange.beautify_file_diff (entry)
3465 def beautify_file_diff(self, filename: str) -> bool: # pragma: no cover
3466 """
3467 Orange: Print the diffs that would resulf from the orange-file command.
3469 Return True if the file would be changed.
3470 """
3471 tag = 'diff-beautify-file'
3472 self.filename = filename
3473 tog = TokenOrderGenerator()
3474 contents, encoding, tokens, tree = tog.init_from_file(filename)
3475 if not contents or not tokens or not tree:
3476 print(f"{tag}: Can not beautify: {filename}")
3477 return False
3478 # fstringify.
3479 results = self.beautify(contents, filename, tokens, tree)
3480 # Something besides newlines must change.
3481 if regularize_nls(contents) == regularize_nls(results):
3482 print(f"{tag}: Unchanged: {filename}")
3483 return False
3484 # Show the diffs.
3485 show_diffs(contents, results, filename=filename)
3486 return True
3487 #@+node:ekr.20200107165250.13: *4* orange: Input token handlers
3488 #@+node:ekr.20200107165250.14: *5* orange.do_comment
3489 in_doc_part = False
3491 def do_comment(self) -> None:
3492 """Handle a comment token."""
3493 val = self.val
3494 #
3495 # Leo-specific code...
3496 if self.node_pat.match(val):
3497 # Clear per-node state.
3498 self.in_doc_part = False
3499 self.verbatim = False
3500 self.decorator_seen = False
3501 # Do *not clear other state, which may persist across @others.
3502 # self.curly_brackets_level = 0
3503 # self.in_arg_list = 0
3504 # self.level = 0
3505 # self.lws = ''
3506 # self.paren_level = 0
3507 # self.square_brackets_stack = []
3508 # self.state_stack = []
3509 else:
3510 # Keep track of verbatim mode.
3511 if self.beautify_pat.match(val):
3512 self.verbatim = False
3513 elif self.nobeautify_pat.match(val):
3514 self.verbatim = True
3515 # Keep trace of @doc parts, to honor the convention for splitting lines.
3516 if self.start_doc_pat.match(val):
3517 self.in_doc_part = True
3518 if self.end_doc_pat.match(val):
3519 self.in_doc_part = False
3520 #
3521 # General code: Generate the comment.
3522 self.clean('blank')
3523 entire_line = self.line.lstrip().startswith('#')
3524 if entire_line:
3525 self.clean('hard-blank')
3526 self.clean('line-indent')
3527 # #1496: No further munging needed.
3528 val = self.line.rstrip()
3529 else:
3530 # Exactly two spaces before trailing comments.
3531 val = ' ' + self.val.rstrip()
3532 self.add_token('comment', val)
3533 #@+node:ekr.20200107165250.15: *5* orange.do_encoding
3534 def do_encoding(self) -> None:
3535 """
3536 Handle the encoding token.
3537 """
3538 pass
3539 #@+node:ekr.20200107165250.16: *5* orange.do_endmarker
3540 def do_endmarker(self) -> None:
3541 """Handle an endmarker token."""
3542 # Ensure exactly one blank at the end of the file.
3543 self.clean_blank_lines()
3544 self.add_token('line-end', '\n')
3545 #@+node:ekr.20200107165250.18: *5* orange.do_indent & do_dedent & helper
3546 def do_dedent(self) -> None:
3547 """Handle dedent token."""
3548 self.level -= 1
3549 self.lws = self.level * self.tab_width * ' '
3550 self.line_indent()
3551 if self.black_mode: # pragma: no cover (black)
3552 state = self.state_stack[-1]
3553 if state.kind == 'indent' and state.value == self.level:
3554 self.state_stack.pop()
3555 state = self.state_stack[-1]
3556 if state.kind in ('class', 'def'):
3557 self.state_stack.pop()
3558 self.handle_dedent_after_class_or_def(state.kind)
3560 def do_indent(self) -> None:
3561 """Handle indent token."""
3562 new_indent = self.val
3563 old_indent = self.level * self.tab_width * ' '
3564 if new_indent > old_indent:
3565 self.level += 1
3566 elif new_indent < old_indent: # pragma: no cover (defensive)
3567 g.trace('\n===== can not happen', repr(new_indent), repr(old_indent))
3568 self.lws = new_indent
3569 self.line_indent()
3570 #@+node:ekr.20200220054928.1: *6* orange.handle_dedent_after_class_or_def
3571 def handle_dedent_after_class_or_def(self, kind: str) -> None: # pragma: no cover (black)
3572 """
3573 Insert blank lines after a class or def as the result of a 'dedent' token.
3575 Normal comment lines may precede the 'dedent'.
3576 Insert the blank lines *before* such comment lines.
3577 """
3578 #
3579 # Compute the tail.
3580 i = len(self.code_list) - 1
3581 tail: List[Token] = []
3582 while i > 0:
3583 t = self.code_list.pop()
3584 i -= 1
3585 if t.kind == 'line-indent':
3586 pass
3587 elif t.kind == 'line-end':
3588 tail.insert(0, t)
3589 elif t.kind == 'comment':
3590 # Only underindented single-line comments belong in the tail.
3591 # @+node comments must never be in the tail.
3592 single_line = self.code_list[i].kind in ('line-end', 'line-indent')
3593 lws = len(t.value) - len(t.value.lstrip())
3594 underindent = lws <= len(self.lws)
3595 if underindent and single_line and not self.node_pat.match(t.value):
3596 # A single-line comment.
3597 tail.insert(0, t)
3598 else:
3599 self.code_list.append(t)
3600 break
3601 else:
3602 self.code_list.append(t)
3603 break
3604 #
3605 # Remove leading 'line-end' tokens from the tail.
3606 while tail and tail[0].kind == 'line-end':
3607 tail = tail[1:]
3608 #
3609 # Put the newlines *before* the tail.
3610 # For Leo, always use 1 blank lines.
3611 n = 1 # n = 2 if kind == 'class' else 1
3612 # Retain the token (intention) for debugging.
3613 self.add_token('blank-lines', n)
3614 for i in range(0, n + 1):
3615 self.add_token('line-end', '\n')
3616 if tail:
3617 self.code_list.extend(tail)
3618 self.line_indent()
3619 #@+node:ekr.20200107165250.20: *5* orange.do_name
3620 def do_name(self) -> None:
3621 """Handle a name token."""
3622 name = self.val
3623 if self.black_mode and name in ('class', 'def'): # pragma: no cover (black)
3624 # Handle newlines before and after 'class' or 'def'
3625 self.decorator_seen = False
3626 state = self.state_stack[-1]
3627 if state.kind == 'decorator':
3628 # Always do this, regardless of @bool clean-blank-lines.
3629 self.clean_blank_lines()
3630 # Suppress split/join.
3631 self.add_token('hard-newline', '\n')
3632 self.add_token('line-indent', self.lws)
3633 self.state_stack.pop()
3634 else:
3635 # Always do this, regardless of @bool clean-blank-lines.
3636 self.blank_lines(2 if name == 'class' else 1)
3637 self.push_state(name)
3638 self.push_state('indent', self.level)
3639 # For trailing lines after inner classes/defs.
3640 self.word(name)
3641 return
3642 #
3643 # Leo mode...
3644 if name in ('class', 'def'):
3645 self.word(name)
3646 elif name in (
3647 'and', 'elif', 'else', 'for', 'if', 'in', 'not', 'not in', 'or', 'while'
3648 ):
3649 self.word_op(name)
3650 else:
3651 self.word(name)
3652 #@+node:ekr.20200107165250.21: *5* orange.do_newline & do_nl
3653 def do_newline(self) -> None:
3654 """Handle a regular newline."""
3655 self.line_end()
3657 def do_nl(self) -> None:
3658 """Handle a continuation line."""
3659 self.line_end()
3660 #@+node:ekr.20200107165250.22: *5* orange.do_number
3661 def do_number(self) -> None:
3662 """Handle a number token."""
3663 self.blank()
3664 self.add_token('number', self.val)
3665 #@+node:ekr.20200107165250.23: *5* orange.do_op
3666 def do_op(self) -> None:
3667 """Handle an op token."""
3668 val = self.val
3669 if val == '.':
3670 self.clean('blank')
3671 prev = self.code_list[-1]
3672 # #2495 & #2533: Special case for 'from .'
3673 if prev.kind == 'word' and prev.value == 'from':
3674 self.blank()
3675 self.add_token('op-no-blanks', val)
3676 elif val == '@':
3677 if self.black_mode: # pragma: no cover (black)
3678 if not self.decorator_seen:
3679 self.blank_lines(1)
3680 self.decorator_seen = True
3681 self.clean('blank')
3682 self.add_token('op-no-blanks', val)
3683 self.push_state('decorator')
3684 elif val == ':':
3685 # Treat slices differently.
3686 self.colon(val)
3687 elif val in ',;':
3688 # Pep 8: Avoid extraneous whitespace immediately before
3689 # comma, semicolon, or colon.
3690 self.clean('blank')
3691 self.add_token('op', val)
3692 self.blank()
3693 elif val in '([{':
3694 # Pep 8: Avoid extraneous whitespace immediately inside
3695 # parentheses, brackets or braces.
3696 self.lt(val)
3697 elif val in ')]}':
3698 # Ditto.
3699 self.rt(val)
3700 elif val == '=':
3701 # Pep 8: Don't use spaces around the = sign when used to indicate
3702 # a keyword argument or a default parameter value.
3703 if self.paren_level:
3704 self.clean('blank')
3705 self.add_token('op-no-blanks', val)
3706 else:
3707 self.blank()
3708 self.add_token('op', val)
3709 self.blank()
3710 elif val in '~+-':
3711 self.possible_unary_op(val)
3712 elif val == '*':
3713 self.star_op()
3714 elif val == '**':
3715 self.star_star_op()
3716 else:
3717 # Pep 8: always surround binary operators with a single space.
3718 # '==','+=','-=','*=','**=','/=','//=','%=','!=','<=','>=','<','>',
3719 # '^','~','*','**','&','|','/','//',
3720 # Pep 8: If operators with different priorities are used,
3721 # consider adding whitespace around the operators with the lowest priority(ies).
3722 self.blank()
3723 self.add_token('op', val)
3724 self.blank()
3725 #@+node:ekr.20200107165250.24: *5* orange.do_string
3726 def do_string(self) -> None:
3727 """Handle a 'string' token."""
3728 # Careful: continued strings may contain '\r'
3729 val = regularize_nls(self.val)
3730 self.add_token('string', val)
3731 self.blank()
3732 #@+node:ekr.20200210175117.1: *5* orange.do_verbatim
3733 beautify_pat = re.compile(
3734 r'#\s*pragma:\s*beautify\b|#\s*@@beautify|#\s*@\+node|#\s*@[+-]others|#\s*@[+-]<<')
3736 def do_verbatim(self) -> None:
3737 """
3738 Handle one token in verbatim mode.
3739 End verbatim mode when the appropriate comment is seen.
3740 """
3741 kind = self.kind
3742 #
3743 # Careful: tokens may contain '\r'
3744 val = regularize_nls(self.val)
3745 if kind == 'comment':
3746 if self.beautify_pat.match(val):
3747 self.verbatim = False
3748 val = val.rstrip()
3749 self.add_token('comment', val)
3750 return
3751 if kind == 'indent':
3752 self.level += 1
3753 self.lws = self.level * self.tab_width * ' '
3754 if kind == 'dedent':
3755 self.level -= 1
3756 self.lws = self.level * self.tab_width * ' '
3757 self.add_token('verbatim', val)
3758 #@+node:ekr.20200107165250.25: *5* orange.do_ws
3759 def do_ws(self) -> None:
3760 """
3761 Handle the "ws" pseudo-token.
3763 Put the whitespace only if if ends with backslash-newline.
3764 """
3765 val = self.val
3766 # Handle backslash-newline.
3767 if '\\\n' in val:
3768 self.clean('blank')
3769 self.add_token('op-no-blanks', val)
3770 return
3771 # Handle start-of-line whitespace.
3772 prev = self.code_list[-1]
3773 inner = self.paren_level or self.square_brackets_stack or self.curly_brackets_level
3774 if prev.kind == 'line-indent' and inner:
3775 # Retain the indent that won't be cleaned away.
3776 self.clean('line-indent')
3777 self.add_token('hard-blank', val)
3778 #@+node:ekr.20200107165250.26: *4* orange: Output token generators
3779 #@+node:ekr.20200118145044.1: *5* orange.add_line_end
3780 def add_line_end(self) -> "Token":
3781 """Add a line-end request to the code list."""
3782 # This may be called from do_name as well as do_newline and do_nl.
3783 assert self.token.kind in ('newline', 'nl'), self.token.kind
3784 self.clean('blank') # Important!
3785 self.clean('line-indent')
3786 t = self.add_token('line-end', '\n')
3787 # Distinguish between kinds of 'line-end' tokens.
3788 t.newline_kind = self.token.kind
3789 return t
3790 #@+node:ekr.20200107170523.1: *5* orange.add_token
3791 def add_token(self, kind: str, value: Any) -> "Token":
3792 """Add an output token to the code list."""
3793 tok = Token(kind, value)
3794 tok.index = self.code_list_index # For debugging only.
3795 self.code_list_index += 1
3796 self.code_list.append(tok)
3797 return tok
3798 #@+node:ekr.20200107165250.27: *5* orange.blank
3799 def blank(self) -> None:
3800 """Add a blank request to the code list."""
3801 prev = self.code_list[-1]
3802 if prev.kind not in (
3803 'blank',
3804 'blank-lines',
3805 'file-start',
3806 'hard-blank', # Unique to orange.
3807 'line-end',
3808 'line-indent',
3809 'lt',
3810 'op-no-blanks',
3811 'unary-op',
3812 ):
3813 self.add_token('blank', ' ')
3814 #@+node:ekr.20200107165250.29: *5* orange.blank_lines (black only)
3815 def blank_lines(self, n: int) -> None: # pragma: no cover (black)
3816 """
3817 Add a request for n blank lines to the code list.
3818 Multiple blank-lines request yield at least the maximum of all requests.
3819 """
3820 self.clean_blank_lines()
3821 prev = self.code_list[-1]
3822 if prev.kind == 'file-start':
3823 self.add_token('blank-lines', n)
3824 return
3825 for i in range(0, n + 1):
3826 self.add_token('line-end', '\n')
3827 # Retain the token (intention) for debugging.
3828 self.add_token('blank-lines', n)
3829 self.line_indent()
3830 #@+node:ekr.20200107165250.30: *5* orange.clean
3831 def clean(self, kind: str) -> None:
3832 """Remove the last item of token list if it has the given kind."""
3833 prev = self.code_list[-1]
3834 if prev.kind == kind:
3835 self.code_list.pop()
3836 #@+node:ekr.20200107165250.31: *5* orange.clean_blank_lines
3837 def clean_blank_lines(self) -> bool:
3838 """
3839 Remove all vestiges of previous blank lines.
3841 Return True if any of the cleaned 'line-end' tokens represented "hard" newlines.
3842 """
3843 cleaned_newline = False
3844 table = ('blank-lines', 'line-end', 'line-indent')
3845 while self.code_list[-1].kind in table:
3846 t = self.code_list.pop()
3847 if t.kind == 'line-end' and getattr(t, 'newline_kind', None) != 'nl':
3848 cleaned_newline = True
3849 return cleaned_newline
3850 #@+node:ekr.20200107165250.32: *5* orange.colon
3851 def colon(self, val: str) -> None:
3852 """Handle a colon."""
3854 def is_expr(node: Node) -> bool:
3855 """True if node is any expression other than += number."""
3856 if isinstance(node, (ast.BinOp, ast.Call, ast.IfExp)):
3857 return True
3858 return (
3859 isinstance(node, ast.UnaryOp)
3860 and not isinstance(node.operand, ast.Num)
3861 )
3863 node = self.token.node
3864 self.clean('blank')
3865 if not isinstance(node, ast.Slice):
3866 self.add_token('op', val)
3867 self.blank()
3868 return
3869 # A slice.
3870 lower = getattr(node, 'lower', None)
3871 upper = getattr(node, 'upper', None)
3872 step = getattr(node, 'step', None)
3873 if any(is_expr(z) for z in (lower, upper, step)):
3874 prev = self.code_list[-1]
3875 if prev.value not in '[:':
3876 self.blank()
3877 self.add_token('op', val)
3878 self.blank()
3879 else:
3880 self.add_token('op-no-blanks', val)
3881 #@+node:ekr.20200107165250.33: *5* orange.line_end
3882 def line_end(self) -> None:
3883 """Add a line-end request to the code list."""
3884 # This should be called only be do_newline and do_nl.
3885 node, token = self.token.statement_node, self.token
3886 assert token.kind in ('newline', 'nl'), (token.kind, g.callers())
3887 # Create the 'line-end' output token.
3888 self.add_line_end()
3889 # Attempt to split the line.
3890 was_split = self.split_line(node, token)
3891 # Attempt to join the line only if it has not just been split.
3892 if not was_split and self.max_join_line_length > 0:
3893 self.join_lines(node, token)
3894 self.line_indent()
3895 # Add the indentation for all lines
3896 # until the next indent or unindent token.
3897 #@+node:ekr.20200107165250.40: *5* orange.line_indent
3898 def line_indent(self) -> None:
3899 """Add a line-indent token."""
3900 self.clean('line-indent')
3901 # Defensive. Should never happen.
3902 self.add_token('line-indent', self.lws)
3903 #@+node:ekr.20200107165250.41: *5* orange.lt & rt
3904 #@+node:ekr.20200107165250.42: *6* orange.lt
3905 def lt(self, val: str) -> None:
3906 """Generate code for a left paren or curly/square bracket."""
3907 assert val in '([{', repr(val)
3908 if val == '(':
3909 self.paren_level += 1
3910 elif val == '[':
3911 self.square_brackets_stack.append(False)
3912 else:
3913 self.curly_brackets_level += 1
3914 self.clean('blank')
3915 prev = self.code_list[-1]
3916 if prev.kind in ('op', 'word-op'):
3917 self.blank()
3918 self.add_token('lt', val)
3919 elif prev.kind == 'word':
3920 # Only suppress blanks before '(' or '[' for non-keyworks.
3921 if val == '{' or prev.value in ('if', 'else', 'return', 'for'):
3922 self.blank()
3923 elif val == '(':
3924 self.in_arg_list += 1
3925 self.add_token('lt', val)
3926 else:
3927 self.clean('blank')
3928 self.add_token('op-no-blanks', val)
3929 #@+node:ekr.20200107165250.43: *6* orange.rt
3930 def rt(self, val: str) -> None:
3931 """Generate code for a right paren or curly/square bracket."""
3932 assert val in ')]}', repr(val)
3933 if val == ')':
3934 self.paren_level -= 1
3935 self.in_arg_list = max(0, self.in_arg_list - 1)
3936 elif val == ']':
3937 self.square_brackets_stack.pop()
3938 else:
3939 self.curly_brackets_level -= 1
3940 self.clean('blank')
3941 self.add_token('rt', val)
3942 #@+node:ekr.20200107165250.45: *5* orange.possible_unary_op & unary_op
3943 def possible_unary_op(self, s: str) -> None:
3944 """Add a unary or binary op to the token list."""
3945 node = self.token.node
3946 self.clean('blank')
3947 if isinstance(node, ast.UnaryOp):
3948 self.unary_op(s)
3949 else:
3950 self.blank()
3951 self.add_token('op', s)
3952 self.blank()
3954 def unary_op(self, s: str) -> None:
3955 """Add an operator request to the code list."""
3956 assert s and isinstance(s, str), repr(s)
3957 self.clean('blank')
3958 prev = self.code_list[-1]
3959 if prev.kind == 'lt':
3960 self.add_token('unary-op', s)
3961 else:
3962 self.blank()
3963 self.add_token('unary-op', s)
3964 #@+node:ekr.20200107165250.46: *5* orange.star_op
3965 def star_op(self) -> None:
3966 """Put a '*' op, with special cases for *args."""
3967 val = '*'
3968 node = self.token.node
3969 self.clean('blank')
3970 if isinstance(node, ast.arguments):
3971 self.blank()
3972 self.add_token('op', val)
3973 return # #2533
3974 if self.paren_level > 0:
3975 prev = self.code_list[-1]
3976 if prev.kind == 'lt' or (prev.kind, prev.value) == ('op', ','):
3977 self.blank()
3978 self.add_token('op', val)
3979 return
3980 self.blank()
3981 self.add_token('op', val)
3982 self.blank()
3983 #@+node:ekr.20200107165250.47: *5* orange.star_star_op
3984 def star_star_op(self) -> None:
3985 """Put a ** operator, with a special case for **kwargs."""
3986 val = '**'
3987 node = self.token.node
3988 self.clean('blank')
3989 if isinstance(node, ast.arguments):
3990 self.blank()
3991 self.add_token('op', val)
3992 return # #2533
3993 if self.paren_level > 0:
3994 prev = self.code_list[-1]
3995 if prev.kind == 'lt' or (prev.kind, prev.value) == ('op', ','):
3996 self.blank()
3997 self.add_token('op', val)
3998 return
3999 self.blank()
4000 self.add_token('op', val)
4001 self.blank()
4002 #@+node:ekr.20200107165250.48: *5* orange.word & word_op
4003 def word(self, s: str) -> None:
4004 """Add a word request to the code list."""
4005 assert s and isinstance(s, str), repr(s)
4006 node = self.token.node
4007 if isinstance(node, ast.ImportFrom) and s == 'import': # #2533
4008 self.clean('blank')
4009 self.add_token('blank', ' ')
4010 self.add_token('word', s)
4011 elif self.square_brackets_stack:
4012 # A previous 'op-no-blanks' token may cancel this blank.
4013 self.blank()
4014 self.add_token('word', s)
4015 elif self.in_arg_list > 0:
4016 self.add_token('word', s)
4017 self.blank()
4018 else:
4019 self.blank()
4020 self.add_token('word', s)
4021 self.blank()
4023 def word_op(self, s: str) -> None:
4024 """Add a word-op request to the code list."""
4025 assert s and isinstance(s, str), repr(s)
4026 self.blank()
4027 self.add_token('word-op', s)
4028 self.blank()
4029 #@+node:ekr.20200118120049.1: *4* orange: Split/join
4030 #@+node:ekr.20200107165250.34: *5* orange.split_line & helpers
4031 def split_line(self, node: Node, token: "Token") -> bool:
4032 """
4033 Split token's line, if possible and enabled.
4035 Return True if the line was broken into two or more lines.
4036 """
4037 assert token.kind in ('newline', 'nl'), repr(token)
4038 # Return if splitting is disabled:
4039 if self.max_split_line_length <= 0: # pragma: no cover (user option)
4040 return False
4041 # Return if the node can't be split.
4042 if not is_long_statement(node):
4043 return False
4044 # Find the *output* tokens of the previous lines.
4045 line_tokens = self.find_prev_line()
4046 line_s = ''.join([z.to_string() for z in line_tokens])
4047 # Do nothing for short lines.
4048 if len(line_s) < self.max_split_line_length:
4049 return False
4050 # Return if the previous line has no opening delim: (, [ or {.
4051 if not any(z.kind == 'lt' for z in line_tokens): # pragma: no cover (defensive)
4052 return False
4053 prefix = self.find_line_prefix(line_tokens)
4054 # Calculate the tail before cleaning the prefix.
4055 tail = line_tokens[len(prefix) :]
4056 # Cut back the token list: subtract 1 for the trailing line-end.
4057 self.code_list = self.code_list[: len(self.code_list) - len(line_tokens) - 1]
4058 # Append the tail, splitting it further, as needed.
4059 self.append_tail(prefix, tail)
4060 # Add the line-end token deleted by find_line_prefix.
4061 self.add_token('line-end', '\n')
4062 return True
4063 #@+node:ekr.20200107165250.35: *6* orange.append_tail
4064 def append_tail(self, prefix: List["Token"], tail: List["Token"]) -> None:
4065 """Append the tail tokens, splitting the line further as necessary."""
4066 tail_s = ''.join([z.to_string() for z in tail])
4067 if len(tail_s) < self.max_split_line_length:
4068 # Add the prefix.
4069 self.code_list.extend(prefix)
4070 # Start a new line and increase the indentation.
4071 self.add_token('line-end', '\n')
4072 self.add_token('line-indent', self.lws + ' ' * 4)
4073 self.code_list.extend(tail)
4074 return
4075 # Still too long. Split the line at commas.
4076 self.code_list.extend(prefix)
4077 # Start a new line and increase the indentation.
4078 self.add_token('line-end', '\n')
4079 self.add_token('line-indent', self.lws + ' ' * 4)
4080 open_delim = Token(kind='lt', value=prefix[-1].value)
4081 value = open_delim.value.replace('(', ')').replace('[', ']').replace('{', '}')
4082 close_delim = Token(kind='rt', value=value)
4083 delim_count = 1
4084 lws = self.lws + ' ' * 4
4085 for i, t in enumerate(tail):
4086 if t.kind == 'op' and t.value == ',':
4087 if delim_count == 1:
4088 # Start a new line.
4089 self.add_token('op-no-blanks', ',')
4090 self.add_token('line-end', '\n')
4091 self.add_token('line-indent', lws)
4092 # Kill a following blank.
4093 if i + 1 < len(tail):
4094 next_t = tail[i + 1]
4095 if next_t.kind == 'blank':
4096 next_t.kind = 'no-op'
4097 next_t.value = ''
4098 else:
4099 self.code_list.append(t)
4100 elif t.kind == close_delim.kind and t.value == close_delim.value:
4101 # Done if the delims match.
4102 delim_count -= 1
4103 if delim_count == 0:
4104 # Start a new line
4105 self.add_token('op-no-blanks', ',')
4106 self.add_token('line-end', '\n')
4107 self.add_token('line-indent', self.lws)
4108 self.code_list.extend(tail[i:])
4109 return
4110 lws = lws[:-4]
4111 self.code_list.append(t)
4112 elif t.kind == open_delim.kind and t.value == open_delim.value:
4113 delim_count += 1
4114 lws = lws + ' ' * 4
4115 self.code_list.append(t)
4116 else:
4117 self.code_list.append(t)
4118 g.trace('BAD DELIMS', delim_count) # pragma: no cover
4119 #@+node:ekr.20200107165250.36: *6* orange.find_prev_line
4120 def find_prev_line(self) -> List["Token"]:
4121 """Return the previous line, as a list of tokens."""
4122 line = []
4123 for t in reversed(self.code_list[:-1]):
4124 if t.kind in ('hard-newline', 'line-end'):
4125 break
4126 line.append(t)
4127 return list(reversed(line))
4128 #@+node:ekr.20200107165250.37: *6* orange.find_line_prefix
4129 def find_line_prefix(self, token_list: List["Token"]) -> List["Token"]:
4130 """
4131 Return all tokens up to and including the first lt token.
4132 Also add all lt tokens directly following the first lt token.
4133 """
4134 result = []
4135 for i, t in enumerate(token_list):
4136 result.append(t)
4137 if t.kind == 'lt':
4138 break
4139 return result
4140 #@+node:ekr.20200107165250.39: *5* orange.join_lines
4141 def join_lines(self, node: Node, token: "Token") -> None:
4142 """
4143 Join preceding lines, if possible and enabled.
4144 token is a line_end token. node is the corresponding ast node.
4145 """
4146 if self.max_join_line_length <= 0: # pragma: no cover (user option)
4147 return
4148 assert token.kind in ('newline', 'nl'), repr(token)
4149 if token.kind == 'nl':
4150 return
4151 # Scan backward in the *code* list,
4152 # looking for 'line-end' tokens with tok.newline_kind == 'nl'
4153 nls = 0
4154 i = len(self.code_list) - 1
4155 t = self.code_list[i]
4156 assert t.kind == 'line-end', repr(t)
4157 # Not all tokens have a newline_kind ivar.
4158 assert t.newline_kind == 'newline'
4159 i -= 1
4160 while i >= 0:
4161 t = self.code_list[i]
4162 if t.kind == 'comment':
4163 # Can't join.
4164 return
4165 if t.kind == 'string' and not self.allow_joined_strings:
4166 # An EKR preference: don't join strings, no matter what black does.
4167 # This allows "short" f-strings to be aligned.
4168 return
4169 if t.kind == 'line-end':
4170 if getattr(t, 'newline_kind', None) == 'nl':
4171 nls += 1
4172 else:
4173 break # pragma: no cover
4174 i -= 1
4175 # Retain at the file-start token.
4176 if i <= 0:
4177 i = 1
4178 if nls <= 0: # pragma: no cover (rare)
4179 return
4180 # Retain line-end and and any following line-indent.
4181 # Required, so that the regex below won't eat too much.
4182 while True:
4183 t = self.code_list[i]
4184 if t.kind == 'line-end':
4185 if getattr(t, 'newline_kind', None) == 'nl': # pragma: no cover (rare)
4186 nls -= 1
4187 i += 1
4188 elif self.code_list[i].kind == 'line-indent':
4189 i += 1
4190 else:
4191 break # pragma: no cover (defensive)
4192 if nls <= 0: # pragma: no cover (defensive)
4193 return
4194 # Calculate the joined line.
4195 tail = self.code_list[i:]
4196 tail_s = tokens_to_string(tail)
4197 tail_s = re.sub(r'\n\s*', ' ', tail_s)
4198 tail_s = tail_s.replace('( ', '(').replace(' )', ')')
4199 tail_s = tail_s.rstrip()
4200 # Don't join the lines if they would be too long.
4201 if len(tail_s) > self.max_join_line_length: # pragma: no cover (defensive)
4202 return
4203 # Cut back the code list.
4204 self.code_list = self.code_list[:i]
4205 # Add the new output tokens.
4206 self.add_token('string', tail_s)
4207 self.add_token('line-end', '\n')
4208 #@-others
4209#@+node:ekr.20200107170847.1: *3* class OrangeSettings
4210class OrangeSettings:
4212 pass
4213#@+node:ekr.20200107170126.1: *3* class ParseState
4214class ParseState:
4215 """
4216 A class representing items in the parse state stack.
4218 The present states:
4220 'file-start': Ensures the stack stack is never empty.
4222 'decorator': The last '@' was a decorator.
4224 do_op(): push_state('decorator')
4225 do_name(): pops the stack if state.kind == 'decorator'.
4227 'indent': The indentation level for 'class' and 'def' names.
4229 do_name(): push_state('indent', self.level)
4230 do_dendent(): pops the stack once or twice if state.value == self.level.
4232 """
4234 def __init__(self, kind: str, value: str) -> None:
4235 self.kind = kind
4236 self.value = value
4238 def __repr__(self) -> str:
4239 return f"State: {self.kind} {self.value!r}" # pragma: no cover
4241 __str__ = __repr__
4242#@+node:ekr.20191231084514.1: *3* class ReassignTokens
4243class ReassignTokens:
4244 """A class that reassigns tokens to more appropriate ast nodes."""
4245 #@+others
4246 #@+node:ekr.20191231084640.1: *4* reassign.reassign
4247 def reassign(self, filename: str, tokens: List["Token"], tree: Node) -> None:
4248 """The main entry point."""
4249 self.filename = filename
4250 self.tokens = tokens
4251 # For now, just handle Call nodes.
4252 for node in ast.walk(tree):
4253 if isinstance(node, ast.Call):
4254 self.visit_call(node)
4255 #@+node:ekr.20191231084853.1: *4* reassign.visit_call
4256 def visit_call(self, node: Node) -> None:
4257 """ReassignTokens.visit_call"""
4258 tokens = tokens_for_node(self.filename, node, self.tokens)
4259 node0, node9 = tokens[0].node, tokens[-1].node
4260 nca = nearest_common_ancestor(node0, node9)
4261 if not nca:
4262 return
4263 # Associate () with the call node.
4264 i = tokens[-1].index
4265 j = find_paren_token(i + 1, self.tokens)
4266 if j is None:
4267 return # pragma: no cover
4268 k = find_paren_token(j + 1, self.tokens)
4269 if k is None:
4270 return # pragma: no cover
4271 self.tokens[j].node = nca
4272 self.tokens[k].node = nca
4273 add_token_to_token_list(self.tokens[j], nca)
4274 add_token_to_token_list(self.tokens[k], nca)
4275 #@-others
4276#@+node:ekr.20191110080535.1: *3* class Token
4277class Token:
4278 """
4279 A class representing a 5-tuple, plus additional data.
4280 """
4282 def __init__(self, kind: str, value: str):
4284 self.kind = kind
4285 self.value = value
4286 #
4287 # Injected by Tokenizer.add_token.
4288 self.five_tuple = None
4289 self.index = 0
4290 # The entire line containing the token.
4291 # Same as five_tuple.line.
4292 self.line = ''
4293 # The line number, for errors and dumps.
4294 # Same as five_tuple.start[0]
4295 self.line_number = 0
4296 #
4297 # Injected by Tokenizer.add_token.
4298 self.level = 0
4299 self.node: Optional[Node] = None
4301 def __repr__(self) -> str: # pragma: no cover
4302 nl_kind = getattr(self, 'newline_kind', '')
4303 s = f"{self.kind:}.{self.index:<3}"
4304 return f"{s:>18}:{nl_kind:7} {self.show_val(80)}"
4306 def __str__(self) -> str: # pragma: no cover
4307 nl_kind = getattr(self, 'newline_kind', '')
4308 return f"{self.kind}.{self.index:<3}{nl_kind:8} {self.show_val(80)}"
4310 def to_string(self) -> str:
4311 """Return the contribution of the token to the source file."""
4312 return self.value if isinstance(self.value, str) else ''
4313 #@+others
4314 #@+node:ekr.20191231114927.1: *4* token.brief_dump
4315 def brief_dump(self) -> str: # pragma: no cover
4316 """Dump a token."""
4317 return (
4318 f"{self.index:>3} line: {self.line_number:<2} "
4319 f"{self.kind:>11} {self.show_val(100)}")
4320 #@+node:ekr.20200223022950.11: *4* token.dump
4321 def dump(self) -> str: # pragma: no cover
4322 """Dump a token and related links."""
4323 # Let block.
4324 node_id = self.node.node_index if self.node else ''
4325 node_cn = self.node.__class__.__name__ if self.node else ''
4326 return (
4327 f"{self.line_number:4} "
4328 f"{node_id:5} {node_cn:16} "
4329 f"{self.index:>5} {self.kind:>11} "
4330 f"{self.show_val(100)}")
4331 #@+node:ekr.20200121081151.1: *4* token.dump_header
4332 def dump_header(self) -> None: # pragma: no cover
4333 """Print the header for token.dump"""
4334 print(
4335 f"\n"
4336 f" node {'':10} token token\n"
4337 f"line index class {'':10} index kind value\n"
4338 f"==== ===== ===== {'':10} ===== ==== =====\n")
4339 #@+node:ekr.20191116154328.1: *4* token.error_dump
4340 def error_dump(self) -> str: # pragma: no cover
4341 """Dump a token or result node for error message."""
4342 if self.node:
4343 node_id = obj_id(self.node)
4344 node_s = f"{node_id} {self.node.__class__.__name__}"
4345 else:
4346 node_s = "None"
4347 return (
4348 f"index: {self.index:<3} {self.kind:>12} {self.show_val(20):<20} "
4349 f"{node_s}")
4350 #@+node:ekr.20191113095507.1: *4* token.show_val
4351 def show_val(self, truncate_n: int) -> str: # pragma: no cover
4352 """Return the token.value field."""
4353 if self.kind in ('ws', 'indent'):
4354 val = str(len(self.value))
4355 elif self.kind == 'string':
4356 # Important: don't add a repr for 'string' tokens.
4357 # repr just adds another layer of confusion.
4358 val = g.truncate(self.value, truncate_n)
4359 else:
4360 val = g.truncate(repr(self.value), truncate_n)
4361 return val
4362 #@-others
4363#@+node:ekr.20191110165235.1: *3* class Tokenizer
4364class Tokenizer:
4366 """Create a list of Tokens from contents."""
4368 results: List[Token] = []
4370 #@+others
4371 #@+node:ekr.20191110165235.2: *4* tokenizer.add_token
4372 token_index = 0
4373 prev_line_token = None
4375 def add_token(self, kind: str, five_tuple: Any, line: str, s_row: int, value: str) -> None:
4376 """
4377 Add a token to the results list.
4379 Subclasses could override this method to filter out specific tokens.
4380 """
4381 tok = Token(kind, value)
4382 tok.five_tuple = five_tuple
4383 tok.index = self.token_index
4384 # Bump the token index.
4385 self.token_index += 1
4386 tok.line = line
4387 tok.line_number = s_row
4388 self.results.append(tok)
4389 #@+node:ekr.20191110170551.1: *4* tokenizer.check_results
4390 def check_results(self, contents: str) -> None:
4392 # Split the results into lines.
4393 result = ''.join([z.to_string() for z in self.results])
4394 result_lines = g.splitLines(result)
4395 # Check.
4396 ok = result == contents and result_lines == self.lines
4397 assert ok, (
4398 f"\n"
4399 f" result: {result!r}\n"
4400 f" contents: {contents!r}\n"
4401 f"result_lines: {result_lines}\n"
4402 f" lines: {self.lines}"
4403 )
4404 #@+node:ekr.20191110165235.3: *4* tokenizer.create_input_tokens
4405 def create_input_tokens(self, contents: str, tokens: Generator) -> List["Token"]:
4406 """
4407 Generate a list of Token's from tokens, a list of 5-tuples.
4408 """
4409 # Create the physical lines.
4410 self.lines = contents.splitlines(True)
4411 # Create the list of character offsets of the start of each physical line.
4412 last_offset, self.offsets = 0, [0]
4413 for line in self.lines:
4414 last_offset += len(line)
4415 self.offsets.append(last_offset)
4416 # Handle each token, appending tokens and between-token whitespace to results.
4417 self.prev_offset, self.results = -1, []
4418 for token in tokens:
4419 self.do_token(contents, token)
4420 # Print results when tracing.
4421 self.check_results(contents)
4422 # Return results, as a list.
4423 return self.results
4424 #@+node:ekr.20191110165235.4: *4* tokenizer.do_token (the gem)
4425 header_has_been_shown = False
4427 def do_token(self, contents: str, five_tuple: Any) -> None:
4428 """
4429 Handle the given token, optionally including between-token whitespace.
4431 This is part of the "gem".
4433 Links:
4435 - 11/13/19: ENB: A much better untokenizer
4436 https://groups.google.com/forum/#!msg/leo-editor/DpZ2cMS03WE/VPqtB9lTEAAJ
4438 - Untokenize does not round-trip ws before bs-nl
4439 https://bugs.python.org/issue38663
4440 """
4441 import token as token_module
4442 # Unpack..
4443 tok_type, val, start, end, line = five_tuple
4444 s_row, s_col = start # row/col offsets of start of token.
4445 e_row, e_col = end # row/col offsets of end of token.
4446 kind = token_module.tok_name[tok_type].lower()
4447 # Calculate the token's start/end offsets: character offsets into contents.
4448 s_offset = self.offsets[max(0, s_row - 1)] + s_col
4449 e_offset = self.offsets[max(0, e_row - 1)] + e_col
4450 # tok_s is corresponding string in the line.
4451 tok_s = contents[s_offset:e_offset]
4452 # Add any preceding between-token whitespace.
4453 ws = contents[self.prev_offset:s_offset]
4454 if ws:
4455 # No need for a hook.
4456 self.add_token('ws', five_tuple, line, s_row, ws)
4457 # Always add token, even if it contributes no text!
4458 self.add_token(kind, five_tuple, line, s_row, tok_s)
4459 # Update the ending offset.
4460 self.prev_offset = e_offset
4461 #@-others
4462#@+node:ekr.20191113063144.1: *3* class TokenOrderGenerator
4463class TokenOrderGenerator:
4464 """
4465 A class that traverses ast (parse) trees in token order.
4467 Overview: https://github.com/leo-editor/leo-editor/issues/1440#issue-522090981
4469 Theory of operation:
4470 - https://github.com/leo-editor/leo-editor/issues/1440#issuecomment-573661883
4471 - http://leoeditor.com/appendices.html#tokenorder-classes-theory-of-operation
4473 How to: http://leoeditor.com/appendices.html#tokenorder-class-how-to
4475 Project history: https://github.com/leo-editor/leo-editor/issues/1440#issuecomment-574145510
4476 """
4478 begin_end_stack: List[str] = []
4479 n_nodes = 0 # The number of nodes that have been visited.
4480 node_index = 0 # The index into the node_stack.
4481 node_stack: List[ast.AST] = [] # The stack of parent nodes.
4483 #@+others
4484 #@+node:ekr.20200103174914.1: *4* tog: Init...
4485 #@+node:ekr.20191228184647.1: *5* tog.balance_tokens
4486 def balance_tokens(self, tokens: List["Token"]) -> int:
4487 """
4488 TOG.balance_tokens.
4490 Insert two-way links between matching paren tokens.
4491 """
4492 count, stack = 0, []
4493 for token in tokens:
4494 if token.kind == 'op':
4495 if token.value == '(':
4496 count += 1
4497 stack.append(token.index)
4498 if token.value == ')':
4499 if stack:
4500 index = stack.pop()
4501 tokens[index].matching_paren = token.index
4502 tokens[token.index].matching_paren = index
4503 else: # pragma: no cover
4504 g.trace(f"unmatched ')' at index {token.index}")
4505 if stack: # pragma: no cover
4506 g.trace("unmatched '(' at {','.join(stack)}")
4507 return count
4508 #@+node:ekr.20191113063144.4: *5* tog.create_links
4509 def create_links(self, tokens: List["Token"], tree: Node, file_name: str='') -> List:
4510 """
4511 A generator creates two-way links between the given tokens and ast-tree.
4513 Callers should call this generator with list(tog.create_links(...))
4515 The sync_tokens method creates the links and verifies that the resulting
4516 tree traversal generates exactly the given tokens in exact order.
4518 tokens: the list of Token instances for the input.
4519 Created by make_tokens().
4520 tree: the ast tree for the input.
4521 Created by parse_ast().
4522 """
4523 # Init all ivars.
4524 self.file_name = file_name # For tests.
4525 self.level = 0 # Python indentation level.
4526 self.node = None # The node being visited.
4527 self.tokens = tokens # The immutable list of input tokens.
4528 self.tree = tree # The tree of ast.AST nodes.
4529 # Traverse the tree.
4530 self.visit(tree)
4531 # Ensure that all tokens are patched.
4532 self.node = tree
4533 self.token('endmarker', '')
4534 # Return [] for compatibility with legacy code: list(tog.create_links).
4535 return []
4536 #@+node:ekr.20191229071733.1: *5* tog.init_from_file
4537 def init_from_file(self, filename: str) -> Tuple[str, str, List["Token"], Node]: # pragma: no cover
4538 """
4539 Create the tokens and ast tree for the given file.
4540 Create links between tokens and the parse tree.
4541 Return (contents, encoding, tokens, tree).
4542 """
4543 self.level = 0
4544 self.filename = filename
4545 encoding, contents = read_file_with_encoding(filename)
4546 if not contents:
4547 return None, None, None, None
4548 self.tokens = tokens = make_tokens(contents)
4549 self.tree = tree = parse_ast(contents)
4550 self.create_links(tokens, tree)
4551 return contents, encoding, tokens, tree
4552 #@+node:ekr.20191229071746.1: *5* tog.init_from_string
4553 def init_from_string(self, contents: str, filename: str) -> Tuple[List["Token"], Node]: # pragma: no cover
4554 """
4555 Tokenize, parse and create links in the contents string.
4557 Return (tokens, tree).
4558 """
4559 self.filename = filename
4560 self.level = 0
4561 self.tokens = tokens = make_tokens(contents)
4562 self.tree = tree = parse_ast(contents)
4563 self.create_links(tokens, tree)
4564 return tokens, tree
4565 #@+node:ekr.20220402052020.1: *4* tog: Syncronizers...
4566 # The synchronizer sync tokens to nodes.
4567 #@+node:ekr.20200110162044.1: *5* tog.find_next_significant_token
4568 def find_next_significant_token(self) -> Optional["Token"]:
4569 """
4570 Scan from *after* self.tokens[px] looking for the next significant
4571 token.
4573 Return the token, or None. Never change self.px.
4574 """
4575 px = self.px + 1
4576 while px < len(self.tokens):
4577 token = self.tokens[px]
4578 px += 1
4579 if is_significant_token(token):
4580 return token
4581 # This will never happen, because endtoken is significant.
4582 return None # pragma: no cover
4583 #@+node:ekr.20191125120814.1: *5* tog.set_links
4584 last_statement_node = None
4586 def set_links(self, node: Node, token: "Token") -> None:
4587 """Make two-way links between token and the given node."""
4588 # Don't bother assigning comment, comma, parens, ws and endtoken tokens.
4589 if token.kind == 'comment':
4590 # Append the comment to node.comment_list.
4591 comment_list: List["Token"] = getattr(node, 'comment_list', [])
4592 node.comment_list = comment_list + [token]
4593 return
4594 if token.kind in ('endmarker', 'ws'):
4595 return
4596 if token.kind == 'op' and token.value in ',()':
4597 return
4598 # *Always* remember the last statement.
4599 statement = find_statement_node(node)
4600 if statement:
4601 self.last_statement_node = statement
4602 assert not isinstance(self.last_statement_node, ast.Module)
4603 if token.node is not None: # pragma: no cover
4604 line_s = f"line {token.line_number}:"
4605 raise AssignLinksError(
4606 f" file: {self.filename}\n"
4607 f"{line_s:>12} {token.line.strip()}\n"
4608 f"token index: {self.px}\n"
4609 f"token.node is not None\n"
4610 f" token.node: {token.node.__class__.__name__}\n"
4611 f" callers: {g.callers()}")
4612 # Assign newlines to the previous statement node, if any.
4613 if token.kind in ('newline', 'nl'):
4614 # Set an *auxilliary* link for the split/join logic.
4615 # Do *not* set token.node!
4616 token.statement_node = self.last_statement_node
4617 return
4618 if is_significant_token(token):
4619 # Link the token to the ast node.
4620 token.node = node
4621 # Add the token to node's token_list.
4622 add_token_to_token_list(token, node)
4623 #@+node:ekr.20191124083124.1: *5* tog.sync_name (aka name)
4624 def sync_name(self, val: str) -> None:
4625 aList = val.split('.')
4626 if len(aList) == 1:
4627 self.sync_token('name', val)
4628 else:
4629 for i, part in enumerate(aList):
4630 self.sync_token('name', part)
4631 if i < len(aList) - 1:
4632 self.sync_op('.')
4634 name = sync_name # for readability.
4635 #@+node:ekr.20220402052102.1: *5* tog.sync_op (aka op)
4636 def sync_op(self, val: str) -> None:
4637 """
4638 Sync to the given operator.
4640 val may be '(' or ')' *only* if the parens *will* actually exist in the
4641 token list.
4642 """
4643 self.sync_token('op', val)
4645 op = sync_op # For readability.
4646 #@+node:ekr.20191113063144.7: *5* tog.sync_token (aka token)
4647 px = -1 # Index of the previously synced token.
4649 def sync_token(self, kind: str, val: str) -> None:
4650 """
4651 Sync to a token whose kind & value are given. The token need not be
4652 significant, but it must be guaranteed to exist in the token list.
4654 The checks in this method constitute a strong, ever-present, unit test.
4656 Scan the tokens *after* px, looking for a token T matching (kind, val).
4657 raise AssignLinksError if a significant token is found that doesn't match T.
4658 Otherwise:
4659 - Create two-way links between all assignable tokens between px and T.
4660 - Create two-way links between T and self.node.
4661 - Advance by updating self.px to point to T.
4662 """
4663 node, tokens = self.node, self.tokens
4664 assert isinstance(node, ast.AST), repr(node)
4665 # g.trace(
4666 # f"px: {self.px:2} "
4667 # f"node: {node.__class__.__name__:<10} "
4668 # f"kind: {kind:>10}: val: {val!r}")
4669 #
4670 # Step one: Look for token T.
4671 old_px = px = self.px + 1
4672 while px < len(self.tokens):
4673 token = tokens[px]
4674 if (kind, val) == (token.kind, token.value):
4675 break # Success.
4676 if kind == token.kind == 'number':
4677 val = token.value
4678 break # Benign: use the token's value, a string, instead of a number.
4679 if is_significant_token(token): # pragma: no cover
4680 line_s = f"line {token.line_number}:"
4681 val = str(val) # for g.truncate.
4682 raise AssignLinksError(
4683 f" file: {self.filename}\n"
4684 f"{line_s:>12} {token.line.strip()}\n"
4685 f"Looking for: {kind}.{g.truncate(val, 40)!r}\n"
4686 f" found: {token.kind}.{token.value!r}\n"
4687 f"token.index: {token.index}\n")
4688 # Skip the insignificant token.
4689 px += 1
4690 else: # pragma: no cover
4691 val = str(val) # for g.truncate.
4692 raise AssignLinksError(
4693 f" file: {self.filename}\n"
4694 f"Looking for: {kind}.{g.truncate(val, 40)}\n"
4695 f" found: end of token list")
4696 #
4697 # Step two: Assign *secondary* links only for newline tokens.
4698 # Ignore all other non-significant tokens.
4699 while old_px < px:
4700 token = tokens[old_px]
4701 old_px += 1
4702 if token.kind in ('comment', 'newline', 'nl'):
4703 self.set_links(node, token)
4704 #
4705 # Step three: Set links in the found token.
4706 token = tokens[px]
4707 self.set_links(node, token)
4708 #
4709 # Step four: Advance.
4710 self.px = px
4712 token = sync_token # For readability.
4713 #@+node:ekr.20191223052749.1: *4* tog: Traversal...
4714 #@+node:ekr.20191113063144.3: *5* tog.enter_node
4715 def enter_node(self, node: Node) -> None:
4716 """Enter a node."""
4717 # Update the stats.
4718 self.n_nodes += 1
4719 # Do this first, *before* updating self.node.
4720 node.parent = self.node
4721 if self.node:
4722 children: List[Node] = getattr(self.node, 'children', [])
4723 children.append(node)
4724 self.node.children = children
4725 # Inject the node_index field.
4726 assert not hasattr(node, 'node_index'), g.callers()
4727 node.node_index = self.node_index
4728 self.node_index += 1
4729 # begin_visitor and end_visitor must be paired.
4730 self.begin_end_stack.append(node.__class__.__name__)
4731 # Push the previous node.
4732 self.node_stack.append(self.node)
4733 # Update self.node *last*.
4734 self.node = node
4735 #@+node:ekr.20200104032811.1: *5* tog.leave_node
4736 def leave_node(self, node: Node) -> None:
4737 """Leave a visitor."""
4738 # begin_visitor and end_visitor must be paired.
4739 entry_name = self.begin_end_stack.pop()
4740 assert entry_name == node.__class__.__name__, f"{entry_name!r} {node.__class__.__name__}"
4741 assert self.node == node, (repr(self.node), repr(node))
4742 # Restore self.node.
4743 self.node = self.node_stack.pop()
4744 #@+node:ekr.20191113081443.1: *5* tog.visit
4745 def visit(self, node: Node) -> None:
4746 """Given an ast node, return a *generator* from its visitor."""
4747 # This saves a lot of tests.
4748 if node is None:
4749 return
4750 if 0: # pragma: no cover
4751 # Keep this trace!
4752 cn = node.__class__.__name__ if node else ' '
4753 caller1, caller2 = g.callers(2).split(',')
4754 g.trace(f"{caller1:>15} {caller2:<14} {cn}")
4755 # More general, more convenient.
4756 if isinstance(node, (list, tuple)):
4757 for z in node or []:
4758 if isinstance(z, ast.AST):
4759 self.visit(z)
4760 else: # pragma: no cover
4761 # Some fields may contain ints or strings.
4762 assert isinstance(z, (int, str)), z.__class__.__name__
4763 return
4764 # We *do* want to crash if the visitor doesn't exist.
4765 method = getattr(self, 'do_' + node.__class__.__name__)
4766 # Don't even *think* about removing the parent/child links.
4767 # The nearest_common_ancestor function depends upon them.
4768 self.enter_node(node)
4769 method(node)
4770 self.leave_node(node)
4771 #@+node:ekr.20191113063144.13: *4* tog: Visitors...
4772 #@+node:ekr.20191113063144.32: *5* tog.keyword: not called!
4773 # keyword arguments supplied to call (NULL identifier for **kwargs)
4775 # keyword = (identifier? arg, expr value)
4777 def do_keyword(self, node: Node) -> None: # pragma: no cover
4778 """A keyword arg in an ast.Call."""
4779 # This should never be called.
4780 # tog.hande_call_arguments calls self.visit(kwarg_arg.value) instead.
4781 filename = getattr(self, 'filename', '<no file>')
4782 raise AssignLinksError(
4783 f"file: {filename}\n"
4784 f"do_keyword should never be called\n"
4785 f"{g.callers(8)}")
4786 #@+node:ekr.20191113063144.14: *5* tog: Contexts
4787 #@+node:ekr.20191113063144.28: *6* tog.arg
4788 # arg = (identifier arg, expr? annotation)
4790 def do_arg(self, node: Node) -> None:
4791 """This is one argument of a list of ast.Function or ast.Lambda arguments."""
4792 self.name(node.arg)
4793 annotation = getattr(node, 'annotation', None)
4794 if annotation is not None:
4795 self.op(':')
4796 self.visit(node.annotation)
4797 #@+node:ekr.20191113063144.27: *6* tog.arguments
4798 # arguments = (
4799 # arg* posonlyargs, arg* args, arg? vararg, arg* kwonlyargs,
4800 # expr* kw_defaults, arg? kwarg, expr* defaults
4801 # )
4803 def do_arguments(self, node: Node) -> None:
4804 """Arguments to ast.Function or ast.Lambda, **not** ast.Call."""
4805 #
4806 # No need to generate commas anywhere below.
4807 #
4808 # Let block. Some fields may not exist pre Python 3.8.
4809 n_plain = len(node.args) - len(node.defaults)
4810 posonlyargs = getattr(node, 'posonlyargs', [])
4811 vararg = getattr(node, 'vararg', None)
4812 kwonlyargs = getattr(node, 'kwonlyargs', [])
4813 kw_defaults = getattr(node, 'kw_defaults', [])
4814 kwarg = getattr(node, 'kwarg', None)
4815 # 1. Sync the position-only args.
4816 if posonlyargs:
4817 for n, z in enumerate(posonlyargs):
4818 # g.trace('pos-only', ast.dump(z))
4819 self.visit(z)
4820 self.op('/')
4821 # 2. Sync all args.
4822 for i, z in enumerate(node.args):
4823 self.visit(z)
4824 if i >= n_plain:
4825 self.op('=')
4826 self.visit(node.defaults[i - n_plain])
4827 # 3. Sync the vararg.
4828 if vararg:
4829 self.op('*')
4830 self.visit(vararg)
4831 # 4. Sync the keyword-only args.
4832 if kwonlyargs:
4833 if not vararg:
4834 self.op('*')
4835 for n, z in enumerate(kwonlyargs):
4836 self.visit(z)
4837 val = kw_defaults[n]
4838 if val is not None:
4839 self.op('=')
4840 self.visit(val)
4841 # 5. Sync the kwarg.
4842 if kwarg:
4843 self.op('**')
4844 self.visit(kwarg)
4845 #@+node:ekr.20191113063144.15: *6* tog.AsyncFunctionDef
4846 # AsyncFunctionDef(identifier name, arguments args, stmt* body, expr* decorator_list,
4847 # expr? returns)
4849 def do_AsyncFunctionDef(self, node: Node) -> None:
4851 if node.decorator_list:
4852 for z in node.decorator_list:
4853 # '@%s\n'
4854 self.op('@')
4855 self.visit(z)
4856 # 'asynch def (%s): -> %s\n'
4857 # 'asynch def %s(%s):\n'
4858 async_token_type = 'async' if has_async_tokens else 'name'
4859 self.token(async_token_type, 'async')
4860 self.name('def')
4861 self.name(node.name) # A string
4862 self.op('(')
4863 self.visit(node.args)
4864 self.op(')')
4865 returns = getattr(node, 'returns', None)
4866 if returns is not None:
4867 self.op('->')
4868 self.visit(node.returns)
4869 self.op(':')
4870 self.level += 1
4871 self.visit(node.body)
4872 self.level -= 1
4873 #@+node:ekr.20191113063144.16: *6* tog.ClassDef
4874 def do_ClassDef(self, node: Node) -> None:
4876 for z in node.decorator_list or []:
4877 # @{z}\n
4878 self.op('@')
4879 self.visit(z)
4880 # class name(bases):\n
4881 self.name('class')
4882 self.name(node.name) # A string.
4883 if node.bases:
4884 self.op('(')
4885 self.visit(node.bases)
4886 self.op(')')
4887 self.op(':')
4888 # Body...
4889 self.level += 1
4890 self.visit(node.body)
4891 self.level -= 1
4892 #@+node:ekr.20191113063144.17: *6* tog.FunctionDef
4893 # FunctionDef(
4894 # identifier name, arguments args,
4895 # stmt* body,
4896 # expr* decorator_list,
4897 # expr? returns,
4898 # string? type_comment)
4900 def do_FunctionDef(self, node: Node) -> None:
4902 # Guards...
4903 returns = getattr(node, 'returns', None)
4904 # Decorators...
4905 # @{z}\n
4906 for z in node.decorator_list or []:
4907 self.op('@')
4908 self.visit(z)
4909 # Signature...
4910 # def name(args): -> returns\n
4911 # def name(args):\n
4912 self.name('def')
4913 self.name(node.name) # A string.
4914 self.op('(')
4915 self.visit(node.args)
4916 self.op(')')
4917 if returns is not None:
4918 self.op('->')
4919 self.visit(node.returns)
4920 self.op(':')
4921 # Body...
4922 self.level += 1
4923 self.visit(node.body)
4924 self.level -= 1
4925 #@+node:ekr.20191113063144.18: *6* tog.Interactive
4926 def do_Interactive(self, node: Node) -> None: # pragma: no cover
4928 self.visit(node.body)
4929 #@+node:ekr.20191113063144.20: *6* tog.Lambda
4930 def do_Lambda(self, node: Node) -> None:
4932 self.name('lambda')
4933 self.visit(node.args)
4934 self.op(':')
4935 self.visit(node.body)
4936 #@+node:ekr.20191113063144.19: *6* tog.Module
4937 def do_Module(self, node: Node) -> None:
4939 # Encoding is a non-syncing statement.
4940 self.visit(node.body)
4941 #@+node:ekr.20191113063144.21: *5* tog: Expressions
4942 #@+node:ekr.20191113063144.22: *6* tog.Expr
4943 def do_Expr(self, node: Node) -> None:
4944 """An outer expression."""
4945 # No need to put parentheses.
4946 self.visit(node.value)
4947 #@+node:ekr.20191113063144.23: *6* tog.Expression
4948 def do_Expression(self, node: Node) -> None: # pragma: no cover
4949 """An inner expression."""
4950 # No need to put parentheses.
4951 self.visit(node.body)
4952 #@+node:ekr.20191113063144.24: *6* tog.GeneratorExp
4953 def do_GeneratorExp(self, node: Node) -> None:
4955 # '<gen %s for %s>' % (elt, ','.join(gens))
4956 # No need to put parentheses or commas.
4957 self.visit(node.elt)
4958 self.visit(node.generators)
4959 #@+node:ekr.20210321171703.1: *6* tog.NamedExpr
4960 # NamedExpr(expr target, expr value)
4962 def do_NamedExpr(self, node: Node) -> None: # Python 3.8+
4964 self.visit(node.target)
4965 self.op(':=')
4966 self.visit(node.value)
4967 #@+node:ekr.20191113063144.26: *5* tog: Operands
4968 #@+node:ekr.20191113063144.29: *6* tog.Attribute
4969 # Attribute(expr value, identifier attr, expr_context ctx)
4971 def do_Attribute(self, node: Node) -> None:
4973 self.visit(node.value)
4974 self.op('.')
4975 self.name(node.attr) # A string.
4976 #@+node:ekr.20191113063144.30: *6* tog.Bytes
4977 def do_Bytes(self, node: Node) -> None:
4979 """
4980 It's invalid to mix bytes and non-bytes literals, so just
4981 advancing to the next 'string' token suffices.
4982 """
4983 token = self.find_next_significant_token()
4984 self.token('string', token.value)
4985 #@+node:ekr.20191113063144.33: *6* tog.comprehension
4986 # comprehension = (expr target, expr iter, expr* ifs, int is_async)
4988 def do_comprehension(self, node: Node) -> None:
4990 # No need to put parentheses.
4991 self.name('for') # #1858.
4992 self.visit(node.target) # A name
4993 self.name('in')
4994 self.visit(node.iter)
4995 for z in node.ifs or []:
4996 self.name('if')
4997 self.visit(z)
4998 #@+node:ekr.20191113063144.34: *6* tog.Constant
4999 def do_Constant(self, node: Node) -> None: # pragma: no cover
5000 """
5001 https://greentreesnakes.readthedocs.io/en/latest/nodes.html
5003 A constant. The value attribute holds the Python object it represents.
5004 This can be simple types such as a number, string or None, but also
5005 immutable container types (tuples and frozensets) if all of their
5006 elements are constant.
5007 """
5008 # Support Python 3.8.
5009 if node.value is None or isinstance(node.value, bool):
5010 # Weird: return a name!
5011 self.token('name', repr(node.value))
5012 elif node.value == Ellipsis:
5013 self.op('...')
5014 elif isinstance(node.value, str):
5015 self.do_Str(node)
5016 elif isinstance(node.value, (int, float)):
5017 self.token('number', repr(node.value))
5018 elif isinstance(node.value, bytes):
5019 self.do_Bytes(node)
5020 elif isinstance(node.value, tuple):
5021 self.do_Tuple(node)
5022 elif isinstance(node.value, frozenset):
5023 self.do_Set(node)
5024 else:
5025 # Unknown type.
5026 g.trace('----- Oops -----', repr(node.value), g.callers())
5027 #@+node:ekr.20191113063144.35: *6* tog.Dict
5028 # Dict(expr* keys, expr* values)
5030 def do_Dict(self, node: Node) -> None:
5032 assert len(node.keys) == len(node.values)
5033 self.op('{')
5034 # No need to put commas.
5035 for i, key in enumerate(node.keys):
5036 key, value = node.keys[i], node.values[i]
5037 self.visit(key) # a Str node.
5038 self.op(':')
5039 if value is not None:
5040 self.visit(value)
5041 self.op('}')
5042 #@+node:ekr.20191113063144.36: *6* tog.DictComp
5043 # DictComp(expr key, expr value, comprehension* generators)
5045 # d2 = {val: key for key, val in d}
5047 def do_DictComp(self, node: Node) -> None:
5049 self.token('op', '{')
5050 self.visit(node.key)
5051 self.op(':')
5052 self.visit(node.value)
5053 for z in node.generators or []:
5054 self.visit(z)
5055 self.token('op', '}')
5056 #@+node:ekr.20191113063144.37: *6* tog.Ellipsis
5057 def do_Ellipsis(self, node: Node) -> None: # pragma: no cover (Does not exist for python 3.8+)
5059 self.op('...')
5060 #@+node:ekr.20191113063144.38: *6* tog.ExtSlice
5061 # https://docs.python.org/3/reference/expressions.html#slicings
5063 # ExtSlice(slice* dims)
5065 def do_ExtSlice(self, node: Node) -> None: # pragma: no cover (deprecated)
5067 # ','.join(node.dims)
5068 for i, z in enumerate(node.dims):
5069 self.visit(z)
5070 if i < len(node.dims) - 1:
5071 self.op(',')
5072 #@+node:ekr.20191113063144.40: *6* tog.Index
5073 def do_Index(self, node: Node) -> None: # pragma: no cover (deprecated)
5075 self.visit(node.value)
5076 #@+node:ekr.20191113063144.39: *6* tog.FormattedValue: not called!
5077 # FormattedValue(expr value, int? conversion, expr? format_spec)
5079 def do_FormattedValue(self, node: Node) -> None: # pragma: no cover
5080 """
5081 This node represents the *components* of a *single* f-string.
5083 Happily, JoinedStr nodes *also* represent *all* f-strings,
5084 so the TOG should *never visit this node!
5085 """
5086 filename = getattr(self, 'filename', '<no file>')
5087 raise AssignLinksError(
5088 f"file: {filename}\n"
5089 f"do_FormattedValue should never be called")
5091 # This code has no chance of being useful...
5093 # conv = node.conversion
5094 # spec = node.format_spec
5095 # self.visit(node.value)
5096 # if conv is not None:
5097 # self.token('number', conv)
5098 # if spec is not None:
5099 # self.visit(node.format_spec)
5100 #@+node:ekr.20191113063144.41: *6* tog.JoinedStr & helpers
5101 # JoinedStr(expr* values)
5103 def do_JoinedStr(self, node: Node) -> None:
5104 """
5105 JoinedStr nodes represent at least one f-string and all other strings
5106 concatentated to it.
5108 Analyzing JoinedStr.values would be extremely tricky, for reasons that
5109 need not be explained here.
5111 Instead, we get the tokens *from the token list itself*!
5112 """
5113 for z in self.get_concatenated_string_tokens():
5114 self.token(z.kind, z.value)
5115 #@+node:ekr.20191113063144.42: *6* tog.List
5116 def do_List(self, node: Node) -> None:
5118 # No need to put commas.
5119 self.op('[')
5120 self.visit(node.elts)
5121 self.op(']')
5122 #@+node:ekr.20191113063144.43: *6* tog.ListComp
5123 # ListComp(expr elt, comprehension* generators)
5125 def do_ListComp(self, node: Node) -> None:
5127 self.op('[')
5128 self.visit(node.elt)
5129 for z in node.generators:
5130 self.visit(z)
5131 self.op(']')
5132 #@+node:ekr.20191113063144.44: *6* tog.Name & NameConstant
5133 def do_Name(self, node: Node) -> None:
5135 self.name(node.id)
5137 def do_NameConstant(self, node: Node) -> None: # pragma: no cover (Does not exist in Python 3.8+)
5139 self.name(repr(node.value))
5141 #@+node:ekr.20191113063144.45: *6* tog.Num
5142 def do_Num(self, node: Node) -> None: # pragma: no cover (Does not exist in Python 3.8+)
5144 self.token('number', node.n)
5145 #@+node:ekr.20191113063144.47: *6* tog.Set
5146 # Set(expr* elts)
5148 def do_Set(self, node: Node) -> None:
5150 self.op('{')
5151 self.visit(node.elts)
5152 self.op('}')
5153 #@+node:ekr.20191113063144.48: *6* tog.SetComp
5154 # SetComp(expr elt, comprehension* generators)
5156 def do_SetComp(self, node: Node) -> None:
5158 self.op('{')
5159 self.visit(node.elt)
5160 for z in node.generators or []:
5161 self.visit(z)
5162 self.op('}')
5163 #@+node:ekr.20191113063144.49: *6* tog.Slice
5164 # slice = Slice(expr? lower, expr? upper, expr? step)
5166 def do_Slice(self, node: Node) -> None:
5168 lower = getattr(node, 'lower', None)
5169 upper = getattr(node, 'upper', None)
5170 step = getattr(node, 'step', None)
5171 if lower is not None:
5172 self.visit(lower)
5173 # Always put the colon between upper and lower.
5174 self.op(':')
5175 if upper is not None:
5176 self.visit(upper)
5177 # Put the second colon if it exists in the token list.
5178 if step is None:
5179 token = self.find_next_significant_token()
5180 if token and token.value == ':':
5181 self.op(':')
5182 else:
5183 self.op(':')
5184 self.visit(step)
5185 #@+node:ekr.20191113063144.50: *6* tog.Str & helper
5186 def do_Str(self, node: Node) -> None:
5187 """This node represents a string constant."""
5188 # This loop is necessary to handle string concatenation.
5189 for z in self.get_concatenated_string_tokens():
5190 self.token(z.kind, z.value)
5191 #@+node:ekr.20200111083914.1: *7* tog.get_concatenated_tokens
5192 def get_concatenated_string_tokens(self) -> List["Token"]:
5193 """
5194 Return the next 'string' token and all 'string' tokens concatenated to
5195 it. *Never* update self.px here.
5196 """
5197 trace = False
5198 tag = 'tog.get_concatenated_string_tokens'
5199 i = self.px
5200 # First, find the next significant token. It should be a string.
5201 i, token = i + 1, None
5202 while i < len(self.tokens):
5203 token = self.tokens[i]
5204 i += 1
5205 if token.kind == 'string':
5206 # Rescan the string.
5207 i -= 1
5208 break
5209 # An error.
5210 if is_significant_token(token): # pragma: no cover
5211 break
5212 # Raise an error if we didn't find the expected 'string' token.
5213 if not token or token.kind != 'string': # pragma: no cover
5214 if not token:
5215 token = self.tokens[-1]
5216 filename = getattr(self, 'filename', '<no filename>')
5217 raise AssignLinksError(
5218 f"\n"
5219 f"{tag}...\n"
5220 f"file: {filename}\n"
5221 f"line: {token.line_number}\n"
5222 f" i: {i}\n"
5223 f"expected 'string' token, got {token!s}")
5224 # Accumulate string tokens.
5225 assert self.tokens[i].kind == 'string'
5226 results = []
5227 while i < len(self.tokens):
5228 token = self.tokens[i]
5229 i += 1
5230 if token.kind == 'string':
5231 results.append(token)
5232 elif token.kind == 'op' or is_significant_token(token):
5233 # Any significant token *or* any op will halt string concatenation.
5234 break
5235 # 'ws', 'nl', 'newline', 'comment', 'indent', 'dedent', etc.
5236 # The (significant) 'endmarker' token ensures we will have result.
5237 assert results
5238 if trace: # pragma: no cover
5239 g.printObj(results, tag=f"{tag}: Results")
5240 return results
5241 #@+node:ekr.20191113063144.51: *6* tog.Subscript
5242 # Subscript(expr value, slice slice, expr_context ctx)
5244 def do_Subscript(self, node: Node) -> None:
5246 self.visit(node.value)
5247 self.op('[')
5248 self.visit(node.slice)
5249 self.op(']')
5250 #@+node:ekr.20191113063144.52: *6* tog.Tuple
5251 # Tuple(expr* elts, expr_context ctx)
5253 def do_Tuple(self, node: Node) -> None:
5255 # Do not call op for parens or commas here.
5256 # They do not necessarily exist in the token list!
5257 self.visit(node.elts)
5258 #@+node:ekr.20191113063144.53: *5* tog: Operators
5259 #@+node:ekr.20191113063144.55: *6* tog.BinOp
5260 def do_BinOp(self, node: Node) -> None:
5262 op_name_ = op_name(node.op)
5263 self.visit(node.left)
5264 self.op(op_name_)
5265 self.visit(node.right)
5266 #@+node:ekr.20191113063144.56: *6* tog.BoolOp
5267 # BoolOp(boolop op, expr* values)
5269 def do_BoolOp(self, node: Node) -> None:
5271 # op.join(node.values)
5272 op_name_ = op_name(node.op)
5273 for i, z in enumerate(node.values):
5274 self.visit(z)
5275 if i < len(node.values) - 1:
5276 self.name(op_name_)
5277 #@+node:ekr.20191113063144.57: *6* tog.Compare
5278 # Compare(expr left, cmpop* ops, expr* comparators)
5280 def do_Compare(self, node: Node) -> None:
5282 assert len(node.ops) == len(node.comparators)
5283 self.visit(node.left)
5284 for i, z in enumerate(node.ops):
5285 op_name_ = op_name(node.ops[i])
5286 if op_name_ in ('not in', 'is not'):
5287 for z in op_name_.split(' '):
5288 self.name(z)
5289 elif op_name_.isalpha():
5290 self.name(op_name_)
5291 else:
5292 self.op(op_name_)
5293 self.visit(node.comparators[i])
5294 #@+node:ekr.20191113063144.58: *6* tog.UnaryOp
5295 def do_UnaryOp(self, node: Node) -> None:
5297 op_name_ = op_name(node.op)
5298 if op_name_.isalpha():
5299 self.name(op_name_)
5300 else:
5301 self.op(op_name_)
5302 self.visit(node.operand)
5303 #@+node:ekr.20191113063144.59: *6* tog.IfExp (ternary operator)
5304 # IfExp(expr test, expr body, expr orelse)
5306 def do_IfExp(self, node: Node) -> None:
5308 #'%s if %s else %s'
5309 self.visit(node.body)
5310 self.name('if')
5311 self.visit(node.test)
5312 self.name('else')
5313 self.visit(node.orelse)
5314 #@+node:ekr.20191113063144.60: *5* tog: Statements
5315 #@+node:ekr.20191113063144.83: *6* tog.Starred
5316 # Starred(expr value, expr_context ctx)
5318 def do_Starred(self, node: Node) -> None:
5319 """A starred argument to an ast.Call"""
5320 self.op('*')
5321 self.visit(node.value)
5322 #@+node:ekr.20191113063144.61: *6* tog.AnnAssign
5323 # AnnAssign(expr target, expr annotation, expr? value, int simple)
5325 def do_AnnAssign(self, node: Node) -> None:
5327 # {node.target}:{node.annotation}={node.value}\n'
5328 self.visit(node.target)
5329 self.op(':')
5330 self.visit(node.annotation)
5331 if node.value is not None: # #1851
5332 self.op('=')
5333 self.visit(node.value)
5334 #@+node:ekr.20191113063144.62: *6* tog.Assert
5335 # Assert(expr test, expr? msg)
5337 def do_Assert(self, node: Node) -> None:
5339 # Guards...
5340 msg = getattr(node, 'msg', None)
5341 # No need to put parentheses or commas.
5342 self.name('assert')
5343 self.visit(node.test)
5344 if msg is not None:
5345 self.visit(node.msg)
5346 #@+node:ekr.20191113063144.63: *6* tog.Assign
5347 def do_Assign(self, node: Node) -> None:
5349 for z in node.targets:
5350 self.visit(z)
5351 self.op('=')
5352 self.visit(node.value)
5353 #@+node:ekr.20191113063144.64: *6* tog.AsyncFor
5354 def do_AsyncFor(self, node: Node) -> None:
5356 # The def line...
5357 # Py 3.8 changes the kind of token.
5358 async_token_type = 'async' if has_async_tokens else 'name'
5359 self.token(async_token_type, 'async')
5360 self.name('for')
5361 self.visit(node.target)
5362 self.name('in')
5363 self.visit(node.iter)
5364 self.op(':')
5365 # Body...
5366 self.level += 1
5367 self.visit(node.body)
5368 # Else clause...
5369 if node.orelse:
5370 self.name('else')
5371 self.op(':')
5372 self.visit(node.orelse)
5373 self.level -= 1
5374 #@+node:ekr.20191113063144.65: *6* tog.AsyncWith
5375 def do_AsyncWith(self, node: Node) -> None:
5377 async_token_type = 'async' if has_async_tokens else 'name'
5378 self.token(async_token_type, 'async')
5379 self.do_With(node)
5380 #@+node:ekr.20191113063144.66: *6* tog.AugAssign
5381 # AugAssign(expr target, operator op, expr value)
5383 def do_AugAssign(self, node: Node) -> None:
5385 # %s%s=%s\n'
5386 op_name_ = op_name(node.op)
5387 self.visit(node.target)
5388 self.op(op_name_ + '=')
5389 self.visit(node.value)
5390 #@+node:ekr.20191113063144.67: *6* tog.Await
5391 # Await(expr value)
5393 def do_Await(self, node: Node) -> None:
5395 #'await %s\n'
5396 async_token_type = 'await' if has_async_tokens else 'name'
5397 self.token(async_token_type, 'await')
5398 self.visit(node.value)
5399 #@+node:ekr.20191113063144.68: *6* tog.Break
5400 def do_Break(self, node: Node) -> None:
5402 self.name('break')
5403 #@+node:ekr.20191113063144.31: *6* tog.Call & helpers
5404 # Call(expr func, expr* args, keyword* keywords)
5406 # Python 3 ast.Call nodes do not have 'starargs' or 'kwargs' fields.
5408 def do_Call(self, node: Node) -> None:
5410 # The calls to op(')') and op('(') do nothing by default.
5411 # Subclasses might handle them in an overridden tog.set_links.
5412 self.visit(node.func)
5413 self.op('(')
5414 # No need to generate any commas.
5415 self.handle_call_arguments(node)
5416 self.op(')')
5417 #@+node:ekr.20191204114930.1: *7* tog.arg_helper
5418 def arg_helper(self, node: Union[Node, str]) -> None:
5419 """
5420 Yield the node, with a special case for strings.
5421 """
5422 if isinstance(node, str):
5423 self.token('name', node)
5424 else:
5425 self.visit(node)
5426 #@+node:ekr.20191204105506.1: *7* tog.handle_call_arguments
5427 def handle_call_arguments(self, node: Node) -> None:
5428 """
5429 Generate arguments in the correct order.
5431 Call(expr func, expr* args, keyword* keywords)
5433 https://docs.python.org/3/reference/expressions.html#calls
5435 Warning: This code will fail on Python 3.8 only for calls
5436 containing kwargs in unexpected places.
5437 """
5438 # *args: in node.args[]: Starred(value=Name(id='args'))
5439 # *[a, 3]: in node.args[]: Starred(value=List(elts=[Name(id='a'), Num(n=3)])
5440 # **kwargs: in node.keywords[]: keyword(arg=None, value=Name(id='kwargs'))
5441 #
5442 # Scan args for *name or *List
5443 args = node.args or []
5444 keywords = node.keywords or []
5446 def get_pos(obj: Any) -> Tuple[int, int, Any]:
5447 line1 = getattr(obj, 'lineno', None)
5448 col1 = getattr(obj, 'col_offset', None)
5449 return line1, col1, obj
5451 def sort_key(aTuple: Tuple) -> int:
5452 line, col, obj = aTuple
5453 return line * 1000 + col
5455 if 0: # pragma: no cover
5456 g.printObj([ast.dump(z) for z in args], tag='args')
5457 g.printObj([ast.dump(z) for z in keywords], tag='keywords')
5459 if py_version >= (3, 9):
5460 places = [get_pos(z) for z in args + keywords]
5461 places.sort(key=sort_key)
5462 ordered_args = [z[2] for z in places]
5463 for z in ordered_args:
5464 if isinstance(z, ast.Starred):
5465 self.op('*')
5466 self.visit(z.value)
5467 elif isinstance(z, ast.keyword):
5468 if getattr(z, 'arg', None) is None:
5469 self.op('**')
5470 self.arg_helper(z.value)
5471 else:
5472 self.arg_helper(z.arg)
5473 self.op('=')
5474 self.arg_helper(z.value)
5475 else:
5476 self.arg_helper(z)
5477 else: # pragma: no cover
5478 #
5479 # Legacy code: May fail for Python 3.8
5480 #
5481 # Scan args for *arg and *[...]
5482 kwarg_arg = star_arg = None
5483 for z in args:
5484 if isinstance(z, ast.Starred):
5485 if isinstance(z.value, ast.Name): # *Name.
5486 star_arg = z
5487 args.remove(z)
5488 break
5489 elif isinstance(z.value, (ast.List, ast.Tuple)): # *[...]
5490 # star_list = z
5491 break
5492 raise AttributeError(f"Invalid * expression: {ast.dump(z)}") # pragma: no cover
5493 # Scan keywords for **name.
5494 for z in keywords:
5495 if hasattr(z, 'arg') and z.arg is None:
5496 kwarg_arg = z
5497 keywords.remove(z)
5498 break
5499 # Sync the plain arguments.
5500 for z in args:
5501 self.arg_helper(z)
5502 # Sync the keyword args.
5503 for z in keywords:
5504 self.arg_helper(z.arg)
5505 self.op('=')
5506 self.arg_helper(z.value)
5507 # Sync the * arg.
5508 if star_arg:
5509 self.arg_helper(star_arg)
5510 # Sync the ** kwarg.
5511 if kwarg_arg:
5512 self.op('**')
5513 self.visit(kwarg_arg.value)
5514 #@+node:ekr.20191113063144.69: *6* tog.Continue
5515 def do_Continue(self, node: Node) -> None:
5517 self.name('continue')
5518 #@+node:ekr.20191113063144.70: *6* tog.Delete
5519 def do_Delete(self, node: Node) -> None:
5521 # No need to put commas.
5522 self.name('del')
5523 self.visit(node.targets)
5524 #@+node:ekr.20191113063144.71: *6* tog.ExceptHandler
5525 def do_ExceptHandler(self, node: Node) -> None:
5527 # Except line...
5528 self.name('except')
5529 if getattr(node, 'type', None):
5530 self.visit(node.type)
5531 if getattr(node, 'name', None):
5532 self.name('as')
5533 self.name(node.name)
5534 self.op(':')
5535 # Body...
5536 self.level += 1
5537 self.visit(node.body)
5538 self.level -= 1
5539 #@+node:ekr.20191113063144.73: *6* tog.For
5540 def do_For(self, node: Node) -> None:
5542 # The def line...
5543 self.name('for')
5544 self.visit(node.target)
5545 self.name('in')
5546 self.visit(node.iter)
5547 self.op(':')
5548 # Body...
5549 self.level += 1
5550 self.visit(node.body)
5551 # Else clause...
5552 if node.orelse:
5553 self.name('else')
5554 self.op(':')
5555 self.visit(node.orelse)
5556 self.level -= 1
5557 #@+node:ekr.20191113063144.74: *6* tog.Global
5558 # Global(identifier* names)
5560 def do_Global(self, node: Node) -> None:
5562 self.name('global')
5563 for z in node.names:
5564 self.name(z)
5565 #@+node:ekr.20191113063144.75: *6* tog.If & helpers
5566 # If(expr test, stmt* body, stmt* orelse)
5568 def do_If(self, node: Node) -> None:
5569 #@+<< do_If docstring >>
5570 #@+node:ekr.20191122222412.1: *7* << do_If docstring >>
5571 """
5572 The parse trees for the following are identical!
5574 if 1: if 1:
5575 pass pass
5576 else: elif 2:
5577 if 2: pass
5578 pass
5580 So there is *no* way for the 'if' visitor to disambiguate the above two
5581 cases from the parse tree alone.
5583 Instead, we scan the tokens list for the next 'if', 'else' or 'elif' token.
5584 """
5585 #@-<< do_If docstring >>
5586 # Use the next significant token to distinguish between 'if' and 'elif'.
5587 token = self.find_next_significant_token()
5588 self.name(token.value)
5589 self.visit(node.test)
5590 self.op(':')
5591 #
5592 # Body...
5593 self.level += 1
5594 self.visit(node.body)
5595 self.level -= 1
5596 #
5597 # Else and elif clauses...
5598 if node.orelse:
5599 self.level += 1
5600 token = self.find_next_significant_token()
5601 if token.value == 'else':
5602 self.name('else')
5603 self.op(':')
5604 self.visit(node.orelse)
5605 else:
5606 self.visit(node.orelse)
5607 self.level -= 1
5608 #@+node:ekr.20191113063144.76: *6* tog.Import & helper
5609 def do_Import(self, node: Node) -> None:
5611 self.name('import')
5612 for alias in node.names:
5613 self.name(alias.name)
5614 if alias.asname:
5615 self.name('as')
5616 self.name(alias.asname)
5617 #@+node:ekr.20191113063144.77: *6* tog.ImportFrom
5618 # ImportFrom(identifier? module, alias* names, int? level)
5620 def do_ImportFrom(self, node: Node) -> None:
5622 self.name('from')
5623 for i in range(node.level):
5624 self.op('.')
5625 if node.module:
5626 self.name(node.module)
5627 self.name('import')
5628 # No need to put commas.
5629 for alias in node.names:
5630 if alias.name == '*': # #1851.
5631 self.op('*')
5632 else:
5633 self.name(alias.name)
5634 if alias.asname:
5635 self.name('as')
5636 self.name(alias.asname)
5637 #@+node:ekr.20220401034726.1: *6* tog.Match* (Python 3.10+)
5638 # Match(expr subject, match_case* cases)
5640 # match_case = (pattern pattern, expr? guard, stmt* body)
5642 # Full syntax diagram: # https://peps.python.org/pep-0634/#appendix-a
5644 def do_Match(self, node: Node) -> None:
5646 cases = getattr(node, 'cases', [])
5647 self.name('match')
5648 self.visit(node.subject)
5649 self.op(':')
5650 for case in cases:
5651 self.visit(case)
5652 #@+node:ekr.20220401034726.2: *7* tog.match_case
5653 # match_case = (pattern pattern, expr? guard, stmt* body)
5655 def do_match_case(self, node: Node) -> None:
5657 guard = getattr(node, 'guard', None)
5658 body = getattr(node, 'body', [])
5659 self.name('case')
5660 self.visit(node.pattern)
5661 if guard:
5662 self.name('if')
5663 self.visit(guard)
5664 self.op(':')
5665 for statement in body:
5666 self.visit(statement)
5667 #@+node:ekr.20220401034726.3: *7* tog.MatchAs
5668 # MatchAs(pattern? pattern, identifier? name)
5670 def do_MatchAs(self, node: Node) -> None:
5671 pattern = getattr(node, 'pattern', None)
5672 name = getattr(node, 'name', None)
5673 if pattern and name:
5674 self.visit(pattern)
5675 self.name('as')
5676 self.name(name)
5677 elif pattern:
5678 self.visit(pattern) # pragma: no cover
5679 else:
5680 self.name(name or '_')
5681 #@+node:ekr.20220401034726.4: *7* tog.MatchClass
5682 # MatchClass(expr cls, pattern* patterns, identifier* kwd_attrs, pattern* kwd_patterns)
5684 def do_MatchClass(self, node: Node) -> None:
5686 cls = node.cls
5687 patterns = getattr(node, 'patterns', [])
5688 kwd_attrs = getattr(node, 'kwd_attrs', [])
5689 kwd_patterns = getattr(node, 'kwd_patterns', [])
5690 self.visit(node.cls)
5691 self.op('(')
5692 for pattern in patterns:
5693 self.visit(pattern)
5694 for i, kwd_attr in enumerate(kwd_attrs):
5695 self.name(kwd_attr) # a String.
5696 self.op('=')
5697 self.visit(kwd_patterns[i])
5698 self.op(')')
5699 #@+node:ekr.20220401034726.5: *7* tog.MatchMapping
5700 # MatchMapping(expr* keys, pattern* patterns, identifier? rest)
5702 def do_MatchMapping(self, node: Node) -> None:
5703 keys = getattr(node, 'keys', [])
5704 patterns = getattr(node, 'patterns', [])
5705 rest = getattr(node, 'rest', None)
5706 self.op('{')
5707 for i, key in enumerate(keys):
5708 self.visit(key)
5709 self.op(':')
5710 self.visit(patterns[i])
5711 if rest:
5712 self.op('**')
5713 self.name(rest) # A string.
5714 self.op('}')
5715 #@+node:ekr.20220401034726.6: *7* tog.MatchOr
5716 # MatchOr(pattern* patterns)
5718 def do_MatchOr(self, node: Node) -> None:
5719 patterns = getattr(node, 'patterns', [])
5720 for i, pattern in enumerate(patterns):
5721 if i > 0:
5722 self.op('|')
5723 self.visit(pattern)
5724 #@+node:ekr.20220401034726.7: *7* tog.MatchSequence
5725 # MatchSequence(pattern* patterns)
5727 def do_MatchSequence(self, node: Node) -> None:
5728 patterns = getattr(node, 'patterns', [])
5729 # Scan for the next '(' or '[' token, skipping the 'case' token.
5730 token = None
5731 for token in self.tokens[self.px + 1 :]:
5732 if token.kind == 'op' and token.value in '([':
5733 break
5734 if is_significant_token(token):
5735 # An implicit tuple: there is no '(' or '[' token.
5736 token = None
5737 break
5738 else:
5739 raise AssignLinksError('Ill-formed tuple') # pragma: no cover
5740 if token:
5741 self.op(token.value)
5742 for i, pattern in enumerate(patterns):
5743 self.visit(pattern)
5744 if token:
5745 self.op(']' if token.value == '[' else ')')
5746 #@+node:ekr.20220401034726.8: *7* tog.MatchSingleton
5747 # MatchSingleton(constant value)
5749 def do_MatchSingleton(self, node: Node) -> None:
5750 """Match True, False or None."""
5751 # g.trace(repr(node.value))
5752 self.token('name', repr(node.value))
5753 #@+node:ekr.20220401034726.9: *7* tog.MatchStar
5754 # MatchStar(identifier? name)
5756 def do_MatchStar(self, node: Node) -> None:
5757 name = getattr(node, 'name', None)
5758 self.op('*')
5759 if name:
5760 self.name(name)
5761 #@+node:ekr.20220401034726.10: *7* tog.MatchValue
5762 # MatchValue(expr value)
5764 def do_MatchValue(self, node: Node) -> None:
5766 self.visit(node.value)
5767 #@+node:ekr.20191113063144.78: *6* tog.Nonlocal
5768 # Nonlocal(identifier* names)
5770 def do_Nonlocal(self, node: Node) -> None:
5772 # nonlocal %s\n' % ','.join(node.names))
5773 # No need to put commas.
5774 self.name('nonlocal')
5775 for z in node.names:
5776 self.name(z)
5777 #@+node:ekr.20191113063144.79: *6* tog.Pass
5778 def do_Pass(self, node: Node) -> None:
5780 self.name('pass')
5781 #@+node:ekr.20191113063144.81: *6* tog.Raise
5782 # Raise(expr? exc, expr? cause)
5784 def do_Raise(self, node: Node) -> None:
5786 # No need to put commas.
5787 self.name('raise')
5788 exc = getattr(node, 'exc', None)
5789 cause = getattr(node, 'cause', None)
5790 tback = getattr(node, 'tback', None)
5791 self.visit(exc)
5792 if cause:
5793 self.name('from') # #2446.
5794 self.visit(cause)
5795 self.visit(tback)
5796 #@+node:ekr.20191113063144.82: *6* tog.Return
5797 def do_Return(self, node: Node) -> None:
5799 self.name('return')
5800 self.visit(node.value)
5801 #@+node:ekr.20191113063144.85: *6* tog.Try
5802 # Try(stmt* body, excepthandler* handlers, stmt* orelse, stmt* finalbody)
5804 def do_Try(self, node: Node) -> None:
5806 # Try line...
5807 self.name('try')
5808 self.op(':')
5809 # Body...
5810 self.level += 1
5811 self.visit(node.body)
5812 self.visit(node.handlers)
5813 # Else...
5814 if node.orelse:
5815 self.name('else')
5816 self.op(':')
5817 self.visit(node.orelse)
5818 # Finally...
5819 if node.finalbody:
5820 self.name('finally')
5821 self.op(':')
5822 self.visit(node.finalbody)
5823 self.level -= 1
5824 #@+node:ekr.20191113063144.88: *6* tog.While
5825 def do_While(self, node: Node) -> None:
5827 # While line...
5828 # while %s:\n'
5829 self.name('while')
5830 self.visit(node.test)
5831 self.op(':')
5832 # Body...
5833 self.level += 1
5834 self.visit(node.body)
5835 # Else clause...
5836 if node.orelse:
5837 self.name('else')
5838 self.op(':')
5839 self.visit(node.orelse)
5840 self.level -= 1
5841 #@+node:ekr.20191113063144.89: *6* tog.With
5842 # With(withitem* items, stmt* body)
5844 # withitem = (expr context_expr, expr? optional_vars)
5846 def do_With(self, node: Node) -> None:
5848 expr: Optional[ast.AST] = getattr(node, 'context_expression', None)
5849 items: List[ast.AST] = getattr(node, 'items', [])
5850 self.name('with')
5851 self.visit(expr)
5852 # No need to put commas.
5853 for item in items:
5854 self.visit(item.context_expr)
5855 optional_vars = getattr(item, 'optional_vars', None)
5856 if optional_vars is not None:
5857 self.name('as')
5858 self.visit(item.optional_vars)
5859 # End the line.
5860 self.op(':')
5861 # Body...
5862 self.level += 1
5863 self.visit(node.body)
5864 self.level -= 1
5865 #@+node:ekr.20191113063144.90: *6* tog.Yield
5866 def do_Yield(self, node: Node) -> None:
5868 self.name('yield')
5869 if hasattr(node, 'value'):
5870 self.visit(node.value)
5871 #@+node:ekr.20191113063144.91: *6* tog.YieldFrom
5872 # YieldFrom(expr value)
5874 def do_YieldFrom(self, node: Node) -> None:
5876 self.name('yield')
5877 self.name('from')
5878 self.visit(node.value)
5879 #@-others
5880#@+node:ekr.20191226195813.1: *3* class TokenOrderTraverser
5881class TokenOrderTraverser:
5882 """
5883 Traverse an ast tree using the parent/child links created by the
5884 TokenOrderGenerator class.
5886 **Important**:
5888 This class is a curio. It is no longer used in this file!
5889 The Fstringify and ReassignTokens classes now use ast.walk.
5890 """
5891 #@+others
5892 #@+node:ekr.20191226200154.1: *4* TOT.traverse
5893 def traverse(self, tree: Node) -> int:
5894 """
5895 Call visit, in token order, for all nodes in tree.
5897 Recursion is not allowed.
5899 The code follows p.moveToThreadNext exactly.
5900 """
5902 def has_next(i: int, node: Node, stack: List[int]) -> bool:
5903 """Return True if stack[i] is a valid child of node.parent."""
5904 # g.trace(node.__class__.__name__, stack)
5905 parent = node.parent
5906 return bool(parent and parent.children and i < len(parent.children))
5908 # Update stats
5910 self.last_node_index = -1 # For visit
5911 # The stack contains child indices.
5912 node, stack = tree, [0]
5913 seen = set()
5914 while node and stack:
5915 # g.trace(
5916 # f"{node.node_index:>3} "
5917 # f"{node.__class__.__name__:<12} {stack}")
5918 # Visit the node.
5919 assert node.node_index not in seen, node.node_index
5920 seen.add(node.node_index)
5921 self.visit(node)
5922 # if p.v.children: p.moveToFirstChild()
5923 children: List[ast.AST] = getattr(node, 'children', [])
5924 if children:
5925 # Move to the first child.
5926 stack.append(0)
5927 node = children[0]
5928 # g.trace(' child:', node.__class__.__name__, stack)
5929 continue
5930 # elif p.hasNext(): p.moveToNext()
5931 stack[-1] += 1
5932 i = stack[-1]
5933 if has_next(i, node, stack):
5934 node = node.parent.children[i]
5935 continue
5936 # else...
5937 # p.moveToParent()
5938 node = node.parent
5939 stack.pop()
5940 # while p:
5941 while node and stack:
5942 # if p.hasNext():
5943 stack[-1] += 1
5944 i = stack[-1]
5945 if has_next(i, node, stack):
5946 # Move to the next sibling.
5947 node = node.parent.children[i]
5948 break # Found.
5949 # p.moveToParent()
5950 node = node.parent
5951 stack.pop()
5952 # not found.
5953 else:
5954 break # pragma: no cover
5955 return self.last_node_index
5956 #@+node:ekr.20191227160547.1: *4* TOT.visit
5957 def visit(self, node: Node) -> None:
5959 self.last_node_index += 1
5960 assert self.last_node_index == node.node_index, (
5961 self.last_node_index, node.node_index)
5962 #@-others
5963#@-others
5964g = LeoGlobals()
5965if __name__ == '__main__':
5966 main() # pragma: no cover
5967#@@language python
5968#@@tabwidth -4
5969#@@pagewidth 70
5970#@-leo