Python 2 scanner doesn't scan unicode.

• 报告人: sforman
• 属主: sforman
• 类型: Bugs
• 状态: 关闭
• 组件: Core
• 里程碑: v0.4.0 (关闭)
• 优先: 5 - Medium
• 严重性: 5 - Medium
• 处理结果: Works For Me
• 文件: 无

Details 回复

(SPF) sforman@bock:~/Desktop/ArchLayer/System/source/Thun$ ipython
Python 3.8.2 (default, Mar 26 2020, 15:53:00) 
Type 'copyright', 'credits' or 'license' for more information
IPython 7.14.0 -- An enhanced Interactive Python. Type '?' for help.

In [1]: import joy.parser                                                                                             

In [2]: joy.parser.text_to_expression('\u2022')                                                                       
Out[2]: (•, ())

In [3]: joy.parser.text_to_expression(u' \u2022 ')                                                                    
Out[3]: (•, ())

In [4]:                                                                                                               
Do you really want to exit ([y]/n)? 
(SPF) sforman@bock:~/Desktop/ArchLayer/System/source/Thun$ conda deactivate
sforman@bock:~/Desktop/ArchLayer/System/source/Thun$ ipython
Python 2.7.12 (default, Apr 15 2020, 17:07:12) 
Type "copyright", "credits" or "license" for more information.

IPython 5.6.0 -- An enhanced Interactive Python.
?         -> Introduction and overview of IPython's features.
%quickref -> Quick reference.
help      -> Python's own help system.
object?   -> Details about 'object', use 'object??' for extra details.

In [1]: import joy.parser

In [2]: joy.parser.text_to_expression('\u2022')
---------------------------------------------------------------------------
ParseError                                Traceback (most recent call last)
<ipython-input-2-664675d3b499> in <module>()
----> 1 joy.parser.text_to_expression('\u2022')

/home/sforman/Desktop/ArchLayer/System/source/Thun/joy/parser.pyc in text_to_expression(text)
     67         :raises ParseError: if the parse fails.
     68 	'''
---> 69         return _parse(_tokenize(text))
     70 
     71 

/home/sforman/Desktop/ArchLayer/System/source/Thun/joy/parser.pyc in _tokenize(text)
     85 		raise ParseError(
     86                         'Scan failed at position %i, %r'
---> 87                         % (len(text) - len(rest), rest[:10])
     88 			)
     89         return tokens

ParseError: Scan failed at position 0, '\\u2022'

Python 2 scanner doesn't scan unicode.

任务单历史 (3/3 Histories)

2020-05-18 00:11 Updated by: sforman

• New Ticket "Python 2 scanner doesn't scan unicode." created

2020-05-18 03:58 Updated by: sforman

• 处理结果 Update from 无 to Works For Me
• 状态 Update from 开启 to 关闭

评论 回复

After wrestling with this for a minute I got it so that eval'ing • (bullet Unicode character) works in GUI in Py 2 & 3, and REPL Py 3, but not REPL Py 2.

Good enough.

2020-05-18 04:24 Updated by: sforman

评论 回复

I was wrong, it doesn't work in GUI in py 2 either.

I think the best thing is to ditch Python 2. As long as 2 compatibility was free I figured "why not?" but if it's causing problems...

This is meant to be living code.

Traceback (most recent call last):
  File "joy/gui/textwidget.py", line 306, in run_command
    self.world.interpret(command)
  File "joy/gui/world.py", line 134, in interpret
    super(StackDisplayWorld, self).interpret(command)
  File "joy/gui/world.py", line 98, in interpret
    self.dictionary,
  File "joy/joy.py", line 77, in run
    expression = text_to_expression(text)
  File "joy/parser.py", line 69, in text_to_expression
    return _parse(_tokenize(text))
  File "joy/parser.py", line 83, in _tokenize
    tokens, rest = _scanner.scan(text)
  File "/usr/lib/python2.7/re.py", line 336, in scan
    action = action(self, m.group())
  File "joy/parser.py", line 119, in <lambda>
    (SYMBOL, lambda _, token: Symbol(token)),
UnicodeEncodeError: 'ascii' codec can't encode character u'\u2022' in position 0: ordinal not in range(128)