CVS commit: pkgsrc/lang/py-uncompyle6

["Adam Ciarcinski" <adam%netbsd.org@localhost>]

Module Name:    pkgsrc
Committed By:   adam
Date:           Thu Feb 13 19:27:36 UTC 2020

Modified Files:
        pkgsrc/lang/py-uncompyle6: Makefile PLIST distinfo

Log Message:
py-uncompyle6: updated to 3.6.4

3.6.4:
plateau

The main focus in this release was fix some of the more glaring problems creapt in from the last release due to that refactor.

uncompyle6 code is at a plateau where what is most needed is a code refactoring. In doing this, until everything refactored and replaced, decomplation may get worse.

Therefore, this release largely serves as a checkpoint before more major upheaval.

The upheaval, in started last release, I believe the pinnicle was around c90ff51 which wasn't a release. I suppose I should tag that.

After c90ff5, I started down the road of redoing control flow in a more comprehensible, debuggable, and scalable way. See The Control Flow Mess.

The bulk of the refactoring going on in the decompyle3 project, but I try to trickle down the changes.

It is tricky because the changes are large and I have to figure decompose things so that little testable pieces can be done. And there is also the problem that what is in decompyle3 is incomplete as 
well.

Other than control flow, another change that will probably happen in the next release is to redo the grammar for lambda expressions. Right now, we treat them as Python statements, you know, things 
with compound statements in them. But lambdas aren't that. And so there is hackery to paper over difference making a statement out of an expression the wrong thing to do. For example, a return of an 
"and" expression can be expressed as nested "if" statements with return inside them, but the "if" variant of the bytecode is not valid in a lambda.

In the decompyle3 code, I've gone down the road making the grammar goal symbol be an expression. This also offers the opportunity to split the grammar making parsing inside lambda not only more 
reliable because the wrong choices don't exist, but also simpler and faster because all those rules just need don't need to exist in parsing.

I cringe in thinking about how the code has lived for so long without noticing such a simple stupidity, and lapse of sufficient thought.

3.6.3:
Martin and Susanne

Of late, every release fixes major gaps and embarrassments of the last release....

And in some cases, like this one, exposes lacuna and rot.

I now have [control] flow under control, even if it isn't the most optimal way.

I now have greatly expanded automated testing.

On the most recent Python versions I regularly decompile thousands of Python programs that are distributed with Python. when it is possible, I then decompile Python's standard test suite distributed 
with Python and run the decompiled source code which basically checks itself. This amounts to about 250 test programs per version. This is in addition to the 3 CI testing services which do different 
things.

Does this mean the decompiler works perfectly? No. There are still a dozen or so failing programs, although the actual number of bugs is probably smaller though.

However, in perparation of a more major refactoring of the parser grammar, this release was born.

In many cases, decompilation is better. But there are some cases where decompilation has gotten worse. For lack of time (and interest) 3.0 bytecode suffered a hit. Possibly some code in the 3.x range 
did too. In time and with cleaner refactored code, this will come back.

Commit c90ff51 was a local maxiumum before, I started reworking the grammar to separate productions that were specific to loops versus those that are not in loops.
In the middle of that I added another grammar simplication to remove singleton productions of the form sstmts-> stmts. These were always was a bit ugly, and complicated output.

At any rate if decompilation fails, you can try c90ff51. Or another decompiler. unpyc37 is pretty good for 3.7. wibiti uncompyle2 is great for 2.7. pycdc is mediocre for Python before 3.5 or so, and 
not that good for the most recent Python. Generally these programs will give some sort of answer even if it isn't correct.

decompyle3 isn't that good for 3.7 and worse for 3.8, but right now it does things no other Python decompiler like unpyc37 or pycdc does. For example, decompyle3 handles variable annotations. As 
always, the issue trackers for the various programs will give you a sense for what needs to be done. For now, I've given up on reporting issues in the other decompilers because there are already 
enough issues reported, and they are just not getting fixed anyway.


To generate a diff of this commit:
cvs rdiff -u -r1.18 -r1.19 pkgsrc/lang/py-uncompyle6/Makefile
cvs rdiff -u -r1.11 -r1.12 pkgsrc/lang/py-uncompyle6/PLIST
cvs rdiff -u -r1.17 -r1.18 pkgsrc/lang/py-uncompyle6/distinfo

Please note that diffs are not public domain; they are subject to the
copyright notices on the relevant files.

Modified files:

Index: pkgsrc/lang/py-uncompyle6/Makefile
diff -u pkgsrc/lang/py-uncompyle6/Makefile:1.18 pkgsrc/lang/py-uncompyle6/Makefile:1.19
--- pkgsrc/lang/py-uncompyle6/Makefile:1.18     Wed Jan  8 15:19:14 2020
+++ pkgsrc/lang/py-uncompyle6/Makefile  Thu Feb 13 19:27:36 2020
@@ -1,6 +1,6 @@
-# $NetBSD: Makefile,v 1.18 2020/01/08 15:19:14 adam Exp $
+# $NetBSD: Makefile,v 1.19 2020/02/13 19:27:36 adam Exp $
 
-DISTNAME=      uncompyle6-3.6.2
+DISTNAME=      uncompyle6-3.6.4
 PKGNAME=       ${PYPKGPREFIX}-${DISTNAME}
 CATEGORIES=    lang python
 MASTER_SITES=  ${MASTER_SITE_PYPI:=u/uncompyle6/}

Index: pkgsrc/lang/py-uncompyle6/PLIST
diff -u pkgsrc/lang/py-uncompyle6/PLIST:1.11 pkgsrc/lang/py-uncompyle6/PLIST:1.12
--- pkgsrc/lang/py-uncompyle6/PLIST:1.11        Wed Jan  8 15:19:14 2020
+++ pkgsrc/lang/py-uncompyle6/PLIST     Thu Feb 13 19:27:36 2020
@@ -1,4 +1,4 @@
-@comment $NetBSD: PLIST,v 1.11 2020/01/08 15:19:14 adam Exp $
+@comment $NetBSD: PLIST,v 1.12 2020/02/13 19:27:36 adam Exp $
 bin/pydisassemble-${PYVERSSUFFIX}
 bin/uncompyle6-${PYVERSSUFFIX}
 ${PYSITELIB}/${EGG_INFODIR}/PKG-INFO
@@ -113,6 +113,51 @@ ${PYSITELIB}/uncompyle6/parsers/parse37b
 ${PYSITELIB}/uncompyle6/parsers/parse38.py
 ${PYSITELIB}/uncompyle6/parsers/parse38.pyc
 ${PYSITELIB}/uncompyle6/parsers/parse38.pyo
+${PYSITELIB}/uncompyle6/parsers/reducecheck/__init__.py
+${PYSITELIB}/uncompyle6/parsers/reducecheck/__init__.pyc
+${PYSITELIB}/uncompyle6/parsers/reducecheck/__init__.pyo
+${PYSITELIB}/uncompyle6/parsers/reducecheck/and_check.py
+${PYSITELIB}/uncompyle6/parsers/reducecheck/and_check.pyc
+${PYSITELIB}/uncompyle6/parsers/reducecheck/and_check.pyo
+${PYSITELIB}/uncompyle6/parsers/reducecheck/and_not_check.py
+${PYSITELIB}/uncompyle6/parsers/reducecheck/and_not_check.pyc
+${PYSITELIB}/uncompyle6/parsers/reducecheck/and_not_check.pyo
+${PYSITELIB}/uncompyle6/parsers/reducecheck/except_handler_else.py
+${PYSITELIB}/uncompyle6/parsers/reducecheck/except_handler_else.pyc
+${PYSITELIB}/uncompyle6/parsers/reducecheck/except_handler_else.pyo
+${PYSITELIB}/uncompyle6/parsers/reducecheck/ifelsestmt.py
+${PYSITELIB}/uncompyle6/parsers/reducecheck/ifelsestmt.pyc
+${PYSITELIB}/uncompyle6/parsers/reducecheck/ifelsestmt.pyo
+${PYSITELIB}/uncompyle6/parsers/reducecheck/iflaststmt.py
+${PYSITELIB}/uncompyle6/parsers/reducecheck/iflaststmt.pyc
+${PYSITELIB}/uncompyle6/parsers/reducecheck/iflaststmt.pyo
+${PYSITELIB}/uncompyle6/parsers/reducecheck/ifstmt.py
+${PYSITELIB}/uncompyle6/parsers/reducecheck/ifstmt.pyc
+${PYSITELIB}/uncompyle6/parsers/reducecheck/ifstmt.pyo
+${PYSITELIB}/uncompyle6/parsers/reducecheck/ifstmts_jump.py
+${PYSITELIB}/uncompyle6/parsers/reducecheck/ifstmts_jump.pyc
+${PYSITELIB}/uncompyle6/parsers/reducecheck/ifstmts_jump.pyo
+${PYSITELIB}/uncompyle6/parsers/reducecheck/or_check.py
+${PYSITELIB}/uncompyle6/parsers/reducecheck/or_check.pyc
+${PYSITELIB}/uncompyle6/parsers/reducecheck/or_check.pyo
+${PYSITELIB}/uncompyle6/parsers/reducecheck/testtrue.py
+${PYSITELIB}/uncompyle6/parsers/reducecheck/testtrue.pyc
+${PYSITELIB}/uncompyle6/parsers/reducecheck/testtrue.pyo
+${PYSITELIB}/uncompyle6/parsers/reducecheck/tryelsestmt.py
+${PYSITELIB}/uncompyle6/parsers/reducecheck/tryelsestmt.pyc
+${PYSITELIB}/uncompyle6/parsers/reducecheck/tryelsestmt.pyo
+${PYSITELIB}/uncompyle6/parsers/reducecheck/tryelsestmtl3.py
+${PYSITELIB}/uncompyle6/parsers/reducecheck/tryelsestmtl3.pyc
+${PYSITELIB}/uncompyle6/parsers/reducecheck/tryelsestmtl3.pyo
+${PYSITELIB}/uncompyle6/parsers/reducecheck/tryexcept.py
+${PYSITELIB}/uncompyle6/parsers/reducecheck/tryexcept.pyc
+${PYSITELIB}/uncompyle6/parsers/reducecheck/tryexcept.pyo
+${PYSITELIB}/uncompyle6/parsers/reducecheck/while1elsestmt.py
+${PYSITELIB}/uncompyle6/parsers/reducecheck/while1elsestmt.pyc
+${PYSITELIB}/uncompyle6/parsers/reducecheck/while1elsestmt.pyo
+${PYSITELIB}/uncompyle6/parsers/reducecheck/while1stmt.py
+${PYSITELIB}/uncompyle6/parsers/reducecheck/while1stmt.pyc
+${PYSITELIB}/uncompyle6/parsers/reducecheck/while1stmt.pyo
 ${PYSITELIB}/uncompyle6/parsers/treenode.py
 ${PYSITELIB}/uncompyle6/parsers/treenode.pyc
 ${PYSITELIB}/uncompyle6/parsers/treenode.pyo

Index: pkgsrc/lang/py-uncompyle6/distinfo
diff -u pkgsrc/lang/py-uncompyle6/distinfo:1.17 pkgsrc/lang/py-uncompyle6/distinfo:1.18
--- pkgsrc/lang/py-uncompyle6/distinfo:1.17     Wed Jan  8 15:19:14 2020
+++ pkgsrc/lang/py-uncompyle6/distinfo  Thu Feb 13 19:27:36 2020
@@ -1,6 +1,6 @@
-$NetBSD: distinfo,v 1.17 2020/01/08 15:19:14 adam Exp $
+$NetBSD: distinfo,v 1.18 2020/02/13 19:27:36 adam Exp $
 
-SHA1 (uncompyle6-3.6.2.tar.gz) = 5f5156784593d127bd8d14327693ef1786fcf39b
-RMD160 (uncompyle6-3.6.2.tar.gz) = e00ad186baa9f58477c52733e2123f175ae79b82
-SHA512 (uncompyle6-3.6.2.tar.gz) = 70fae57ed0edc2cc2f8f3701d190a72c9a5ce0ee4ca60f04f6f56a300edb7bacb5ab81b3184461eef27258d556b56379ef0a2ad670259935dd381e03c8907b26
-Size (uncompyle6-3.6.2.tar.gz) = 2307106 bytes
+SHA1 (uncompyle6-3.6.4.tar.gz) = e139e9a28428c924003af93d0de39ffa626b5d50
+RMD160 (uncompyle6-3.6.4.tar.gz) = 14a173b8c2a5cc4d29b5ed3222a949b4c18e2884
+SHA512 (uncompyle6-3.6.4.tar.gz) = 431da503cc6fac0d5a1118f9473be36b446a821f571c47f65f5d7fd2021a68657be6babc2693091340cae3bbd635a1e302e6c74230c5f6afe4961726dcbb98a3
+Size (uncompyle6-3.6.4.tar.gz) = 2347670 bytes