CVS commit: pkgsrc/www/py-scrapy

["Adam Ciarcinski" <adam%netbsd.org@localhost>]

Module Name:    pkgsrc
Committed By:   adam
Date:           Thu Jan  4 21:31:41 UTC 2018

Modified Files:
        pkgsrc/www/py-scrapy: Makefile PLIST distinfo

Log Message:
py-scrapy: updated to 1.5.0

Scrapy 1.5.0:
This release brings small new features and improvements across the codebase.
Some highlights:

* Google Cloud Storage is supported in FilesPipeline and ImagesPipeline.
* Crawling with proxy servers becomes more efficient, as connections
  to proxies can be reused now.
* Warnings, exception and logging messages are improved to make debugging
  easier.
* scrapy parse command now allows to set custom request meta via
  --meta argument.
* Compatibility with Python 3.6, PyPy and PyPy3 is improved;
  PyPy and PyPy3 are now supported officially, by running tests on CI.
* Better default handling of HTTP 308, 522 and 524 status codes.
* Documentation is improved, as usual.

Backwards Incompatible Changes
* Scrapy 1.5 drops support for Python 3.3.
* Default Scrapy User-Agent now uses https link to scrapy.org.
  **This is technically backwards-incompatible**; override
  :setting:USER_AGENT if you relied on old value.
* Logging of settings overridden by custom_settings is fixed;
  **this is technically backwards-incompatible** because the logger
  changes from [scrapy.utils.log] to [scrapy.crawler]. If you're
  parsing Scrapy logs, please update your log parsers.
* LinkExtractor now ignores m4v extension by default, this is change
  in behavior.
* 522 and 524 status codes are added to RETRY_HTTP_CODES

New features
- Support <link> tags in Response.follow
- Support for ptpython REPL
- Google Cloud Storage support for FilesPipeline and ImagesPipeline
- New --meta option of the "scrapy parse" command allows to pass additional
  request.meta
- Populate spider variable when using shell.inspect_response
- Handle HTTP 308 Permanent Redirect
- Add 522 and 524 to RETRY_HTTP_CODES
- Log versions information at startup
- scrapy.mail.MailSender now works in Python 3 (it requires Twisted 17.9.0)
- Connections to proxy servers are reused
- Add template for a downloader middleware
- Explicit message for NotImplementedError when parse callback not defined
- CrawlerProcess got an option to disable installation of root log handler
- LinkExtractor now ignores m4v extension by default
- Better log messages for responses over :setting:DOWNLOAD_WARNSIZE and
  :setting:DOWNLOAD_MAXSIZE limits
- Show warning when a URL is put to Spider.allowed_domains instead of
  a domain.

Bug fixes
- Fix logging of settings overridden by custom_settings;
  **this is technically backwards-incompatible** because the logger
  changes from [scrapy.utils.log] to [scrapy.crawler], so please
  update your log parsers if needed
- Default Scrapy User-Agent now uses https link to scrapy.org.
  **This is technically backwards-incompatible**; override
  :setting:USER_AGENT if you relied on old value.
- Fix PyPy and PyPy3 test failures, support them officially
- Fix DNS resolver when DNSCACHE_ENABLED=False
- Add cryptography for Debian Jessie tox test env
- Add verification to check if Request callback is callable
- Port extras/qpsclient.py to Python 3
- Use getfullargspec under the scenes for Python 3 to stop DeprecationWarning
- Update deprecated test aliases
- Fix SitemapSpider support for alternate links


To generate a diff of this commit:
cvs rdiff -u -r1.4 -r1.5 pkgsrc/www/py-scrapy/Makefile
cvs rdiff -u -r1.2 -r1.3 pkgsrc/www/py-scrapy/PLIST
cvs rdiff -u -r1.3 -r1.4 pkgsrc/www/py-scrapy/distinfo

Please note that diffs are not public domain; they are subject to the
copyright notices on the relevant files.

Modified files:

Index: pkgsrc/www/py-scrapy/Makefile
diff -u pkgsrc/www/py-scrapy/Makefile:1.4 pkgsrc/www/py-scrapy/Makefile:1.5
--- pkgsrc/www/py-scrapy/Makefile:1.4   Mon Sep  4 18:08:30 2017
+++ pkgsrc/www/py-scrapy/Makefile       Thu Jan  4 21:31:41 2018
@@ -1,6 +1,6 @@
-# $NetBSD: Makefile,v 1.4 2017/09/04 18:08:30 wiz Exp $
+# $NetBSD: Makefile,v 1.5 2018/01/04 21:31:41 adam Exp $
 
-DISTNAME=      Scrapy-1.4.0
+DISTNAME=      Scrapy-1.5.0
 PKGNAME=       ${PYPKGPREFIX}-${DISTNAME:tl}
 CATEGORIES=    www python
 MASTER_SITES=  ${MASTER_SITE_PYPI:=S/Scrapy/}
@@ -10,10 +10,16 @@ HOMEPAGE=   https://scrapy.org/
 COMMENT=       High-level Web Crawling and Web Scraping framework
 LICENSE=       modified-bsd
 
-DEPENDS+=      ${PYPKGPREFIX}-queuelib-[0-9]*:../../devel/py-queuelib
+DEPENDS+=      ${PYPKGPREFIX}-OpenSSL-[0-9]*:../../security/py-OpenSSL
+DEPENDS+=      ${PYPKGPREFIX}-cssselect-[0-9]*:../../textproc/py-cssselect
+DEPENDS+=      ${PYPKGPREFIX}-lxml-[0-9]*:../../textproc/py-lxml
 DEPENDS+=      ${PYPKGPREFIX}-parsel>=1.1.0:../../www/py-parsel
 DEPENDS+=      ${PYPKGPREFIX}-pydispatcher>=2.0.5:../../devel/py-pydispatcher
+DEPENDS+=      ${PYPKGPREFIX}-queuelib-[0-9]*:../../devel/py-queuelib
+DEPENDS+=      ${PYPKGPREFIX}-service_identity-[0-9]*:../../security/py-service_identity
+DEPENDS+=      ${PYPKGPREFIX}-six>=1.5.2:../../lang/py-six
 DEPENDS+=      ${PYPKGPREFIX}-twisted>=13.1.0:../../net/py-twisted
+DEPENDS+=      ${PYPKGPREFIX}-w3lib>=1.17.0:../../www/py-w3lib
 
 post-install:
        ${MV} ${DESTDIR}${PREFIX}/bin/scrapy \

Index: pkgsrc/www/py-scrapy/PLIST
diff -u pkgsrc/www/py-scrapy/PLIST:1.2 pkgsrc/www/py-scrapy/PLIST:1.3
--- pkgsrc/www/py-scrapy/PLIST:1.2      Sat May 20 06:25:36 2017
+++ pkgsrc/www/py-scrapy/PLIST  Thu Jan  4 21:31:41 2018
@@ -1,4 +1,4 @@
-@comment $NetBSD: PLIST,v 1.2 2017/05/20 06:25:36 adam Exp $
+@comment $NetBSD: PLIST,v 1.3 2018/01/04 21:31:41 adam Exp $
 bin/scrapy${PYVERSSUFFIX}
 ${PYSITELIB}/${EGG_INFODIR}/PKG-INFO
 ${PYSITELIB}/${EGG_INFODIR}/SOURCES.txt
@@ -729,6 +729,9 @@ ${PYSITELIB}/scrapy/utils/trackref.pyo
 ${PYSITELIB}/scrapy/utils/url.py
 ${PYSITELIB}/scrapy/utils/url.pyc
 ${PYSITELIB}/scrapy/utils/url.pyo
+${PYSITELIB}/scrapy/utils/versions.py
+${PYSITELIB}/scrapy/utils/versions.pyc
+${PYSITELIB}/scrapy/utils/versions.pyo
 ${PYSITELIB}/scrapy/xlib/__init__.py
 ${PYSITELIB}/scrapy/xlib/__init__.pyc
 ${PYSITELIB}/scrapy/xlib/__init__.pyo

Index: pkgsrc/www/py-scrapy/distinfo
diff -u pkgsrc/www/py-scrapy/distinfo:1.3 pkgsrc/www/py-scrapy/distinfo:1.4
--- pkgsrc/www/py-scrapy/distinfo:1.3   Sat May 20 06:25:36 2017
+++ pkgsrc/www/py-scrapy/distinfo       Thu Jan  4 21:31:41 2018
@@ -1,6 +1,6 @@
-$NetBSD: distinfo,v 1.3 2017/05/20 06:25:36 adam Exp $
+$NetBSD: distinfo,v 1.4 2018/01/04 21:31:41 adam Exp $
 
-SHA1 (Scrapy-1.4.0.tar.gz) = 24222debf2e6b9220a91a56c476c208ac5ecb8e5
-RMD160 (Scrapy-1.4.0.tar.gz) = ef20b9288851962fb552c1045e297c8917a74d17
-SHA512 (Scrapy-1.4.0.tar.gz) = eedcd7003c51f45a580f160b4e5f428c01713e4ecb5b64e35570bc750fc03bef7cc991e318ef4ff9c96e12a2d21cc32d0f07ce278486ea2e65f08e53c3e4a8f1
-Size (Scrapy-1.4.0.tar.gz) = 898159 bytes
+SHA1 (Scrapy-1.5.0.tar.gz) = 466a6e502585507f0bdd711043a5474ba0f3899d
+RMD160 (Scrapy-1.5.0.tar.gz) = 083f584cbe11a9382eef6314829f891f3b2a3b9d
+SHA512 (Scrapy-1.5.0.tar.gz) = b2fb3bc58ab2fe64b8527c9b33478e9bb5239a15c793147d7e1af2827daf2de219c506e07596cdd5ff1ed51a2f489028b29f9ffa8b729125098892dea35d8b50
+Size (Scrapy-1.5.0.tar.gz) = 905439 bytes