2月 – 2016 – 📔独行看客的笔记本

在Mac中用pip安装Scrapy

薛定谔的nil2016年2月11日 0 Comments

遇到了两个大问题，第一个在这里已经解决。

第二个问题是：

creating build/temp.macosx-10.11-intel-2.7/src/lxml
cc -fno-strict-aliasing -fno-common -dynamic -arch x86_64 -arch i386 -g -Os -pipe -fno-common -fno-strict-aliasing -fwrapv -DENABLE_DTRACE -DMACOSX -DNDEBUG -Wall -Wstrict-prototypes -Wshorten-64-to-32 -DNDEBUG -g -fwrapv -Os -Wall -Wstrict-prototypes -DENABLE_DTRACE -arch x86_64 -arch i386 -pipe -I/usr/include/libxml2 -Isrc/lxml/includes -I/System/Library/Frameworks/Python.framework/Versions/2.7/include/python2.7 -c src/lxml/lxml.etree.c -o build/temp.macosx-10.11-intel-2.7/src/lxml/lxml.etree.o -w -flat_namespace
In file included from src/lxml/lxml.etree.c:323:
src/lxml/includes/etree_defs.h:14:10: fatal error: 'libxml/xmlversion.h' file not found
#include "libxml/xmlversion.h"
^
1 error generated.
Compile failed: command 'cc' failed with exit status 1
creating tmp
cc -I/usr/include/libxml2 -I/usr/include/libxml2 -c /tmp/xmlXPathInitb43sL5.c -o tmp/xmlXPathInitb43sL5.o
/tmp/xmlXPathInitb43sL5.c:1:10: fatal error: 'libxml/xpath.h' file not found
#include "libxml/xpath.h"
^
1 error generated.
*********************************************************************************
Could not find function xmlCheckVersion in library libxml2. Is libxml2 installed?
Perhaps try: xcode-select --install
*********************************************************************************
error: command 'cc' failed with exit status 1

----------------------------------------
Command "/usr/bin/python -u -c "import setuptools, tokenize;__file__='/private/tmp/pip-build-PmvVPX/lxml/setup.py';exec(compile(getattr(tokenize, 'open', open)(__file__).read().replace('\r\n', '\n'), __file__, 'exec'))" install --record /tmp/pip-uPgqEz-record/install-record.txt --single-version-externally-managed --compile" failed with error code 1 in /private/tmp/pip-build-PmvVPX/lxml

creating build/temp.macosx-10.11-intel-2.7/src/lxml

cc -fno-strict-aliasing -fno-common -dynamic -arch x86_64 -arch i386 -g -Os -pipe -fno-common -fno-strict-aliasing -fwrapv -DENABLE_DTRACE -DMACOSX -DNDEBUG -Wall -Wstrict-prototypes -Wshorten-64-to-32 -DNDEBUG -g -fwrapv -Os -Wall -Wstrict-prototypes -DENABLE_DTRACE -arch x86_64 -arch i386 -pipe -I/usr/include/libxml2 -Isrc/lxml/includes -I/System/Library/Frameworks/Python.framework/Versions/2.7/include/python2.7 -c src/lxml/lxml.etree.c -o build/temp.macosx-10.11-intel-2.7/src/lxml/lxml.etree.o -w -flat_namespace

In file included from src/lxml/lxml.etree.c:323:

src/lxml/includes/etree_defs.h:14:10: fatal error: 'libxml/xmlversion.h' file not found

#include "libxml/xmlversion.h"

1 error generated.

Compile failed: command 'cc' failed with exit status 1

creating tmp

cc -I/usr/include/libxml2 -I/usr/include/libxml2 -c /tmp/xmlXPathInitb43sL5.c -o tmp/xmlXPathInitb43sL5.o

/tmp/xmlXPathInitb43sL5.c:1:10: fatal error: 'libxml/xpath.h' file not found

#include "libxml/xpath.h"

1 error generated.

*********************************************************************************

Could not find function xmlCheckVersion in library libxml2. Is libxml2 installed?

Perhaps try: xcode-select --install

*********************************************************************************

error: command 'cc' failed with exit status 1

----------------------------------------

Command "/usr/bin/python -u -c "import setuptools, tokenize;__file__='/private/tmp/pip-build-PmvVPX/lxml/setup.py';exec(compile(getattr(tokenize, 'open', open)(__file__).read().replace('\r\n', '\n'), __file__, 'exec'))" install --record /tmp/pip-uPgqEz-record/install-record.txt --single-version-externally-managed --compile" failed with error code 1 in /private/tmp/pip-build-PmvVPX/lxml

我使用了这个文章里的解决办法：

MacBook-Pro:~ EverybodyLies$ sudo STATIC_DEPS=true pip install lxml
Password:
The directory '/Users/EverybodyLies/Library/Caches/pip/http' or its parent directory is not owned by the current user and the cache has been disabled. Please check the permissions and owner of that directory. If executing pip with sudo, you may want sudo's -H flag.
The directory '/Users/EverybodyLies/Library/Caches/pip' or its parent directory is not owned by the current user and caching wheels has been disabled. check the permissions and owner of that directory. If executing pip with sudo, you may want sudo's -H flag.
Collecting lxml
Downloading lxml-3.5.0.tar.gz (3.8MB)
100% |████████████████████████████████| 3.8MB 120kB/s
Installing collected packages: lxml
Running setup.py install for lxml ... done
Successfully installed lxml-3.5.0

MacBook-Pro:~ EverybodyLies$ sudo STATIC_DEPS=true pip install lxml

Password:

The directory '/Users/EverybodyLies/Library/Caches/pip/http' or its parent directory is not owned by the current user and the cache has been disabled. Please check the permissions and owner of that directory. If executing pip with sudo, you may want sudo's -H flag.

The directory '/Users/EverybodyLies/Library/Caches/pip' or its parent directory is not owned by the current user and caching wheels has been disabled. check the permissions and owner of that directory. If executing pip with sudo, you may want sudo's -H flag.

Collecting lxml

Downloading lxml-3.5.0.tar.gz (3.8MB)

100% |████████████████████████████████| 3.8MB 120kB/s

Installing collected packages: lxml

Running setup.py install for lxml ... done

Successfully installed lxml-3.5.0

成功安装lxml后，再运行：

MacBook-Pro:~ EverybodyLies$ sudo pip install Scrapy
The directory '/Users/EverybodyLies/Library/Caches/pip/http' or its parent directory is not owned by the current user and the cache has been disabled. Please check the permissions and owner of that directory. If executing pip with sudo, you may want sudo's -H flag.
The directory '/Users/EverybodyLies/Library/Caches/pip' or its parent directory is not owned by the current user and caching wheels has been disabled. check the permissions and owner of that directory. If executing pip with sudo, you may want sudo's -H flag.
Collecting Scrapy
Downloading Scrapy-1.0.5-py2-none-any.whl (291kB)
100% |████████████████████████████████| 294kB 257kB/s
Requirement already satisfied (use --upgrade to upgrade): cssselect>=0.9 in /Library/Python/2.7/site-packages (from Scrapy)
Requirement already satisfied (use --upgrade to upgrade): queuelib in /Library/Python/2.7/site-packages (from Scrapy)
Requirement already satisfied (use --upgrade to upgrade): pyOpenSSL in /Library/Python/2.7/site-packages (from Scrapy)
Requirement already satisfied (use --upgrade to upgrade): w3lib>=1.8.0 in /Library/Python/2.7/site-packages (from Scrapy)
Requirement already satisfied (use --upgrade to upgrade): lxml in /Library/Python/2.7/site-packages (from Scrapy)
Collecting Twisted>=10.0.0 (from Scrapy)
Downloading Twisted-15.5.0.tar.bz2 (3.1MB)
100% |████████████████████████████████| 3.1MB 105kB/s
Requirement already satisfied (use --upgrade to upgrade): six>=1.5.2 in /Library/Python/2.7/site-packages (from Scrapy)
Requirement already satisfied (use --upgrade to upgrade): service-identity in /Library/Python/2.7/site-packages (from Scrapy)
Requirement already satisfied (use --upgrade to upgrade): cryptography>=0.7 in /Library/Python/2.7/site-packages (from pyOpenSSL->Scrapy)
Requirement already satisfied (use --upgrade to upgrade): zope.interface>=3.6.0 in /System/Library/Frameworks/Python.framework/Versions/2.7/Extras/lib/python (from Twisted>=10.0.0->Scrapy)
Requirement already satisfied (use --upgrade to upgrade): characteristic>=14.0.0 in /Library/Python/2.7/site-packages (from service-identity->Scrapy)
Requirement already satisfied (use --upgrade to upgrade): pyasn1-modules in /Library/Python/2.7/site-packages (from service-identity->Scrapy)
Requirement already satisfied (use --upgrade to upgrade): pyasn1 in /Library/Python/2.7/site-packages (from service-identity->Scrapy)
Requirement already satisfied (use --upgrade to upgrade): enum34 in /Library/Python/2.7/site-packages (from cryptography>=0.7->pyOpenSSL->Scrapy)
Requirement already satisfied (use --upgrade to upgrade): ipaddress in /Library/Python/2.7/site-packages (from cryptography>=0.7->pyOpenSSL->Scrapy)
Requirement already satisfied (use --upgrade to upgrade): setuptools>=1.0 in /Library/Python/2.7/site-packages (from cryptography>=0.7->pyOpenSSL->Scrapy)
Requirement already satisfied (use --upgrade to upgrade): idna>=2.0 in /Library/Python/2.7/site-packages (from cryptography>=0.7->pyOpenSSL->Scrapy)
Requirement already satisfied (use --upgrade to upgrade): cffi>=1.4.1 in /Library/Python/2.7/site-packages (from cryptography>=0.7->pyOpenSSL->Scrapy)
Requirement already satisfied (use --upgrade to upgrade): pycparser in /Library/Python/2.7/site-packages (from cffi>=1.4.1->cryptography>=0.7->pyOpenSSL->Scrapy)
Installing collected packages: Twisted, Scrapy
Running setup.py install for Twisted ... done
Successfully installed Scrapy-1.0.5 Twisted-15.5.0

MacBook-Pro:~ EverybodyLies$ sudo pip install Scrapy

Collecting Scrapy

Downloading Scrapy-1.0.5-py2-none-any.whl (291kB)

100% |████████████████████████████████| 294kB 257kB/s

Requirement already satisfied (use --upgrade to upgrade): cssselect>=0.9 in /Library/Python/2.7/site-packages (from Scrapy)

Requirement already satisfied (use --upgrade to upgrade): queuelib in /Library/Python/2.7/site-packages (from Scrapy)

Requirement already satisfied (use --upgrade to upgrade): pyOpenSSL in /Library/Python/2.7/site-packages (from Scrapy)

Requirement already satisfied (use --upgrade to upgrade): w3lib>=1.8.0 in /Library/Python/2.7/site-packages (from Scrapy)

Requirement already satisfied (use --upgrade to upgrade): lxml in /Library/Python/2.7/site-packages (from Scrapy)

Collecting Twisted>=10.0.0 (from Scrapy)

Downloading Twisted-15.5.0.tar.bz2 (3.1MB)

100% |████████████████████████████████| 3.1MB 105kB/s

Requirement already satisfied (use --upgrade to upgrade): six>=1.5.2 in /Library/Python/2.7/site-packages (from Scrapy)

Requirement already satisfied (use --upgrade to upgrade): service-identity in /Library/Python/2.7/site-packages (from Scrapy)

Requirement already satisfied (use --upgrade to upgrade): cryptography>=0.7 in /Library/Python/2.7/site-packages (from pyOpenSSL->Scrapy)

Requirement already satisfied (use --upgrade to upgrade): zope.interface>=3.6.0 in /System/Library/Frameworks/Python.framework/Versions/2.7/Extras/lib/python (from Twisted>=10.0.0->Scrapy)

Requirement already satisfied (use --upgrade to upgrade): characteristic>=14.0.0 in /Library/Python/2.7/site-packages (from service-identity->Scrapy)

Requirement already satisfied (use --upgrade to upgrade): pyasn1-modules in /Library/Python/2.7/site-packages (from service-identity->Scrapy)

Requirement already satisfied (use --upgrade to upgrade): pyasn1 in /Library/Python/2.7/site-packages (from service-identity->Scrapy)

Requirement already satisfied (use --upgrade to upgrade): enum34 in /Library/Python/2.7/site-packages (from cryptography>=0.7->pyOpenSSL->Scrapy)

Requirement already satisfied (use --upgrade to upgrade): ipaddress in /Library/Python/2.7/site-packages (from cryptography>=0.7->pyOpenSSL->Scrapy)

Requirement already satisfied (use --upgrade to upgrade): setuptools>=1.0 in /Library/Python/2.7/site-packages (from cryptography>=0.7->pyOpenSSL->Scrapy)

Requirement already satisfied (use --upgrade to upgrade): idna>=2.0 in /Library/Python/2.7/site-packages (from cryptography>=0.7->pyOpenSSL->Scrapy)

Requirement already satisfied (use --upgrade to upgrade): cffi>=1.4.1 in /Library/Python/2.7/site-packages (from cryptography>=0.7->pyOpenSSL->Scrapy)

Requirement already satisfied (use --upgrade to upgrade): pycparser in /Library/Python/2.7/site-packages (from cffi>=1.4.1->cryptography>=0.7->pyOpenSSL->Scrapy)

Installing collected packages: Twisted, Scrapy

Running setup.py install for Twisted ... done

Successfully installed Scrapy-1.0.5 Twisted-15.5.0

安装成功。

在OS X 10.11 (El Capitan)中用pip安装Scrapy遇到无法更新six包的错误

薛定谔的nil2016年2月11日 0 Comments

在OSX 10.11 (El Capitan) 中用pip安装Scrapy时遇到了这样的问题：

MacBook-Pro:~ EverybodyLies$ sudo pip install Scrapy
The directory '/Users/EverybodyLies/Library/Caches/pip/http' or its parent directory is not owned by the current user and the cache has been disabled. Please check the permissions and owner of that directory. If executing pip with sudo, you may want sudo's -H flag.
The directory '/Users/EverybodyLies/Library/Caches/pip' or its parent directory is not owned by the current user and caching wheels has been disabled. check the permissions and owner of that directory. If executing pip with sudo, you may want sudo's -H flag.
Collecting Scrapy
Downloading Scrapy-1.0.5-py2-none-any.whl (291kB)
100% |████████████████████████████████| 294kB 1.5MB/s
Collecting cssselect>=0.9 (from Scrapy)
Downloading cssselect-0.9.1.tar.gz
Collecting queuelib (from Scrapy)
Downloading queuelib-1.4.2-py2.py3-none-any.whl
Requirement already satisfied (use --upgrade to upgrade): pyOpenSSL in /System/Library/Frameworks/Python.framework/Versions/2.7/Extras/lib/python (from Scrapy)
Collecting w3lib>=1.8.0 (from Scrapy)
Downloading w3lib-1.13.0-py2.py3-none-any.whl
Collecting lxml (from Scrapy)
Downloading lxml-3.5.0.tar.gz (3.8MB)
100% |████████████████████████████████| 3.8MB 106kB/s
Collecting Twisted>=10.0.0 (from Scrapy)
Downloading Twisted-15.5.0.tar.bz2 (3.1MB)
100% |████████████████████████████████| 3.1MB 127kB/s
Collecting six>=1.5.2 (from Scrapy)
Downloading six-1.10.0-py2.py3-none-any.whl
Requirement already satisfied (use --upgrade to upgrade): service-identity in /Library/Python/2.7/site-packages (from Scrapy)
Requirement already satisfied (use --upgrade to upgrade): zope.interface>=3.6.0 in /System/Library/Frameworks/Python.framework/Versions/2.7/Extras/lib/python (from Twisted>=10.0.0->Scrapy)
Requirement already satisfied (use --upgrade to upgrade): characteristic>=14.0.0 in /Library/Python/2.7/site-packages (from service-identity->Scrapy)
Requirement already satisfied (use --upgrade to upgrade): pyasn1-modules in /Library/Python/2.7/site-packages (from service-identity->Scrapy)
Requirement already satisfied (use --upgrade to upgrade): pyasn1 in /Library/Python/2.7/site-packages (from service-identity->Scrapy)
Requirement already satisfied (use --upgrade to upgrade): setuptools in /System/Library/Frameworks/Python.framework/Versions/2.7/Extras/lib/python (from zope.interface>=3.6.0->Twisted>=10.0.0->Scrapy)
Installing collected packages: cssselect, queuelib, six, w3lib, lxml, Twisted, Scrapy
Running setup.py install for cssselect ... done
Found existing installation: six 1.4.1
DEPRECATION: Uninstalling a distutils installed project (six) has been deprecated and will be removed in a future version. This is due to the fact that uninstalling a distutils project will only partially uninstall the project.
Uninstalling six-1.4.1:
Exception:
Traceback (most recent call last):
File "/Library/Python/2.7/site-packages/pip/basecommand.py", line 209, in main
status = self.run(options, args)
File "/Library/Python/2.7/site-packages/pip/commands/install.py", line 317, in run
prefix=options.prefix_path,
File "/Library/Python/2.7/site-packages/pip/req/req_set.py", line 725, in install
requirement.uninstall(auto_confirm=True)
File "/Library/Python/2.7/site-packages/pip/req/req_install.py", line 752, in uninstall
paths_to_remove.remove(auto_confirm)
File "/Library/Python/2.7/site-packages/pip/req/req_uninstall.py", line 115, in remove
renames(path, new_path)
File "/Library/Python/2.7/site-packages/pip/utils/__init__.py", line 266, in renames
shutil.move(old, new)
File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/shutil.py", line 302, in move
copy2(src, real_dst)
File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/shutil.py", line 131, in copy2
copystat(src, dst)
File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/shutil.py", line 103, in copystat
os.chflags(dst, st.st_flags)
OSError: [Errno 1] Operation not permitted: '/tmp/pip-2O_yX_-uninstall/System/Library/Frameworks/Python.framework/Versions/2.7/Extras/lib/python/six-1.4.1-py2.7.egg-info'

MacBook-Pro:~ EverybodyLies$ sudo pip install Scrapy

Collecting Scrapy

Downloading Scrapy-1.0.5-py2-none-any.whl (291kB)

100% |████████████████████████████████| 294kB 1.5MB/s

Collecting cssselect>=0.9 (from Scrapy)

Downloading cssselect-0.9.1.tar.gz

Collecting queuelib (from Scrapy)

Downloading queuelib-1.4.2-py2.py3-none-any.whl

Requirement already satisfied (use --upgrade to upgrade): pyOpenSSL in /System/Library/Frameworks/Python.framework/Versions/2.7/Extras/lib/python (from Scrapy)

Collecting w3lib>=1.8.0 (from Scrapy)

Downloading w3lib-1.13.0-py2.py3-none-any.whl

Collecting lxml (from Scrapy)

Downloading lxml-3.5.0.tar.gz (3.8MB)

100% |████████████████████████████████| 3.8MB 106kB/s

Collecting Twisted>=10.0.0 (from Scrapy)

Downloading Twisted-15.5.0.tar.bz2 (3.1MB)

100% |████████████████████████████████| 3.1MB 127kB/s

Collecting six>=1.5.2 (from Scrapy)

Downloading six-1.10.0-py2.py3-none-any.whl

Requirement already satisfied (use --upgrade to upgrade): service-identity in /Library/Python/2.7/site-packages (from Scrapy)

Requirement already satisfied (use --upgrade to upgrade): zope.interface>=3.6.0 in /System/Library/Frameworks/Python.framework/Versions/2.7/Extras/lib/python (from Twisted>=10.0.0->Scrapy)

Requirement already satisfied (use --upgrade to upgrade): characteristic>=14.0.0 in /Library/Python/2.7/site-packages (from service-identity->Scrapy)

Requirement already satisfied (use --upgrade to upgrade): pyasn1-modules in /Library/Python/2.7/site-packages (from service-identity->Scrapy)

Requirement already satisfied (use --upgrade to upgrade): pyasn1 in /Library/Python/2.7/site-packages (from service-identity->Scrapy)

Requirement already satisfied (use --upgrade to upgrade): setuptools in /System/Library/Frameworks/Python.framework/Versions/2.7/Extras/lib/python (from zope.interface>=3.6.0->Twisted>=10.0.0->Scrapy)

Installing collected packages: cssselect, queuelib, six, w3lib, lxml, Twisted, Scrapy

Running setup.py install for cssselect ... done

Found existing installation: six 1.4.1

DEPRECATION: Uninstalling a distutils installed project (six) has been deprecated and will be removed in a future version. This is due to the fact that uninstalling a distutils project will only partially uninstall the project.

Uninstalling six-1.4.1:

Exception:

Traceback (most recent call last):

File "/Library/Python/2.7/site-packages/pip/basecommand.py", line 209, in main

status = self.run(options, args)

File "/Library/Python/2.7/site-packages/pip/commands/install.py", line 317, in run

prefix=options.prefix_path,

File "/Library/Python/2.7/site-packages/pip/req/req_set.py", line 725, in install

requirement.uninstall(auto_confirm=True)

File "/Library/Python/2.7/site-packages/pip/req/req_install.py", line 752, in uninstall

paths_to_remove.remove(auto_confirm)

File "/Library/Python/2.7/site-packages/pip/req/req_uninstall.py", line 115, in remove

renames(path, new_path)

File "/Library/Python/2.7/site-packages/pip/utils/__init__.py", line 266, in renames

shutil.move(old, new)

File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/shutil.py", line 302, in move

copy2(src, real_dst)

File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/shutil.py", line 131, in copy2

copystat(src, dst)

File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/shutil.py", line 103, in copystat

os.chflags(dst, st.st_flags)

OSError: [Errno 1] Operation not permitted: '/tmp/pip-2O_yX_-uninstall/System/Library/Frameworks/Python.framework/Versions/2.7/Extras/lib/python/six-1.4.1-py2.7.egg-info'

去网上查找解决方法，看到了这个：“OSError: [Errno 1] Operation not permitted” when installing Scrapy in OSX 10.11 (El Capitan) (System Integrity Protection)，发现问题一模一样，于是就按照回答开始尝试。

按照最高票回答的方法尝试手动更新six：

MacBook-Pro:~ EverybodyLies$ sudo -H pip install --ignore-installed six
Password:
Collecting six
Using cached six-1.10.0-py2.py3-none-any.whl
Installing collected packages: six
Successfully installed six-1.4.1

MacBook-Pro:~ EverybodyLies$ sudo -H pip install --ignore-installed six

Password:

Collecting six

Using cached six-1.10.0-py2.py3-none-any.whl

Installing collected packages: six

Successfully installed six-1.4.1

然而并没有成功更新！于是去看了：Source: github.com/pypa/pip/issues/3165

按照里面的某些讨论进行尝试：

MacBook-Pro:~ EverybodyLies$ sudo -H pip install Scrapy --upgrade --ignore-installed six

1	MacBook-Pro:~ EverybodyLies$ sudo -H pip install Scrapy --upgrade --ignore-installed six

的确没有再次报错，但是我觉得并没有解决这个问题（后来检查日志应该是更新成功了）。

另一个同学在尝试中使用easy_install升级six，解决了这个问题，我也进行了这种尝试：

MacBook-Pro:~ EverybodyLies$ sudo easy_install "six>=1.5.2"
Password:
Searching for six>=1.5.2
Best match: six 1.10.0
Adding six 1.10.0 to easy-install.pth file

Using /Library/Python/2.7/site-packages
Processing dependencies for six>=1.5.2
Finished processing dependencies for six>=1.5.2

MacBook-Pro:~ EverybodyLies$ sudo easy_install "six>=1.5.2"

Password:

Searching for six>=1.5.2

Best match: six 1.10.0

Adding six 1.10.0 to easy-install.pth file

Using /Library/Python/2.7/site-packages

Processing dependencies for six>=1.5.2

Finished processing dependencies for six>=1.5.2

非常顺利就能完成six的升级。

一些思考：

在解决这个问题的过程中我一开始觉得是10.11中SIP的问题，而且很多文章也说是SIP的问题。

但是后来另一个同学用easy_install很容易的就升级了six，我突然意识到可能是six这个包本来是用easy_install安装的，因此pip才无法卸载six。

Month: 2月 2016

在Mac中用pip安装Scrapy

在OS X 10.11 (El Capitan)中用pip安装Scrapy遇到无法更新six包的错误

其他操作