Python
From FarmShare
Contents |
Q
How to create a fresh virtualenv with the latest python?
A
After you login, create a directory where you would like to store your python3.4 environment. You can put this in your AFS home directory, or you can point it at your /farmshare/user_data/sunetid directory, which has a bit more space. For exampe; 'mkdir /farmshare/user_data/chekh/python3-env' then you can run 'virtualenv -p python3.4 /farmshare/user_data/chekh/python3-env' and it will create some files and a directory structure for you. Once that command has completed, run 'source /farmshare/user_data/chekh/python3-env/bin/activate' and then your fresh python environment will be active. You can check what you have installed with pip3 list, and then you can go ahead and install pysam with 'pip3 install pysam'
Q
Is there a way to have a module added to corn/barley’s Python, or is there another way I can include that module for these scripts? (Alternatively, I’ve thought about running a local version of Python and using that, instead of the global version, but that seems unnecessarily redundant.)
A
Date: Wed, 13 Mar 2013 12:32:24 -0700 (PDT) From: Jason Bishop <bishopj@stanford.edu> To: Open discussion for users of FarmShare <farmshare-discuss@lists.stanford.edu>
Hi Jeremy, I can think of 3 options off the top of my head. If anybody has a clever idea to improve please let me know.
first, a little bit of background: Since ubuntu provides a good up-to-date python environment, this is the main python environment we provide on farmshare. (This is not written in stone however, I have been considering looking to see if we can get any more speed out of numpy/scipy using ACML).
As such, python libraries can be managed in 3 ways:
1) The ubuntu provided package can be installed. This is easy, just send us an email research-computing-support@stanford.edu
There are quite a few packages, so there is a good chance your package is there:
$ aptitude search python | wc -l 3992
2) If the ubuntu package does not exist or is the wrong version, you can install it in your home directory.
For example, there is no ubuntu package for globusonline-transfer-api-client, so you can use pip to install it with --user flag to have it installed in your home directory (in .local)
$ pip install --user globusonline-transfer-api-client
Downloading/unpacking globusonline-transfer-api-client
Downloading globusonline-transfer-api-client-0.10.13.tar.gz Running setup.py egg_info for package globusonline-transfer-api-client
Installing collected packages: globusonline-transfer-api-client
Running setup.py install for globusonline-transfer-api-client
Successfully installed globusonline-transfer-api-client Cleaning up...
check to see if it is working (it is): $ python -c "from globusonline.transfer import api_client; print api_client.__path__" ['/afs/ir/users/b/i/bishopj/.local/lib/python2.7/site-packages/globusonline/transfer/api_client']
Taking a look in .local, I can see all of the packages installed with easy_install or pip:
$ ls .local/lib/python2.7/site-packages/ boto globusonline grapher.pyc lib boto-2.8.0.egg-info globusonline_transfer_api_client-0.10.13.egg-info ioapps.so tai64n-0.3.3-py2.7.egg easy-install.pth grapher.py ioprofiler-1.0.egg-info
3) If you have alot of packages which you want to keep as a self-contained group you can use virtualenv. This sounds like the situation you describe where you have python eggs and scripts from the paper. virtualenv also provides a nice way to collect and use the exact versions of python packages you want to use.
Virtualenv sets up a new python area with its own packages, but shares all of the python interpreter and library environment from the original.
For example, we will clean the home directory install of the globusonline api we just did:
$ rm -r /afs/ir.stanford.edu/users/b/i/bishopj/.local/lib/python2.7/site-packages/globusonline*
And then setup a virtualenv environment called mysandbox
$ virtualenv --system-site-packages mysandbox New python executable in mysandbox/bin/python Installing distribute.............................................................................................................................................................................................done. Installing pip...............done.
Then set your path to point to this environment so that python will execute from your mysandbox environment:
$ export PATH=$PWD/mysandbox/bin:$PATH
$ which python /mnt/glusterfs/bishopj/vee/mysandbox/bin/python
Now pip install will work fine, installing everything into mysandbox:
$ pip install globusonline-transfer-api-client Downloading/unpacking globusonline-transfer-api-client
Downloading globusonline-transfer-api-client-0.10.13.tar.gz Running setup.py egg_info for package globusonline-transfer-api-client
Installing collected packages: globusonline-transfer-api-client
Running setup.py install for globusonline-transfer-api-client
Successfully installed globusonline-transfer-api-client Cleaning up...
$ python -c "from globusonline.transfer import api_client; print api_client.__path__" ['/mnt/glusterfs/bishopj/vee/mysandbox/local/lib/python2.7/site-packages/globusonline/transfer/api_client']
You can install eggs too:
$ easy_install DDN_SFA_API-1.5.3.0.g_r12955-py2.7.egg Processing DDN_SFA_API-1.5.3.0.g_r12955-py2.7.egg creating /mnt/glusterfs/bishopj/vee/mysandbox/lib/python2.7/site-packages/DDN_SFA_API-1.5.3.0.g_r12955-py2.7.egg Extracting DDN_SFA_API-1.5.3.0.g_r12955-py2.7.egg to /mnt/glusterfs/bishopj/vee/mysandbox/lib/python2.7/site-packages Adding DDN-SFA-API 1.5.3.0.g-r12955 to easy-install.pth file
Installed /mnt/glusterfs/bishopj/vee/mysandbox/lib/python2.7/site-packages/DDN_SFA_API-1.5.3.0.g_r12955-py2.7.egg Processing dependencies for DDN-SFA-API==1.5.3.0.g-r12955 Searching for python-ntlm Reading http://pypi.python.org/simple/python-ntlm/ Reading http://code.google.com/p/python-ntlm Best match: python-ntlm 1.0.1 Downloading http://pypi.python.org/packages/source/p/python-ntlm/python-ntlm-1.0.1.tar.gz#md5=3b7ccdb2fd601596fea0f92d06f4fdaf Processing python-ntlm-1.0.1.tar.gz Running python-ntlm-1.0.1/setup.py -q bdist_egg --dist-dir /tmp/easy_install-V6gue6/python-ntlm-1.0.1/egg-dist-tmp-Q8g0mm Adding python-ntlm 1.0.1 to easy-install.pth file Installing ntlm_example_extended script to /mnt/glusterfs/bishopj/vee/mysandbox/bin Installing ntlm_example_simple script to /mnt/glusterfs/bishopj/vee/mysandbox/bin
Installed /mnt/glusterfs/bishopj/vee/mysandbox/lib/python2.7/site-packages/python_ntlm-1.0.1-py2.7.egg Searching for zope.interface>=3.8.0 Reading http://pypi.python.org/simple/zope.interface/ Best match: zope.interface 4.0.5 Downloading http://pypi.python.org/packages/source/z/zope.interface/zope.interface-4.0.5.zip#md5=caf26025ae1b02da124a58340e423dfe Processing zope.interface-4.0.5.zip Running zope.interface-4.0.5/setup.py -q bdist_egg --dist-dir /tmp/easy_install-zVwcnX/zope.interface-4.0.5/egg-dist-tmp-r8gb1L warning: no previously-included files matching '*.dll' found anywhere in distribution warning: no previously-included files matching '*.pyc' found anywhere in distribution warning: no previously-included files matching '*.pyo' found anywhere in distribution warning: no previously-included files matching '*.so' found anywhere in distribution Adding zope.interface 4.0.5 to easy-install.pth file
Installed /mnt/glusterfs/bishopj/vee/mysandbox/lib/python2.7/site-packages/zope.interface-4.0.5-py2.7-linux-x86_64.egg Finished processing dependencies for DDN-SFA-API==1.5.3.0.g-r12955
