SAP Data Services and Python - how to install a library? (PIP missing?)

[stvndw]

Hello,

Love the fact that python is available in SAP Data Services.

But does anybody know how to add python libs to data services.

In a normal python environment this would be done by a 'pip isntall…'

But with data services is less clear (Where is pip??) 😞

Python is a powerful addon to the tool.

But without the possibility to add the vast availability of Python libs it loses a lot of power.

Many thanks for the help!

Regards,

Steven Dauw

[former_member635361]

Hi Alex

1. On your local PC, install Python and the necessary libraries (pip or easy_install)

2. Navigate to your local Python install folder and find the "Lib\site-packages" folder. You should see a folder for the package you just installed (e.g. certifi-2021.5.30-py3.7.egg)

3. Copy whatever folder your package is in and put it on the job server's equivalent folder, but rename the folder to only the name of the package (e.g. certifi)

4. Write a batch file that calls your Python program

"F:\Program Files (x86)\SAP\Data Services\DataQuality\python\python" F:\MyFolder\MyPython.py

5. In a DS script, call a batch file

exec('cmd','\\\\myserver\MyFolder\mybatfile.bat',1);

Hope that helps.

[former_member762669]

Graeme, thanks

[former_member427802]

Hi, Steven.

You can use User_Defined Transform in Data Services for this purpose.

You can use classes and functions from custom python module in the following way:

1. Install python library locally using pip.
2. Put library to the file server at an accessible location.
3. Use this library in python script.

Note: When you import library, that was put to the file server, don't forget add to sys.path relative to the current running script.

import sys
import os
sys.path.insert(0, os.path.dirname(__file__))

4. Insert User_Defined Transform in BODS Job and import your python script using "imp" module and use classes and functions from your custom script.

import imp
myscript = imp.load_source('myscript', '//job_server/lib_path/myscript.py')
myscript.myFunction()

[former_member762669]

Hi, Igor!

I don't know how to write classes and functions in python, but I can write a simple python script to transform data.

if I want to write python code in the UDT editor, can I write a link to external python libraries in UDT?

I want to learn how to use the UDT editor.

now this scheme does not work.

[jmuiruri]

Greetings,

SAP Data Services libraries are located in this directory $LINK_DIR/DataQuality/python/Lib/site-packages. You can install 3rd party libraries for use with SAP Data services to this directory by following the steps shown below

1. Make sure you have python pip installed.
   • You can download and install python pip as explained in the pip documentation in the link below
     • PIP Installation
2. Once you have pip installed you can go ahead and install 3rd party libraries just like you do it in python as shown below

pip install --target=$LINK_DIR/DataQuality/python/Lib/site-packages package_name

Please see the following KBA

• 2536123 - How to install python libraries for use with SAP Data Services - SAP Data Services 4.x

Best Regards,

Joseph

[former_member635361]

I've never used user defined transforms, only calling Python via a script.

To change the version of Python on the job server, your administrator would need to install a newer version of BODS. I believe version 14.2.14 upgrades to Python 3.8

[jmuiruri]

Hello zordon ,

SAP Data services comes with a customized version of python and as a result you can't just download any python version.

• SAP Data Services 4.2 SP14 the python version is 3.8.13
• SAP Data Services 4.3 SP1 the python version is 3.9.13

Regarding your other question from 2021,SAP Data Services libraries are located in this directory $LINK_DIR/DataQuality/python/Lib/site-packages. You can install third party libraries to this directory as shown below

pip install --target=$LINK_DIR/DataQuality/python/Lib/site-packages package_name

After installing them you can import them directly to a UserDefined transform just like you import them in python.

Best Regards

Joseph

SAP Support

[former_member635361]

Hi Alex. Did you come right getting libraries to work?

[former_member762669]

Hi Graeme.
I didn't understand how to install python libraries in DS, here:(%LINK_DIR%\DataQuality\python\Lib\site-packages).

I use python libraries installed locally on the job server.

Now I want to learn how to use a User-defined Transformation in DS!

If you can help me update Python 2>3, and install libraries in DS, I will be grateful.

[former_member635361]

Found that Python comes pre-installed on the DS job server (version 2.7), although without pip. To install a 3rd party library you can do one of the following:

Option 1: Download the source (.tar.gz), uncompress it, go into the now uncompressed folder and run python setup.py install (on the DS job server)

Option 2: Install the necessary 3rd party library on your personal computer, then copy the necessary library from your Python's Lib\site-packages to the same location on the DS job server (assuming you have write access to the server).

You can then either call Python via a script or the UDT and use whatever libraries you installed.

[former_member762669]

Graeme hello)

Can you describe in more detail the process of installing Python libraries in SAP Data Services?

Thanks.

[former_member635361]

Do I need to install Python on the job server in order to add 3rd party Python modules? If so, do I need to ensure it's the same version as the one used by the UDT?

Could I also install the latest version of Python and instead of using the UDT, simply call my Python program in a DS script?

[former_member562213]

Hi Ihor,

I see your solution. So, there seems to be no way other than installing python locally on the job server and then installing the 3rd party libs using pip.