Share corporate info with an LLM using Embeddings

[AronMac]

NOTE: The views and opinions expressed in this blog are my own

So you’re diving into the world of generative AI  for your Enterprise, but you have doubts.

You’ve asked ChatGPT  questions and it’s awesome, but you quickly realize it doesn’t know anything about your business.

Unfortunately every other off the shelf  LLM you try  isn’t  a perfect fit for your needs either.

You’ve heard about “Fine-tuning”  an LLM but you just aren’t ready for a more bespoke solution yet.

Then recently you’ve heard about “Embeddings”.  A magical way of sharing your corporate info with an LLM, to help provide better context to your questions, improving the LLM's responses.

Fantastic  … problem solved….. Unfortunately that’s just the start of the journey:

DALL-E helped with this.. click to enlarge

Never mind all that,  I heard OpenAI   has the model text-embedding-ada-002 .  Why not just use that?

Firstly did you know you’ll need to send, via their api, all the corporate info you want to turn into vectors  as text payloads?

It’s also currently only ranked 38th on the Massive Text Embedding Benchmark (MTEB) Leaderboard

What if  OpenAI decommissioned that model (perhaps in April 2025), would you have to convert all your corporate info again?

How comfortable do you feel about that simple choice now?

The good news is that with SAP AI Core and SAP Generative AI  you have options to take more control with your sensitive Enterprise Information.

As some brief background embedding  is simply a process of converting text data into numerical representations (vectors) that a machine can use and interpret.

Let’s say we have 3 animals. E.g.  a Brown Labrador , Red Tabby Cat  and a Black German Sheppard.

Embedding models need to turn these animal descriptions  into N dimension Vectors, subject to the size of the model.

For easier understanding let’s say we have a model with only 2 dimensions, perhaps it might convert them as:

Brown Labrador                                -> [1, 1]

Red Tabby Cat                                   -> [-1, -1]

Black German Shepherd                    -> [1.5, 1.5]

Now we add a new animal , a Black Panther  [1.5, -0.5]  

We can easily visual 2 dimensions vectors on a chart as:

click to enlarge

With the animals stored as vectors then mathematical formulas can be applied to determine similarity.

Is the Black Panther more similar to the dogs or the cat?

But we are getting ahead of ourselves.  In Q1 SAP Hana Cloud will soon have a new Vector Engine so we will be able to answer those questions.

For now, let’s use SAP AI CORE to generate various Embeddings:

• Azure OpenAI  text-embedding-ada-002   [1536 Dimensions] 

Utilises a remote call to OpenAI models running on Azure

Pre-reqs: AI CORE extended plan, setup  config for ada model and deploy . See models-and-scenarios-in-generative-ai-hub 

• Ollama with Phi-2 model  [2560 Dimensions] 

Tiny open source LLM, running on AI CORE with GPU infer.s

Pre-reqs: follow the steps on Gunters excellent blog It's Christmas! Ollama+Phi-2 on SAP AI Core

Before continuing ensure in AI CORE you have 2 running deployments similar to:

click to enlarge

Next double check you have a valid service key for SAP AI CORE.  See here for more details.

A useful way to interact with AI Core is via the Python SDK

!pip install ai-api-client-sdk 

ai_api_client = AIAPIV2Client(
base_url=’<details from AI Core Service Key>’, 
 auth_url= ’<details from AI Core Service Key>’,
client_id= ’<details from AI Core Service Key>’,
client_secret= ’<details from AI Core Service Key>’,, 
 resource_group= 'default'
)
print(ai_api_client.rest_client.get_token())

RESPONSE:    Bearer …..

At this point you can stay in python and continue to use the ai_api_client.rest_client to interact with the deployments or take the Bear token and use it with curl or other REST tools.

Now lets see what "Black Panther" looks like as an OpenAI 1536 Dimension Vector:

POST <your deployment url here>/embeddings?api-version=2023-05-15
Content-Type: application/json
Authorization: Bearer ………
ai-resource-group: default

{
    "input" : "black panther"
}

It returns:

click to enlarge

Now lets see the output of the opensource LLM Phi-2 embeddings (running on AI CORE) which has 2560 Dimensions:

POST <your deployment url>/v1/api/embeddings
Content-Type: application/json
Authorization: Bearer ......
ai-resource-group: default

{
  "model": "phi",
  "prompt":"black panther",
  "stream": false
}

It returns:

click to enlarge

So now we know we can generate different kinds of embeddings but how do we choose the right one for our use cases?

In my next blog  I will deploy 2 additional models (Word2Vec  and WhereIsAI/UAE-Large-V1)  on AI Core utilizing CPU and the Basic resource plan and then start to compare them with a slightly more complex use case.

In the meantime I welcome your feedback below.

SAP notes that posts about potential uses of generative AI and large language models are merely the individual poster’s ideas and opinions, and do not represent SAP’s official position or future development roadmap. SAP has no legal obligation or other commitment to pursue any course of business, or develop or release any functionality, mentioned in any post or related content on this website.