Deployment of Seamless M4T v2 models on SAP AI Core

[YuikoKoike26]

Introduction

Have you ever faced language barriers while perusing documents or participating in meetings? Language differences can be a significant obstacle to communication. However, there's a fantastic solution: Seamless M4T. This translation system enables people from diverse linguistic backgrounds to share knowledge and experiences more seamlessly. The AI model is designed to provide high-quality translations. 

Consider this use case: by integrating SeamlessM4T with SAP AI Core, you can overcome the language gap among delivery personnel, ensuring clear communication even when they do not share a common language. Seamless M4T is effective for SAP users worldwide!

In this post, I will share how I deployed Seamless M4T on the SAP Business Technology Platform (BTP) AI Core!

What is Seamless M4T?

Seamless M4T is a foundational multilingual and multitask model developed by Facebook AI Research, with the latest version being Seamless M4T-v2, released on November 30th, 2023.

This AI translation model supports speech recognition, speech-to-text translation, speech-to-speech translation, text-to-text translation, and text-to-speech translation for nearly 100 input languages and 35 (+ English) output languages. I will demonstrate how it works later in this blog.

Process of the Project

1. Running the official demo app of Seamless M4T v2 models locally

2. Building a Docker image of Seamless M4T

3. Deploying a Seamless M4T container on SAP AI Core

Getting Started

Pre-requisites

Only for Linux x86-64 and macOS: 
Seamless M4T is available only for Linux x86-64 and macOS because fairseq2, one of the prerequisites, does not have native support for Windows. If you are a Windows user, you can install fairseq2 via the Windows Subsystem for Linux (WSL).

Recommend using Python ~3.11: 
Due to compatibility issues with the latest Python version (3.12), it is advisable to use versions prior to Python 3.11. For this guide, Python 3.11.8 was used.

1. Running the Official Demo App of Seamless M4T v2 Models Locally

1. Clone the official Seamless M4T repository from [here] (seamless_communication/demo/m4tv2/)
    
2. Create a folder named ‘app’ at the root of the cloned directory and move the following three files into this new folder: app.py, lang_list.py, and requirements.txt.
   
   
3. Update the checkpoint path (line33) in app.py to build docker image.
   

Before:
CHECKPOINTS_PATH = pathlib.Path(os.getenv("CHECKPOINTS_PATH", f"/home/{user}/app/models"))

After:
CHECKPOINTS_PATH = pathlib.Path(os.getenv("CHECKPOINTS_PATH", f"{pathlib.Path.home()}/models"))

Since there is no default home directory in Docker, this change allows for more flexibility in determining the path to the user's home directory across different operating systems. 

4. Update the contents of requirements.txt as follows:

Before:

gradio~=4.5.0
omegaconf~=2.3.0
torch~=2.1.0
torchaudio~=2.1.0
fairseq2~=0.2.0

...

After:

git+https://github.com/facebookresearch/seamless_communication.git
fairseq2==0.2.0
gradio==4.21.0
httpx==0.27.0
omegaconf==2.3.0
pydub==0.25.1
sentencepiece==0.2.0

5. To run the demo app, execute the following commands:

$ cd app
$ pip install -r requirements.txt
$ python app.py​

If it starts successfully, you should see logs similar to these:

Using the cached tokenizer of seamlessM4T_v2_large. Set `force` to `True` to download again.
Using the cached tokenizer of seamlessM4T_v2_large. Set `force` to `True` to download again.
Using the cached etox dataset. Set `force` to `True` to download again.
Using the cached tokenizer of mintox. Set `force` to `True` to download again.
…
2024-04-10 17:34:57,454 INFO -- httpx: HTTP Request: HEAD http://127.0.0.1:8000/ "HTTP/1.1 200 OK"

To create a public link, set `share=True` in `launch()`.
2024-04-10 17:34:58,322 INFO -- httpx: HTTP Request: POST https://api.gradio.app/gradio-launched-telemetry/ "HTTP/1.1 200 OK"​

By click http://127.0.0.1:8000/, you can launch and try the demo app.

2. Building a Docker image of Seamless M4T

1. Create a file named Dockerfile at the root of the folder and add the following content:
   

# Set base image (host OS)
FROM python:3.11.8

# Set environment variables for Matplotlib and Fontconfig cache directories
ENV MPLCONFIGDIR=/app/matplotlib_cache
ENV FONTCONFIG_PATH=/app/fontconfig

# Create the necessary directories within your Docker image
RUN mkdir -p /nonexistent /app/src/ /app/matplotlib_cache /app/fontconfig  

# Set permissions for the created directories
RUN chmod -R 777 /nonexistent /app 

# Copy your application's code
COPY ./app/requirements.txt /app/
COPY ./app/app.py /app/src/
COPY ./app/lang_list.py /app/src/

# Install Python dependencies within your Docker image
RUN pip install -r /app/requirements.txt

# Set the working directory
WORKDIR /app/src/

# Expose the port your app runs on
EXPOSE 8000

# Set the entry point command
CMD ["python", "app.py"]

2. Create the Docker image using the following command & Upload the Docker image to the cloud:

$ docker buildx build -t <your docker image and tag> .
$ docker push <your docker image and tag>

3. Test the Docker image by running the following command:

AI Core uses kserve to serve the models - all containers run as user "nobody". The user "nobody" is a non-root user with minimal privileges, suitable for running processes and performing tasks that do not require administrative permissions.

$ docker run --user nobody -p 8000:8000 -d <your docker image and tag>

Similar to running app.py locally, it takes 1-2 hours to finish. If it runs successfully, you will see the corresponding logs.

3. Deploying a Seamless M4T container on SAP AI core

You can also follow SAP Learning with SAP AI Core:
- [Preparation for Deployment]( Build a House Price Predictor with SAP AI Core | SAP Tutorials )
- [How to Deploy Your App with SAP AI Core]( Make Predictions for House Prices with SAP AI Core | SAP Tutorials )

1. Create a folder named "kyma-deployment" and add a YAML file with the following content:

apiVersion: ai.sap.com/v1alpha1
kind: ServingTemplate
metadata:
  name: seamlessm4t-on-aicore
  annotations:
    scenarios.ai.sap.com/description: "Run a SeamlessM4Tm4t scenario on AI Core"
    scenarios.ai.sap.com/name: "seamless-scenario"
    executables.ai.sap.com/description: "Run a SeamlessM4T executable on AI Core"
    executables.ai.sap.com/name: "seamlessm4t-executable"
  labels:
    scenarios.ai.sap.com/id: "seamlessm4t-server"
    ai.sap.com/version: "0.1"
spec:
  template:
    apiVersion: "serving.kserve.io/v1beta1"
    metadata:
      annotations: |
        autoscaling.knative.dev/metric: concurrency
        autoscaling.knative.dev/target: 1
        autoscaling.knative.dev/targetBurstCapacity: 0
      labels: |
        ai.sap.com/resourcePlan: infer.l
    spec: |
      predictor:
        minReplicas: 1
        maxReplicas: 1
        containers:
        - name: kserve-container
          image: <your docker image and tag>
          ports:
            - containerPort: 8000
              protocol: TCP
          env:
            - name: SEAMLESSM4T_MODEL
              value: "medium"
          resources:
            requests:
              memory: "64Mi"
              cpu: "250m"
            limits:
              memory: "128Mi"
              cpu: "500m"

<Lack of free disk space>
Seamless M4T v2 is a large size model that may exceed the default disk storage size limit for nodes within SAP AI Core. Consequently, you might need to increase the disk size by creating or updating a volume. (See [Here])

Recommendations:
The SeamlessM4T-Large v2 model introduced here has 2.3 billion parameters. If this model is too large, medium (1.2B) and small (281M) models are available as on-device options, but note that these are SeamlessM4T v1 models.

Before proceeding to the next step, ensure that all the necessary directories and files have been pushed to your GitHub repository, and that your repository's visibility is set to "private".
 
2. Add Your GitHub Repository to SAP AI Launchpad

After selecting the Workspaces you wish to use, click Add under Git Repositories. Then, enter the details of your GitHub repository and click Add.

URL: The URL of Your GitHub repository. 
Name: The name you wish to assign to the repository in AI Launchpad. 
User Name: Your GitHub account username. 
Access Token: Your GitHub Personal Access Token. 

3. Create an Application on AI Launchpad

Click Applications> Create and fill in the details of your application.

Application Name: The name you wish to assign to your application in AI Launchpad. 
Repository: Select the Git Repository you added in the previous step.
Path in Repository: The folder in your GitHub where your workflow YAML file is located, which you added in the previous step. This time, it's 'kyma-deployment'.
Revision: The specific ID of your GitHub commit. Set this to "HEAD" to automatically refer to the latest commit.

4. Check the Sync Status of Your Workflow in Applications and Scenarios

After waiting 3 minutes post-creation of your applications and navigating to Applications > your application, you can view the status of your workflow. Verify it against your workflow YAML file.

In ML Operations, click Scenarios and compare the names of scenarios to those in your workflow YAML file.

5. Create a Configuration for Your Workflow

Click Workspaces and select the 'default' resource group. After selecting the resource group, navigate to Configurations under ML Operations and click Create. Enter the details of your Configuration.

🚨Remember, you cannot delete a Configuration once created, so be careful not to enter incorrect information!🚨

Configuration Name: The name you wish to assign to your configuration in AI Launchpad
Scenario: Select ‘seamless-scenario’
Version: Select ‘0.1’
Executable: Select ‘seamlessm4t-executable’ (refers to the Executable Name)

Click Next in 2: Input Parameters and 3: Input Artifacts to review the remaining steps, and then click Create.

6. Start a Deployment on AI Launchpad

On the configuration details page of your configuration, click Create Deployment.

Click the refresh icon on the far right.
The deployment status will change from UNKNOWN to RUNNING.

If it runs successfully, you will be able to view the logs.

Success Deployment

Try Seamless M4T

Let's explore the capabilities of Seamless M4T!

Speech-to-Speech (S2ST)

I uploaded an audio file in Mandarin Chinese and received a translation in English with remarkably natural-sounding audio.

Text-to-Speech (T2ST)

I then attempted to translate from English to Japanese. The translation was accurate, though the resulting voice sounded somewhat robotic compared to the English audio. In comparison to English, Spanish, German, and certain other languages, the translation accuracy into various languages is not yet perfect.

Automatic Speech Recognition (ASR)

Using the same Mandarin Chinese audio file as in the S2ST example, Seamless M4T automatically recognized the language I uploaded and translated it into German as follows.

 

Conclusion

Seamless M4T v2 enables us to overcome major language barriers globally within SAP environments.
Give it a try and discover how it can enhance your projects!

Reference 

AI core:

• https://help.sap.com/docs/sap-ai-core
  
• https://developers.sap.com/tutorials/ai-core-code.html
• https://developers.sap.com/tutorials/ai-core-deploy.html

Seamless M4T: 

• https://github.com/facebookresearch/seamless_communication
• https://ai.meta.com/research/seamless-communication/