AI Automated Multilingual Support

AI Automated Multilingual Support

  • Posted by Mariam Elbaz
  • On August 21, 2023

There are more than 7,100 languages and dialects in the world, nearly 135 of which are supported by online translation engines. It surely is a big gap, but it still means that your application can come in a range of 135 languages! Now is there really anything technology can’t do? As of the 21st century, multilingual user interfaces were introduced for better inclusivity of the global population.

 

As you may well be aware, there are many ways in which you can include multiple language support in your application, some more tedious than others. But first, why even go through with this? Are there ways to make it less tedious? Keep reading to find out!

 

Benefits of a Multilingual Application

 

While English is the most spoken language in the world, it only accounts for 17% of the world’s population. Surely that means that if you want your application to reach for the stars, you have got to include a little cultural diversity! Soon enough, you will get to enjoy what’s listed below.

 

  • Reach larger Audience

Think of the remaining 83% of the population, or that portion of the English speaking 17% who is more comfortable with another language.

  • Reduce Bounce Rate

Bounce rate is the percentage of users who leave your website or application after viewing only 1 page. Likely enough, that could be due to disregard of their mother tongue.

  • Strengthen User Experience

People tend to give more credit to information, if presented to them in their preferred language. They will also most likely process it better and hence, make more informed decisions.

  • Enhance Search Engine Optimization (SEO)

Give your application or website more odds to appear in search engine results, regardless of the language used while searching.

 

While there are many more benefits you could enjoy, it is now time to get to the “How?” behind all this.

 

Most Common Practices

 

Regardless of the stack your application/website is using, any attempt to internationalize it must include a library or two depending on your choice of stack. In all cases, there will likely be two or more files, in accordance with the number of supported languages, to include all necessary translations.

 

As an example, an angular-based application could use the third-party package ngx-translate to implement an internationalized English-Arabic website. This will require two files, namely ar.json and en.json, to locate the needed translation.

 

Those two files may look something like this:

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

Via interpolation or tag property, the package will then utilize these two files and proceed to localize the website. However, did you stop to think about populating these files? What if there are more? What if you want to include all 135 supported languages? Pretty tiresome, correct?

 

It is about time to let Artificial Intelligence (AI) come to the rescue and save us plenty of time. Hang tight to find out how!

 

How AI Saves the Day

 

Let me present to you a very modernized solution to the problem that is populating the JSON files like above. With the aid of a brief python script, I was able to facilitate this whole translation process to take about 1 – 2 minutes at most, rather than hours and hours of error-prone work!

 

The actual translation of the text was done using a python library named google trans (version 4.0.0), which is free, unlimited, and the most stable version released to this day. It uses the public Google Translate Ajax API to make calls as needed.

 

BEWARE:

 

  • The maximum character length for a single translation is 15K characters.
  • If the API repeatedly fails, it is likely because your IP has been banned for the day. This may happen if you make an excessive number of calls in a short period of time. In such case, you may wait till the next day and proceed normally, or you may use Google Collaboratory Notebooks and change the google account if it has been banned.

 

Having set the stones for localizing your website, you must have at least one JSON file to start with. At that point, you only need to feed this file into the script and specify the language you need the values to be translated to. Note that there is no need to specify the source language, since the google translate API provides a detection call.

 

Below is the code I used to translate an ar.json file to an en.json file, feel free to change the source or destination languages. I started by reading the JSON file into a pandas dataframe, which I then looped over and called the translation API for the single JSON data entry iteratively. Lastly, it is worth noting that the solution provided below is recursive in nature, and that is to account for nesting or grouping within the input JSON file, as can be seen in the aforementioned files.

 

 

You can add this script to your application and run it as a prerequisite using any command file.

 

Do not forget to add the necessary imports and install googletrans==4.0.0rc1 beforehand!

 

 !pip install googletrans==4.0.0rc1

 

 

import pandas as pd

import json

import googletrans

from googletrans import Translator

 

 

 

 # Recursive Solution

def

# Read data in the json object as key and value

data = {k:[v] for k,v in json_df.items()}

 

# Put the data in a pandas DataFrame

data_df = pd.DataFrame(data)

 

# Initialize empty object for recursion

sub_data = {}

 

# Loop over the keys in the data frame

for idx, obj in enumerate(data_df.keys()):

# Key is always at index 0

key = data_df[obj][0]

# Check if key is not empty and of type string

if (key != None and type(key) == str):

# Remove whitespace from the key, if any

key = key.strip()

try:

if(key == ""):

# If key is empty then value will also be empty

sub_data[obj] = ""

else:

# If key is not empty use it for translation

sub_data[obj] = translator.translate(key, dest='en').text

# Recursion occurs if the value for the key is another json object

if (isObject == False):

en_json[obj]=sub_data[obj]
# Catching the exception that may occur due to reasons mentioned above

except AttributeError as e:

print("Failed on key: " + key + " with error: " + str(e))

json_data = json.dumps(en_json, indent=4)

# Save the translated data before breaking

with open("en.json", "w") as outfile:

outfile.write(json_data)

break

else:

# Recursive call for the nested data

if (isObject):

# Save data in a sub object in case there is more nesting

sub_data[obj] = translate_json(key, en_json, translator, True, obj)

else:

# Save data directly in case this is final nesting

en_json[obj] = translate_json(key, en_json, translator, True, obj)

return sub_data

 

# Google Translate API

translator = Translator()

 

# Open the file to be translated

f = open('ar.json')

 

# Returns the data in that file as a json object

data = json.load(f)

 

# Initialize empty object for the translated data

en_json = {}

 

# Function call

translate_json(data, en_json, translator, False, None)

 

# Load the object into a json string to be formatted using a 4-tab indentation

json_data = json.dumps(en_json, indent=4)

 

# Write the data to output file

with open("en.json", "w") as outfile:

outfile.write(json_data)

 

 

Client-Side Solution

 

You can also directly use your client-side code to access google translate API and use it to your benefit. Same as above, an ar.json file can be looped over and iteratively fed into the API to get the translation as needed. This can be done by directly calling the API using the HttpClient provided by angular, yet in order to do that you will need an API key, which can be retrieved from the Google Cloud Console using a billing account.

 

Tip: You can use your work email to get a 4-month free trial to kick start your experience.

 

Below you will find a code snippet that does the same function on client-side:

 

 

 // Service class to host the google translate API call

export class GoogleTranslateService {
// The URL of the API
url = 'https://translation.googleapis.com/language/translate/v2?key=';

// The API Key retrieved from Google Cloud Console
key = '';

// Constructor with an HttpClient injection to call the API
constructor(private http: HttpClient) { }

translate(obj: GoogleObj) {
return this.http.post(this.url + this.key, obj);
}
}

Note: The GoogleObj is just a model that puts the data into the format needed by the API. (Provided below)

export interface GoogleObj {
q: string[]; // string(s) to be translated
target: string; // target language
}

 

// Function that uses the injected GoogleTranslateService as “google” and does the translation

iterateOverJsonObj(obj: any) {

// Looping over the json file keys
for (var k in obj) {

// If the value to the key is an object => Recursion
if (typeof obj[k] == "object" && obj[k] !== null) {
this.iterateOverJsonObj(obj[k]);
} else {

// Construct the GoogleObj and call the API
const googleObj: GoogleObj = {
q: [obj[k]],
target: 'en'
};
this.google.translate(googleObj).subscribe(
(res: any) => {
obj[k] = res.data.translations[0].translatedText;
}, err => {
console.log(err);
}
);
}
}
}

You cannot, however, save the resulting object to a json file, yet you can manipulate and use it however you like within your code, one way of doing so may be caching the results.