🎉 My colleagues and members of the language community have released SMOL, a new open-source dataset (CC-BY-4) designed for machine translation research. SMOL includes professionally translated parallel text for over 115 low-resource languages, with a significant representation of over 50 African languages. This dataset is intended to provide a valuable resource for researchers working on machine translation for under-represented languages.
Kindly check the paper for more details including limitations of this dataset.
Afar Acoli Afrikaans Alur Amharic Bambara Baoulé Bemba (Zambia) Berber Chiga Dinka Dombe Dyula Efik Ewe Fon Fulfulde Ga Hausa Igbo Kikuyu Kongo Kanuri Krio Kituba (DRC) Lingala Luo Kiluba (Luba-Katanga) Malagasy Mossi North Ndebele Ndau Nigerian Pidgin Oromo Rundi Kinyarwanda Sepedi Shona Somali South Ndebele Susu Swati Swahili Tamazight Tigrinya Tiv Tsonga Tumbuka Tswana Twi Venda Wolof Xhosa Yoruba Zulu
Credits:
Isaac Caswell and Elizabeth Nielsen and Jiaming Luo and Colin Cherry and Geza Kovacs and Hadar Shemtov and Partha Talukdar and Dinesh Tewari and Baba Mamadi Diane and Koulako Moussa Doumbouya and Djibrila Diane and Solo Farabado Cissé. SMOL: Professionally translated parallel data for 115 under-represented languages.
Important: This modifies the Content Security Policy (CSP) to allow the web console to function. To understand the security implications and best practices for CSP, it’s highly recommended to read more about it here: https://developer.chrome.com/docs/privacy-security/csp
Note: This CSP configuration is suitable for local development and testing. When deploying a production application, you should carefully review and adjust the CSP to ensure proper security and privacy measures are in place.
Google Translate has taken a significant step towards greater inclusivity by adding 110 new languages to their service, including 31 from Africa. This expansion means that millions of people who previously lacked access to translation services now have a tool to communicate and connect with a wider world.
This achievement is the result of a concerted multi years effort by Isaac Caswell, the Google Translate Research team, and numerous community collaborators. They faced unique challenges, as building high-quality translations for languages with limited digital resources is complex. To address this, they developed an approach that relies on “monolingual” data – text in a single language – instead of solely relying on translated text. This method, called “zero-shot learning,” allows for the creation of translations for languages not explicitly trained on, though it’s important to remember that this is still a developing technology.
What does this mean?
More Accessibility: People speaking these newly included languages now have a tool to access information, communicate, and break down language barriers.
Language Preservation: This expansion helps preserve and promote less commonly used languages, which is crucial for maintaining linguistic diversity.
New Opportunities: The inclusion of languages like Punjabi and Romani opens doors for those communities, aiding in digital navigation and accessing information.
While this is a significant step, it’s important to note that Google Translate, while utilizing powerful AI technology, is not a replacement for the expertise of professional translators, however the app is useful for millions of people. This new approach, powered by the Palm 2 language model will require ongoing refinement and feedback to improve accuracy.
The languages added are significant:
A few notables ones:
Punjabi (Shahmukhi): This language, written in the Shahmukhi script, is spoken by millions in Pakistan and India. Its inclusion expands communication and access to information for a large community.
Romani: Romani, the language of romani communities with presence in many european countries, has historically been underrepresented in technology. Its inclusion in Google Translate is a step towards recognizing and supporting this community.
N’ko: Created in the 1940s, N’ko uses a unique script to unify Manding languages in West Africa. This addition supports literacy and cultural preservation efforts.
Tamazight (with Tifinagh script): Tamazight is spoken by millions of Berber people in North Africa. Its inclusion acknowledges their cultural diversity and language heritage.
This expansion is a positive step towards a more inclusive digital world, but it’s important to be aware of its limitations. While AI is an exciting tool, it requires continual development and feedback to refine its capabilities. The addition of these new languages is a testament to the potential of technology to foster communication and understanding, but it’s crucial to remember that it’s a journey, not a destination.
