Romanized Corpus
As a part of the “Bangla Syntactic Treebank Corpus with Processing Pipeline and Distribution” project, a romanized dataset, commonly referred to as Banglish will be collected and processed. This dataset consists of Bangla text written in English or Roman script. The corpus is expected to comprise at least 2 million words, ensuring it is balanced and representative across various domains and types, collected from real-life usage, and linked with the source.
Data Cardinality
| Type | Count |
|---|---|
| average text length | 63.701 |
| median text length | 47 |
| mode text length | 30 |
| max text length | 6,712 |
| min text length | 8 |
| number of words | 20,01,169 |
| number of unique words | 2,54,379 |
| number of sentences | 1,76,295 |
Metadata
The Banglish dataset will be available in JSON format, has multiple fields in each instance. The explanation of each field is described in the table below:
| Field Name | Description |
|---|---|
| text_id | Unique identification string of the data entry |
| text | Banglish text for the dataset requirement |
| title | Title of the text |
| categories | Category of the data |
| domain | The domain names from where the data belongs to |
| source | Source name from where data is collected |
| source_type | The type of source the data belongs to |
| publish_date | The date when the data was published |
N.B. There are some additional fields in some specific collections. Like there is a "Fb_page" field in "facebook.com", "tags" field in newspapers like "bd-protidin.com" etc. These fields are prime information of source and the data.
Source Distribution
The dataset comprises Banglish data from different sources. While the majority of the data originates from "facebook.com" and "youtube.com", other sources also contribute valuable Banglish content. The distribution of word counts across different sources is presented below for reference:
Sentence Length Distribution:
As the dataset consists of real world data, it encompasses a diverse range of sentence lengths. To analyze the distribution of sentence lengths, the following representation illustrates the count of sentences corresponding to different word counts:
Time Distribution
While gathering the data, priority was given to more recent sources to ensure data quality, even though the dataset contains information from various years. An overview of the distribution of data counts across different years is presented below: