This module contains the bits required to use the fastai DataBlock API and/or mid-level data processing pipelines to organize your data for causal and masked language modeling tasks. This includes things like training BERT from scratch or fine-tuning a particular pre-trained LM on your own corpus.
torch.cuda.set_device(1)
print(f'Using GPU #{torch.cuda.current_device()}: {torch.cuda.get_device_name()}')
Using GPU #1: GeForce GTX 1080 Ti

Causal LM

wiki_path = untar_data(URLs.WIKITEXT_TINY)
wiki_path.ls()
(#2) [Path('/home/wgilliam/.fastai/data/wikitext-2/train.csv'),Path('/home/wgilliam/.fastai/data/wikitext-2/test.csv')]
train_df = pd.read_csv(wiki_path/'train.csv', header=None)
valid_df = pd.read_csv(wiki_path/'test.csv', header=None)

print(len(train_df), len(valid_df))
train_df.head()
615 47
0
0 \n = 2013 – 14 York City F.C. season = \n \n The 2013 – 14 season was the <unk> season of competitive association football and 77th season in the Football League played by York City Football Club , a professional football club based in York , North Yorkshire , England . Their 17th @-@ place finish in 2012 – 13 meant it was their second consecutive season in League Two . The season ran from 1 July 2013 to 30 June 2014 . \n Nigel Worthington , starting his first full season as York manager , made eight permanent summer signings . By the turn of the year York were only above the relegation z...
1 \n = Big Boy ( song ) = \n \n " Big Boy " <unk> " I 'm A Big Boy Now " was the first single ever recorded by the Jackson 5 , which was released by Steeltown Records in January 1968 . The group played instruments on many of their Steeltown compositions , including " Big Boy " . The song was neither a critical nor commercial success , but the Jackson family were delighted with the outcome nonetheless . \n The Jackson 5 would release a second single with Steeltown Records before moving to Motown Records . The group 's recordings at Steeltown Records were thought to be lost , but they were re...
2 \n = The Remix ( Lady Gaga album ) = \n \n The Remix is a remix album by American recording artist Lady Gaga . Released in Japan on March 3 , 2010 , it contains remixes of the songs from her first studio album , The Fame ( 2008 ) , and her third extended play , The Fame Monster ( 2009 ) . A revised version of the track list was prepared for release in additional markets , beginning with Mexico on May 3 , 2010 . A number of recording artists have produced the songs , including Pet Shop Boys , Passion Pit and The Sound of Arrows . The remixed versions feature both uptempo and <unk> composit...
3 \n = New Year 's Eve ( Up All Night ) = \n \n " New Year 's Eve " is the twelfth episode of the first season of the American comedy television series Up All Night . The episode originally aired on NBC in the United States on January 12 , 2012 . It was written by Erica <unk> and was directed by Beth McCarthy @-@ Miller . The episode also featured a guest appearance from Jason Lee as Chris and Reagan 's neighbor and Ava 's boyfriend , Kevin . \n During Reagan ( Christina Applegate ) and Chris 's ( Will <unk> ) first New Year 's Eve game night , Reagan 's competitiveness comes out causing Ch...
4 \n = Geopyxis carbonaria = \n \n Geopyxis carbonaria is a species of fungus in the genus Geopyxis , family <unk> . First described to science in 1805 , and given its current name in 1889 , the species is commonly known as the charcoal loving elf @-@ cup , dwarf <unk> cup , <unk> <unk> cup , or pixie cup . The small , <unk> @-@ shaped fruitbodies of the fungus are reddish @-@ brown with a whitish fringe and measure up to 2 cm ( 0 @.@ 8 in ) across . They have a short , tapered stalk . Fruitbodies are commonly found on soil where brush has recently been burned , sometimes in great numbers ....
train_df['is_valid'] = False
valid_df['is_valid'] = True

df = pd.concat([train_df, valid_df])
df.head()
0 is_valid
0 \n = 2013 – 14 York City F.C. season = \n \n The 2013 – 14 season was the <unk> season of competitive association football and 77th season in the Football League played by York City Football Club , a professional football club based in York , North Yorkshire , England . Their 17th @-@ place finish in 2012 – 13 meant it was their second consecutive season in League Two . The season ran from 1 July 2013 to 30 June 2014 . \n Nigel Worthington , starting his first full season as York manager , made eight permanent summer signings . By the turn of the year York were only above the relegation z... False
1 \n = Big Boy ( song ) = \n \n " Big Boy " <unk> " I 'm A Big Boy Now " was the first single ever recorded by the Jackson 5 , which was released by Steeltown Records in January 1968 . The group played instruments on many of their Steeltown compositions , including " Big Boy " . The song was neither a critical nor commercial success , but the Jackson family were delighted with the outcome nonetheless . \n The Jackson 5 would release a second single with Steeltown Records before moving to Motown Records . The group 's recordings at Steeltown Records were thought to be lost , but they were re... False
2 \n = The Remix ( Lady Gaga album ) = \n \n The Remix is a remix album by American recording artist Lady Gaga . Released in Japan on March 3 , 2010 , it contains remixes of the songs from her first studio album , The Fame ( 2008 ) , and her third extended play , The Fame Monster ( 2009 ) . A revised version of the track list was prepared for release in additional markets , beginning with Mexico on May 3 , 2010 . A number of recording artists have produced the songs , including Pet Shop Boys , Passion Pit and The Sound of Arrows . The remixed versions feature both uptempo and <unk> composit... False
3 \n = New Year 's Eve ( Up All Night ) = \n \n " New Year 's Eve " is the twelfth episode of the first season of the American comedy television series Up All Night . The episode originally aired on NBC in the United States on January 12 , 2012 . It was written by Erica <unk> and was directed by Beth McCarthy @-@ Miller . The episode also featured a guest appearance from Jason Lee as Chris and Reagan 's neighbor and Ava 's boyfriend , Kevin . \n During Reagan ( Christina Applegate ) and Chris 's ( Will <unk> ) first New Year 's Eve game night , Reagan 's competitiveness comes out causing Ch... False
4 \n = Geopyxis carbonaria = \n \n Geopyxis carbonaria is a species of fungus in the genus Geopyxis , family <unk> . First described to science in 1805 , and given its current name in 1889 , the species is commonly known as the charcoal loving elf @-@ cup , dwarf <unk> cup , <unk> <unk> cup , or pixie cup . The small , <unk> @-@ shaped fruitbodies of the fungus are reddish @-@ brown with a whitish fringe and measure up to 2 cm ( 0 @.@ 8 in ) across . They have a short , tapered stalk . Fruitbodies are commonly found on soil where brush has recently been burned , sometimes in great numbers .... False
task = HF_TASKS_AUTO.CausalLM

pretrained_model_name = "gpt2"
hf_arch, hf_config, hf_tokenizer, hf_model = BLURR_MODEL_HELPER.get_hf_objects(pretrained_model_name, task=task)

# some tokenizers like gpt and gpt2 do not have a pad token, so we add it here mainly for the purpose
# of setting the "labels" key appropriately (see below)
if (hf_tokenizer.pad_token is None): hf_tokenizer.pad_token = '[PAD]'

hf_tokenizer.pad_token, hf_tokenizer.pad_token_id
Using pad_token, but it is not set yet.
('[PAD]', 50256)

class HF_CausalLMBeforeBatchTransform[source]

HF_CausalLMBeforeBatchTransform(hf_arch, hf_config, hf_tokenizer, hf_model, max_length=None, padding=True, truncation=True, is_split_into_words=False, ignore_token_id=-100, tok_kwargs={}, text_gen_kwargs={}, **kwargs) :: HF_BeforeBatchTransform

Handles everything you need to assemble a mini-batch of inputs and targets, as well as decode the dictionary produced as a byproduct of the tokenization process in the encodes method.

Our HF_CausalLMBeforeBatchTransform allows us to update the input's labels and our targets appropriately given a causal LM task.

The labels argument allows you to forgo calculating the loss yourself by letting huggingface return it for you should you choose to do that. Padding tokens are set to -100 by default (e.g., CrossEntropyLossFlat().ignore_index) and prevent cross entropy loss from considering token prediction for tokens it should ... i.e., the padding tokens. For more information on the meaning of this argument, see the huggingface glossary entry for "Labels"

before_batch_tfm = HF_CausalLMBeforeBatchTransform(hf_arch, hf_config, hf_tokenizer, hf_model)
blocks = (HF_Seq2SeqBlock(before_batch_tfm=before_batch_tfm), noop)

dblock = DataBlock(blocks=blocks, get_x=ColReader(0), splitter=ColSplitter(col='is_valid'))
dls = dblock.dataloaders(df, bs=4)
b = dls.one_batch()
b[0]['input_ids'].shape, b[0]['labels'].shape, b[1].shape
(torch.Size([4, 1024]), torch.Size([4, 1024]), torch.Size([4, 1024]))
explode_types(b)
{tuple: [dict, torch.Tensor]}
dls.show_batch(dataloaders=dls, max_n=2, trunc_at=500)
text target
0 \n = Bob Dylan = \n \n Bob Dylan ( / <unk> / ; born Robert Allen Zimmerman, May 24, 1941 ) is an American singer @-@ songwriter, artist and writer. He has been influential in popular music and culture for more than five decades. Much of his most celebrated work dates from the 1960s when his songs chronicled social unrest, although Dylan repudiated suggestions from journalists that he was a spokesman for his generation. Nevertheless, early songs such as " Blowin'in the Wind " and " The Times They Are a @-@ <unk>'" became anthems for the American civil rights and anti @-@ war movements. After he left his initial base in the American folk music revival, his six @-@ minute single " Like a Rolling Stone " altered the range of popular music in 1965. His mid @-@ 1960s recordings, backed by rock musicians, reached the top end of the United States music charts while also attracting <unk> and criticism from others in the folk movement. \n Dylan's lyrics have incorporated various political, social, philosophical, and literary influences. They defied existing pop music conventions and appealed to the burgeoning counterculture. Initially inspired by the performances of Little Richard and the songwriting of Woody Guthrie, Robert Johnson, and Hank Williams, Dylan has amplified and <unk> musical genres. His recording career, spanning more than 50 years, has explored the traditions in American song — from folk, blues, and country to gospel, rock and roll, and rockabilly to English, Scottish, and Irish folk music, embracing even jazz and the Great American Songbook. Dylan performs with guitar, keyboards, and harmonica. Backed by a changing lineup of musicians, he has toured steadily since the late 1980s on what has been dubbed the Never Ending Tour. His accomplishments as a recording artist and performer have been central to his career, but songwriting is considered his greatest contribution. \n Since 1994, Dylan has published six books of drawings and paintings, and his work has been exhibited in major art galleries. As a musician, Dylan has sold more than 100 million records, making him one of the best @-@ selling artists of all time. He has also received numerous awards including eleven Grammy Awards, a Golden Globe Award, and an Academy Award. Dylan has been inducted into the Rock and Roll Hall of Fame, Minnesota Music Hall of Fame, Nashville Songwriters Hall of Fame, and Songwriters Hall of Fame. The Pulitzer Prize jury in 2008 awarded him a special citation for " his profound impact on popular music and American culture, marked by lyrical compositions of extraordinary poetic power. " In May 2012, Dylan received the Presidential Medal of Freedom from President Barack Obama. \n \n = = Life and career = = \n \n \n = = = Origins and musical beginnings = = = \n \n Bob Dylan was born Robert Allen Zimmerman ( Hebrew name <unk> <unk> <unk> <unk> [ Shabtai <unk> ben <unk> ] ) in St Mary's Hospital on May 24, 1941, in Duluth, Minnesota, and raised in Hibbing, Minnesota, on the <unk> Range west of Lake Superior. He has a younger brother, David. Dylan's paternal grandparents, <unk> and Anna Zimmerman, emigrated from Odessa, in the Russian Empire ( now Ukraine ), to the United States following the anti @-@ Semitic <unk> of 1905. His maternal grandparents, Ben and Florence Stone, were Lithuanian Jews who arrived in the United States in 1902. In his autobiography, Chronicles : Volume One, Dylan wrote that his paternal grandmother's maiden name was <unk> and her family originated from <unk> district of <unk> Province in northeastern Turkey. \n Dylan's parents, <unk> Zimmerman and <unk> " Beatty " Stone, were part of a small but close @-@ knit Jewish community. They lived in Duluth until Robert was six, when his father had <unk> and the family returned to his mother's hometown, Hibbing, where they lived for the rest of Robert's childhood. In his early years he listened to the radio — first to blues and country stations from Shreveport, Louisiana and later, when he was a teenager, to rock and roll. He formed several bands while attending Hibbing High School. In the Golden <unk>, he performed covers of songs by Little Richard and Elvis Presley. Their performance of Danny & the Juniors'" Rock and Roll Is Here to Stay " at their high school talent show was so loud that the principal cut the microphone. In 1959, his high school <unk> carried the caption " Robert Zimmerman : to join'Little Richard '. " The same year, as <unk> <unk> [ sic ], he performed two dates with Bobby <unk>, playing piano and clapping. \n Zimmerman moved to Minneapolis in September 1959 and \n = Bob Dylan = \n \n Bob Dylan ( / <unk> / ; born Robert Allen Zimmerman, May 24, 1941 ) is an American singer @-@ songwriter, artist and writer. He has been influential in popular music and culture for more than five decades. Much of his most celebrated work dates from the 1960s when his songs chronicled social unrest, although Dylan repudiated suggestions from journalists that he was a spokesman for his generation. Nevertheless, early songs such as " Blowin'in the Wind " and " The Times They Are a @-@ <unk>'" became anthems for the American civil rights and anti @-@ war movements. After he left his initial base in the American folk music revival, his six @-@ minute single " Like a Rolling Stone " altered the range of popular music in 1965. His mid @-@ 1960s recordings, backed by rock musicians, reached the top end of the United States music charts while also attracting <unk> and criticism from others in the folk movement. \n Dylan's lyrics have incorporated various political, social, philosophical, and literary influences. They defied existing pop music conventions and appealed to the burgeoning counterculture. Initially inspired by the performances of Little Richard and the songwriting of Woody Guthrie, Robert Johnson, and Hank Williams, Dylan has amplified and <unk> musical genres. His recording career, spanning more than 50 years, has explored the traditions in American song — from folk, blues, and country to gospel, rock and roll, and rockabilly to English, Scottish, and Irish folk music, embracing even jazz and the Great American Songbook. Dylan performs with guitar, keyboards, and harmonica. Backed by a changing lineup of musicians, he has toured steadily since the late 1980s on what has been dubbed the Never Ending Tour. His accomplishments as a recording artist and performer have been central to his career, but songwriting is considered his greatest contribution. \n Since 1994, Dylan has published six books of drawings and paintings, and his work has been exhibited in major art galleries. As a musician, Dylan has sold more than 100 million records, making him one of the best @-@ selling artists of all time. He has also received numerous awards including eleven Grammy Awards, a Golden Globe Award, and an Academy Award. Dylan has been inducted into the Rock and Roll Hall of Fame, Minnesota Music Hall of Fame, Nashville Songwriters Hall of Fame, and Songwriters Hall of Fame. The Pulitzer Prize jury in 2008 awarded him a special citation for " his profound impact on popular music and American culture, marked by lyrical compositions of extraordinary poetic power. " In May 2012, Dylan received the Presidential Medal of Freedom from President Barack Obama. \n \n = = Life and career = = \n \n \n = = = Origins and musical beginnings = = = \n \n Bob Dylan was born Robert Allen Zimmerman ( Hebrew name <unk> <unk> <unk> <unk> [ Shabtai <unk> ben <unk> ] ) in St Mary's Hospital on May 24, 1941, in Duluth, Minnesota, and raised in Hibbing, Minnesota, on the <unk> Range west of Lake Superior. He has a younger brother, David. Dylan's paternal grandparents, <unk> and Anna Zimmerman, emigrated from Odessa, in the Russian Empire ( now Ukraine ), to the United States following the anti @-@ Semitic <unk> of 1905. His maternal grandparents, Ben and Florence Stone, were Lithuanian Jews who arrived in the United States in 1902. In his autobiography, Chronicles : Volume One, Dylan wrote that his paternal grandmother's maiden name was <unk> and her family originated from <unk> district of <unk> Province in northeastern Turkey. \n Dylan's parents, <unk> Zimmerman and <unk> " Beatty " Stone, were part of a small but close @-@ knit Jewish community. They lived in Duluth until Robert was six, when his father had <unk> and the family returned to his mother's hometown, Hibbing, where they lived for the rest of Robert's childhood. In his early years he listened to the radio — first to blues and country stations from Shreveport, Louisiana and later, when he was a teenager, to rock and roll. He formed several bands while attending Hibbing High School. In the Golden <unk>, he performed covers of songs by Little Richard and Elvis Presley. Their performance of Danny & the Juniors'" Rock and Roll Is Here to Stay " at their high school talent show was so loud that the principal cut the microphone. In 1959, his high school <unk> carried the caption " Robert Zimmerman : to join'Little Richard '. " The same year, as <unk> <unk> [ sic ], he performed two dates with Bobby <unk>, playing piano and clapping. \n Zimmerman moved to Minneapolis in September 1959 and
1 \n = Ireland = \n \n Ireland ( / <unk> / ; Irish : <unk> [ <unk> ] ; Ulster @-@ Scots : <unk> [ <unk> ] ) is an island in the North Atlantic. It is separated from Great Britain to its east by the North Channel, the Irish Sea, and St George's Channel. Ireland is the second @-@ largest island of the British Isles, the third @-@ largest in Europe, and the twentieth @-@ largest on Earth. \n <unk>, Ireland is divided between the Republic of Ireland ( officially named Ireland ), which covers five @-@ <unk> of the island, and Northern Ireland, which is part of the United Kingdom, in the northeast of the island. In 2011 the population of Ireland was about 6 @.@ 4 million, ranking it the second @-@ most populous island in Europe after Great Britain. Just under 4 @.@ 6 million live in the Republic of Ireland and just over 1 @.@ 8 million live in Northern Ireland. \n The island's geography comprises relatively low @-@ lying mountains surrounding a central plain, with several navigable rivers extending inland. The island has lush vegetation, a product of its mild but changeable climate which avoids extremes in temperature. <unk> woodlands covered the island until the Middle Ages. As of 2013, the amount of land that is wooded in Ireland is about 11 % of the total, compared with a European average of 35 %. There are twenty @-@ six extant mammal species native to Ireland. The Irish climate is very moderated and classified as oceanic. As a result, winters are <unk> than expected for such a northerly area. However, summers are cooler than those in Continental Europe. Rainfall and cloud cover are abundant. \n The earliest evidence of human presence in Ireland is dated at 10 @,@ 500 BC. Gaelic Ireland had emerged by the 1st century AD and lasted until the First World War. The island was <unk> from the 5th century onward. Following the Norman invasion in the 12th century, England claimed sovereignty over Ireland. However, English rule did not extend over the whole island until the 16th – 17th century Tudor conquest, which led to colonisation by settlers from Britain. In the <unk>, a system of Protestant English rule was designed to materially disadvantage the Catholic majority and Protestant dissenters, and was extended during the 18th century. With the Acts of Union in 1801, Ireland became a part of the United Kingdom. A war of independence in the early 20th century was followed by the partition of the island, creating the Irish Free State, which became increasingly sovereign over the following decades, and Northern Ireland, which remained a part of the United Kingdom. Northern Ireland saw much civil unrest from the late 1960s until the 1990s. This subsided following a political agreement in 1998. In 1973 the Republic of Ireland joined the European Economic Community while the United Kingdom, and Northern Ireland, as part of it, did the same. \n Irish culture has had a significant influence on other cultures, especially in the fields of literature. Alongside mainstream Western culture, a strong indigenous culture exists, as expressed through Gaelic games, Irish music, and the Irish language. The culture of the island also shares many features with that of Great Britain, including the English language, and sports such as association football, rugby, horse racing, and golf. \n \n = = Name = = \n \n Ireland consists of Old Irish <unk> + English land. <unk> derives from Proto @-@ Celtic * <unk> ( compare Welsh <unk> ), which is also the source of Latin <unk>. <unk> derives from a root meaning " fat, prosperous ". \n \n = = History = = \n \n \n = = = Prehistoric Ireland = = = \n \n During the last glacial period, and up until about 9000 years ago, most of Ireland was covered with ice, most of the time. Sea levels were lower and Ireland, like Great Britain, formed part of continental Europe. By 12 @,@ 000 BC, rising sea levels due to ice melting caused Ireland to become separated from Great Britain. Later, around <unk> BC, Great Britain itself became separated from continental Europe. The earliest evidence of human presence in Ireland is dated at 10 @,@ 500 BC. Until recently the earliest evidence of humans in Ireland were Mesolithic people who arrived by boat from Britain between 8000 BC and 7000 BC. \n From about <unk> BC, Neolithic settlers arrived introducing cereal <unk>, a housing culture ( similar to those of the same period in Scotland ) and stone monuments. A more advanced agriculture was to develop. At the <unk> Fields, preserved beneath a blanket of peat in present @-@ day County Mayo, is an extensive field system, arguably the oldest in the \n = Ireland = \n \n Ireland ( / <unk> / ; Irish : <unk> [ <unk> ] ; Ulster @-@ Scots : <unk> [ <unk> ] ) is an island in the North Atlantic. It is separated from Great Britain to its east by the North Channel, the Irish Sea, and St George's Channel. Ireland is the second @-@ largest island of the British Isles, the third @-@ largest in Europe, and the twentieth @-@ largest on Earth. \n <unk>, Ireland is divided between the Republic of Ireland ( officially named Ireland ), which covers five @-@ <unk> of the island, and Northern Ireland, which is part of the United Kingdom, in the northeast of the island. In 2011 the population of Ireland was about 6 @.@ 4 million, ranking it the second @-@ most populous island in Europe after Great Britain. Just under 4 @.@ 6 million live in the Republic of Ireland and just over 1 @.@ 8 million live in Northern Ireland. \n The island's geography comprises relatively low @-@ lying mountains surrounding a central plain, with several navigable rivers extending inland. The island has lush vegetation, a product of its mild but changeable climate which avoids extremes in temperature. <unk> woodlands covered the island until the Middle Ages. As of 2013, the amount of land that is wooded in Ireland is about 11 % of the total, compared with a European average of 35 %. There are twenty @-@ six extant mammal species native to Ireland. The Irish climate is very moderated and classified as oceanic. As a result, winters are <unk> than expected for such a northerly area. However, summers are cooler than those in Continental Europe. Rainfall and cloud cover are abundant. \n The earliest evidence of human presence in Ireland is dated at 10 @,@ 500 BC. Gaelic Ireland had emerged by the 1st century AD and lasted until the First World War. The island was <unk> from the 5th century onward. Following the Norman invasion in the 12th century, England claimed sovereignty over Ireland. However, English rule did not extend over the whole island until the 16th – 17th century Tudor conquest, which led to colonisation by settlers from Britain. In the <unk>, a system of Protestant English rule was designed to materially disadvantage the Catholic majority and Protestant dissenters, and was extended during the 18th century. With the Acts of Union in 1801, Ireland became a part of the United Kingdom. A war of independence in the early 20th century was followed by the partition of the island, creating the Irish Free State, which became increasingly sovereign over the following decades, and Northern Ireland, which remained a part of the United Kingdom. Northern Ireland saw much civil unrest from the late 1960s until the 1990s. This subsided following a political agreement in 1998. In 1973 the Republic of Ireland joined the European Economic Community while the United Kingdom, and Northern Ireland, as part of it, did the same. \n Irish culture has had a significant influence on other cultures, especially in the fields of literature. Alongside mainstream Western culture, a strong indigenous culture exists, as expressed through Gaelic games, Irish music, and the Irish language. The culture of the island also shares many features with that of Great Britain, including the English language, and sports such as association football, rugby, horse racing, and golf. \n \n = = Name = = \n \n Ireland consists of Old Irish <unk> + English land. <unk> derives from Proto @-@ Celtic * <unk> ( compare Welsh <unk> ), which is also the source of Latin <unk>. <unk> derives from a root meaning " fat, prosperous ". \n \n = = History = = \n \n \n = = = Prehistoric Ireland = = = \n \n During the last glacial period, and up until about 9000 years ago, most of Ireland was covered with ice, most of the time. Sea levels were lower and Ireland, like Great Britain, formed part of continental Europe. By 12 @,@ 000 BC, rising sea levels due to ice melting caused Ireland to become separated from Great Britain. Later, around <unk> BC, Great Britain itself became separated from continental Europe. The earliest evidence of human presence in Ireland is dated at 10 @,@ 500 BC. Until recently the earliest evidence of humans in Ireland were Mesolithic people who arrived by boat from Britain between 8000 BC and 7000 BC. \n From about <unk> BC, Neolithic settlers arrived introducing cereal <unk>, a housing culture ( similar to those of the same period in Scotland ) and stone monuments. A more advanced agriculture was to develop. At the <unk> Fields, preserved beneath a blanket of peat in present @-@ day County Mayo, is an extensive field system, arguably the oldest in the

Masked LM

Coming Soon

Cleanup