This module contains custom models, custom splitters, etc... for both causal and MLM language modeling tasks. This includes things like training BERT from scratch or fine-tuning a particular pre-trained LM on your own corpus.
 
What we're running with at the time this documentation was generated:
torch: 1.9.0+cu102
fastai: 2.5.2
transformers: 4.10.0
wiki_path = untar_data(URLs.WIKITEXT_TINY)

train_df = pd.read_csv(wiki_path/'train.csv', header=None)
valid_df = pd.read_csv(wiki_path/'test.csv', header=None)

train_df['is_valid'] = False
valid_df['is_valid'] = True

df = pd.concat([train_df, valid_df])

print(len(df))
df.head()
662
0 is_valid
0 \n = 2013 – 14 York City F.C. season = \n \n The 2013 – 14 season was the <unk> season of competitive association football and 77th season in the Football League played by York City Football Club , a professional football club based in York , North Yorkshire , England . Their 17th @-@ place finish in 2012 – 13 meant it was their second consecutive season in League Two . The season ran from 1 July 2013 to 30 June 2014 . \n Nigel Worthington , starting his first full season as York manager , made eight permanent summer signings . By the turn of the year York were only above the relegation z... False
1 \n = Big Boy ( song ) = \n \n " Big Boy " <unk> " I 'm A Big Boy Now " was the first single ever recorded by the Jackson 5 , which was released by Steeltown Records in January 1968 . The group played instruments on many of their Steeltown compositions , including " Big Boy " . The song was neither a critical nor commercial success , but the Jackson family were delighted with the outcome nonetheless . \n The Jackson 5 would release a second single with Steeltown Records before moving to Motown Records . The group 's recordings at Steeltown Records were thought to be lost , but they were re... False
2 \n = The Remix ( Lady Gaga album ) = \n \n The Remix is a remix album by American recording artist Lady Gaga . Released in Japan on March 3 , 2010 , it contains remixes of the songs from her first studio album , The Fame ( 2008 ) , and her third extended play , The Fame Monster ( 2009 ) . A revised version of the track list was prepared for release in additional markets , beginning with Mexico on May 3 , 2010 . A number of recording artists have produced the songs , including Pet Shop Boys , Passion Pit and The Sound of Arrows . The remixed versions feature both uptempo and <unk> composit... False
3 \n = New Year 's Eve ( Up All Night ) = \n \n " New Year 's Eve " is the twelfth episode of the first season of the American comedy television series Up All Night . The episode originally aired on NBC in the United States on January 12 , 2012 . It was written by Erica <unk> and was directed by Beth McCarthy @-@ Miller . The episode also featured a guest appearance from Jason Lee as Chris and Reagan 's neighbor and Ava 's boyfriend , Kevin . \n During Reagan ( Christina Applegate ) and Chris 's ( Will <unk> ) first New Year 's Eve game night , Reagan 's competitiveness comes out causing Ch... False
4 \n = Geopyxis carbonaria = \n \n Geopyxis carbonaria is a species of fungus in the genus Geopyxis , family <unk> . First described to science in 1805 , and given its current name in 1889 , the species is commonly known as the charcoal loving elf @-@ cup , dwarf <unk> cup , <unk> <unk> cup , or pixie cup . The small , <unk> @-@ shaped fruitbodies of the fungus are reddish @-@ brown with a whitish fringe and measure up to 2 cm ( 0 @.@ 8 in ) across . They have a short , tapered stalk . Fruitbodies are commonly found on soil where brush has recently been burned , sometimes in great numbers .... False

class LM_MetricsCallback[source]

LM_MetricsCallback(**kwargs) :: Callback

A fastai friendly metric implemented as a callback so that we can handle use cases where we don't want to count tokens marked to be ignored or else not count batches where there are no targs

Parameters:

  • kwargs : <class 'inspect._empty'>

Causal Language Modeling

In causal language modeling, we are attempting to predict the next token given those before it.

model_cls = AutoModelForCausalLM

pretrained_model_name = "gpt2"
hf_arch, hf_config, hf_tokenizer, hf_model = BLURR.get_hf_objects(pretrained_model_name, model_cls=model_cls)

if (hf_tokenizer.pad_token is None): hf_tokenizer.pad_token = '[PAD]'
Using pad_token, but it is not set yet.
bbtfm = HF_LMBeforeBatchTransform(hf_arch, hf_config, hf_tokenizer, hf_model, lm_strategy_cls=CausalLMStrategy)
blocks = (HF_TextBlock(before_batch_tfm=bbtfm, input_return_type=HF_CausalLMInput), noop)

dblock = DataBlock(blocks=blocks, get_x=ColReader(0), splitter=ColSplitter(col='is_valid'))
dls = dblock.dataloaders(df, bs=2)
b = dls.one_batch()
b[0]['input_ids'].shape, b[0]['labels'].shape, b[1].shape
(torch.Size([2, 1024]), torch.Size([2, 1024]), torch.Size([2, 1024]))
dls.show_batch(dataloaders=dls, max_n=2, trunc_at=500)
text target
0 \n = Bob Dylan = \n \n Bob Dylan ( / <unk> / ; born Robert Allen Zimmerman, May 24, 1941 ) is an American singer @-@ songwriter, artist and writer. He has been influential in popular music and culture for more than five decades. Much of his most celebrated work dates from the 1960s when his songs chronicled social unrest, although Dylan repudiated suggestions from journalists that he was a spokesman for his generation. Nevertheless, early songs such as " Blowin'in the Wind " and " The Times They A \n = Bob Dylan = \n \n Bob Dylan ( / <unk> / ; born Robert Allen Zimmerman, May 24, 1941 ) is an American singer @-@ songwriter, artist and writer. He has been influential in popular music and culture for more than five decades. Much of his most celebrated work dates from the 1960s when his songs chronicled social unrest, although Dylan repudiated suggestions from journalists that he was a spokesman for his generation. Nevertheless, early songs such as " Blowin'in the Wind " and " The Times They Ar
1 \n = Wales national rugby union team = \n \n The Wales national rugby union team ( Welsh : <unk> <unk> <unk> <unk> <unk> Cymru ) represent Wales in international rugby union. They compete annually in the Six Nations Championship with England, France, Ireland, Italy and Scotland. Wales have won the Six Nations and its predecessors 26 times outright, one less than England. Wales'most recent championship win came in 2013. \n The governing body, the Welsh Rugby Union ( WRU ), was established in 1881, t \n = Wales national rugby union team = \n \n The Wales national rugby union team ( Welsh : <unk> <unk> <unk> <unk> <unk> Cymru ) represent Wales in international rugby union. They compete annually in the Six Nations Championship with England, France, Ireland, Italy and Scotland. Wales have won the Six Nations and its predecessors 26 times outright, one less than England. Wales'most recent championship win came in 2013. \n The governing body, the Welsh Rugby Union ( WRU ), was established in 1881, th
model = HF_BaseModelWrapper(hf_model)
fit_cbs = [LM_MetricsCallback()]

learn = Learner(dls, 
                model,
                opt_func=partial(Adam),
                loss_func=HF_PreCalculatedLoss(),
                cbs=[HF_BaseModelCallback],
                metrics=[perplexity],
                splitter=hf_splitter).to_fp16()

learn.freeze()
learn.summary()
# preds = learn.model(b[0])
# len(preds),preds[0], preds[1].shape
print(len(learn.opt.param_groups))
5
learn.lr_find(suggest_funcs=[minimum, steep, valley, slide])
/home/wgilliam/miniconda3/envs/blurr/lib/python3.9/site-packages/fastai/callback/schedule.py:270: UserWarning: color is redundantly defined by the 'color' keyword argument and the fmt string "ro" (-> color='r'). The keyword argument will take precedence.
  ax.plot(val, idx, 'ro', label=nm, c=color)
SuggestedLRs(minimum=0.19054607152938843, steep=1.5848932266235352, valley=0.0002754228771664202, slide=0.2089296132326126)
learn.fit_one_cycle(1, lr_max=3e-3, cbs=fit_cbs)
epoch train_loss valid_loss perplexity lm_accuracy time
0 3.044529 2.811769 16.639324 0.456156 01:46
learn.show_results(learner=learn, trunc_at=250)
text target prediction
0 \n = Military history of Australia = \n \n The military history of Australia spans the nation's 220 @-@ year modern history, from the early Australian frontier wars between <unk> and Europeans to the ongoing conflicts in Iraq and Afghanistan in the ear \n = Military history of Australia = \n \n The military history of Australia spans the nation's 220 @-@ year modern history, from the early Australian frontier wars between <unk> and Europeans to the ongoing conflicts in Iraq and Afghanistan in the earl \n = / = the = \n \n Australian history of Australia is the period froms history- 1@ century history history, from the early days colonies to to theunk> and < to the early conflict between the and Afghanistan. the late 20st century. the history is no
1 \n = Air Rhodesia Flight <unk> = \n \n Air Rhodesia Flight <unk> was a scheduled passenger flight that was shot down by the Zimbabwe People's Revolutionary Army ( <unk> ) on 3 September 1978, during the Rhodesian Bush War. The aircraft involved, a Vick \n = Air Rhodesia Flight <unk> = \n \n Air Rhodesia Flight <unk> was a scheduled passenger flight that was shot down by the Zimbabwe People's Revolutionary Army ( <unk> ) on 3 September 1978, during the Rhodesian Bush War. The aircraft involved, a Vicke \n = Force = = # /> = \n \n Rhodesia Flight (unk> was a flight flight flight from was scheduled down by a Uan'ss Liberation Army ( Zunk> ) on July August 2014. and the countryian militaryi. plane was was the Boeing- Fickersount, " "unk>, was a from <
learn.blurr_generate('Blurr is fun to work with because', max_length=50, do_sample=True, top_k=25)
[' Blurr is fun to work with because he can do some amazing stuff on a guitar while making it sound really good.\n\nWhen asked whether he had ever performed in his "Stones " set to be recorded for an upcoming album , the']

Masked Language Modeling

In masked language modeling (MLM), we are attempting to predict the masked tokens. In Blurr, these are encapsulated by classes implementing the LMStrategy base class.

For a list of some of the more common strategies, see table 3 of the Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer paper. When fine-tuning a MLM model. you'll want to make sure you use the same approach as the model authors should you be looking to reproduce their results ... but our approach here makes it easy to play with different strategies regardless.

In the example below, we'll tell Blurr we want to use the BERT-style masking strategy.

model_cls = AutoModelForMaskedLM

pretrained_model_name = "distilroberta-base"
hf_arch, hf_config, hf_tokenizer, hf_model = BLURR.get_hf_objects(pretrained_model_name, model_cls=model_cls)
bbtfm = HF_LMBeforeBatchTransform(hf_arch, hf_config, hf_tokenizer, hf_model, lm_strategy_cls=BertMLMStrategy)
blocks = (HF_TextBlock(before_batch_tfm=bbtfm, input_return_type=HF_MLMInput), noop)

dblock = DataBlock(blocks=blocks, get_x=ColReader(0), splitter=ColSplitter(col='is_valid'))
dls = dblock.dataloaders(df, bs=2)
b = dls.one_batch()
b[0]['input_ids'].shape, b[0]['labels'].shape, b[1].shape
(torch.Size([2, 512]), torch.Size([2, 512]), torch.Size([2, 512]))
dls.show_batch(dataloaders=dls, max_n=2, trunc_at=250)
text target
0 [ =] <mask> <mask> = \n <mask> \n Bob Dylan ( <mask> <mask> ; born Robert Allen Zimmerman <mask> May 24 , 1941 ) is an <mask> singer @ - @ song writer , artist and writer . He <mask> been influential in popular music and culture for more than five decades . Much of his most celebrated work dates <mask> the 1960 <mask> when his songs chronic led social unrest , although Dylan repud iated suggestions from journalists that he was a spokesman for his generation [ (_] Nevertheless , early songs such <mask> " Blow in ' in the Wind " <mask> " The Times They Are a @ <mask> @ ' " became an the ms for the American civil rights and <mask> @ - @ war movements . [ Delay] he left his initial base in the American folk music revival , [ his] six @ - @ minute <mask> " Like a Rolling [ motions] " altered the range of popular <mask> in [papers] . <mask> mid @ - @ 1960 s recordings , <mask> by rock <mask> , reached <mask> <mask> end of the United States music charts while also attracting and criticism from others <mask> the folk <mask> . \n Dylan ' s <mask> have incorporated various political , social , philosophical [,] and literary influences . They <mask> existing pop music conventions and appealed to <mask> burgeoning counter culture . Initially inspired <mask> the performances <mask> Little Richard and the [ =] [ Bob] [ Dylan] = \n [ ] \n Bob Dylan ( [ /] [ /] ; born Robert Allen Zimmerman [,] May 24 , 1941 ) is an [ American] singer @ - @ song writer , artist and writer . He [ has] been influential in popular music and culture for more than five decades . Much of his most celebrated work dates [ from] the 1960 [s] when his songs chronic led social unrest , although Dylan repud iated suggestions from journalists that he was a spokesman for his generation [.] Nevertheless , early songs such [ as] " Blow in ' in the Wind " [ and] " The Times They Are a @ [-] @ ' " became an the ms for the American civil rights and [ anti] @ - @ war movements . [ After] he left his initial base in the American folk music revival , [ his] six @ - @ minute [ single] " Like a Rolling [ Stone] " altered the range of popular [ music] in [ 1965] . [ His] mid @ - @ 1960 s recordings , [ backed] by rock [ musicians] , reached [ the] [ top] end of the United States music charts while also attracting and criticism from others [ in] the folk [ movement] . \n Dylan ' s [ lyrics] have incorporated various political , social , philosophical [,] and literary influences . They [ defied] existing pop music conventions and appealed to [ the] burgeoning counter culture . Initially inspired [ by] the performances [ of] Little Richard and the
1 = Missouri River = \n \n The Missouri River is the longest river in North America . Rising in the Rocky Mountains of western Montana <mask> the Missouri flows east and south for 2 @ , @ 341 miles ( 3 <mask> , @ 7 67 km <mask> before entering the Mississippi River north of St . Louis , Missouri . The river takes drainage from a sparse ly populated [atomic] semi @ - @ ar id watershed of more than <mask> a million square miles [ (] 1 @ , @ 300 <mask> <mask> @ <mask> km <mask> <mask> , which includes parts of ten U . S . states and two Canadian provinces <mask> <mask> combined with the lower Mississippi River , it <mask> the world ' s fourth longest <mask> system . \n For <mask> <mask> @ , <mask> 000 years , <mask> <mask> depended on the Missouri River and its t <mask> aries as a source of sust [285] and transportation . More than ten major groups of Native Americans populated the watershed <mask> most leading a nom [adic] lifestyle and [ongevity] on enormous buffalo herds that once ro amed <mask> the Great Plains . <mask> <mask> Europeans encountered the river in the late sevent eenth century , and the region <mask> through Spanish and French hands before finally becoming part of the United States through the <mask> Purchase . The Missouri was long believed to be part <mask> the <mask> <mask> = Missouri River = \n \n The Missouri River is the longest river in North America . Rising in the Rocky Mountains of western Montana [,] the Missouri flows east and south for 2 @ , @ 341 miles ( 3 [ @] , @ 7 67 km [ )] before entering the Mississippi River north of St . Louis , Missouri . The river takes drainage from a sparse ly populated [,] semi @ - @ ar id watershed of more than [ half] a million square miles [ (] 1 @ , @ 300 [ @] [,] @ [ 000] km [2] [ )] , which includes parts of ten U . S . states and two Canadian provinces [.] [ When] combined with the lower Mississippi River , it [ forms] the world ' s fourth longest [ river] system . \n For [ over] [ 12] @ , [@] 000 years , [ people] [ have] depended on the Missouri River and its t [ribut] aries as a source of sust [enance] and transportation . More than ten major groups of Native Americans populated the watershed [,] most leading a nom [adic] lifestyle and [ dependent] on enormous buffalo herds that once ro amed [ through] the Great Plains . [ The] [ first] Europeans encountered the river in the late sevent eenth century , and the region [ passed] through Spanish and French hands before finally becoming part of the United States through the [ Louisiana] Purchase . The Missouri was long believed to be part [ of] the [ Northwest] [ Passage]
model = HF_BaseModelWrapper(hf_model)
fit_cbs = [LM_MetricsCallback()]

learn = Learner(dls, 
                model,
                opt_func=partial(Adam, decouple_wd=True),
                loss_func= HF_PreCalculatedLoss(),
                cbs=[HF_BaseModelCallback],
                metrics=[perplexity],
                splitter=hf_splitter).to_fp16()

learn.freeze()
learn.summary()
print(len(learn.opt.param_groups))
3
learn.lr_find(suggest_funcs=[minimum, steep, valley, slide])
/home/wgilliam/miniconda3/envs/blurr/lib/python3.9/site-packages/fastai/callback/schedule.py:270: UserWarning: color is redundantly defined by the 'color' keyword argument and the fmt string "ro" (-> color='r'). The keyword argument will take precedence.
  ax.plot(val, idx, 'ro', label=nm, c=color)
SuggestedLRs(minimum=0.00036307806149125097, steep=9.12010818865383e-07, valley=0.00010964782268274575, slide=0.00015848931798245758)
learn.fit_one_cycle(1, lr_max=1e-4, cbs=fit_cbs)
epoch train_loss valid_loss perplexity lm_accuracy time
0 1.842754 1.759045 5.806890 0.634658 00:30
learn.show_results(learner=learn, trunc_at=250)
text target prediction
0 = Military history <mask> Australia = \n <mask> The military history of Australia spans the nation ' s 220 @ - @ year modern history , <mask> the early Australian frontier wars between and <mask> <mask> <mask> [abel] conflicts in Iraq and Afghanistan <mask> the early 21 st century . Although this history is short when compared to that <mask> many other nations [ escorted] Australia has been involved in numerous conflicts <mask> wars , and war and military service have been significant <mask> <mask> [LER] society and national identity , including the An zac spirit <mask> The relationship between war and Australian society has [ also] been shaped by the enduring themes of Australian strategic culture and <mask> unique security [ dilemma] . \n As British , the Australian colonies participated in [ Britain] ' s small wars of the 19 th century , while later as a dominion [ workplaces] and then an independent <mask> , Australia fought in the <mask> World War and Second <mask> War , as well as in the wars in <mask> , Mal aya , B orne o and Vietnam during the Cold War . In the <mask> @ - @ <mask> era Australian <mask> have been involved <mask> numerous international peace keeping missions <mask> through the United <mask> and other agencies , including in the Sinai , <mask> Gulf , <mask> , Somalia , East <mask> <mask> and <mask> Solomon Islands , while more recently they have also fought = Military history [ of] Australia = \n [\n] The military history of Australia spans the nation ' s 220 @ - @ year modern history , [ from] the early Australian frontier wars between and [ Europeans] [ to] [ the] [ ongoing] conflicts in Iraq and Afghanistan [ in] the early 21 st century . Although this history is short when compared to that [ of] many other nations [,] Australia has been involved in numerous conflicts [ and] wars , and war and military service have been significant [ influences] [ on] [ Australian] society and national identity , including the An zac spirit [.] The relationship between war and Australian society has [ also] been shaped by the enduring themes of Australian strategic culture and [ its] unique security [ dilemma] . \n As British , the Australian colonies participated in [ Britain] ' s small wars of the 19 th century , while later as a dominion [,] and then an independent [ nation] , Australia fought in the [ First] World War and Second [ World] War , as well as in the wars in [ Korea] , Mal aya , B orne o and Vietnam during the Cold War . In the [ Post] @ - @ [ Vietnam] era Australian [ forces] have been involved [ in] numerous international peace keeping missions [,] through the United [ Nations] and other agencies , including in the Sinai , [ Persian] Gulf , [ ] , Somalia , East [ Tim] [or] and [ the] Solomon Islands , while more recently they have also fought = Military history [ and] Australia = \n [\n] The military history of Australia spans the nation ' s 220 @ - @ year modern history , [ including] the early Australian frontier wars between and [ ] [ ] [ ] [ and] conflicts in Iraq and Afghanistan [ in] the early 21 st century . Although this history is short when compared to that [,] many other nations [,] Australia has been involved in numerous conflicts [ of] wars , and war and military service have been significant [ influences] [ to] [ Australian] society and national identity , including the An zac spirit [.] The relationship between war and Australian society has [ also] been shaped by the enduring themes of Australian strategic culture and [ a] unique security [ dilemma] . \n As British , the Australian colonies participated in [ Britain] ' s small wars of the 19 th century , while later as a dominion [ state] and then an independent [ nation] , Australia fought in the [ First] World War and Second [ World] War , as well as in the wars in [ Cambodia] , Mal aya , B orne o and Vietnam during the Cold War . In the [ ] @ - @ [ ] era Australian [ troops] have been involved [ in] numerous international peace keeping missions [,] through the United [ Nations] and other agencies , including in the Sinai , [ Persian] Gulf , [ South] , Somalia , East [ Ang] [,] and [ the] Solomon Islands , while more recently they have also fought
1 = <mask> Rhodes ia Flight [ =] \n \n Air Rhodes ia Flight was a scheduled passenger flight <mask> was shot down by the <mask> People ' s Revolutionary Army ( <mask> on 3 September 1978 , during [ tumultuous] <mask> ian Bush War . The aircraft <mask> , a V ickers <mask> isc ount <mask> the , was flying the last leg of Air Rhodes ia ' s regular <mask> service <mask> Victoria [Production] to <mask> <mask> Sal isbury , via the resort [ town] of <mask> \n Soon after Flight took off , a group of gu err illas scored a direct hit on its star board wing with a Soviet [ RL] - @ made 2 surface @ - @ to @ - @ <mask> infrared missile , critically damaging the aircraft [ and] forcing an emergency landing . An attempted belly landing in a cotton field just <mask> of was by an unseen ditch <mask> which caused <mask> plane to cart wheel <mask> break up . Of the 52 passengers and <mask> crew , 38 died in this crash ; the insurgents then approached the <mask> , rounded up the 10 <mask> they could see and massac [ Hilton] them with <mask> gunfire . <mask> passengers survived by hiding in <mask> surrounding bush , while a further five lived because they had gone to <mask> for water before the <mask> <mask> illas arrived . = [ Air] Rhodes ia Flight [ =] \n \n Air Rhodes ia Flight was a scheduled passenger flight [ that] was shot down by the [ Zimbabwe] People ' s Revolutionary Army ( [ )] on 3 September 1978 , during [ the] [ Rhodes] ian Bush War . The aircraft [ involved] , a V ickers [ V] isc ount [ named] the , was flying the last leg of Air Rhodes ia ' s regular [ scheduled] service [ from] Victoria [ Falls] to [ the] [ capital] Sal isbury , via the resort [ town] of [.] \n Soon after Flight took off , a group of gu err illas scored a direct hit on its star board wing with a Soviet [ @] - @ made 2 surface @ - @ to @ - @ [ air] infrared missile , critically damaging the aircraft [ and] forcing an emergency landing . An attempted belly landing in a cotton field just [ west] of was by an unseen ditch [,] which caused [ the] plane to cart wheel [ and] break up . Of the 52 passengers and [ four] crew , 38 died in this crash ; the insurgents then approached the [ wreckage] , rounded up the 10 [ survivors] they could see and massac [red] them with [ automatic] gunfire . [ Three] passengers survived by hiding in [ the] surrounding bush , while a further five lived because they had gone to [ look] for water before the [ gu] [err] illas arrived . = [ ] Rhodes ia Flight [ =] \n \n Air Rhodes ia Flight was a scheduled passenger flight [ that] was shot down by the [ Zimbabwe] People ' s Revolutionary Army ( [ )] on 3 September 1978 , during [ the] [ Rhodes] ian Bush War . The aircraft [ carrier] , a V ickers [ V] isc ount [ of] the , was flying the last leg of Air Rhodes ia ' s regular [ passenger] service [ from] Victoria [,] to [ ] [ of] Sal isbury , via the resort [ town] of [.] \n Soon after Flight took off , a group of gu err illas scored a direct hit on its star board wing with a Soviet [ RL] - @ made 2 surface @ - @ to @ - @ [ an] infrared missile , critically damaging the aircraft [ and] forcing an emergency landing . An attempted belly landing in a cotton field just [ south] of was by an unseen ditch [,] which caused [ the] plane to cart wheel [ and] break up . Of the 52 passengers and [ the] crew , 38 died in this crash ; the insurgents then approached the [ aircraft] , rounded up the 10 [ people] they could see and massac [red] them with [ live] gunfire . [ Six] passengers survived by hiding in [ the] surrounding bush , while a further five lived because they had gone to [ look] for water before the [ gu] [err] illas arrived .

While Learner.blurr_generate will work well for causal LMs designed for text generation, it won't for MLM models designed to predict masked tokens. To accomodate the later, we add Learner.blurr_fill_mask ...

Learner.blurr_fill_mask[source]

Learner.blurr_fill_mask(inp:Union[List[int], str], n_preds:int=1, **kwargs)

For MLM models

Parameters:

  • inp : typing.Union[typing.List[int], str]

    Your input_ids or raw text string with a `hf_tokenizer.mask_token`

  • n_preds : <class 'int'>, optional

    The number of predictions you want to return for the [MASK]ed token

  • kwargs : <class 'inspect._empty'>
learn.blurr_fill_mask(f'The best place on earth is {hf_tokenizer.mask_token}.', n_preds=5)
['The best place on earth is here.',
 'The best place on earth is Antarctica.',
 'The best place on earth is home.',
 'The best place on earth is America.',
 'The best place on earth is heaven.']

BlearnerForLM

We can use the BlearnerForLM for either Causal or Masked language models. With one line of code, we get our DataBlock, DataLoaders, and Blearner with sensible defaults and ready for training

class BlearnerForLM[source]

BlearnerForLM(dls:DataLoaders, hf_model:PreTrainedModel, base_model_cb:HF_BaseModelCallback=HF_BaseModelCallback, loss_func=None, opt_func=Adam, lr=0.001, splitter=trainable_params, cbs=None, metrics=None, path=None, model_dir='models', wd=None, wd_bn_bias=False, train_bn=True, moms=(0.95, 0.85, 0.95)) :: Blearner

Group together a model, some dls and a loss_func to handle training

Parameters:

  • dls : <class 'fastai.data.core.DataLoaders'>

  • hf_model : <class 'transformers.modeling_utils.PreTrainedModel'>

  • kwargs : <class 'inspect._empty'>

Causal language modeling ...

learn = BlearnerForLM.from_dataframe(df, 'gpt2', text_attr=0, dblock_splitter=RandomSplitter(), 
                                     dl_kwargs={'bs':2}).to_fp16()
Using pad_token, but it is not set yet.
learn.dls.show_batch(dataloaders=learn.dls, max_n=2, trunc_at=500)
text target
0 \n = Military history of Australia = \n \n The military history of Australia spans the nation's 220 @-@ year modern history, from the early Australian frontier wars between <unk> and Europeans to the ongoing conflicts in Iraq and Afghanistan in the early 21st century. Although this history is short when compared to that of many other nations, Australia has been involved in numerous conflicts and wars, and war and military service have been significant influences on Australian society and national \n = Military history of Australia = \n \n The military history of Australia spans the nation's 220 @-@ year modern history, from the early Australian frontier wars between <unk> and Europeans to the ongoing conflicts in Iraq and Afghanistan in the early 21st century. Although this history is short when compared to that of many other nations, Australia has been involved in numerous conflicts and wars, and war and military service have been significant influences on Australian society and national i
1 \n = Ireland = \n \n Ireland ( / <unk> / ; Irish : <unk> [ <unk> ] ; Ulster @-@ Scots : <unk> [ <unk> ] ) is an island in the North Atlantic. It is separated from Great Britain to its east by the North Channel, the Irish Sea, and St George's Channel. Ireland is the second @-@ largest island of the British Isles, the third @-@ largest in Europe, and the twentieth @-@ largest on Earth. \n <unk>, Ireland is divided between the Republic of Ireland ( officially named Ireland ), which covers five @-@ <un \n = Ireland = \n \n Ireland ( / <unk> / ; Irish : <unk> [ <unk> ] ; Ulster @-@ Scots : <unk> [ <unk> ] ) is an island in the North Atlantic. It is separated from Great Britain to its east by the North Channel, the Irish Sea, and St George's Channel. Ireland is the second @-@ largest island of the British Isles, the third @-@ largest in Europe, and the twentieth @-@ largest on Earth. \n <unk>, Ireland is divided between the Republic of Ireland ( officially named Ireland ), which covers five @-@ <unk
learn.fit_one_cycle(1, lr_max=3e-3, cbs=[BlearnerForLM.get_metrics_cb()])
epoch train_loss valid_loss perplexity lm_accuracy time
0 3.654257 3.266997 26.232441 0.416250 01:42
learn.show_results(learner=learn, trunc_at=250)
text target prediction
0 \n = Laurence Olivier = \n \n Laurence Kerr Olivier, Baron Olivier, <unk> ( / <unk> <unk> <unk> / ; 22 May 1907 – 11 July 1989 ) was an English actor who, along with his contemporaries Ralph Richardson and John Gielgud, dominated the British stage of t \n = Laurence Olivier = \n \n Laurence Kerr Olivier, Baron Olivier, <unk> ( / <unk> <unk> <unk> / ; 22 May 1907 – 11 July 1989 ) was an English actor who, along with his contemporaries Ralph Richardson and John Gielgud, dominated the British stage of th \n\n =nce L\n\n\n =\n =nce Olivier, is aess, Baron <>, unkunk> )unk> )unk> ) < ) ) 18 ) < : 2007 ), a American novelist and was in with his wife, Wald and William Richardsonelkesud, was the British theatre and the 1920- @19 periodth century. He was was as
1 \n = Eva Perón = \n \n María Eva Duarte de Perón ( 7 May 1919 – 26 July 1952 ) was the second wife of Argentine President Juan Perón ( 1895 – 1974 ) and served as the First Lady of Argentina from 1946 until her death in 1952. She is usually referred to \n = Eva Perón = \n \n María Eva Duarte de Perón ( 7 May 1919 – 26 July 1952 ) was the second wife of Argentine President Juan Perón ( 1895 – 1974 ) and served as the First Lady of Argentina from 1946 until her death in 1952. She is usually referred to \n\n =-is <\n\n =\n =ía Per Perarte = laón =Eva ) 18 ) 7 May 2014 ) was a first daughter of Eva President Jose Perón and 7 –1945 ) and the as the first Vice of Argentina during 18 to her death in 1975. She was a depicted to as Eva Perón, ) Spanish <> ]u
learn.blurr_generate('Blurr is fun to work with because', max_length=50, do_sample=True, top_k=25)
[' Blurr is fun to work with because it helps me get hands on a one at a time for my work. It also helps me get my hands on the newbie tools for a newbie hobby."\n "I']

Masked language modeling

learn = BlearnerForLM.from_dataframe(df, 'bert-base-cased', lm_strategy_cls=BertMLMStrategy, 
                                     text_attr=0, dblock_splitter=RandomSplitter(), 
                                     dl_kwargs={'bs':2}).to_fp16()
learn.dls.show_batch(dataloaders=learn.dls, max_n=2, trunc_at=250)
text target
0 [MASK] Military history of Australia [MASK] The military history of Australia [MASK] the nation ' s 220 @ - @ year modern history , from the early Australian frontier wars between < [MASK] [MASK] > and Europeans [MASK] the ongoing conflicts in Iraq and Afghanistan in the [MASK] [MASK] [MASK] . Although this history is short when [MASK] to that of many other nations , [MASK] has been involved in numerous conflicts and wars , and war [MASK] military [MASK] have been significant influences [MASK] [MASK] society and national identity , including the An ##zac spirit . The relationship between war and Australian society has [MASK] been shaped by the enduring themes of [MASK] strategic culture and [335] unique [security] di ##lemma . As British < un ##k > , [MASK] Australian colonies participated [ὶ] Britain ['] [MASK] small wars of the 19th century , while later as a < un ##k [>] do ##mini ##on [MASK] and then an independent nation , Australia fought in the First World War and Second World War , as well as in the wars in Korea [MASK] Malaya , Borneo and Vietnam during the Cold War . In the Post [MASK] - [MASK] Vietnam era Australian forces have [MASK] involved in numerous international peace ##keeping missions , through the United Nations and other agencies , including in the Sinai , Persian Gulf , < [MASK] ##k > [MASK] [MASK] , [MASK] Timor and the [Deborah] Islands , while more recently [MASK] have also [=] Military history of Australia [=] The military history of Australia [spans] the nation ' s 220 @ - @ year modern history , from the early Australian frontier wars between < [un] [##k] > and Europeans [to] the ongoing conflicts in Iraq and Afghanistan in the [early] [21st] [century] . Although this history is short when [compared] to that of many other nations , [Australia] has been involved in numerous conflicts and wars , and war [and] military [service] have been significant influences [on] [Australian] society and national identity , including the An ##zac spirit . The relationship between war and Australian society has [also] been shaped by the enduring themes of [Australian] strategic culture and [its] unique [security] di ##lemma . As British < un ##k > , [the] Australian colonies participated [in] Britain ['] [s] small wars of the 19th century , while later as a < un ##k [>] do ##mini ##on [,] and then an independent nation , Australia fought in the First World War and Second World War , as well as in the wars in Korea [,] Malaya , Borneo and Vietnam during the Cold War . In the Post [@] - [@] Vietnam era Australian forces have [been] involved in numerous international peace ##keeping missions , through the United Nations and other agencies , including in the Sinai , Persian Gulf , < [un] ##k > [,] [Somalia] , [East] Timor and the [Solomon] Islands , while more recently [they] have also
1 = Air Rhodesia Flight [MASK] un ##k > = Air Rhodesia Flight < un [MASK] > was a scheduled passenger flight that was shot down by the [MASK] People [MASK] s Revolutionary Army ( < un ##k > ) on 3 September 1978 , [MASK] the Rhodesia ##n Bush War . [The] aircraft involved , a Vickers Viscount named the < un [MASK] > , [MASK] flying the last [emphasizing] of Air Rhodesia ' s regular [MASK] service from Victoria Falls to the capital Salisbury , [MASK] the resort town [MASK] < un ##k > [MASK] Soon after Flight < [MASK] [MASK] [MASK] took off , a [MASK] of [elite] un ##k > guerrilla ##s scored a direct hit on its star ##board [MASK] with a Soviet @ - @ made < un ##k > 2 surface [MASK] [MASK] @ [MASK] @ - @ air infrared < un ##k > missile , critically damaging the aircraft and forcing an emergency [MASK] . An attempted belly landing in a cotton field just west of < un ##k [MASK] was [MASK] un ##k > by an unseen [MASK] [MASK] which caused the plane [to] [##bo] ##wheel and break up . Of the 52 passengers and four [MASK] [MASK] 38 died in this [MASK] ; the insurgents then approached [the] wreckage , rounded [MASK] the [10] [MASK] they could see [and] massacre ##d them with [fluent] gunfire . Three passengers survived by hiding in the surrounding bush , while a further five lived = Air Rhodesia Flight [<] un ##k > = Air Rhodesia Flight < un [##k] > was a scheduled passenger flight that was shot down by the [Zimbabwe] People ['] s Revolutionary Army ( < un ##k > ) on 3 September 1978 , [during] the Rhodesia ##n Bush War . [The] aircraft involved , a Vickers Viscount named the < un [##k] > , [was] flying the last [leg] of Air Rhodesia ' s regular [scheduled] service from Victoria Falls to the capital Salisbury , [via] the resort town [of] < un ##k > [.] Soon after Flight < [un] [##k] [>] took off , a [group] of [<] un ##k > guerrilla ##s scored a direct hit on its star ##board [wing] with a Soviet @ - @ made < un ##k > 2 surface [@] [-] @ [to] @ - @ air infrared < un ##k > missile , critically damaging the aircraft and forcing an emergency [landing] . An attempted belly landing in a cotton field just west of < un ##k [>] was [<] un ##k > by an unseen [ditch] [,] which caused the plane [to] [cart] ##wheel and break up . Of the 52 passengers and four [crew] [,] 38 died in this [crash] ; the insurgents then approached [the] wreckage , rounded [up] the [10] [survivors] they could see [and] massacre ##d them with [automatic] gunfire . Three passengers survived by hiding in the surrounding bush , while a further five lived
learn.fit_one_cycle(1, lr_max=6e-4, cbs=[BlearnerForLM.get_metrics_cb()])
epoch train_loss valid_loss perplexity lm_accuracy time
0 2.623405 2.556298 12.888012 0.569083 00:40
learn.show_results(learner=learn, trunc_at=250)
text target prediction
0 = 2010 Haiti earthquake = The 2010 [MASK] earthquake [MASK] French : < un ##k [MASK] de [MASK] [à] < un ##k > ; Haitian C ##reole [Sino] < un ##k > [12] < un [MASK] > 2010 < un [MASK] > < un ##k > < un ##k > [MASK] was a [catastrophic] magnitude 7 @ [MASK] @ 0 M ##w earthquake , [MASK] an epic ##enter near the town of < [MASK] ##k [yoga] ( < un ##k > [MASK] , approximately [MASK] kilometres ( 16 [MASK] ) west of Port @ - @ au @ - @ [MASK] , Haiti ' s capital . [MASK] earthquake occurred at [MASK] : 53 local time [MASK] 21 : 53 UTC ) on Tuesday , 12 [MASK] 2010 . By 24 January , at [##aic] 52 after ##sh ##ocks measuring [MASK] @ . [MASK] 5 or greater had been recorded . An estimated [MASK] million people were affected by the [MASK] ##ua ##ke [feared] Death toll estimates range from 100 @ [,] @ 000 to about 160 @ , @ 000 to Haitian government figures from 220 @ , @ 000 to 316 [MASK] , @ 000 that have [MASK] widely characterized as deliberately in [MASK] ##lated by the Haitian government [rotating] The [government] of Haiti [MASK] that [MASK] [MASK] , @ 000 residences and 30 @ , @ 000 commercial buildings had [MASK] [MASK] were severely damaged . [There] has [MASK] a history of national debt , < un = 2010 Haiti earthquake = The 2010 [Haiti] earthquake [(] French : < un ##k [>] de [2010] [à] < un ##k > ; Haitian C ##reole [:] < un ##k > [12] < un [##k] > 2010 < un [##k] > < un ##k > < un ##k > [)] was a [catastrophic] magnitude 7 @ [.] @ 0 M ##w earthquake , [with] an epic ##enter near the town of < [un] ##k [>] ( < un ##k > [)] , approximately [25] kilometres ( 16 [mi] ) west of Port @ - @ au @ - @ [Prince] , Haiti ' s capital . [The] earthquake occurred at [16] : 53 local time [(] 21 : 53 UTC ) on Tuesday , 12 [January] 2010 . By 24 January , at [least] 52 after ##sh ##ocks measuring [4] @ . [@] 5 or greater had been recorded . An estimated [three] million people were affected by the [q] ##ua ##ke [.] Death toll estimates range from 100 @ [,] @ 000 to about 160 @ , @ 000 to Haitian government figures from 220 @ , @ 000 to 316 [@] , @ 000 that have [been] widely characterized as deliberately in [##f] ##lated by the Haitian government [.] The [government] of Haiti [estimated] that [250] [@] , @ 000 residences and 30 @ , @ 000 commercial buildings had [collapsed] [or] were severely damaged . [There] has [been] a history of national debt , < un = 2010 Haiti earthquake = The 2010 [Haiti] earthquake [(] French : < un ##k [>] de [la] [à] < un ##k > ; Haitian C ##reole [:] < un ##k > [12] < un [##k] > 2010 < un [##k] > < un ##k > < un ##k > [)] was a [catastrophic] magnitude 7 @ [5] @ 0 M ##w earthquake , [with] an epic ##enter near the town of < [un] ##k [>] ( < un ##k > [)] , approximately [25] kilometres ( 16 [mi] ) west of Port @ - @ au @ - @ [Prince] , Haiti ' s capital . [The] earthquake occurred at [19] : 53 local time [(] 21 : 53 UTC ) on Tuesday , 12 [January] 2010 . By 24 January , at [least] 52 after ##sh ##ocks measuring [7] @ . [@] 5 or greater had been recorded . An estimated [one] million people were affected by the [q] ##ua ##ke [.] Death toll estimates range from 100 @ [,] @ 000 to about 160 @ , @ 000 to Haitian government figures from 220 @ , @ 000 to 316 [@] , @ 000 that have [been] widely characterized as deliberately in [##f] ##lated by the Haitian government [.] The [government] of Haiti [estimates] that [the] [@] , @ 000 residences and 30 @ , @ 000 commercial buildings had [been] [or] were severely damaged . [There] has [been] a history of national debt , < un
1 = Roger Fed ##erer = Roger Fed ##erer ( German : [ < un ##k [MASK] ] born 8 August 1981 ) is a Swiss professional tennis player who is currently [MASK] world No . 3 by the Association of Tennis < un ##k > ( ATP [MASK] . [His] accomplishments in professional tennis have [haunted] to him being regarded by [MASK] as [MASK] greatest tennis player of all time . Fed ##erer turned professional in 1998 and has been continuously ranked in the top 10 since October 2002 . [MASK] ##erer holds several records of the Open Era : holding the [world] No [.] 1 position for [MASK] weeks ( including 237 consecutive weeks [MASK] ; [MASK] 17 Grand Slam singles [MASK] [##ler] reaching each Grand Slam final at least five times ( an all @ - @ time record ) [MASK] and [MASK] the Wimbledon [MASK] ten times . He is among the eight men ( and among the [MASK] in [MASK] Era ) to capture a career Grand Slam . [MASK] ##erer shares an Open Era [MASK] [Crane] most titles at [MASK] with Pete Sam ##pra ##s ( seven [MASK] and at the US Open [Viscount] [MASK] Connor ##s and Sam ##pra ##s ( five ) . [MASK] is the only male player to win five [MASK] US Open titles . Fed ##erer has reached 27 [MASK] [François] s singles Grand [enough] finals [MASK] [MASK] 10 in a row from the 2005 Wimbledon Championships to the 2007 = Roger Fed ##erer = Roger Fed ##erer ( German : [ < un ##k [>] ] born 8 August 1981 ) is a Swiss professional tennis player who is currently [ranked] world No . 3 by the Association of Tennis < un ##k > ( ATP [)] . [His] accomplishments in professional tennis have [led] to him being regarded by [many] as [the] greatest tennis player of all time . Fed ##erer turned professional in 1998 and has been continuously ranked in the top 10 since October 2002 . [Fed] ##erer holds several records of the Open Era : holding the [world] No [.] 1 position for [302] weeks ( including 237 consecutive weeks [)] ; [winning] 17 Grand Slam singles [titles] [;] reaching each Grand Slam final at least five times ( an all @ - @ time record ) [;] and [reaching] the Wimbledon [final] ten times . He is among the eight men ( and among the [five] in [Open] Era ) to capture a career Grand Slam . [Fed] ##erer shares an Open Era [record] [for] most titles at [Wimbledon] with Pete Sam ##pra ##s ( seven [)] and at the US Open [with] [Jimmy] Connor ##s and Sam ##pra ##s ( five ) . [He] is the only male player to win five [consecutive] US Open titles . Fed ##erer has reached 27 [men] ['] s singles Grand [Slam] finals [,] [including] 10 in a row from the 2005 Wimbledon Championships to the 2007 = Roger Fed ##erer = Roger Fed ##erer ( German : [ < un ##k [>] ] born 8 August 1981 ) is a Swiss professional tennis player who is currently [ranked] world No . 3 by the Association of Tennis < un ##k > ( ATP [)] . [His] accomplishments in professional tennis have [led] to him being regarded by [many] as [the] greatest tennis player of all time . Fed ##erer turned professional in 1998 and has been continuously ranked in the top 10 since October 2002 . [Fed] ##erer holds several records of the Open Era : holding the [world] No [.] 1 position for [38] weeks ( including 237 consecutive weeks [)] ; [his] 17 Grand Slam singles [travel] [##ler] reaching each Grand Slam final at least five times ( an all @ - @ time record ) [;] and [winning] the Wimbledon [Championships] ten times . He is among the eight men ( and among the [youngest] in [Open] Era ) to capture a career Grand Slam . [Fed] ##erer shares an Open Era [with] [the] most titles at [Wimbledon] with Pete Sam ##pra ##s ( seven [)] and at the US Open [with] [##s] Connor ##s and Sam ##pra ##s ( five ) . [He] is the only male player to win five [consecutive] US Open titles . Fed ##erer has reached 27 [of] ['] s singles Grand [Slam] finals [,] [the] 10 in a row from the 2005 Wimbledon Championships to the 2007
batch_tfm = first_blurr_tfm(learn.dls)
learn.blurr_fill_mask(f'The best place on earth is {batch_tfm.hf_tokenizer.mask_token}.', n_preds=5)
['The best place on earth is heaven.',
 'The best place on earth is here.',
 'The best place on earth is Heaven.',
 'The best place on earth is paradise.',
 'The best place on earth is Egypt.']

Summary

Whether you're using the low, mid, or high-level APIs, the modeling.language_modeling provides everything you need to train causual and masked language models