참고Clickhereto download the full example codeT5-Base Model for Summarization, Sentiment Classification, and Translation¶Authors:Pendo Abbo,Joe CummingsOverviewThis tutorial demonstrates how to use a pretrained T5 Model for summarization, sentiment classification, and
translation tasks. We will demonstrate how to use the torchtext library to:Build a text preprocessing pipeline for a T5 modelInstantiate a pretrained T5 model with base configurationRead in the CNNDM, IMDB, and Multi30k datasets and preprocess their texts in preparation for the modelPerform text summarization, sentiment classification, and translation참고This tutorial requires PyTorch 2.0.0 or later.The T5 model does not work with raw text. Instead, it requires the text to be transformed into numerical form
in order to perform training and inference. The following transformations are required for the T5 model:Tokenize textConvert tokens into (integer) IDsTruncate the sequences to a specified maximum lengthAdd end-of-sequence (EOS) and padding token IDsT5 uses aSentencePiecemodel for text tokenization. Below, we use a pretrainedSentencePiecemodel to build
the text preprocessing pipeline using torchtext’s T5Transform. Note that the transform supports both
batched and non-batched text input (for example, one can either pass a single sentence or a list of sentences), however the T5 model expects the input to be batched.fromtorchtext.modelsimportT5Transformpadding_idx=0eos_idx=1max_seq_len=512t5_sp_model_path="https://download.pytorch.org/models/text/t5_tokenizer_base.model"transform=T5Transform(sp_model_path=t5_sp_model_path,max_seq_len=max_seq_len,eos_idx=eos_idx,padding_idx=padding_idx,)Alternatively, we can also use the transform shipped with the pretrained models that does all of the above out-of-the-boxfromtorchtext.modelsimportT5_BASE_GENERATIONtransform=T5_BASE_GENERATION.transform()Model Preparation¶torchtext provides SOTA pretrained models that can be used directly for NLP tasks or fine-tuned on downstream tasks. Below
we use the pretrained T5 model with standard base configuration to perform text summarization, sentiment classification, and
translation. For additional details on available pretrained models, seethe torchtext documentationfromtorchtext.modelsimportT5_BASE_GENERATIONt5_base=T5_BASE_GENERATIONtransform=t5_base.transform()model=t5_base.get_model()model.eval()UsingGenerationUtils¶We can use torchtext’sGenerationUtilsto produce an output sequence based on the input sequence provided. This calls on the
model’s encoder and decoder, and iteratively expands the decoded sequences until the end-of-sequence token is generated
for all sequences in the batch. Thegeneratemethod shown below uses greedy search to generate the sequences. Beam search and
other decoding strategies are also supported.fromtorchtext.prototype.generateimportGenerationUtilssequence_generator=GenerationUtils(model)Datasets¶torchtext provides several standard NLP datasets. For a complete list, refer to the documentation
athttps://pytorch.org/text/stable/datasets.html. These datasets are built using composable torchdata
datapipes and hence support standard flow-control and mapping/transformation using user defined
functions and transforms.Below we demonstrate how to preprocess the CNNDM dataset to include the prefix necessary for the
model to identify the task it is performing. The CNNDM dataset has a train, validation, and test
split. Below we demo on the test split.The T5 model uses the prefix 《summarize》 for text summarization. For more information on task
prefixes, please visit Appendix D of theT5 Paper참고Using datapipes is still currently subject to a few caveats. If you wish
to extend this example to include shuffling, multi-processing, or
distributed learning, please seethis notefor further instructions.fromfunctoolsimportpartialfromtorch.utils.dataimportDataLoaderfromtorchtext.datasetsimportCNNDMcnndm_batch_size=5cnndm_datapipe=CNNDM(split="test")task="summarize"defapply_prefix(task,x):returnf"{task}: "+x[0],x[1]cnndm_datapipe=cnndm_datapipe.map(partial(apply_prefix,task))cnndm_datapipe=cnndm_datapipe.batch(cnndm_batch_size)cnndm_datapipe=cnndm_datapipe.rows2columnar(["article","abstract"])cnndm_dataloader=DataLoader(cnndm_datapipe,shuffle=True,batch_size=None)Alternately, we can also use batched API, for example, apply the prefix on the whole batch:defbatch_prefix(task,x):return{"article":[f'{task}: '+yforyinx["article"]],"abstract":x["abstract"]}cnndm_batch_size=5cnndm_datapipe=CNNDM(split="test")task='summarize'cnndm_datapipe=cnndm_datapipe.batch(cnndm_batch_size).rows2columnar(["article","abstract"])cnndm_datapipe=cnndm_datapipe.map(partial(batch_prefix,task))cnndm_dataloader=DataLoader(cnndm_datapipe,batch_size=None)We can also load the IMDB dataset, which will be used to demonstrate sentiment classification using the T5 model.
This dataset has a train and test split. Below we demo on the test split.The T5 model was trained on the SST2 dataset (also available in torchtext) for sentiment classification using the
prefixsst2sentence. Therefore, we will use this prefix to perform sentiment classification on the IMDB dataset.fromtorchtext.datasetsimportIMDBimdb_batch_size=3imdb_datapipe=IMDB(split="test")task="sst2 sentence"labels={"1":"negative","2":"positive"}defprocess_labels(labels,x):returnx[1],labels[str(x[0])]imdb_datapipe=imdb_datapipe.map(partial(process_labels,labels))imdb_datapipe=imdb_datapipe.map(partial(apply_prefix,task))imdb_datapipe=imdb_datapipe.batch(imdb_batch_size)imdb_datapipe=imdb_datapipe.rows2columnar(["text","label"])imdb_dataloader=DataLoader(imdb_datapipe,batch_size=None)Finally, we can also load the Multi30k dataset to demonstrate English to German translation using the T5 model.
This dataset has a train, validation, and test split. Below we demo on the test split.The T5 model uses the prefix 《translate English to German》 for this task.fromtorchtext.datasetsimportMulti30kmulti_batch_size=5language_pair=("en","de")multi_datapipe=Multi30k(split="test",language_pair=language_pair)task="translate English to German"multi_datapipe=multi_datapipe.map(partial(apply_prefix,task))multi_datapipe=multi_datapipe.batch(multi_batch_size)multi_datapipe=multi_datapipe.rows2columnar(["english","german"])multi_dataloader=DataLoader(multi_datapipe,batch_size=None)Generate Summaries¶We can put all of the components together to generate summaries on the first batch of articles in the CNNDM test set
using a beam size of 1.batch=next(iter(cnndm_dataloader))input_text=batch["article"]target=batch["abstract"]beam_size=1model_input=transform(input_text)model_output=sequence_generator.generate(model_input,eos_idx=eos_idx,num_beams=beam_size)output_text=transform.decode(model_output.tolist())foriinrange(cnndm_batch_size):print(f"Example{i+1}:\n")print(f"prediction:{output_text[i]}\n")print(f"target:{target[i]}\n\n")Summarization Output¶Summarization output might vary since we shuffle the dataloader.Example1:prediction:the24-year-oldhasbeentattooedforoveradecade.hehaslandedinaustraliatostartworkonanewcampaign.hesaysheis'taking it in your stride'tobehonest.target:London-basedmodelStephenJamesHendryfamedforhisfullbodytattoo.ThesupermodelisinSydneyforanewmodellingcampaign.Australianfansunderstoodtohavealreadylocatedhimathishotel.The24-year-oldheartthrobisrecentlysingle.Example2:prediction:astraypoochhasusedupatleastthreeofherownafterbeinghitbyacarandburiedinafield.thedogmanagedtostaggertoanearbyfarm,dirt-coveredandemaciated,whereshewasfound.shesufferedadislocatedjaw,leginjuriesandacaved-insinuscavity--andstillrequiressurgerytohelpherbreathe.target:Theia,abullybreedmix,wasapparentlyhitbyacar,whackedwithahammerandburiedinafield."She's a true miracle dog and she deserves a good life,"saysSaraMellado,whoislookingforahomeforTheia.Example3:prediction:mohammadJavadZarifarrivedinIranonasunnyfridaymorning.hehasgonealongwaytobringIraninfromthecoldandallowittorejointheinternationalcommunity.buttherearesomefactsabouthimthatarelesswell-known.target:MohammadJavadZarifhasspentmoretimewithJohnKerrythananyotherforeignminister.HeonceparticipatedinatakeoveroftheIranianConsulateinSanFrancisco.TheIranianforeignministertweetsinEnglish.Example4:prediction:fiveamericansweremonitoredforthreeweeksafterbeingexposedtoEbolainwestafrica.oneofthefivehadaheart-relatedissueandhasbeendischargedbuthasn'tleftthearea.theyarecliniciansforPartnersinHealth,aBoston-basedaidgroup.target:17AmericanswereexposedtotheEbolaviruswhileinSierraLeoneinMarch.AnotherpersonwasdiagnosedwiththediseaseandtakentohospitalinMaryland.NationalInstitutesofHealthsaysthepatientisinfairconditionafterweeksoftreatment.Example5:prediction:thestudentwasidentifiedduringaninvestigationbycampuspoliceandtheofficeofstudentaffairs.headmittedtoplacingthenooseonthetreeearlyWednesdaymorning.theincidentisoneofseveralrecentracisteventstoaffectcollegestudents.target:StudentisnolongeronDukeUniversitycampusandwillfacedisciplinaryreview.Schoolofficialsidentifiedstudentduringinvestigationandthepersonadmittedtohangingthenoose,Dukesays.Thenoose,madeofrope,wasdiscoveredoncampusabout2a.m.Generate Sentiment Classifications¶Similarly, we can use the model to generate sentiment classifications on the first batch of reviews from the IMDB test set
using a beam size of 1.batch=next(iter(imdb_dataloader))input_text=batch["text"]target=batch["label"]beam_size=1model_input=transform(input_text)model_output=sequence_generator.generate(model_input,eos_idx=eos_idx,num_beams=beam_size)output_text=transform.decode(model_output.tolist())foriinrange(imdb_batch_size):print(f"Example{i+1}:\n")print(f"input_text:{input_text[i]}\n")print(f"prediction:{output_text[i]}\n")print(f"target:{target[i]}\n\n")Sentiment Output¶Example1:

input_text:sst2sentence:Ilovesci-fiandamwillingtoputupwithalot.Sci-fi
movies/TVareusuallyunderfunded,under-appreciatedandmisunderstood.Itriedtolike
this,Ireallydid,butitistogoodTVsci-fiasBabylon5istoStarTrek(theoriginal).
Sillyprosthetics,cheapcardboardsets,stilteddialogues,CGthatdoesn't match thebackground, and painfully one-dimensional characters cannot be overcome with a 'sci-fi'setting. (I'msuretherearethoseofyououttherewhothinkBabylon5isgoodsci-fiTV.
It's not. It'sclichédanduninspiring.)WhileUSviewersmightlikeemotionandcharacter
development,sci-fiisagenrethatdoesnottakeitselfseriously(cf.StarTrek).Itmay
treatimportantissues,yetnotasaseriousphilosophy.It's really difficult to care aboutthe characters here as they are not simply foolish, just missing a spark of life. Theiractions and reactions are wooden and predictable, often painful to watch. The makers of EarthKNOW it'srubbishastheyhavetoalwayssay"Gene Roddenberry's Earth..."otherwisepeople
wouldnotcontinuewatching.Roddenberry's ashes must be turning in their orbit as this dull,cheap, poorly edited (watching it without advert breaks really brings this home) trudgingTrabant of a show lumbers into space. Spoiler. So, kill off a main character. And then bringhim back as another actor. Jeeez. Dallas all over again.prediction: negativetarget: negativeExample 2:input_text: sst2 sentence: Worth the entertainment value of a rental, especially if you likeaction movies. This one features the usual car chases, fights with the great Van Damme kickstyle, shooting battles with the 40 shell load shotgun, and even terrorist style bombs. Allof this is entertaining and competently handled but there is nothing that really blows youaway if you'veseenyoursharebefore.<br/><br/>Theplotismadeinterestingbythe
inclusionofarabbit,whichiscleverbuthardlyprofound.Manyofthecharactersare
heavilystereotyped--theangryveterans,theterrifiedillegalaliens,thecrookedcops,
theindifferentfeds,thebitchytoughladystationhead,thecrookedpolitician,thefat
federalewholookslikehewastypecastastheMexicaninaHollywoodmoviefromthe1940s.
Allpassablyactedbutagainnothingspecial.<br/><br/>Ithoughtthemainvillainswere
prettywelldoneandfairlywellacted.Bytheendofthemovieyoucertainlyknewwhothe
goodguyswereandweren't. There was an emotional lift as the really bad ones got their justdeserts. Very simplistic, but then you weren'texpectingHamlet,right?TheonlythingIfound
reallyannoyingwastheconstantcutstoVDsdaughterduringthelastfightscene.<br/><br/>
Notbad.Notgood.Passable4.

prediction:positive

target:negative


Example3:

input_text:sst2sentence:itsatotallyaveragefilmwithafewsemi-alrightactionsequences
thatmaketheplotseemalittlebetterandremindthevieweroftheclassicvandamfilms.
partsoftheplotdon't make sense and seem to be added in to use up time. the end plot is thatof a very basic type that doesn'tleavetheviewerguessingandanytwistsareobviousfromthe
beginning.theendscenewiththeflaskbacksdon't make sense as they are added in and seem tohave little relevance to the history of van dam'scharacter.notreallyworthwatchingagain,
bitdisappointedintheendproduction,eventhoughitisapparentitwasshotonalowbudget
certainshotsandsectionsinthefilmareofpoordirectedquality.

prediction:negative

target:negativeGenerate Translations¶Finally, we can also use the model to generate English to German translations on the first batch of examples from the Multi30k
test set.batch=next(iter(multi_dataloader))input_text=batch["english"]target=batch["german"]model_input=transform(input_text)model_output=sequence_generator.generate(model_input,eos_idx=eos_idx,num_beams=beam_size)output_text=transform.decode(model_output.tolist())foriinrange(multi_batch_size):print(f"Example{i+1}:\n")print(f"input_text:{input_text[i]}\n")print(f"prediction:{output_text[i]}\n")print(f"target:{target[i]}\n\n")Translation Output¶Example1:

input_text:translateEnglishtoGerman:Amaninanorangehatstarringatsomething.

prediction:EinMannineinemorangenHut,deranetwasschaut.

target:EinMannmiteinemorangefarbenenHut,deretwasanstarrt.


Example2:

input_text:translateEnglishtoGerman:ABostonTerrierisrunningonlushgreengrassinfrontofawhitefence.

prediction:EinBostonTerrierläuftaufüppigemgrünemGrasvoreinemweißenZaun.

target:EinBostonTerrierläuftübersaftig-grünesGrasvoreinemweißenZaun.


Example3:

input_text:translateEnglishtoGerman:Agirlinkarateuniformbreakingastickwithafrontkick.

prediction:EinMädcheninKarate-UniformbrichteinenStöckmiteinemFrontkick.

target:EinMädchenineinemKarateanzugbrichteinBrettmiteinemTritt.


Example4:

input_text:translateEnglishtoGerman:Fivepeoplewearingwinterjacketsandhelmetsstandinthesnow,withsnowmobilesinthebackground.

prediction:FünfMenschenmitWinterjackenundHelmenstehenimSchnee,mitSchneemobilenimHintergrund.

target:FünfLeuteinWinterjackenundmitHelmenstehenimSchneemitSchneemobilenimHintergrund.


Example5:

input_text:translateEnglishtoGerman:Peoplearefixingtheroofofahouse.

prediction:DieLeutefixierendasDacheinesHauses.

target:LeuteReparierendasDacheinesHauses.Total running time of the script:( 0 minutes  0.000 seconds)DownloadPythonsourcecode:t5_tutorial.pyDownloadJupyternotebook:t5_tutorial.ipynbGallery generated by Sphinx-Gallery