[{"_1":2,"_152":-5,"_153":-5},"loaderData",{"_3":4,"_13":14,"_21":22,"_131":132},"root",{"_5":-5,"_6":-7,"_7":8,"_9":-7,"_10":11,"_12":-7},"currentUser","primaryEmail","siteVerification","6LeIxAcTAAAAAJcZVRqyHh71UMIEGNQ_MXjiZKhI","toast","enableDev",false,"incidentStatus","routes/_layout",{"_15":-5,"_16":17,"_18":17,"_19":11,"_20":11},"username","isAskGordonEnabled",true,"isUsingNewTemplate","showMinimalLanding","showHoneypotTrap","routes/_layout.r.$namespace.$reponame",{"_23":24,"_25":-5,"_26":11,"_27":11,"_28":29,"_30":31,"_32":-5,"_33":11,"_34":35,"_84":-5,"_85":86,"_124":-5,"_125":11,"_126":17,"_127":11,"_128":-5,"_129":130},"canonicalUrl","https://hub.docker.com/r/code2k13/nlphose/tags","mcpServer","canSeeDockerfile","canSeeAutobuilds","namespace","code2k13","reponame","nlphose","cloudBuildSource","isAutobuildsAvailable","repo",{"_36":37,"_38":39,"_41":42,"_43":44,"_45":11,"_46":47,"_48":49,"_50":51,"_53":31,"_28":29,"_54":55,"_56":40,"_57":58,"_59":60,"_61":62,"_63":-5,"_64":58,"_65":66,"_67":11,"_68":29,"_69":70,"_75":11,"_76":77,"_81":29,"_82":83},"categories",[],"content_types",[40],"image","date_registered","2021-07-03T11:02:37.21794Z","description","","is_private","last_updated","2021-12-13T06:10:09.560117Z","last_modified","2024-10-16T13:48:34.145251Z","media_types",[52],"application/vnd.docker.container.image.v1+json","name","pull_count",1115,"repository_type","star_count",0,"status",1,"status_description","active","source","collaborator_count","full_description","### Project's Github URL : https://github.com/code2k13/nlphose\n### My website : https://ashishware.com\n\n### Table of Contents\n- [Introduction](#introduction)\n- [Installing and running with Docker](#installing-and-running-with-docker)\n- [Usage Examples](#usage-examples)\n- [Stopping the pipeline without loosing data](#stopping-the-pipeline-without-loosing-data)\n- [Monitoring progress](#monitoring-progress)\n- [Acknowledgements](#acknowledgements)\n\n## Introduction\n

\n \nThis is a collection of python scripts that perform NLP tasks on output of [twint](https://github.com/twintproject/twint).\nIdea is to simply pipe output from twint to these \nscripts. Every script perfroms a different type of NLP operation on tweets, and it's output can be in turn piped to another script.\n\nCurrently following scripts are present:\n* twint2json.py - converts from default twint format to json\n* senti.py - perfroms AFINN based sentiement analysis on the json formatted tweet and appends *afinn_score* field with AFINN afinn_score\n* entity.py - performs named entity recognition on json fromatted tweets using [spacy](https://github.com/explosion/spaCy)\n* lang.py - performs language identificiation using FastText and [lid.176.ftz](https://dl.fbaipublicfiles.com/fasttext/supervised-models/lid.176.ftz) model\n* chunk.py - extracts chunks using [NLTK's regex based chunking](https://www.nltk.org/book_1ed/ch07.html) feature. You need to supply a regex and a name for the regex\n* xformer.py - Adds power of transformers to the tool. It currently supports these three tasks: 'sentiment-analysis',\n 'question-answering' and 'zero-shot-classification'\n\n Examples of usage:\n * For 'sentiment-analysis' task, the --pipeline has to be specified as 'sentiment-analysis'\n ```\n ... | ./xformer.py --pipeline sentiment-analysis --param \"sports#news#history\"\n ```\n * For 'question-answering' task, the --param should be the question we want to ask\n ```\n ... | ./xformer.py --pipeline question-answering --param 'By how many runs did India win ?'\n ```\n * For 'zero-shot-classification' the --param should be a single string containing labels seperated by '#'\n ```\n ... | ./xformer.py --pipeline zero-shot-classification --param \"sports#news#history\"\n ``` \n Additionally, a GPU device can be specified by adding a optional parameter '--device'\n\n\n## Installing and running with Docker\nSimply run following command\n```\ndocker run --rm -it code2k13/nlphose:latest\n```\n\nthen start [running examples](#Usage-Examples)\n\n\n\n## Usage Examples\n\nGet positive tweets containing term netflix (AFINN) :\n```shell\ntwint -s netflix |\\\n./twint2json.py |\\\n./senti.py |\\\n./entity.py |\\\njq 'if (.afinn_score) > 5 then .text else empty end'\n```\nGet positive tweets containing term netflix (transformers) :\n```shell\ntwint -s netflix |\\\n./twint2json.py |\\\n./xformer.py --pipeline sentiment-analysis |\\\njq 'if (.xfrmr_sentiment_analysis[0].label) == \"POSITIVE\" then .text else empty end'\n```\nGet works of art (TV shows or movie names) in positive tweets containing term netflix :\n```shell\ntwint -s netflix |\\\n./twint2json.py |\\\n./senti.py |\\\n./entity.py |\\\njq 'if (.afinn_score) > 5 then .entities|.[]| select(.label == \"WORK_OF_ART\") | .entity else empty end'\n```\n\nGet works of art (TV shows or movie names) in negative tweets containing term netflix :\n```shell\ntwint -s netflix |\\\n./twint2json.py |\\\n./senti.py |\\\n./entity.py |\\\njq 'if (.afinn_score) < -5 then .entities|.[]| select(.label == \"WORK_OF_ART\") | .entity else empty end'\n```\n\nGet tweet and people mentioned in postive tweets about premierleague :\n```shell\ntwint -s premierleague |\\\n./twint2json.py |\\\n./senti.py |\\\n./entity.py |\\\njq ' if (.afinn_score) > 5 then . as $parent | .entities|.[]| select((.label == \"PERSON\") and .entity != \"Netflix\") | [$parent.text,.entity] else empty end'\n```\nGet tweets about India in hindi\n```shell\ntwint -s india |\\\n./twint2json.py |\\\n./lang.py |\\\njq ' if .lang == \"hi\" then .text else empty end'\n```\nGet noun phrases that match (adjective * noun/ noun * noun) in all tweets containing foldscope\n```shell\ntwint -s foldscope |\\\n./twint2json.py |\\\n./chunk.py observation '{|*}' |\\\njq ' if (.chunks | length) > 0 then .chunks else empty end'\n```\n\nGet all tweets containing the word 'rainfall' and extract location which experienced rainfall using quantion-answering task\n```shell\ntwint -s 'rainfall' |\\\n./twint2json.py |\\\n./xformer.py --pipeline question-answering --param 'where did it rain' |\\\njq '{\"text\":.text,\"answer\":.xfrmr_question_answering.answer}'\n```\n\nGet all tweets containing the word 'food' and classify if the tweet was about Indian, Italian, Mexican or Chinese food\n```shell\ntwint -s 'food' |\\\n./twint2json.py |\\\n./xformer.py --pipeline zero-shot-classification --param 'indian#italian#mexican#chinese' \n```\n\n## Stopping the pipeline without loosing data\nBecause the output of programs is piped to each other there is always a 'performace mismatch'. Some scripts run faster than others, so buffering needs to be used.\nThankfully, this is automatically handled by the operating system, we need not worry. However, this will lead to increased memory usage over time.\nTo stop the piped command without loosing any data, you just need to terminate the first command, which in above cases is *twint*.\n\nFirst we need to find the process id of the process running twint using below command:\n```shell\nps -aux |grep twint\n```\n\nthen kill it using:\n```shell\nkill - KILL xxxxx\n```\nEven if the first command is killed, the data is processed by subsequent commands. DO NOT stop the piped command using CTRL+C if you care about data loss. \n\n## Monitoring progress\nEvery command in the pipeline will have different rate of processing data. You can easily monitor the progress of each command in the pipeline \nusing the *pv* command.\n\nFirst you will have to download and install *pv* if you dont have it :\n```shell\napt-get install pv\n```\nOnce you have the *pv* command installed you need to modify your pipeline to use it at points where you want to monitor output/progress. \n\nFor example, let us look at a sample NLP pipeline:\n```shell\ntwint -s premierleague |\\\n./twint2json.py |\\\n./senti.py |\\\n./entity.py |\\\n./chunk.py observation '{|*}' > output.txt\n```\nThe above pipeline finds tweets about *premierleague* and performs sentiment analysis, named entity recognition and chunk extraction before saving \nthe output to *output.txt*. Now let us say we want to know how many tweets are being read per second and how many per second complete processing.\nTo do that we can use *pv -N INPUT -i 2 -l* command. This command reads count of lines *(-l)* piped into it and shows count of lines received per seconds.\nThe display is refreshed after two seconds *(-i 2)*. The count is shown with a unique label i.e *INPUT* in this case. \n\nWe can use the *pv* command at multiple positions in our pipes (with different labels), so that we can monitor the performance of overall pipeline at various points.\nIn the sample below, I have used *pv* command twice, with labels *INPUT* (for incoming tweets) and *PROCESSED* (for processed records).\n```shell\ntwint -s premierleague |\\\npv -N INPUT -i 2 -l|\\\n./twint2json.py |\\\n./senti.py |\\\n./entity.py |\\\n./chunk.py observation '{|*}' |\\\npv -N PROCESSED -i 2 -l > output.txt\n```\nYou should see similar output, when you run the command :\n\n![nlphose.gif](nlphosepv.gif)\n\n## Acknowledgements\n\n> Finn Årup Nielsen, “A new ANEW: evaluation of a word list for sentiment analysis in microblogs”, Proceedings of the ESWC2011 Workshop on ‘Making Sense of Microposts’: Big things come in small packages. Volume 718 in CEUR Workshop Proceedings: 93-98. 2011 May. Matthew Rowe, Milan Stankovic, Aba-Sah Dadzie, Mariann Hardey (editors)\n\n[1] A. Joulin, E. Grave, P. Bojanowski, T. Mikolov, [Bag of Tricks for Efficient Text Classification](https://arxiv.org/abs/1607.01759)\n>@article{joulin2016bag,\n title={Bag of Tricks for Efficient Text Classification},\n author={Joulin, Armand and Grave, Edouard and Bojanowski, Piotr and Mikolov, Tomas},\n journal={arXiv preprint arXiv:1607.01759},\n year={2016}\n}\n\n\n\n[2] A. Joulin, E. Grave, P. Bojanowski, M. Douze, H. Jégou, T. Mikolov, [FastText.zip: Compressing text classification models](https://arxiv.org/abs/1612.03651)\n>@article{joulin2016fasttext,\n title={FastText.zip: Compressing text classification models},\n author={Joulin, Armand and Grave, Edouard and Bojanowski, Piotr and Douze, Matthijs and J{\\'e}gou, H{\\'e}rve and Mikolov, Tomas},\n journal={arXiv preprint arXiv:1612.03651},\n year={2016}\n}\n","has_starred","hub_user","immutable_tags_settings",{"_71":11,"_72":73},"enabled","rules",[74],".*","is_automated","permissions",{"_78":17,"_79":11,"_80":11},"read","write","admin","user","storage_size",4445099846,"orgData","recentTags",[87],{"_88":40,"_89":90,"_91":92,"_93":94,"_95":96,"_46":113,"_114":90,"_115":29,"_116":52,"_53":117,"_118":119,"_120":104,"_121":113,"_122":62,"_123":17},"content_type","creator",13430375,"full_size",1089113804,"id",160679857,"images",[97],{"_98":99,"_100":101,"_102":44,"_103":104,"_105":106,"_107":108,"_109":44,"_110":-5,"_111":92,"_59":62,"_112":-5},"architecture","amd64","digest","sha256:7424138f1a98bf17f3e1efe76b6ed857eca4e6092197b3f22077d02470419c36","features","last_pulled","2026-09-09T14:08:49.102061241Z","last_pushed","2021-12-13T06:10:10Z","os","linux","os_features","os_version","size","variant","2021-12-13T06:10:09.257483Z","last_updater","last_updater_username","media_type","latest","repository",14809682,"tag_last_pulled","tag_last_pushed","tag_status","v2","sbxKitMetadata","isPotentialSpam","isHubSbxKitsEnabled","displaySecurityMetadata","tagAttestations","tagsHaveAttestations",[],"routes/_layout.r.$namespace.$reponame.tags",{"_34":133,"_140":141,"_129":150,"_151":11},{"_36":134,"_38":135,"_41":42,"_43":44,"_45":11,"_46":47,"_48":49,"_50":136,"_53":31,"_28":29,"_54":55,"_56":40,"_57":58,"_59":60,"_61":62,"_63":-5,"_64":58,"_65":66,"_67":11,"_68":29,"_69":137,"_75":11,"_76":139,"_81":29,"_82":83},[],[40],[52],{"_71":11,"_72":138},[74],{"_78":17,"_79":11,"_80":11},"tags",{"_142":60,"_143":-5,"_144":-5,"_145":146},"count","next","previous","results",[147],{"_88":40,"_89":90,"_91":92,"_93":94,"_95":148,"_46":113,"_114":90,"_115":29,"_116":52,"_53":117,"_118":119,"_120":104,"_121":113,"_122":62,"_123":17},[149],{"_98":99,"_100":101,"_102":44,"_103":104,"_105":106,"_107":108,"_109":44,"_110":-5,"_111":92,"_59":62,"_112":-5},[],"canDeleteTags","actionData","errors"]

code2k13/nlphose

Tags for code2k13/nlphose