自然语言处理:汉语分词

news/2024/6/28 22:27:13

NLPIR/ICTCLAS 汉语分词系统(http://ictclas.nlpir.org)
PyNLPIR 是该汉语分词系统的 python 封装版(http://pynlpir.readthedocs.io...)

安装步骤:
① pip install pynlpir
② pynlpir update

官方文档的汉语分词示例:

import pynlpir
pynlpir.open()str = '欢迎科研人员、技术工程师、企事业单位与个人参与 NLPIR 平台的建设工作。'
result = pynlpir.segment(str)print(result)# output: [('欢迎', 'verb'), ('科研', 'noun'), ('人员', 'noun'), ('、', 'punctuation mark'), ('技术', 'noun'), ('工程师', 'noun'), ('、', 'punctuation mark'), ('企事业', 'noun'), ('单位', 'noun'), ('与', 'conjunction'), ('个人', 'noun'), ('参与', 'verb'), ('NLPIR', 'noun'), ('平台', 'noun'), ('的', 'particle'), ('建设', 'verb'), ('工作', 'verb'), ('。', 'punctuation mark')]

可能遇到的问题:
① raise RuntimeError("NLPIR function 'NLPIR_Init' failed.")

解决方案:
访问 https://github.com/NLPIR-team... 仓库,
下载 license 例如 NLPIR-ICTCLAS 分词系统授权中的 NLPIR.user 文件,
替换路径 path_to_local_python/Lib/site-packages/pynlpir/Data 下的同名文件以更新授权。

中文停用词表:

["啊","阿","哎","哎呀","哎哟","唉","俺","俺们","按","按照","吧","吧哒","把","罢了","被","本","本着","比","比方","比如","鄙人","彼","彼此","边","别","别的","别说","并","并且","不比","不成","不单","不但","不独","不管","不光","不过","不仅","不拘","不论","不怕","不然","不如","不特","不惟","不问","不只","朝","朝着","趁","趁着","乘","冲","除","除此之外","除非","除了","此","此间","此外","从","从而","打","待","但","但是","当","当着","到","得","的","的话","等","等等","地","第","叮咚","对","对于","多","多少","而","而况","而且","而是","而外","而言","而已","尔后","反过来","反过来说","反之","非但","非徒","否则","嘎","嘎登","该","赶","个","各","各个","各位","各种","各自","给","根据","跟","故","故此","固然","关于","管","归","果然","果真","过","哈","哈哈","呵","和","何","何处","何况","何时","嘿","哼","哼唷","呼哧","乎","哗","还是","还有","换句话说","换言之","或","或是","或者","极了","及","及其","及至","即","即便","即或","即令","即若","即使","几","几时","己","既","既然","既是","继而","加之","假如","假若","假使","鉴于","将","较","较之","叫","接着","结果","借","紧接着","进而","尽","尽管","经","经过","就","就是","就是说","据","具体地说","具体说来","开始","开外","靠","咳","可","可见","可是","可以","况且","啦","来","来着","离","例如","哩","连","连同","两者","了","临","另","另外","另一方面","论","嘛","吗","慢说","漫说","冒","么","每","每当","们","莫若","某","某个","某些","拿","哪","哪边","哪儿","哪个","哪里","哪年","哪怕","哪天","哪些","哪样","那","那边","那儿","那个","那会儿","那里","那么","那么些","那么样","那时","那些","那样","乃","乃至","呢","能","你","你们","您","宁","宁可","宁肯","宁愿","哦","呕","啪达","旁人","呸","凭","凭借","其","其次","其二","其他","其它","其一","其余","其中","起","起见","岂但","恰恰相反","前后","前者","且","然而","然后","然则","让","人家","任","任何","任凭","如","如此","如果","如何","如其","如若","如上所述","若","若非","若是","啥","上下","尚且","设若","设使","甚而","甚么","甚至","省得","时候","什么","什么样","使得","是","是的","首先","谁","谁知","顺","顺着","似的","虽","虽然","虽说","虽则","随","随着","所","所以","他","他们","他人","它","它们","她","她们","倘","倘或","倘然","倘若","倘使","腾","替","通过","同","同时","哇","万一","往","望","为","为何","为了","为什么","为着","喂","嗡嗡","我","我们","呜","呜呼","乌乎","无论","无宁","毋宁","嘻","吓","相对而言","像","向","向着","嘘","呀","焉","沿","沿着","要","要不","要不然","要不是","要么","要是","也","也罢","也好","一","一般","一旦","一方面","一来","一切","一样","一则","依","依照","矣","以","以便","以及","以免","以至","以至于","以致","抑或","因","因此","因而","因为","哟","用","由","由此可见","由于","有","有的","有关","有些","又","于","于是","于是乎","与","与此同时","与否","与其","越是","云云","哉","再说","再者","在","在下","咱","咱们","则","怎","怎么","怎么办","怎么样","怎样","咋","照","照着","者","这","这边","这儿","这个","这会儿","这就是说","这里","这么","这么点儿","这么些","这么样","这时","这些","这样","正如","吱","之","之类","之所以","之一","只是","只限","只要","只有","至","至于","诸位","着","着呢","自","自从","自个儿","自各儿","自己","自家","自身","综上所述","总的来看","总的来说","总的说来","总而言之","总之","纵","纵令","纵然","纵使","遵照","作为","兮","呃","呗","咚","咦","喏","啐","喔唷","嗬","嗯","嗳","啊哈","啊呀","啊哟","挨次","挨个","挨家挨户","挨门挨户","挨门逐户","挨着","按理","按期","按时","按说","暗地里","暗中","暗自","昂然","八成","白白","半","梆","保管","保险","饱","背地里","背靠背","倍感","倍加","本人","本身","甭","比起","比如说","比照","毕竟","必","必定","必将","必须","便","别人","并非","并肩","并没","并没有","并排","并无","勃然","不","不必","不常","不大","不得","不得不","不得了","不得已","不迭","不定","不对","不妨","不管怎样","不会","不仅仅","不仅仅是","不经意","不可开交","不可抗拒","不力","不了","不料","不满","不免","不能不","不起","不巧","不然的话","不日","不少","不胜","不时","不是","不同","不能","不要","不外","不外乎","不下","不限","不消","不已","不亦乐乎","不由得","不再","不择手段","不怎么","不曾","不知不觉","不止","不止一次","不至于","才","才能","策略地","差不多","差一点","常","常常","常言道","常言说","常言说得好","长此下去","长话短说","长期以来","长线","敞开儿","彻夜","陈年","趁便","趁机","趁热","趁势","趁早","成年","成年累月","成心","乘机","乘胜","乘势","乘隙","乘虚","诚然","迟早","充分","充其极","充其量","抽冷子","臭","初","出","出来","出去","除此","除此而外","除此以外","除开","除去","除却","除外","处处","川流不息","传","传说","传闻","串行","纯","纯粹","此后","此中","次第","匆匆","从不","从此","从此以后","从古到今","从古至今","从今以后","从宽","从来","从轻","从速","从头","从未","从无到有","从小","从新","从严","从优","从早到晚","从中","从重","凑巧","粗","存心","达旦","打从","打开天窗说亮话","大","大不了","大大","大抵","大都","大多","大凡","大概","大家","大举","大略","大面儿上","大事","大体","大体上","大约","大张旗鼓","大致","呆呆地","带","殆","待到","单","单纯","单单","但愿","弹指之间","当场","当儿","当即","当口儿","当然","当庭","当头","当下","当真","当中","倒不如","倒不如说","倒是","到处","到底","到了儿","到目前为止","到头","到头来","得起","得天独厚","的确","等到","叮当","顶多","定","动不动","动辄","陡然","都","独","独自","断然","顿时","多次","多多","多多少少","多多益善","多亏","多年来","多年前","而后","而论","而又","尔等","二话不说","二话没说","反倒","反倒是","反而","反手","反之亦然","反之则","方","方才","方能","放量","非常","非得","分期","分期分批","分头","奋勇","愤然","风雨无阻","逢","弗","甫","嘎嘎","该当","概","赶快","赶早不赶晚","敢","敢情","敢于","刚","刚才","刚好","刚巧","高低","格外","隔日","隔夜","个人","各式","更","更加","更进一步","更为","公然","共","共总","够瞧的","姑且","古来","故而","故意","固","怪","怪不得","惯常","光","光是","归根到底","归根结底","过于","毫不","毫无","毫无保留地","毫无例外","好在","何必","何尝","何妨","何苦","何乐而不为","何须","何止","很","很多","很少","轰然","后来","呼啦","忽地","忽然","互","互相","哗啦","话说","还","恍然","会","豁然","活","伙同","或多或少","或许","基本","基本上","基于","极","极大","极度","极端","极力","极其","极为","急匆匆","即将","即刻","即是说","几度","几番","几乎","几经","既...又","继之","加上","加以","间或","简而言之","简言之","简直","见","将才","将近","将要","交口","较比","较为","接连不断","接下来","皆可","截然","截至","藉以","借此","借以","届时","仅","仅仅","谨","进来","进去","近","近几年来","近来","近年来","尽管如此","尽可能","尽快","尽量","尽然","尽如人意","尽心竭力","尽心尽力","尽早","精光","经常","竟","竟然","究竟","就此","就地","就算","居然","局外","举凡","据称","据此","据实","据说","据我所知","据悉","具体来说","决不","决非","绝","绝不","绝顶","绝对","绝非","均","喀","看","看来","看起来","看上去","看样子","可好","可能","恐怕","快","快要","来不及","来得及","来讲","来看","拦腰","牢牢","老","老大","老老实实","老是","累次","累年","理当","理该","理应","历","立","立地","立刻","立马","立时","联袂","连连","连日","连日来","连声","连袂","临到","另方面","另行","另一个","路经","屡","屡次","屡次三番","屡屡","缕缕","率尔","率然","略","略加","略微","略为","论说","马上","蛮","满","没","没有","每逢","每每","每时每刻","猛然","猛然间","莫","莫不","莫非","莫如","默默地","默然","呐","那末","奈","难道","难得","难怪","难说","内","年复一年","凝神","偶而","偶尔","怕","砰","碰巧","譬如","偏偏","乒","平素","颇","迫于","扑通","其后","其实","奇","齐","起初","起来","起首","起头","起先","岂","岂非","岂止","迄","恰逢","恰好","恰恰","恰巧","恰如","恰似","千","万","千万","千万千万","切","切不可","切莫","切切","切勿","窃","亲口","亲身","亲手","亲眼","亲自","顷","顷刻","顷刻间","顷刻之间","请勿","穷年累月","取道","去","权时","全都","全力","全年","全然","全身心","然","人人","仍","仍旧","仍然","日复一日","日见","日渐","日益","日臻","如常","如此等等","如次","如今","如期","如前所述","如上","如下","汝","三番两次","三番五次","三天两头","瑟瑟","沙沙","上","上来","上去","一.","一一","一下","一个","一些","一何","一则通过","一天","一定","一时","一次","一片","一番","一直","一致","一起","一转眼","一边","一面","上升","上述","上面","下","下列","下去","下来","下面","不一","不久","不变","不可","不够","不尽","不尽然","不敢","不断","不若","不足","与其说","专门","且不说","且说","严格","严重","个别","中小","中间","丰富","为主","为什麽","为止","为此","主张","主要","举行","乃至于","之前","之后","之後","也就是说","也是","了解","争取","二来","云尔","些","亦","产生","人","人们","什麽","今","今后","今天","今年","今後","介于","从事","他是","他的","代替","以上","以下","以为","以前","以后","以外","以後","以故","以期","以来","任务","企图","伟大","似乎","但凡","何以","余外","你是","你的","使","使用","依据","依靠","便于","促进","保持","做到","傥然","儿","允许","元/吨","先不先","先后","先後","先生","全体","全部","全面","共同","具体","具有","兼之","再","再其次","再则","再有","再次","再者说","决定","准备","凡","凡是","出于","出现","分别","则甚","别处","别是","别管","前此","前进","前面","加入","加强","十分","即如","却","却不","原来","又及","及时","双方","反应","反映","取得","受到","变成","另悉","只","只当","只怕","只消","叫做","召开","各人","各地","各级","合理","同一","同样","后","后者","后面","向使","周围","呵呵","咧","唯有","啷当","喽","嗡","嘿嘿","因了","因着","在于","坚决","坚持","处在","处理","复杂","多么","多数","大力","大多数","大批","大量","失去","她是","她的","好","好的","好象","如同","如是","始而","存在","孰料","孰知","它们的","它是","它的","安全","完全","完成","实现","实际","宣布","容易","密切","对应","对待","对方","对比","小","少数","尔","尔尔","尤其","就是了","就要","属于","左右","巨大","巩固","已","已矣","已经","巴","巴巴","帮助","并不","并不是","广大","广泛","应当","应用","应该","庶乎","庶几","开展","引起","强烈","强调","归齐","当前","当地","当时","形成","彻底","彼时","往往","後来","後面","得了","得出","得到","心里","必然","必要","怎奈","怎麽","总是","总结","您们","您是","惟其","意思","愿意","成为","我是","我的","或则","或曰","战斗","所在","所幸","所有","所谓","扩大","掌握","接著","数/","整个","方便","方面","无","无法","既往","明显","明确","是不是","是以","是否","显然","显著","普通","普遍","曾","曾经","替代","最","最后","最大","最好","最後","最近","最高","有利","有力","有及","有所","有效","有时","有点","有的是","有着","有著","末##末","本地","来自","来说","构成","某某","根本","欢迎","欤","正值","正在","正巧","正常","正是","此地","此处","此时","此次","每个","每天","每年","比及","比较","没奈何","注意","深入","清楚","满足","然後","特别是","特殊","特点","犹且","犹自","现代","现在","甚且","甚或","甚至于","用来","由是","由此","目前","直到","直接","相似","相信","相反","相同","相对","相应","相当","相等","看出","看到","看看","看见","真是","真正","眨眼","矣乎","矣哉","知道","确定","种","积极","移动","突出","突然","立即","竟而","第二","类如","练习","组成","结合","继后","继续","维持","考虑","联系","能否","能够","自后","自打","至今","至若","致","般的","良好","若夫","若果","范围","莫不然","获得","行为","行动","表明","表示","要求","规定","觉得","譬喻","认为","认真","认识","许多","设或","诚如","说明","说来","说说","诸","诸如","谁人","谁料","贼死","赖以","距","转动","转变","转贴","达到","迅速","过去","过来","运用","还要","这一来","这次","这点","这种","这般","这麽","进入","进步","进行","适应","适当","适用","逐步","逐渐","通常","造成","遇到","遭到","遵循","避免","那般","那麽","部分","采取","里面","重大","重新","重要","针对","问题","防止","附近","限制","随后","随时","随著","难道说","集中","需要","非特","非独","高兴","若果 "]

http://lihuaxi.xjx100.cn/news/240787.html

相关文章

显示来自多个表的数据——JOIN

表关系简介 一、语法 SELECT 字段列表FROM TABLE1 [CROSS JOIN TABLE2 ] | [NATURAL JOIN TABLE2 ] | [JOIN TABLE2 USING (字段名) ] | [JOIN TABLE2 ON (TABLE.COLUMN_NAME TABLE2.COLUMN_NAME) ] | [(LEFT | RIGHT | FULL OUT) JOIN TABLE2 ON (TABLE1.COLUMN_NAME TABL…

JavaScript封装一个注册函数解决兼容问题

我们知道JavaScript注册(绑定)事件主要有两类方式,第一类传统方式具有注册事件的唯一性,即对于同一元素的同一事件,不会出现两个处理函数,如下 var btn document.querySelector(button);btn.onclick function(){document.body.s…

排序(一)归并、快排、优先队列等(图文具体解释)

排序(一) 0基础排序算法 选择排序 思想:首先,找到数组中最小的那个元素。其次,将它和数组的第一个元素交换位置。再次。在剩下的元素中找到最小的元素。将它与数组的第二个元素交换位置。如此往复,直到将整个数组排序。 【图例】 …

练习 MongoDB 操作 —— 备份篇(三)

2019独角兽企业重金招聘Python工程师标准>>> mongodb数据备份和还原主要分为二种, 一种是针对于库的mongodump和mongorestore, 一种是针对库中表的mongoexport和mongoimport。 mongodb 数据库操作--备份 还原 导出 导入 导入与导入 导入与导出…

RabbitMQ死信队列、延时队列

介绍: 消息被消费⽅否定确认,使⽤ channel.basicNack 或 channel.basicReject ,并且此时 requeue 属性被设置为 false 。消息在队列的存活时间超过设置的TTL时间。消息队列的消息数量已经超过最⼤队列⻓度。那么该消息将成为“死信”。“死信…

JavaScript实现智能搜索框

应用场景: 1. 搜索框在页面中占据的空间过小,希望无论浏览到什么位置,可以轻易地回到并聚焦搜索框。 2. 搜索框里面的文字大小过小,希望能够在上方开辟一块空间放大内容 解决思路: 1. 对整个页面添加键盘事件keyup…

phpstorm failed to create jvm:error code -6 解决办法 解决方法

phpStorm 软件打开运行提示 failed to create JVM的解决办法。 修改文件 D:\Program Files (x86)\JetBrains\PhpStorm 7.1.3\bin\PhpStorm.exe.vmoptions 把内存值改成标准值,文件全部内容如下: [plain] view plaincopy -server -Xms128m -Xmx512m -X…

CentOS 6.x中目录结构及用途

目录名称作用/根目录,一般根目录下只存放目录,不要存放文件/bin存放普通用户可执行的系统命令/sbin存放的是超级管理员执行的系统命令/etc系统管理和配置文件存放路径/etc/rc.d启动的配置文件和脚本存放目录/etc/passwd系统用户密码文件,包含…