基于yelps评论数据的情感分析系统baseline

目标：

建立一个简单的情感分析系统，支持Aspect-Based Sentiment Analysis，即不仅要考虑到 整体的情感，也需要考虑到用户对产品的每个方面(aspects)的评价也需要抽取出来。而且一个用户评价中可能会存在多个 aspect，比如”我对这款产品的电池比较满意，但它太贵了!” 从这句话里我们可以得出: ‘电池’:正面， ‘价格’:负面”。

输出内容：

Business Name: XXXXX

Overall Rating: X

Detailed Rating:

aspect1: { rating: XXX, pos: [XXX], neg: [XXX]} 

aspect2: {rating: XXX, pos: [XXX], neg: [XXX]} 

aspect3: {rating: XXX, pos: [XXX], neg: [XXX]}

aspect5: {rating: XXX, pos:[xxx], neg:[xxxx]}

具体内容可见我最后的项目输出(引用内容)

过程：

1.准备： 数据集下载/ python工具包

数据集：https://www.yelp.com/dataset/download

yelp_academic_dataset_business.json: 用来描述一个 business，包括地理位置，属性，邮编等信息 

yelp_academic_dataset_review.json: 一个用户对一个 business 的评价，这里包括具体的评价文本还有 stars。

python: nltk(其中nltk的corpus语料最好挂梯子下载)

2. 设计：项目结构

--data/

            /yelp_academic_dataset_business.json

            /yelp_academic_dataset_review.json

--main.py: 项目主流程

--model_trainning.py：用于训练并生成判断评论极性的分类器

--sentence.py：用于封装并标准化句子的类模块(LEMMATIZER, ASPECT_EXTRACTOR，WORD_TOKENIZER，POS_tag )

--model.pickle：生成好的 分类器的序列化文件


3. Tricks：方法实现和部分捷径

1. 内存管理：

由于review评论文件比较大(9G左右)，在吃进内存的策略上可能需要多注意，不然内存泄漏就会经常光顾你

其中   1. Dataframe.append(dataframe )如果使用循环来append数据行，效率会非常非常低

            2. np.array强转dataframe内存消耗很大，请不要在load数据时使用

            3. python中大对象并不会自动启动gc机制，此处我为了保证内存不出现泄漏，使用策略：每隔50000行手动启动一次gc.collect() 来手动销毁循环中建立的字符串对象

2.抽取核心实体- 特性(aspects) : 

基于规则进行抽取，即只对评论中的名词及名词复数进行抽取，并形成aspect标签

3. 抽取特性对应评论中的子文本：

由于大部分评论会很长，且评论会带有多个aspect，所以此处需要将多个aspect分离出来，并分别对其进行情感极性的判断。其中aspect对应着的子文本分离办法 为使用正则进行提取（相邻标签之间含有aspect单词的子文本）

4. 模型的技术选型：

使用nltk的朴素贝叶斯做为baseline： 原因有三 1. 时间及内存开销少 2. 整个项目的业务建模方案，在未引入词向量和句向量的前提下，基于统计的贝叶斯方法直观上也能比较好的契合实际。3. 实验效果可行

4. 待改进的，未实现的

1. 文本的清洗：1. 大量非评论类的文本未清洗，例如链接，表情 2.单词的纠错统一

2. aspects的抽取: 1. 模糊概念的实体统一 2. 动名词短语的捕获

3. aspect对应短文本的抽取: 抽取规则最好是设定在相邻两个aspects之间的最长子文本(被常用标点所分割的)

4. 相同aspect下正向及负向评论集合的内容多样性的考量


产出结果范例

{

    'id': '--1UhMGODdWsrMastO9DZw',

    'content': {

        'biz_name': 'The Spicy Amigos',

        'stars': '4.0',

        'summary': {

            'taco': {

                'stars': 3.8747578276357726,

                'aspect_pos_review': [

                    '. we will definitely be coming back to get our taco fix!',

                    ', tacos,',

                    '. the shrimp especiale taco is unreal.',

                    ', tried the tacos,',

                    'we were in the mood for tacos,',

                    'i have been in search of good grilled steak tacos here in calgary for 2 years.'],

                'aspect_neg_review': [

                    ". the menus not extremely extensive so don't expect pages of choice but they have a great variety of tacos."]

            },

            'food': {

                'stars': 4.249734391600525,

                'aspect_pos_review': [

                    'if you are looking for authentic mexican street food,',

                    '.  very fresh and tasty mexican food.',

                    ', authentic mexican street food that gives appropriate portions relative to the prices.',

                    'great food,',

                    '! the decor is amazing and the food is to die for.',

                    ', but the food is still delicious.'],

                'aspect_neg_review': [

                    ". the sorry excuses for mexican food i've found in canada so far."]

            },

            'price': {

                'stars': 4.555049438951228,

                'aspect_pos_review': [

                    '.  a little on the pricey side but worth it.',

                    ', authentic mexican street food that gives appropriate portions relative to the prices.',

                    ', prices are great for real cooked food.',

                    ', service and price.',

                    ', authentic mexican food for a great price.'],

                'aspect_neg_review': [

                    '.  honestly i could have taken 1/2 home for later which is why i say good value for the price.']

            },

            'lunch': {

                'stars': 4.374453193350831,

                'aspect_pos_review': [

                    'fantastic spot for lunch with great value for your money.',

                    '! this spot opens at 11 and i can visualize it backed out the door for lunch.',

                    ", it might be a decent lunch spot but nothing spectacular and definitely wouldn't go again for dinner."],

                'aspect_neg_review': [

                ]

            }

        }

    }

}

{

    'id': '--6MefnULPED_I942VcFNA',

    'content': {

        'biz_name': "John's Chinese BBQ Restaurant",

        'stars': '3.0',

        'summary': {

            'pork': {

                'stars': 3.4998970618511223,

                'aspect_pos_review': [

                    'the bbq pork is very juicy and i only come here for that.',

                    ', is their bbq pork.',

                    '. the signature roasted pork was juicy and moist with the sweet tangy taste.',

                    ', minced pork meat pie,',

                    'if you want a quick fix for a scrumptious char siu or chinese pork bbq,',

                    ', the best roast pork and bbq pork on highway 7.'],

                'aspect_neg_review': [

                    'service with boss lady is horrible but the bbq pork is really tasty!']

            },

            'bbq': {

                'stars': 3.3527439562378683,

                'aspect_pos_review': [

                    'the bbq pork is very juicy and i only come here for that.',

                    '.  the decor itself is your usual chinese bbq house/restaurant and is nothing to go crazy over,',

                    'if you want a quick fix for a scrumptious char siu or chinese pork bbq,',

                    ', the best roast pork and bbq pork on highway 7.',

                    ". they're well known for the bbq pork,",

                    '.  the honey deep fried oysters and bbq pork were excellent too.'],

                'aspect_neg_review': [

                    'service with boss lady is horrible but the bbq pork is really tasty!',

                    'i walked by the restaurant more than 5 years ago when i witnessed from the window one of the employees drop a bbq chicken wing on the floor,']

            },

            'place': {

                'stars': 2.9998965552911967,

                'aspect_pos_review': [

                    '. the place was a little outdated,',

                    ', this is theeee place.',

                    '.this place has,',

                    'this place is a restaurant and a chinese bbq restaurant.',

                    '.  for the prices this place charges,',

                    '. compare to other places that sell bbq dishes,'],

                'aspect_neg_review': [

                ]

            },

            'chines': {

                'stars': 2.9992501874531365,

                'aspect_pos_review': [

                    'if you want a quick fix for a scrumptious char siu or chinese pork bbq,',

                    'this place is a restaurant and a chinese bbq restaurant.',

                    ', the chinese spare rib with onion,',

                    '. beef with gai lan (chinese broccoli?'],

                'aspect_neg_review': [

                ]

            }

        }

    }

}

{

    'id': '--7zmmkVg-IMGaXbuVd0SQ',

    'content': {

        'biz_name': 'Primal Brewery',

        'stars': '4.0',

        'summary': {

            'beer': {

                'stars': 3.9826899536214895,

                'aspect_pos_review': [

                    '. primal had great beer,',

                    ', fantastic beer (try the grim creeper!',

                    "if you're a harry potter fan then you will enjoy their variety of butter beers.",

                    ". \n\ni wasn't in the mood for beer,",

                    'the hubby and i stopped in for a quick beer before going home.',

                    ', the beer.'],

                'aspect_neg_review': [

                    "the beer is horrible and dave doesn't know the difference between a cat and a dog.",

                    '.  i sat at the bar and asked about the type of beer i prefer.']

            },

            'breweri': {

                'stars': 3.817834742296155,

                'aspect_pos_review': [

                    ', especially if you combine it with a trip to some of the breweries in cornelius.',

                    ". it's also conveniently located not far up the same road from crafty beer guys where you can find some of primal's and other local breweries on tap.",

                    'on a recent tour of lake norman area breweries,',

                    '. \n\nalso i love it when breweries have "activities".'],

                'aspect_neg_review': [

                ]

            },

            'food': {

                'stars': 3.6664629742792063,

                'aspect_pos_review': [

                    '.\n\nthey have a food truck on site,',

                    '.  there are often a few food trucks present,',

                    ", out front there's a few umbrella-ed tables and a slew of full-sun seats over by the food truck/corn hole arena if you're so inclined.",

                    "! the fried pickle chips at the food truck outside were some of the best i've ever had (i've had a lot).",

                    '. there was no food truck when we were there but it was raining.',

                    ', food trucks,'],

                'aspect_neg_review': [

                ]

            },

            'place': {

                'stars': 3.9998571479590015,

                'aspect_pos_review': [

                    'this is exactly the type of place that huntersville has always needed.',

                    '. a little popcorn as i sit by the fireplace drinking my beer!',

                    '. relaxing atmosphere with a fire place!',

                    ', as they have a delightful fireplace and sitting area.',

                    ', a firepit and indoor fireplace,',

                    'primal brewery is a quaint and small place up in huntersville/cornelius.'],

                'aspect_neg_review': [

                    "i'm a tad reluctant to write a review as i run the risk of spoiling one of the things i love about the place,",

                    "i've been meaning to write a review for this place for a while but now seems fitting."]

            }

        }

    }

}

{

    'id': '--9QQLMTbFzLJ_oT-ON3Xw',

    'content': {

        'biz_name': 'Great Clips',

        'stars': '3.0',

        'summary': {

            'hair': {

                'stars': 2.8459349280824555,

                'aspect_pos_review': [

                    '. affordable haircuts and pleasant hair cutters.',

                    ', i was greeted by the hair stylist and she asked,'],

                'aspect_neg_review': [

                    '. would it be ok for me to bring her in to get her hair washed and cut?',

                    ". ask for a hair cut i got seated at stephanie's corals chair.",

                    ". ask for a hair cut i got seated at stephanie's corals chair."]

            },

            'cut': {

                'stars': 2.7998133457769483,

                'aspect_pos_review': [

                    '. affordable haircuts and pleasant hair cutters.'],

                'aspect_neg_review': [

                    ". she told me that they do not cut extensions and that's great clips policy .",

                    'haircut was good and stylist nice but the manager was unpleasant from the time i stepped in until i left.',

                    '. would it be ok for me to bring her in to get her hair washed and cut?',

                    ", blah blah) she seemed uninterested on my answers (it's fine in a boring person) but she would get upset when i keep asking to cut more on the top,",

                    ". ask for a hair cut i got seated at stephanie's corals chair.",

                    ". ask for a hair cut i got seated at stephanie's corals chair."]

            },

            'time': {

                'stars': 2.3330741028774584,

                'aspect_pos_review': [

                    'tried this place one more time.'],

                'aspect_neg_review': [

                    'haircut was good and stylist nice but the manager was unpleasant from the time i stepped in until i left.',

                    'service was quick even though it said waiting time to be 30 mins,',

                    '. she slaps some shampoo on the top of my head several times in the same spot and barely moved it around with her finger tips as if she was grossed out by me.',

                    '. she slaps some shampoo on the top of my head several times in the same spot and barely moved it around with her finger tips as if she was grossed out by me.']

            },

            'didnt': {

                'stars': 0.0,

                'aspect_pos_review': [

                ],

                'aspect_neg_review': [

                ]

            }

        }

    }

}

最后编辑于 ：
©著作权归作者所有,转载或内容合作请联系作者
  • 人面猴
    序言：七十年代末，一起剥皮案震惊了整个滨河市，随后出现的几起案子，更是在滨河造成了极大的恐慌，老刑警刘岩，带你破解...
    沈念sama阅读 217,406评论 6赞 503
  • 死咒
    序言：滨河连续发生了三起死亡事件，死亡现场离奇诡异，居然都是意外死亡，警方通过查阅死者的电脑和手机，发现死者居然都...
    沈念sama阅读 92,732评论 3赞 393
  • 救了他两次的神仙让他今天三更去死
    文/潘晓璐 我一进店门，熙熙楼的掌柜王于贵愁眉苦脸地迎上来，“玉大人，你说我怎么就摊上这事。” “怎么了？”我有些...
    开封第一讲书人阅读 163,711评论 0赞 353
  • 道士缉凶录：失踪的卖姜人
    文/不坏的土叔 我叫张陵，是天一观的道长。 经常有香客问我，道长，这世上最难降的妖魔是什么？ 我笑而不...
    开封第一讲书人阅读 58,380评论 1赞 293
  • 港岛之恋（遗憾婚礼）
    正文 为了忘掉前任，我火速办了婚礼，结果婚礼上，老公的妹妹穿的比我还像新娘。我一直安慰自己，他们只是感情好，可当我...
    茶点故事阅读 67,432评论 6赞 392
  • 恶毒庶女顶嫁案：这布局不是一般人想出来的
    文/花漫 我一把揭开白布。 她就那样静静地躺着，像睡着了一般。 火红的嫁衣衬着肌肤如雪。 梳的纹丝不乱的头发上，一...
    开封第一讲书人阅读 51,301评论 1赞 301
  • 城市分裂传说
    那天，我揣着相机与录音，去河边找鬼。 笑死，一个胖子当着我的面吹牛，可吹牛的内容都是我干的。 我是一名探鬼主播，决...
    沈念sama阅读 40,145评论 3赞 418
  • 双鸳鸯连环套：你想象不到人心有多黑
    文/苍兰香墨 我猛地睁开眼，长吁一口气：“原来是场噩梦啊……” “哼！你这毒妇竟也来了？” 一声冷哼从身侧响起，我...
    开封第一讲书人阅读 39,008评论 0赞 276
  • 万荣杀人案实录
    序言：老挝万荣一对情侣失踪，失踪者是张志新（化名）和其女友刘颖，没想到半个月后，有当地人在树林里发现了一具尸体，经...
    沈念sama阅读 45,443评论 1赞 314
  • 护林员之死
    正文 独居荒郊野岭守林人离奇死亡，尸身上长有42处带血的脓包…… 初始之章·张勋 以下内容为张勋视角 年9月15日...
    茶点故事阅读 37,649评论 3赞 334
  • 白月光启示录
    正文 我和宋清朗相恋三年，在试婚纱的时候发现自己被绿了。 大学时的朋友给我发了我未婚夫和他白月光在一起吃饭的照片。...
    茶点故事阅读 39,795评论 1赞 347
  • 活死人
    序言：一个原本活蹦乱跳的男人离奇死亡，死状恐怖，灵堂内的尸体忽然破棺而出，到底是诈尸还是另有隐情，我是刑警宁泽，带...
    沈念sama阅读 35,501评论 5赞 345
  • 日本核电站爆炸内幕
    正文 年R本政府宣布，位于F岛的核电站，受9级特大地震影响，放射性物质发生泄漏。R本人自食恶果不足惜，却给世界环境...
    茶点故事阅读 41,119评论 3赞 328
  • 男人毒药：我在死后第九天来索命
    文/蒙蒙 一、第九天 我趴在偏房一处隐蔽的房顶上张望。 院中可真热闹，春花似锦、人声如沸。这庄子的主人今日做“春日...
    开封第一讲书人阅读 31,731评论 0赞 22
  • 一桩弑父案，背后竟有这般阴谋
    文/苍兰香墨 我抬头看了看天上的太阳。三九已至，却和暖如春，着一层夹袄步出监牢的瞬间，已是汗流浃背。 一阵脚步声响...
    开封第一讲书人阅读 32,865评论 1赞 269
  • 情欲美人皮
    我被黑心中介骗来泰国打工， 没想到刚下飞机就差点儿被人妖公主榨干…… 1. 我叫王不留，地道东北人。 一个月前我还...
    沈念sama阅读 47,899评论 2赞 370
  • 代替公主和亲
    正文 我出身青楼，却偏偏与公主长得像，于是被迫代替她去往敌国和亲。 传闻我的和亲对象是个残疾皇子，可洞房花烛夜当晚...
    茶点故事阅读 44,724评论 2赞 354