Integrating Sentiment Analysis, Text Analytics, and Keyword Extraction into Your Web Scraping Operation

hi everyone this is Sarah McKenna at


um we it looks like we have a really

good crowd today thank you so much for

joining us we're going to go through how

to integrate

um uh

sentiment analysis text analytics and

keyword extraction into your web

scraping operation

um and with me today as always is zijang

who's our New York City Tech lead he's

going to lead you through how to do that

using some of the common API services

that are out there that provide

sentiment analysis Etc so it'll also be

a little bit little tutorial on how to


third-party apis into your

workflow before we start I wanted to ask

is are there any particular questions

um that you have or that you'd like to

see covered in today's

webinar we have a a very specific agenda

we're going to go through quickly

so if there's anything else that you'd

like us to cover please do

um go ahead and and ask a question I'll

be monitoring these questions throughout

the presentation and

um and we'll be happy to to get those

answered for you today

with that I'm going to hand it over

no it's okay they can they I don't see

any questions

um I'm going to go ahead and hand this

over to Z

going to make you presenter

there we go

all right

I'm gonna go on mute can you see my


nope okay hi guys um welcome to a

webinar on uh integrating

Microsoft cognitive Services into um

into a Content wrapper so before we



you would have to sign up for like a key


Microsoft Azure in order to um use in

our agent I've already set one up but

then there's um you can set up like a

free model or just like different

pricing tiers for uh different Services

as well

um differences between like the

different Services is pretty much a

company cars you can have up to a minute

and how many calls you can have for a


like on a monthly basis right now I'm

using the free tier and I'm just going

to be using the key for the free tier

um to show how to integrate this into

content grabber

so I'm over here I've actually already

set up an agent that uses a Microsoft

cognitive services in order to analyze

some text that I have extracted from


so then first I created a calculated

value with my key

and then afterwards I just created a

data field here with

um some of my text I extracted from

Twitter that I'm just going to be using

for this example

just a bunch of text and then inside the

text for the system analysis

the one for the sentiment URL

um if if you go on to Azure if you log

into the portal they're going to give

you a specific endpoint that you would

need to um have as well as a specific


so then um the endpoint that I'm using

right now is the text analytics SC uh


and then after that you just need to

navigate to text Analytics

uh V2 and then from here we're just

going to be using the sentiment API

and then afterwards you just all you

need to do is insert in your key

and your text that you're going to use

inside a Json post

so from here um under actions I'm

configuring the browser here to load up

a Json parser because the information

that we get is returning in the Json and

if I execute this command directly you

can see that uh

there's a score that appears that

basically analyzes the text that was

given to um this cognitive Services a

score of 0.5 basically means that it's

neutral it's like it's not positive or

negative in any way

and then the second one is a is a key


analysis it's also relatively the same

as setting up the sentiment analysis

except from here which we're going to

use the same endpoint

um texting out text analytics

sequentum.cognitive service at

except from here which is going to

change sentiment to key phrases

and then we're also going to still put

in the key in the text as well and now

if I execute this directly

you can see that there's a bunch of key

phrases that gets extracted

as well and then right now

um I'm basically concatenating all of uh

these separate phrases into one with

like a

pipe delimited deliminator from here so

now if I um just debug this agent real


I'm just going to stop it now

you can see that for some of the texts

for some of the text over here

um you're getting like a sentiment score

of 0.5 which specifies that um

this tweet here was kind of neutral like

it doesn't it's not positive or negative

in any way and then a sentiment score of

0.74 specify that this was like a


tweet pretty much for the topic and a

score of 0.25 or score lower than 0.5

basically means like more of a negative

View for the tweet

and then for the key phrases you can see

that uh it extracted key phrases that

were used inside the text

like uh Rugby World Cup and BBC News and

so forth

and um that is how we will go about

um implementing

Microsoft's cognitive services like

sentiment analysis and keywords

extractions into our agent

would there be anything

else there or

um I don't see any questions

um I I will add thank you very much Z

for showing us

um how to how that works and I will add

that uh you know there's more and more

apis out there that allow you to do text


um uh I was just on a panel with uh

tomek from IBM Watson and uh they he he

let me know that uh and something I

didn't know is uh they do they not only

provide Google sorry not Google they not

only provide translation services

language translation services but unlike


they do not

um in their in their uh engage your

their legal and great engagement they do

not take ownership of the content that

you're running through their API

uh which is great

um so you really have a lot of choices

you have Microsoft you have IBM Watson

you have Google apis

um you know obviously you know you can

integrate location map

translation sentiment any text analytics

or entity extraction you can do keyword

extraction if you're if you're pulling

news articles for example

um really the sky's the limit so it's a

very very powerful way to integrate

third-party apis to add

intelligence and artificial intelligence

machine learning and natural language

processing into your web scraping

workflow I still don't see any any

questions up there so I'm assuming that

this is uh this is all you needed to

know on this topic I'm going to give you

one more minute to ask some questions

and if there are none then we'll uh

we'll go ahead and and finalize this

this webinar for today

I'm going to be silent for a minute

while I give you a moment to to think if

you have any other questions

all right all I think that's it for

today thanks so much for joining oh wait


you raised your hand


um would you like to come off mute let's

see can I you are unmuted now oh but if

you have a question

yeah yeah hi



uh how it would be in Bissell driper for


I'm sorry

and if I'm using a web Bissell Reaper

that's house from your company how it

would be realized this uh

the extraction of the keywords for


so I'm sorry to tell you about it

deports this um it's our first

generation product

um it's it's it's built around the

Microsoft Internet Explorer 11 browser

which was sunset by Microsoft is no

longer supported by Microsoft so we're

really focused

um as a as a company on the third

generation product which is CG


um and uh and and I I don't believe that

you can

integrate easily into apis from Visual

Web Ripper


sorry I don't have an answer for you

there we do encourage our our Legacy

software licensees to upgrade

um you know to stay in line with all of

the all of the features and capabilities

that we're continuously added to our

product line

now okay thank you yep

all right I see Patrick you just joined

uh we that we concluded our

demonstration and on on how to integrate

um text analytics sentiment analysis

into your uh web scripting workflow you

can see the uh you can see the

the recording of the webinar

um you know probably we'll probably

release that later today but in the

meantime I wanted to give you uh the the

opportunity to ask any questions you

might have


question page okay okay so you're good

so you look for the video okay excellent

so I think that concludes it for today

that was a quick demonstration and

hopefully that's useful for you

um any other questions or follow-ups

feel free to email us at sales at really appreciate your

time and uh happy holidays

bye everyone