CG Enterpise Webinar

Center in a little bit um so let's see let's go ahead on over

to the agent control center so now I think I'm going running a little bit long so I'm

not going to spend too much time here but basically you've got this aging

control center where you set up your organizations and your users and who has access to what

um and then you've set up your you check in your agents and we have again it's a

purpose-built Version Control repository so all the all the things the Myriad

things that go into your agent so your inputs um you if it's you're writing the same

job every day your all your reference data your third-party libraries

um your uh you know your your SQL schemas all those things that your

engineer has to track independently and somehow to coordinate

that that tracking um so that all the changes to these agents move forward in lockstep we have

built a purpose uh purpose-built Version Control repository that allows you to

package up all of those different things into a single agent version check it in

to the agent control repository and you can actually see uh the versions of that

um let me see if I can show you versions let's see this doesn't really have a lot of versions so let me see

download the latest no that's not what I want um

let's see no I don't I only have one version in here so you can basically

um track all the different versions of your agent see who made those changes you can

have comments based on each change and you can get the latest um version or you can deploy an earlier

version just from from right clicking it's brilliant and based on top of that

we can set schedules we can set rate limits for these agents so what we do as

a service we'll go to Sim a website like which tells you what the

monthly visits are on average for that website and what the page loads are per

monthly visit so then we just basically calculate an average daily volume and

we'll stay under one percent of that volume for our customers and so you know

we'll set rate limits and make sure that we stay within those those ranges and

that's how we make sure that we never put excessive volume on a website and

because we're tracking Version Control and we have an audit log through our agent control center

we can basically uh you know just prove to a compliance

group or a manager um or anyone else that uh that these rate limits have been followed in the

operation um you can also set up your servers and your clusters

um and yours and the Aging control center will even add servers to the cluster if you're starting to get really

busy which is great if you're in a cloud setting um again you can add your providers and

your pools and there's tickets based on each agent so one thing I didn't show you was when

you're creating your agent you also configure the uh ticketing system into

the agent so that when something goes wrong it can go ahead and create a ticket we have our own ticketing system

that ships with the agent control center but you don't have to use it you can use jira or whatever your internal system is

if it has a rest API that you can configure this to communicate with you can do it that way whatever works for

your operation you are not hemmed in um what else can I show you

um so that that's basically it and then you can see um based on uh you know what all the

runs are in the Aging control center you can you can kick off a run or you can

um look at the job settings here's you know more ways to set a page load limit

um uh success criteria you can set the level of the job you can look at the job schedules you can look at the job

history um you know Etc

um and then when you actually go into um the job itself you can actually see

let's see what have I got here um I should be able to get some data

there is seated there we go so this is sorry the agent

control center is brand new this month it's really my first webinar showing it

so I can download the data directly from this web portal which is incredibly

useful um for teams that have uh diverse skill

sets so some are programmers and some are um just Ops managers basically everyone

has access to the data everyone has access to the files there's really no

site we're not creating any silos here um everyone can see it

um so everyone has access so if this were outputting to 10 different formats you

would get in 10 different formats that's fine um and if also if you were writing it to S3 for example you would have it show up

there um you know it'd be very easy for everyone to access the data the other thing about

this um product is that we have our manual written in zendesk so in addition to

having a knowledge base and a fact we have a very detailed help manual

um which uh it's actually public now to anyone who has the link but it's not

officially supposed to be done until June 30 so you're getting a little preview

um and you know in here they have a section on

the API which I want to share with you so this agent control center the way

that the let's go back to this diagram the way that the desktops communicate

with the servers is through the Aging Control Center and you saw the portal and you saw the ticketing system and you

saw all these components of the Aging control center but actually everything's connected via

API and you also have access to make those API calls

um so you basically um you know you use you configure that

agent for API access you get your API key

um your token and this basically walks you through the steps you need to do to

make every single API call work and if you prefer

you know this is written out for you in great detail but if you prefer you can just open those calls in Postman and you

can go ahead and test them out and run them in Postman against your you know evaluation ACC version or against your

own local installation of the server the ACC comes bundled with the server

and so any CG Enterprise server will have this

capability and you'll be able to use these API calls to integrate very deeply

with it um so at that now I'm going to look

again if there are any questions and I'd love to I'm going to unmute

everyone uh somehow

I don't see any questions let's see how do I unmute everyone

okay there we go all right everyone is no longer muted

um looks like James MacArthur has a question James why don't you go ahead

question is pain so you see the SQL integration yes so

it's actually not an ACC to SQL integration in in our Paradigm your SQL

integration comes from how you configure your database so if your export Target

is an actual database then you can enable database export to

you know any of these different databases SQL Server MySQL Oracle ol any

oledb like you know postgres or whatever um mongodb Azure Cosmos or of course you

can write any script that you want um it's the and so there's an internal

database which is what it's using on the Fly

um so for example if you're in Azure you're going to want to use the Azure SQL Server you're going to have to make

sure that um you know you've got enough you know dtus in there to to handle the volume of

agents that are all pointing to the same database um and and then you're good to go so

there's an internal database and there's an external database

um and so that's that's basically how you do it you can integrate in in either or or both

can you show a little more on the ACC SQL integration

um let me see oh gosh I don't know how to make this bigger oh it's tiny hello

hi honey yeah we have a question here okay

we're wondering where is this a scg hosted can it be hosted in the cloud or

is a software as a service well yes um all of the above so it can be

installed on premise it can be hosted by us in the cloud or it can be hosted by

you in the cloud the software is licensed um so if we sell you a license you'll

pay a single annual fee for that license and then you'll operate it on your own infrastructure or you can have us set it

up for you and then we'll manage it for you and that's the SAS offering

so for the SAS are offering meaning to say it's not only sdg the entire like

the entire uh the desktop version also will be need to be hosted I mean managed

by by your side right yeah so if we we have uh three pillars

to our business one of them is software licensing the second is Services where we provide

TurnKey services okay and the third is data products so

okay if you wanted us to write everything for you and provide services we do that on a daily basis if you

wanted us to consult we would charge 150 an hour and it would take us you know as

you see it would take us probably two to four hours to write a simple agent um end to end you know then we could

write the agents and you could run them or the other way around we could license

you the desktop software and you could write the agents and then we could run them in our cloud

there's a lot of flexibility right all right thank you

um let's see and and the other thing about the the cloud is that it basically it's just

Windows software so it can run in any cloud it can run in Azure it can run a Google Cloud AWS we have customers in all of

those clouds what we normally do is we will run it we will run the agent

control center in the cloud on highly available infrastructure and then you know highly

redundant and then we'll run our servers on Data Center

physical servers there's a huge cost difference

and then we'll we'll have our you know databases and file systems in the cloud and that that really helps us so we have

full redundancy we've rolled over because the AG control center is managing that and then we have

um we we have our low-cost Hardware plus we have Hardware in the area in the region

of the world where we need proxies so we'll set some up in Australia we'll set some up in in China or the UK

and we won't have to spend a lot on proxies so here's a question from James oh a

little more on this ACC SQL engine integration so James did I answer your question on

the on adequately on the SQL integration

for some reason I'm having a hard time opening this question box

hi how you going this is James here how you doing today hi James how are you

good thank you I am I was just actually just trying to get a bit more of a follow-up to the the SQL integration

um I've built 20 agents in a test profile now each of them seem to go to a

different table what I'm trying to figure out is if there is a way to scrape multiple create multiple agents

but then have all of the data going back to a single table because the multiple agents are all getting the same

information just from different websites oh yeah so whatever

yeah what I would recommend is first of all when you um when you set up your internal

database I would actually set up a separate export database because as you see with this when you when you set up

oh sorry I never ran this is that right

to run it um to create the tables but when it creates the tables it it's creating data

structures behind every parent child um tab in the workflow and then it's

creating tables to hold the data collected from each capture command so you've got I

don't know this is it thinks it has proxies no it's not running and device setup proxies yes I did stop

that it didn't fully set them up um so let me just so there's the

internal tables that are set up and then the internal tables when they're exported they go to that single export

database so when I go to that export database you can either configure a different export

database and point them all at the same database or um you can write a custom export script

that puts them all in the same database the the thing that you're wrestling with

is that by default the tool is assuming

that you don't want to mix data from multiple agents even though you may have the same structure so it's keeping track

of the unique agent um do it internally and it's keeping those things

separate so in that case you're probably going to want to just write a custom script and

to put them all into the same table and our support can supply a sample of

that that's right we can still do it externally anyway it was just a matter

as if if there was an easy option somewhere that's not a humongous problem no it sounds like it it could be a

feature request and we could add in a you know a screen to configure that

it is something we run into every now and then uh obviously I can't say too much about

a competitor but the the current system that we use now if we used their version of an ACC H agent that we manage we go

into the portal and we say okay uh the the job ID number from this from this

agent goes into this column of the SQ called um table the product description from

each of the agents goes into the the product column of that SQL table and and so on and so forth so it was just a

matter of just if there was some sort of parity so if there's a feature that could be worked on later that'd be great but it's not life-threatening to our to

our product or services ah so it sounds like it's uh there's a some some data mapping screens like

pipes yep exactly who are actually configuring to show how the data flows

Downstream and then where it ends up correct so mapping's probably the perfect word for it we map where each of

the data points from H to the agent goes into a SQL database and and it's Associated tables

oh yeah that's nice that's very nice it would be a great

great feature to have but it's it's it was just more if it already existed that it'd be great if not we we can just

write some script internally that's not a problem at all right

um I mean this tool is automatic automating a lot of that for you right it's yes

this thing is yeah it's it's it's very powerful this

tool and it's basically do you don't have to think about you don't even have to think about that step because it's

automatically exporting your data in the in the format of the capture command so

let's see let's say I didn't like the uh

uh let's say I didn't like the way that these fields were getting um captured I wanted to change the order I'm just

dragging and dropping the order and that's going to change the way that it gets written out at the end

and I could just click here and change the name and it's automatically tracking that on the back end in SQL and it's

going to come out that way you know so you there's a lot of stuff that you can do you just don't realize

it's so intuitive and obvious in the UI you don't realize that you're achieving the same the same purpose without having

a big mapping exercise yes I wish we'd found this software about three years ago before we wrote

everything out it's very very as a new customer I can tell you that we're very impressed with what it's done so far oh

we're so excited I literally was talking to a customer of ours earlier today who

was so thrilled he asked if he could be uh an investor

buying five like five new licenses he's been growing a team and buying like five

new licenses every other month I mean it's just really exciting yeah I mean I joined this company in 2017 but

um you know these our Founders are in Australia they've been at it since 2010 and it's it's definitely

um very exciting to see very mature yeah um sometimes in subtle ways but anyway

let me look at these other questions saw that earlier I've made several books and go to oh yeah this is your question

I can't seem to get this this uh uh

this window of questions to get any bigger for some reason but

is there a way to send foreign

way to send multiple agents to a single table okay we covered that one it doesn't look like there are any other

questions um I have one more if you do if you have

some time yeah absolutely um the only other sort of feedback or

problem that I've had with this software is what to do if the pagination option

doesn't come up so I've got a website at the moment with 10 pages it's got a just

a normal next button at the bottom of the screen but when I click it the

pagination option in the pop-up window doesn't appear and I can't seem to find

a way to to ask it to consider a pagination option it plays when I debug it basically just starts running

so then you can just go into into the top level um parent on that page and click this

plus sign add command and then you get this window that pops up

and you can scroll down and click on any of these commands including navigation

I know how to do that um it's not doing it um it's not it's

I'm just wondering if there's an extra way or something a little bit more advanced when it's not working so I can

add a pagination it's not protecting the X path properly or uniquely

um so let's say there isn't of course pagination on this particular field but let's say there is and I've gone and

clicked on single next link and now I'm trying to figure out what the

what the um you know I'm just pointing and clicking on that next button you can

go over here and see what the XPath is that's been collected and then you can

go down here and see in the selection count how many elements on that page are

mapping to that XPath and you can see if there's um if the the XPath is not properly or

cleanly identifying that single next button then you can look at it in visual

debug mode and you can go page to page and you can pause and you can see if

it's actually getting the correct next button when it goes to

the next page because sometimes they they'll do that they'll make it you know

just a little bit harder for you they'll change the next button you know from

page to page they'll change the XPath so you actually have to use a different identifier than the default that the

tools okay so sometimes it'll it'll click next on the first page but the second one has

something slightly different so you'll you'll have to look for the next uh

button image you know or something like that okay awesome

I can definitely yeah um so you can basically you know play with that and and if you have any

trouble please send your questions to support at

what's up yes I've already spoken with your support team for uh for a bit of code

that would I can if anyone else is listening this company really does look after you with their support as well yeah it's in about six hours

oh that's great yeah no they um they're really much better than I am and they

um they're just very committed yep I mean it's it's a nice tool people kind

of fall in love with the products they work on and that definitely I think um

you know our support staff is uh we look after them and they seem to look after everyone else so

you know let us know if that's ever not the case as I said as a new customer can say I'm

very very happy with the server so far oh I'm so glad what company are you from

uh core logic in Brisbane Australia all right very nice that's fantastic

what a company um well what else can I answer for you today I I

think I've given you kind of a broad tour um are there any other things that you're

working on that we can help with um well from I don't know how many other

people are listening for me it would be handy if you had any instructions or a read-over of the ACC I've got the

instructions for the the desktop software but not the ACC as of yet

um we're just trying to figure out if there's a business case for that to be useful for us sure

um this is our support guide

so this isn't technically going live for another 12 days so that's funny they

kill me that I'm sending it to you but I think that it looks pretty done there's obviously

um things we know of that we're still adding and massaging but it's it's uh

it's going to give you a good idea of where we're headed with our support and if there are any topics that you feel

are missing or questions that are not answered let us know and we will just add it we've got

um you know a growing knowledge base a growing fact and the manual has been has gotten the you know the Lion's Share of

our Focus over the last couple of months um but yeah there's a section in here at

the end for the agent control center um and it just explains you know how to

set up your organizations how to set up your servers and clusters your cluster deployment what to do with jobs right

there's a concept of runs and there's you know multiple concurrent sessions in a run and then and then you can have

multiple runs that go into a job um you know so that's an important sort

of structure to understand um and you know if you wanted to do a single server deployment schedules

ticketing you know all of that kind of thing the thing for us we run very large scale

um you know web scraping operations so the Aging control center is critical to us and the reason why it's it's critical

is really because of this um agent control Repository

um so let me just pause this for a second get into the Asian control repository

yeah that's fine um just never know exactly what folder I'm

in um so for this uh this folder I'm going to show that there are multiple versions

for this agent that have been checked in um you know there's a little comment

with each of these and I can basically go roll back I can get a previous version or compare with the local

version um you know Etc and then once I have my versions in there then I can configure

all of these things I can configure job settings including rate limits

um schedules I can look at job history you know Etc I can look at that all the

information for the run I can I can um you know basically deploy the latest

to a cluster version so for example let's say you know if you're like us you have thousands of agents that you're

running and you've got uh you know all of these individual agent developers

that you have a whole process you want to follow with a Dev and a QA and a production server

you know you want to make sure that your data collection you know all those risks all those areas where your data

collection could go wrong or mitigated so this tool is basically helping to manage that whole process so you package

up your entire agent check it into the Version Control repository and then you have controls over

you know what gets deployed what version gets deployed where and all of the dependent

um you know all the dependencies that go with that agent that changes to your database schema and you know the newer

version of the you know tesseractocr library or whatever you're using

um you know that those things are are updated maybe you have inputs you have a list of cities and or zip codes and

you've updated them those inputs you know they they can be you know packaged

up with the same agent those are all things that are areas that really go go wrong and then managing your your

servers and deployments is the next big thing um there's an audit Trail so that you can

go back and figure out like what actually went wrong um you know if if anything did go wrong

so you can see in detail exactly what happened um there's a

um you know you can move uh these Agents from one repository to another or one

folder to another folder um Etc you can also go in and configure

um let's see so I'm logged in as demo so I can't see a lot of

information um so there's there's servers behind

here that I can't actually manage the leg Finish Line let's see

okay so I'm sorry the the tool is a bit new um and uh I can't what I'm trying to

show you um and I'm not I'm failing to show you is you can actually um like imagine you have huge beefy

servers and they're running hundreds of Agents on each um and then you need get to a place

where you need to upgrade your server software so how do you manage that process

um we can you can just literally on a server basis because it's all database backed you can right click pause

collection on that server right click upgrade the content Grabber

software right it supports remote upgrades of yours of your software

and then again just right click and resume and you know yeah so it's it's it's just

for us it's incredibly easy to manage our operation and the proxies you know

the proxy providers they all have different problems at different times so we always configure

them all separately and then we set up all of our pools um you know specific to you know what

the what the agent needs so if they need Au residential then they'll be an au residential pool

and then separately we'll be managing all the providers and the proxies so sometimes certain providers go down and

we'll just take them out of the pool and uh and we'd make that change in one place and all of the agents are updated

you know so it's so the way that we've componentized everything and and centralized the management of each each

of the different pieces really helps us um the other thing I wanted to say is if

you look in here in this folder we've got an agent we've got a config file

which um let me show you um so in here you can basically set up

in your config file you can set up all the types of things that you'll want to

set up on you know maybe there's maybe there's 50 agents in this folder but

they're all using the same config um so you can basically set up a config file per folder so this really helps you

manage your environment standardize your environment and enforce standards if this is how you're working because if

there is a config file the agent is going to inherit from the config file

um so this is a really nice thing as well to help manage a large scale

operation and then of course any credentials this this agent happens to

be writing to um you know an S3 bucket

um you know so I've got my S3 credentials stored here in the agent

repository but none of my agent developers actually

know what they are right so we have these hedge fund

um clients for example and they um they you know that they're they get

tremendous value out of the data that we provide um but you know the only the first

person who puts these credentials in here is is going to be the one who knows how to access that S3 bucket to get all

the data we don't you know for those guys we don't store the data we just offload it every day

um you know so we don't you know we don't have to worry about you know anybody making making a

bad choice we just don't don't really have to think about it we have them the credentials in the agent repository and

they're encrypted and uh and that's it the agent is going

to use that yeah thank you it's just all these

really convenient things and and things you know just to support a large scale operation and we're we're you know we

launched it originally last August and we've had an amazing response

um and we this is our second version it's basically taking the ACC out of the desktop software and putting it in the

browser um and that has had a great response and we're just gonna you know keep

innovating and keep um you know stretching ourselves and growing so if you have ideas on what uh

areas we could help you with and and we took in your feedback on on mapping and

pipes um that seems like an interesting module that might be useful uh

um I've got lots of feedback for no time unfortunately yeah well anytime you have

feedback just send it over and we'll we'll keep track of

it um just again just quickly before I go but thank you so much for the demo

um one one thing that would be incredibly handy to have now I don't know if it's just because I'm still on a

trial before I talk with Tony um is the how do I say this

uh some common Transformations just in the the transformation tool just you

know a couple of common ones so like uh something like split on a dollar symbol

to a another column or something like just some very basic transformations

can do that feel right now um let's go to let me get this out of the way are

you still seeing my screen yes let me well first of all let me get rid of this

navigation group so it doesn't matter um

actually it does because I want it to run well I'll show you so let's go down to something let's see

there's no price in here is there uh uh let's see

if you can do something else something simple something simple with a price

I'm looking at all these so I know I know about the Highlight certain things that you keep and then like there is

sort of some some magic in the background that happens um there's more when you go to do the

transformation screen itself like with your regex and c-sharp and everything like that if there was you don't have to

do that you don't have to do C sharp or anything like that let me see

um if I can just load these

uh now this is old basically what you're gonna do is you're gonna go into

um let me open this Amazon thing again you're going to go into here and you're

going to go into one of these um one of these things and you're

basically going to let's see if there's anything in here oh let's take out let's take out

let's work with the job path one and first of all I'm going to basically copy and then I'm going to paste

and then let's see I have to paste it up here don't I

want to paste it and then I'm going to you know put them next to each other so I don't get confused

okay and in the first one I'm gonna go in and I'm gonna write a simple uh

transformation and I'm basically going to say you know I don't want

you know I want to pull uh well again here it's like if you wanna if you want

to just pull this out you can basically write the regex for that now if you wanted to strip out all of the

um you know slashes from here and just have the text um I don't actually in that I think you

have to basically write um uh some simple regex for which I don't

actually uh have in my hand um and then so so one of these things could

be just you know you could pull out just the currency symbol or just the the

Slash and then the next one would be um you know without so you'd basically

duplicate the the currency field and then one of them you would you would

highlight that currency symbol generate the transformation pulling that out and the next one you would highlight only

the price without the currency symbol and um and generate the regex automatically

for that and highlight that out so you'd have basically you'd be splitting it into two Fields but um extracting you

know the two different portions of the price yeah so the feedback would be is that

all of all of that technically works it would be good if we had a repository of

just commonly used ones on sort of like examples or power um one of the ones that I've had to

write by hand is to join two strings together to to create a file name um so I had to do a bit of playing

around but just if there was somewhere that a repository of codes that your guys have you know had in but the past

or even if we have you know string and code examples on on the FAQ or development page

um because I have found that out in the greater world uh you you can get C sharp

and you can get regex but it sometimes doesn't quite work with what we've got here so if you had some internalized you

know scripts and samples that'd be just somewhere that'd be fantastic as well yep

will do anything else I I unfortunately no

unfortunately I have to run for another meeting but thank you so much the the demo on that was fantastic we're very very very happy

oh great oh I'm so glad please keep the questions coming and let us know if there's any other way we can help you

not a problem at all thank you very much for taking the time to share me that yep thank you glad to do it Heidi how about

you do you have anything else any other questions

so far no thank you for your demo okay thank you so much let us know if you

have any follow-on questions yeah sure okay good night

good night