Digital Disruption

00;00;01;01 - 00;00;03;28

These days, the vast majority of attention

00;00;03;28 - 00;00;07;21

is on generative AI,

and disproportionately so.

00;00;07;21 - 00;00;08;10

This is my

00;00;08;10 - 00;00;12;05

extraordinarily subversive statement,

and I hope everyone's sitting down.

00;00;12;07 - 00;00;15;22

Most organizations should be investing

at least as much in predictive

00;00;15;22 - 00;00;19;11

AI or predictive analytics

as they do in generative AI.

00;00;21;01 - 00;00;24;22

Generative AI is where the attention

and most of the investment is going.

00;00;25;00 - 00;00;26;14

So why does Eric Siegel

00;00;26;14 - 00;00;29;24

think companies may be overlooking

an equally important opportunity?

00;00;30;07 - 00;00;33;07

I'm Geoff Nielson

and this is digital disruption.

00;00;33;16 - 00;00;37;21

Today we get past the incredibly vague

label of AI

00;00;37;24 - 00;00;42;01

and ask what underlying technology

is actually doing something valuable.

00;00;42;11 - 00;00;45;11

Let's find out.

00;00;47;06 - 00;00;48;13

Eric, super excited to have you.

00;00;48;13 - 00;00;49;27

Thanks for being here.

00;00;49;27 - 00;00;51;02

Maybe just to jump into it.

00;00;51;02 - 00;00;52;15

You know, I was reflecting.

00;00;52;15 - 00;00;55;26

There's this, like, bad habit that a lot

of people in this space are guilty of.

00;00;55;26 - 00;00;59;05

And I'm one of them,

which is using this, like, umbrella term

00;00;59;05 - 00;01;03;02

AI versus

actually clarifying what you mean, right?

00;01;03;02 - 00;01;06;06

Because there's, you know,

different components under there.

00;01;06;15 - 00;01;09;07

And so this is something that that, you

know, I've heard you talk about before.

00;01;09;07 - 00;01;12;10

So under AI, what are kind of

00;01;12;10 - 00;01;15;13

the different macro types of AI.

00;01;15;13 - 00;01;17;00

And can you unpack those a little bit?

00;01;18;02 - 00;01;18;11

Sure.

00;01;18;11 - 00;01;21;11

It's a great question Geoff, and thanks

for having me on the show, by the way.

00;01;22;03 - 00;01;23;21

Yeah. AI's not a monolith.

00;01;23;21 - 00;01;24;23

It never has been.

00;01;24;23 - 00;01;27;23

It's always sort of been an umbrella term

for collection of things.

00;01;28;15 - 00;01;31;26

But these days, you know,

without when it's not qualified

00;01;31;26 - 00;01;33;22

and it's general usage, it usually refers

00;01;33;22 - 00;01;37;02

to a use of large language models

or generative AI.

00;01;37;29 - 00;01;41;20

But indeed, there is at least one

other main kind of AI,

00;01;41;20 - 00;01;44;23

which is very much still in operation,

which is predictive AI and predictive.

00;01;44;23 - 00;01;49;03

I learned from data to predict

for each individual who's going to click,

00;01;49;03 - 00;01;51;23

buy, lie or die, commit

an act of fraud, any outcome

00;01;51;23 - 00;01;54;22

or behavior for which there's value

for the organization

00;01;54;22 - 00;01;57;06

to improve its existing

large scale operations.

00;01;57;06 - 00;01;58;12

So that's the technology.

00;01;58;12 - 00;02;02;23

Unlike generative AI that you turn to

for improving existing large scale

00;02;02;23 - 00;02;04;27

operations,

targeting marketing fraud detection,

00;02;04;27 - 00;02;07;27

which satellite should, I suspect of maybe

running out of a battery?

00;02;07;27 - 00;02;09;29

Where should I drill for oil?

00;02;09;29 - 00;02;10;19

Any and all.

00;02;10;19 - 00;02;14;15

The largest scale existing operations

where you want to tip the balance

00;02;14;15 - 00;02;16;07

in the numbers game. That is business.

00;02;16;07 - 00;02;17;19

We don't have a magic crystal ball.

00;02;17;19 - 00;02;19;28

We can predict better than guessing.

00;02;19;28 - 00;02;22;17

And that's generally more than sufficient

to improve those.

00;02;22;17 - 00;02;25;14

So that turns out to be a very different

00;02;25;14 - 00;02;28;14

kind of discipline or project

00;02;28;19 - 00;02;33;19

than the kinds of, large language

models, type use cases and applications,

00;02;33;19 - 00;02;37;15

those types of generative AI projects,

they're really still very distinct.

00;02;38;11 - 00;02;42;10

And these days, the vast

majority of attention

00;02;42;10 - 00;02;46;11

is on generative

AI and disproportionately so.

00;02;46;11 - 00;02;47;12

So this is my

00;02;47;12 - 00;02;51;07

extraordinarily subversive statement,

and I hope everyone's sitting down.

00;02;51;26 - 00;02;55;18

Or most organizations should be investing

at least as much in predictive

00;02;55;18 - 00;02;59;08

AI or predictive analytics

as they do in generative AI.

00;03;01;24 - 00;03;03;06

So I want to I want to

00;03;03;06 - 00;03;06;09

dive right into that, but I also want to

I want to clarify something as well

00;03;06;10 - 00;03;07;13

because so

00;03;07;13 - 00;03;11;07

predictive AI to me, as I understand

it, is almost sort of like the OG AI,

00;03;11;08 - 00;03;15;00

like there is some version of this around

for a lot longer than decades.

00;03;15;03 - 00;03;16;12

LMS yeah,

00;03;16;12 - 00;03;20;02

but one of the things that I think

is maybe confusing some people as well.

00;03;20;02 - 00;03;24;09

And, you know, I want to ask you about it

is it seems like more and more

00;03;24;21 - 00;03;29;06

LMS and generative AI are almost

masquerading as predictive AI.

00;03;29;07 - 00;03;33;09

Like they try and plug in somewhere

to something that feels like

00;03;33;09 - 00;03;37;24

it's predictive, even if it's not

necessarily predictive itself.

00;03;37;29 - 00;03;38;26

Do you know what I mean by that?

00;03;38;26 - 00;03;43;01

Like it seems it seems like Al Alam

LMS are not just

00;03;43;15 - 00;03;46;03

positioned in the game of,

hey, here's a block of text.

00;03;46;03 - 00;03;48;17

They're trying to get

into the autonomous business.

00;03;48;17 - 00;03;53;21

They're trying to get into the hey,

I'm giving you data or an answer business,

00;03;53;21 - 00;03;56;09

and they're not necessarily doing that

on their own. Do I have that right?

00;03;56;09 - 00;03;57;12

Yeah. Let me answer that.

00;03;57;12 - 00;03;59;18

And also address

I can hear in the back of my head

00;03;59;18 - 00;04;04;12

the screams of protests from a, a good

a good portion of the listeners,

00;04;05;05 - 00;04;08;14

all machine learning predicts,

including large language models

00;04;08;14 - 00;04;09;13

predicting the next word.

00;04;09;13 - 00;04;12;13

That's what machine learning essentially

does is are different ways of using it.

00;04;12;21 - 00;04;16;03

You can use a large language model

for a predictive AI project.

00;04;16;10 - 00;04;17;24

You can use it to predict.

00;04;17;24 - 00;04;20;16

It's typically overkill.

00;04;20;16 - 00;04;21;20

Maybe not, though.

00;04;21;20 - 00;04;22;17

It can help.

00;04;22;17 - 00;04;23;23

There's diminishing returns

00;04;23;23 - 00;04;25;18

as you increase

the complexity of the model,

00;04;25;18 - 00;04;27;28

whether it's a large language model

or one of the standard

00;04;27;28 - 00;04;31;06

classical machine learning methods

like decision trees, log linear

00;04;31;06 - 00;04;34;14

regression, and even smaller

neural networks and ensemble models.

00;04;36;16 - 00;04;39;09

But is diminishing

returns because we can't expect

00;04;39;09 - 00;04;42;17

to have highly confident prediction about

who's going to click buy, live or die.

00;04;42;17 - 00;04;45;17

Too often

we're predicting for humans or, or

00;04;46;16 - 00;04;49;02

enterprise customers or transactions,

00;04;49;02 - 00;04;52;03

which kind of turn out to be which ones

are going to turn out to be fraudulent.

00;04;52;15 - 00;04;56;07

We don't have a magic crystal ball, so

we don't have high confidence, but we can.

00;04;56;11 - 00;04;58;26

We have the next best thing,

which is probabilities,

00;04;58;26 - 00;05;02;09

which is putting the odds on those numbers

so we can tip the odds

00;05;02;09 - 00;05;06;00

as we play these numbers games

and treat each case accordingly.

00;05;06;08 - 00;05;09;11

So the difference between generative

AI and predictive

00;05;09;11 - 00;05;13;02

I used to be known as predictive

analytics.

00;05;13;02 - 00;05;14;17

It's older but not old school.

00;05;14;17 - 00;05;16;22

Most of its value

is still largely untapped.

00;05;16;22 - 00;05;19;22

The difference isn't so much

about the core technology,

00;05;20;03 - 00;05;22;06

but about the the way it's being used.

00;05;22;06 - 00;05;26;09

So these are two

main categories of use cases,

00;05;27;24 - 00;05;29;10

of machine learning.

00;05;29;10 - 00;05;33;21

And they're destined to remain distinct

because they're inherently there's

00;05;33;21 - 00;05;37;17

inherently different thing

for those types of projects

00;05;37;17 - 00;05;40;26

where you're trying to target

large scale operations more effectively,

00;05;40;26 - 00;05;44;29

whether it's marketing or any kind of risk

management or insurance decisions,

00;05;44;29 - 00;05;49;29

pricing selection, fraud detection, it's

there's there's a an embarrassingly long

00;05;49;29 - 00;05;54;25

it's embarrassingly, horizontal

is an extremely long tail of use cases.

00;05;56;04 - 00;06;00;06

And the type of sort of discipline

and endeavor when you're putting odds

00;06;00;06 - 00;06;03;09

and then you're systematically acting on

00;06;03;09 - 00;06;07;08

those probabilities, those predictive

scores of things, output by the model.

00;06;07;08 - 00;06;07;20

Right.

00;06;07;20 - 00;06;11;14

So in, in sort of a technical

point is when you're using

00;06;11;14 - 00;06;13;01

a large language model, is predicting

00;06;13;01 - 00;06;15;17

which of the next word

be or on that level of detail, right.

00;06;15;17 - 00;06;19;05

The next token,

which is incredible what it can do.

00;06;19;16 - 00;06;23;01

Whereas for the sort of enterprise machine

learning projects that I'm referring to

00;06;23;01 - 00;06;26;01

as predictive AI or predictive analytics,

00;06;26;01 - 00;06;29;03

you're predicting on the enterprise

a unit.

00;06;29;05 - 00;06;32;09

So, you know, for which customer health

00;06;32;10 - 00;06;36;21

care, patient company

transaction, automobile, whatever it is.

00;06;36;21 - 00;06;41;17

So it's on that level rather than on a

per word level with a language model.

00;06;42;21 - 00;06;43;21

And so when we come back to the

00;06;43;21 - 00;06;48;02

so that was a big word salad or you know,

do you think that I helped clarify there.

00;06;48;07 - 00;06;50;01

Well let's let let's keep that.

00;06;50;01 - 00;06;52;10

It was helpful. Let's keep down

that clarification journey.

00;06;52;10 - 00;06;58;02

Because as I mentioned before, people

just often say AI and they leave it there.

00;06;58;02 - 00;06;59;25

Maybe that's deliberate, maybe it's not.

00;06;59;25 - 00;07;03;05

But when when people say

AI is going to cure cancer

00;07;03;05 - 00;07;06;10

or these Patriot missiles, or using AI,

00;07;07;11 - 00;07;08;15

what are they talking about there?

00;07;08;15 - 00;07;11;13

Are they talking about generative AI,

or are they talking about predictive AI?

00;07;11;13 - 00;07;14;00

Or is there

something more nuanced than that?

00;07;14;00 - 00;07;17;00

And, you know, it's often the case

that they're talking about predictive,

00;07;17;00 - 00;07;20;15

but it's in the same conversation

that I've started with, generative.

00;07;20;15 - 00;07;23;20

On the other hand, the two areas

you just described could be either, right.

00;07;23;20 - 00;07;25;16

There's ways

in which they both can pertain.

00;07;26;26 - 00;07;29;14

So it just

depends on the very particular project,

00;07;29;14 - 00;07;33;16

the problem you're trying to solve,

the value proposition.

00;07;33;16 - 00;07;33;23

Right.

00;07;33;23 - 00;07;39;07

Are you making

probabilistic determinations of an unknown

00;07;39;07 - 00;07;41;04

whether this is going

to turn out to be fraudulent,

00;07;41;04 - 00;07;45;07

whether this health care patient is going

to turn out to have this diagnosis?

00;07;46;18 - 00;07;49;09

And you know, who's going to click buy

Live or die, right?

00;07;49;09 - 00;07;52;09

Or for or cancel their subscription.

Right.

00;07;52;22 - 00;07;55;07

Whatever. Those are predictive use cases.

00;07;55;07 - 00;07;58;10

And those tend to be the,

I mean, predictions, the holy grail

00;07;58;10 - 00;08;01;23

for improving existing

large scale operations. So,

00;08;02;24 - 00;08;06;00

I've heard you in other episodes say,

hey, look, is it

00;08;06;00 - 00;08;11;03

turning out that AI is mostly valuable

for really unsexy things?

00;08;11;15 - 00;08;15;17

And my answer is mostly yes, kind of.

00;08;15;21 - 00;08;19;28

It's a very subjective concept,

but certainly on the scale

00;08;20;06 - 00;08;24;22

in, in, in, in comparison to the way

generative AI is hyped,

00;08;25;14 - 00;08;28;05

this stuff is extremely unsexy.

00;08;28;05 - 00;08;31;00

But to me, as someone who has been in love

00;08;31;00 - 00;08;34;01

with the concept of machine

learning for more than 30 years

00;08;34;19 - 00;08;38;14

and just as much loves its actual viable

00;08;38;14 - 00;08;42;09

operationalization, its use,

its deployment, its application, right?

00;08;42;09 - 00;08;46;01

Not just this awesome technology

with all technology, including

00;08;46;01 - 00;08;47;14

both kinds of eyes.

00;08;47;14 - 00;08;51;19

There are two wows

and we almost always get to the first wow.

00;08;51;19 - 00;08;57;01

And and very rarely more often

not than do get to the second wow.

00;08;57;01 - 00;08;58;23

The first wow is the technology.

00;08;58;23 - 00;09;03;29

The core capability learning from data

to predict or being able to do

00;09;03;29 - 00;09;08;04

an amazing demo with a large language

model, a pilot, right?

00;09;08;04 - 00;09;11;02

Sort of a a broad proof of concept.

00;09;11;02 - 00;09;11;25

That's the first wow.

00;09;11;25 - 00;09;14;28

The second Wow comes with deployment

operationalization, right?

00;09;14;28 - 00;09;20;03

With the first wow, the technology

you have created, potential value

00;09;20;07 - 00;09;23;04

you haven't necessarily captured

or realize that value.

00;09;23;04 - 00;09;26;07

And indeed you definitely have

not until you actually deploy it.

00;09;28;18 - 00;09;30;05

If you're a technology leader,

00;09;30;05 - 00;09;33;17

Infotech is a name you need to know

for getting ahead with AI.

00;09;34;04 - 00;09;37;12

Infotech helps leaders

deliver on top technology initiatives,

00;09;37;17 - 00;09;41;23

and one of our most popular programs

right now is AI strategy.

00;09;42;03 - 00;09;45;19

We've helped hundreds of organizations

build out an AI strategy

00;09;45;19 - 00;09;48;19

and roadmap

using our step by step methodology.

00;09;49;05 - 00;09;52;12

Infotech road tested

approach gives you a trusted blueprint

00;09;52;12 - 00;09;55;12

to execute faster

and avoid costly mistakes.

00;09;55;18 - 00;09;59;11

And what you walk away

with is a boardroom ready AI strategy

00;09;59;15 - 00;10;02;15

tailored to your needs

and ready to execute.

00;10;02;21 - 00;10;05;21

What sets Infotech apart

is how we deliver.

00;10;05;24 - 00;10;09;25

Unlike traditional research or AI tools,

we don't stop at guidance.

00;10;10;01 - 00;10;13;08

Infotech works with your team

to build the strategy.

00;10;13;19 - 00;10;17;27

You can do this through a focused one week

workshop, ongoing analyst support,

00;10;18;05 - 00;10;22;10

or by using our full methodology

and toolkit to get it done yourself.

00;10;22;27 - 00;10;26;20

The outcome is an AI strategy

ready for the boardroom and a team

00;10;26;20 - 00;10;28;09

that's ready to deliver.

00;10;28;09 - 00;10;30;15

To learn more about our

AI strategy approach,

00;10;30;15 - 00;10;33;18

check out the link below

and don't forget to like and subscribe!

00;10;34;17 - 00;10;37;05

I want to come back to something

you said earlier, because I think we're

00;10;37;05 - 00;10;40;05

we're very close to your thesis here,

which is,

00;10;40;22 - 00;10;45;05

you know, basically what I heard you say

is that most organizations are over

00;10;45;05 - 00;10;49;20

indexed right now on generative

AI and under indexed on predictive AI.

00;10;49;23 - 00;10;51;04

And so I wanted to

00;10;51;04 - 00;10;54;09

I wanted to come back to what you were

just saying and ask, like, why is that?

00;10;54;10 - 00;11;00;21

What is it about predictive AI that that

has more opportunity or potential for you?

00;11;01;22 - 00;11;04;29

You know, around the time

this episode is releasing, I'm

00;11;05;04 - 00;11;08;20

releasing an article on CDO magazine,

which I make the case that,

00;11;09;06 - 00;11;12;13

yeah, it's overindex,

at least in terms of venture

00;11;12;13 - 00;11;16;29

investments and R&D,

but in terms of actual enterprise

00;11;16;29 - 00;11;19;06

deployment and projects,

maybe on the surface,

00;11;19;06 - 00;11;21;16

maybe in terms of the headlines,

maybe in terms of the way

00;11;21;16 - 00;11;24;17

executives talk to the board

and such, it's very under indexed,

00;11;24;24 - 00;11;28;20

but predictive AI is still,

it turns out, very much alive.

00;11;28;20 - 00;11;30;19

And I give a bunch of evidence

to that effect.

00;11;30;19 - 00;11;35;04

For example, when I did a broad call

for speakers for the conference series,

00;11;35;04 - 00;11;38;22

I've been running, for a long time

called Machine Learning Week,

00;11;39;01 - 00;11;42;03

and I didn't specify, hey,

we prefer predictive versus generative.

00;11;42;03 - 00;11;44;03

And it ended up being about 5050.

00;11;44;03 - 00;11;45;23

I mean, there's lots of predictive power

00;11;45;23 - 00;11;49;17

and and also just informally like

this isn't legit data,

00;11;49;17 - 00;11;54;19

but the buzz behind the scenes so often

we hear people saying,

00;11;54;25 - 00;11;57;25

you know, there's a lot of pressure

to use generative AI,

00;11;57;25 - 00;12;00;28

but really the thing that's

still delivering value is predictive.

00;12;00;29 - 00;12;05;02

So even though the bad news might be that,

hey, look, it almost looks like

00;12;05;02 - 00;12;07;28

we're throwing out the baby

with the bathwater, much to the detriment

00;12;07;28 - 00;12;11;25

of the organization and sacrificing

all this potential value because

00;12;12;10 - 00;12;16;26

the overwhelming spotlight and attention

is being given on generative AI.

00;12;17;12 - 00;12;20;29

The good news is,

it turns out that hasn't fully taken hold.

00;12;20;29 - 00;12;25;02

And there's still I

mean, there is something sexy about value.

00;12;27;09 - 00;12;28;28

For for sure.

00;12;28;28 - 00;12;31;28

And I'm curious, like to use your words.

00;12;32;04 - 00;12;34;09

Where does that second wild live?

00;12;34;09 - 00;12;37;22

Like where are we seeing that right now

versus where are people,

00;12;37;22 - 00;12;39;02

you know all talking about it.

00;12;39;02 - 00;12;41;06

But it's kind of hot air.

00;12;41;06 - 00;12;44;00

Well, to answer that question,

I kind of have two answers separately

00;12;44;00 - 00;12;45;09

for generative versus predictive.

00;12;45;09 - 00;12;48;23

They both have this very parallel story

where, you know,

00;12;48;29 - 00;12;51;29

pilots

don't go to production and most don't.

00;12;52;04 - 00;12;53;21

But some do, right?

00;12;53;21 - 00;12;57;18

So if it's 5 to 15% depending.

00;12;57;18 - 00;12;58;06

Right.

00;12;58;06 - 00;13;01;23

And this is a moving target and there's a

there's a lot sort of conflicting data

00;13;01;23 - 00;13;04;03

about it, it's

hard to put an exact number on.

00;13;04;03 - 00;13;06;00

But I was involved with survey,

00;13;06;00 - 00;13;09;06

industry surveys for predictive

and that looked like it was around 15%.

00;13;12;18 - 00;13;14;23

What happens is with predictive AI.

00;13;14;23 - 00;13;18;07

So if it's, you know,

if it's 5 to 15% of many, many projects,

00;13;18;07 - 00;13;19;24

that's still a lot of successes.

00;13;19;24 - 00;13;22;00

And predictive

AI has been around for decades, right?

00;13;22;00 - 00;13;24;19

I mean, in the 60s,

they started targeting,

00;13;24;19 - 00;13;27;19

marketing and doing credit risk,

00;13;28;00 - 00;13;29;29

prediction with regression

00;13;29;29 - 00;13;32;29

and like sort of primordial machine

learning models.

00;13;34;23 - 00;13;37;10

So there's a been a long track record,

00;13;37;10 - 00;13;42;03

and even if, despite its age, predictive

AI still hasn't fully mature

00;13;42;03 - 00;13;45;02

and reached that level

of professionalization,

00;13;46;00 - 00;13;47;12

there's still a lot of success.

00;13;47;12 - 00;13;52;18

However, the potential is generally

untapped, and we could do a lot better.

00;13;52;18 - 00;13;55;08

And it's not on both accounts

generative and predictive.

00;13;55;08 - 00;13;58;25

It's not a tech problem

so much as an organizational one.

00;13;59;10 - 00;14;03;01

In the case of generative,

the problem is unrealistic

00;14;03;01 - 00;14;06;19

expectations in feasibility

and that is to say reliability.

00;14;06;19 - 00;14;10;11

So it's very, very easy

to conceive of a project to scope

00;14;10;16 - 00;14;14;23

and intended use of what people often

you use the word agent.

00;14;14;23 - 00;14;15;08

I feel that

00;14;16;17 - 00;14;18;04

word isn't helping

00;14;18;04 - 00;14;21;04

avoid the over promising. But,

00;14;21;24 - 00;14;23;06

whatever you want to call it, right?

00;14;23;06 - 00;14;27;11

The system, can't be deployed

because even though its error

00;14;27;11 - 00;14;32;02

rate is only 5% and then 95% success rate

is incredible, right?

00;14;32;02 - 00;14;32;29

Unprecedented.

00;14;32;29 - 00;14;35;19

Couldn't even have imagined it

five years ago.

00;14;35;19 - 00;14;37;09

The error rates too frequent.

00;14;37;09 - 00;14;41;03

So it's not viable,

at least with its intended current

00;14;41;03 - 00;14;44;12

and currently intended usage

of being fully autonomous.

00;14;44;19 - 00;14;47;21

And therefore it's going to collect dust

nobody can operationalize.

00;14;47;21 - 00;14;50;20

It therefore doesn't provide any value

that's generative.

00;14;50;20 - 00;14;55;09

In the case of predictive, it's

not that it's got a reliability problem.

00;14;55;09 - 00;14;58;28

Nobody and nobody who's familiar

enough with this area,

00;14;59;16 - 00;15;01;07

or anybody who listens to the following

00;15;01;07 - 00;15;04;22

sentence, is going to expect too much

because it's not about high.

00;15;05;03 - 00;15;06;18

Well, the word accuracy is already,

00;15;07;19 - 00;15;10;19

often thrown around, but what that is

usually means

00;15;10;19 - 00;15;14;25

is that it's really confident about

both positive and negative predictions.

00;15;15;15 - 00;15;18;15

That is to say, really confident

about every prediction.

00;15;18;17 - 00;15;20;12

It's not a magic crystal ball.

00;15;20;12 - 00;15;24;04

It's simply a lot better

than guessing that tends to be valuable.

00;15;24;04 - 00;15;28;17

So it's not that it's un

it needs more technical reliability,

00;15;28;28 - 00;15;32;08

but there's an understandability

or complexity issue.

00;15;32;14 - 00;15;35;09

It's not a rocket science level

type of understanding.

00;15;35;09 - 00;15;38;00

It's sort of a semi technical

accessible now.

00;15;38;00 - 00;15;40;24

But stakeholders need to have, an

00;15;40;24 - 00;15;44;04

understanding and applied just the right

particular arithmetic to understand.

00;15;44;04 - 00;15;45;26

Hey, here are my operations.

00;15;45;26 - 00;15;47;05

Here's how often we, you know,

00;15;47;05 - 00;15;48;29

we fall victim

to fraud in the transactions

00;15;48;29 - 00;15;51;05

or whatever problem

you're trying to solve.

00;15;51;05 - 00;15;54;14

And here's the math that shows,

hey, if we take this action in these

00;15;54;14 - 00;15;57;14

high risk cases, we'll be wrong in

the other direction.

00;15;57;14 - 00;16;00;02

We'll have false positives.

Let's do the arithmetic.

00;16;00;02 - 00;16;02;01

Let's figure out what the bottom line is.

00;16;02;01 - 00;16;06;21

That level of complexity

often eludes, business stakeholders.

00;16;06;21 - 00;16;09;14

So there's sort of a cultural catch up.

00;16;09;14 - 00;16;13;22

Yeah, it's already been 40 or 50 years,

but you know, we're so close

00;16;13;22 - 00;16;14;17

and yet so far.

00;16;14;17 - 00;16;18;11

So just to summarize, generative

AI has a reliability issue.

00;16;18;11 - 00;16;20;10

Predictive AI has a complexity issue.

00;16;25;19 - 00;16;26;22

There's an angle here that

00;16;26;22 - 00;16;29;22

I want to talk about around the hype

there.

00;16;29;22 - 00;16;31;15

And and tell me if you disagree,

00;16;31;15 - 00;16;34;15

because I don't even know

that I agree with myself right now.

00;16;35;12 - 00;16;37;07

One of the things

00;16;37;07 - 00;16;41;22

that comes to mind hearing about both of

those is it feels to me like predictive

00;16;41;22 - 00;16;47;03

AI is probably a really effective

optimization tool for any business.

00;16;47;03 - 00;16;50;03

It can help you squeeze

more juice from the same fruit.

00;16;50;04 - 00;16;53;18

The promise that we're starting to hear

about from generative AI,

00;16;54;14 - 00;16;55;29

you know, more so seems to be like,

00;16;55;29 - 00;16;58;11

this is going to transform

the way you do business.

00;16;58;11 - 00;17;00;03

Forget the fruit you've got right now.

00;17;00;03 - 00;17;04;04

Like this is going to take you from

selling oranges to, you know, a smoothie

00;17;04;04 - 00;17;04;16

stand.

00;17;04;16 - 00;17;07;16

I've, I think I've stretched this metaphor

about as far as it can go,

00;17;07;21 - 00;17;11;09

but but it seems to be a very different

narrative around it right now.

00;17;12;06 - 00;17;14;24

Do you buy my framing and do you buy that?

00;17;14;24 - 00;17;17;04

That's actually. True.

00;17;17;04 - 00;17;19;06

You mean the that's that it's true

00;17;19;06 - 00;17;22;05

that that's the way in which it's over

promised and overhyped.

00;17;22;24 - 00;17;23;29

Yeah.

00;17;23;29 - 00;17;27;07

Are those the conversations

that people are having and

00;17;27;19 - 00;17;30;06

is that actually

true about the technology?

00;17;31;09 - 00;17;31;19

Yeah.

00;17;31;19 - 00;17;38;03

I mean, look, I,

I've been annoyed by hype for a long time.

00;17;38;21 - 00;17;41;24

And in the last few years,

two things have happened in once

00;17;42;03 - 00;17;46;13

large language

models came out and knocked my socks off.

00;17;46;27 - 00;17;47;14

Right.

00;17;47;14 - 00;17;50;04

So to be specific, like during my six

00;17;50;04 - 00;17;53;04

years of PhD work at Columbia,

00;17;53;17 - 00;17;57;21

before I was on the faculty there,

I was in the natural language processing

00;17;57;21 - 00;18;00;08

or computational linguistics

group, and I never saw

00;18;00;08 - 00;18;02;09

and I see what we see

today, in my lifetime.

00;18;02;09 - 00;18;04;05

So I'm like, Holy cow.

00;18;04;05 - 00;18;07;10

And then yet the world is sort

of a hundred times more excited than I am.

00;18;07;10 - 00;18;10;10

And I'm like, hey, whoa, it's

not a deity, right?

00;18;10;17 - 00;18;11;15

Let let you know.

00;18;11;15 - 00;18;13;07

Calm down a little bit here.

00;18;13;07 - 00;18;16;12

It's so it's it's incredible.

00;18;16;17 - 00;18;18;20

But the promising is saying,

00;18;20;17 - 00;18;23;03

you know, is saying is promising extreme

00;18;23;03 - 00;18;26;03

autonomy, otherwise known

as artificial general intelligence.

00;18;26;03 - 00;18;30;18

Autonomy as in doing what the person

can do without a human in the loop.

00;18;31;18 - 00;18;35;08

So and that hype was always there

and the potential for it

00;18;35;08 - 00;18;40;08

and the sort of sci fi narratives,

the way predictive I was often over

00;18;40;08 - 00;18;43;23

promised was it was

you would use the word highly accurate,

00;18;44;03 - 00;18;48;07

and it would be framed in a way

that implied that it was really confident

00;18;48;18 - 00;18;50;26

about every output,

about every prediction.

00;18;50;26 - 00;18;51;27

It could tell who was pregnant.

00;18;51;27 - 00;18;54;22

That was a target thing,

00;18;54;22 - 00;18;55;14

things like that.

00;18;55;14 - 00;18;58;14

And we're sort of,

00;18;58;25 - 00;19;02;09

expressed in terms of,

you know, it's just expectation.

00;19;02;09 - 00;19;05;05

Hype is always a matter of mismanaged

expectation.

00;19;05;05 - 00;19;07;28

So, no, it doesn't predict

like a magic crystal ball.

00;19;07;28 - 00;19;10;06

Unfortunately,

we don't have magic crystal balls.

00;19;10;06 - 00;19;15;14

And no matter how advanced and complex

the technology gets, we we still won't.

00;19;16;14 - 00;19;17;23

But even then, before

00;19;17;23 - 00;19;20;24

generative AI, there would also be

the word intelligence thrown around

00;19;20;24 - 00;19;24;16

and these sort of these promises of, hey,

it's going to become super autonomous.

00;19;24;25 - 00;19;26;20

And that always drove me crazy.

00;19;26;20 - 00;19;30;00

Can you imagine how I feel now

when that's the prevalent narrative.

00;19;31;01 - 00;19;34;12

While and that I completely agree with

it feels like

00;19;34;19 - 00;19;38;19

autonomy has become,

you know, the the word of the day.

00;19;38;19 - 00;19;42;09

And, you know, you mentioned AGI,

artificial general intelligence.

00;19;42;09 - 00;19;44;26

You mentioned you mentioned agents.

00;19;44;26 - 00;19;50;22

And all of these seem to be thrown around

as though the capability is already here.

00;19;50;22 - 00;19;52;22

It's just around the corner.

00;19;52;22 - 00;19;55;07

You know, speaking about all of those,

you know, I hear on

00;19;55;07 - 00;19;58;07

a spectrum from AGI is already here.

00;19;58;07 - 00;20;00;25

We've got autonomous

AI that can do whatever a human

00;20;00;25 - 00;20;04;15

can in execution form

to actually laser a dead end.

00;20;04;17 - 00;20;06;22

And they're not ever going to take us

to AGI.

00;20;06;22 - 00;20;09;22

Like, really,

truly opposite ends of the spectrum.

00;20;09;27 - 00;20;13;05

And so I'm curious,

I mean, you've got a much deeper machine

00;20;13;05 - 00;20;16;07

learning background than most people

I speak with.

00;20;16;19 - 00;20;18;00

Where do you fall on that?

00;20;18;00 - 00;20;22;27

What what level of autonomy

are we at right now with these tools

00;20;22;27 - 00;20;26;10

and what's even realistic to get to

in the, you know, foreseeable future?

00;20;28;13 - 00;20;29;24

What level of autonomy are

00;20;29;24 - 00;20;33;00

we on where at 80%.

00;20;34;01 - 00;20;35;28

Okay,

so I'm that's a little tongue in cheek.

00;20;35;28 - 00;20;37;13

Okay.

00;20;37;13 - 00;20;40;03

It could be 90%,

but there's there's diminishing

00;20;40;03 - 00;20;43;29

returns to the last few percent

that are almost impossible. So.

00;20;44;03 - 00;20;48;22

And in fact, as I address your question,

let me simultaneously say that.

00;20;48;22 - 00;20;52;09

Look, hybridizing predictive

AI and generative AI together,

00;20;52;09 - 00;20;56;23

using predictive AI as reliability layer,

I believe that's

00;20;56;23 - 00;21;00;24

the next killer app for predictive AI

and the saving grace for this hype, right?

00;21;00;24 - 00;21;05;16

It's this audacious promise of supreme

autonomy that's generally infeasible

00;21;05;22 - 00;21;10;19

unless you have like a really constrained,

like customer service agent

00;21;10;19 - 00;21;13;24

that only can answer questions about,

you know, 20

00;21;13;24 - 00;21;17;20

pages of fake level information

and it can't conduct transactions.

00;21;17;20 - 00;21;20;20

Like if you constrain the scope

well enough, then,

00;21;20;20 - 00;21;23;13

you know, we're very quickly

getting to the point where it is feasible,

00;21;23;13 - 00;21;26;13

even for end consumer facing uses.

00;21;27;04 - 00;21;31;06

That even the most vain

or cosmetically conscious,

00;21;31;13 - 00;21;34;28

large corporations

has potential, to deploy.

00;21;35;02 - 00;21;36;28

But once you start allowing it to do

00;21;36;28 - 00;21;41;03

transactions or deal with sensitive data

and all sorts and sort of open the doors,

00;21;41;03 - 00;21;45;02

all things that to high sensitivity

to failure and errors,

00;21;46;11 - 00;21;47;03

it's it

00;21;47;03 - 00;21;50;05

the viability goes out the window

very, very quickly.

00;21;50;05 - 00;21;52;09

And I believe that

00;21;52;09 - 00;21;56;03

just from first principles, the way

that that's going to be reckoned with

00;21;56;16 - 00;22;01;26

is to play the odds predictably and say,

hey, look, here's a high risk case.

00;22;01;26 - 00;22;05;24

The system has predicted

that this is relatively high, five times

00;22;05;24 - 00;22;08;28

more likely than average, ten times

more likely than average to

00;22;08;28 - 00;22;12;10

go off the rails to to do, to hallucinate

00;22;12;10 - 00;22;15;19

or to divulge sense of data

or do a bad transfer or whatever.

00;22;15;19 - 00;22;17;17

There's a lot of different things

that can go wrong.

00;22;17;17 - 00;22;20;08

Let's now, escalate to a human.

00;22;20;08 - 00;22;24;07

And that decision on the fly to go to that

more expensive human in the loop.

00;22;24;07 - 00;22;24;25

Right.

00;22;24;25 - 00;22;27;21

If it were the if, if,

if the system were to do that

00;22;27;21 - 00;22;31;20

for the top,

let's say, 20% of interactions, then,

00;22;32;21 - 00;22;35;21

potentially the net error rate

00;22;36;02 - 00;22;39;01

after doing so,

even if only a small portion of that

00;22;39;01 - 00;22;42;07

20 were relatively small portion,

let's say only two out of those 20

00;22;43;00 - 00;22;46;18

points were actually a grave error,

but the human was on those.

00;22;47;03 - 00;22;50;15

Now we're expanding

where we're not reaching full autonomy.

00;22;51;11 - 00;22;54;11

We still have to spend money on humans.

00;22;54;16 - 00;22;57;13

But we have achieved 80% of that otherwise

00;22;57;13 - 00;23;00;15

extraordinarily

audacious promise of autonomy.

00;23;01;11 - 00;23;03;11

A lot better than the 0% that

00;23;03;11 - 00;23;06;11

we realize if we can't launch the system.

00;23;08;18 - 00;23;09;24

Well, and your

00;23;09;24 - 00;23;13;04

your point is well taken,

because if I'm looking at that through,

00;23;13;04 - 00;23;17;10

like an actual execution and, like,

I don't know, business process lens,

00;23;17;21 - 00;23;24;06

what that's telling me is that at 80%,

we can still re-architect

00;23;24;06 - 00;23;26;17

a lot of our systems,

a lot of our processes

00;23;26;17 - 00;23;30;19

in a big way that we couldn't do

without these in the system,

00;23;30;19 - 00;23;32;00

even if it's not full autonomy.

00;23;32;00 - 00;23;33;18

And maybe if it's full autonomy,

you know, you

00;23;33;18 - 00;23;37;12

you re-architect them again

and you squeeze even more juice out of it.

00;23;37;12 - 00;23;40;19

But yeah, I mean, to your point,

when do we get there?

00;23;40;19 - 00;23;45;07

So I don't know, I guess what

I, what I heard and I'm curious to play it

00;23;45;07 - 00;23;49;25

back to you is, yeah, there's

a lot of really good stuff here right now.

00;23;49;25 - 00;23;50;27

So maybe the,

00;23;50;27 - 00;23;55;03

maybe the full autonomy doesn't matter

as much as the hype is making it out to,

00;23;57;14 - 00;23;58;22

Well, no, it does matter

00;23;58;22 - 00;24;02;11

if we could achieve

artificial general intelligence,

00;24;02;19 - 00;24;06;23

that would be amazing

and makes for great science fiction movie.

00;24;06;23 - 00;24;09;07

And I love science fiction.

00;24;09;07 - 00;24;10;21

I enjoy it.

00;24;10;21 - 00;24;13;21

But let's get real here, folks.

00;24;13;21 - 00;24;17;08

I mean, there's a reason

that that that, promise

00;24;17;08 - 00;24;20;08

is being thrown around so haphazardly.

00;24;20;28 - 00;24;22;18

It's making a lot of people

a lot of money.

00;24;22;18 - 00;24;25;18

It's it's bolstering,

00;24;25;23 - 00;24;26;29

valuations.

00;24;26;29 - 00;24;29;06

And it's

00;24;29;06 - 00;24;30;06

it makes me groan.

00;24;30;06 - 00;24;31;23

It makes me snicker. Right.

00;24;31;23 - 00;24;34;05

It makes me make lots of,

00;24;34;05 - 00;24;36;06

kind of pessimistic

00;24;36;06 - 00;24;39;06

jokes

behind closed doors with my wife and,

00;24;40;06 - 00;24;40;17

right.

00;24;40;17 - 00;24;44;26

But it's really hard to address the hype

without saying something negative.

00;24;44;26 - 00;24;48;21

Or they say people who address the hype,

you know, sound smart, and people

00;24;48;21 - 00;24;51;21

who enable the hype get rich.

00;24;52;17 - 00;24;55;07

But but also, I don't want to lose tab

00;24;55;07 - 00;24;58;12

with the fact that there's a lot

to be very positive and excited about.

00;24;58;12 - 00;25;01;09

I'm generally a, a tech optimist.

00;25;01;09 - 00;25;04;09

And I've,

I've thought for a very long time

00;25;04;16 - 00;25;08;24

that, learning from data to predict

and all the ways that that can be used

00;25;09;00 - 00;25;12;22

is the next best thing since sliced bread,

both in terms of geeking out

00;25;12;22 - 00;25;14;19

and how cool the first of two wows.

00;25;14;19 - 00;25;16;28

You know just how amazing it is.

00;25;16;28 - 00;25;19;26

This idea of what's called, induction.

Right?

00;25;19;26 - 00;25;23;15

Deriving generalizations

from some limited number of examples.

00;25;23;15 - 00;25;25;03

And you may have many training examples.

00;25;25;03 - 00;25;27;16

You may have millions of them,

but it's a small number compared

00;25;27;16 - 00;25;31;07

to the number of possible situations

you could ever possibly encounter,

00;25;31;12 - 00;25;34;26

such that you derive generalizations

that hold that turn out

00;25;34;26 - 00;25;39;08

to have truth about the world,

because they turn out to pertain to,

00;25;39;29 - 00;25;44;04

previously unseen

novel cases and situations and examples.

00;25;44;25 - 00;25;48;20

So that ability to learn from data

geeking out the first.

00;25;48;20 - 00;25;50;09

Wow, that's really awesome.

00;25;50;09 - 00;25;54;13

And indeed it is extremely valuable

if you actually can get it deployed.

00;25;54;14 - 00;25;55;13

That's the second. Well,

00;25;56;21 - 00;25;59;23

and everything I just said applies

for both predictive and generative,

00;25;59;23 - 00;26;03;12

because the generative

is one of the ultimate so far,

00;26;03;24 - 00;26;05;27

manifestations of that concept. Right?

00;26;05;27 - 00;26;07;10

I mean, the ability to predict

00;26;07;10 - 00;26;11;19

what should be the next word,

the next token, right, is only possible

00;26;11;29 - 00;26;16;28

because over the previous

two paragraphs or 200 pages or whatever,

00;26;17;16 - 00;26;20;18

you know,

the prompt was for this next step

00;26;20;18 - 00;26;24;06

in the basically generating of text,

right?

00;26;24;29 - 00;26;28;29

It had to derive some real sense

of meaning

00;26;30;09 - 00;26;32;05

about the language

00;26;32;05 - 00;26;36;07

in order to do that as well as it does,

and so that as it continues to do that,

00;26;36;22 - 00;26;41;13

what it writes coherent and more often

than not actually correct, miraculously.

00;26;41;13 - 00;26;41;25

Right.

00;26;41;25 - 00;26;46;27

So it's it's really, really,

really amazing and worthy of geeking out.

00;26;46;27 - 00;26;50;11

And yet at the same time,

I do believe that the difference between

00;26;50;11 - 00;26;55;09

what it's capable of and what humans are

will become increasingly, clear.

00;26;59;07 - 00;27;00;24

Do you want to,

00;27;00;24 - 00;27;01;29

elucidate that a little more?

00;27;01;29 - 00;27;04;22

I'd love I'd love to hear more about that.

Like, let's clarify.

00;27;04;22 - 00;27;07;14

What that's that's that's

that's the ultimate challenge.

00;27;07;14 - 00;27;07;19

Right.

00;27;07;19 - 00;27;11;17

And I'm I'm constantly looking for someone

who can put their finger on it.

00;27;14;06 - 00;27;19;18

So it is uncanny how

seemingly humanlike it is.

00;27;19;21 - 00;27;19;27

Right?

00;27;19;27 - 00;27;24;10

It seems to work with concepts oftentimes

very much the same as people.

00;27;25;02 - 00;27;29;10

It often sort of reasons about things

the way you might expect a person to.

00;27;30;06 - 00;27;34;20

So it is actually very,

very difficult to clearly define

00;27;34;20 - 00;27;37;20

what's the difference between

what a large language model can do.

00;27;38;00 - 00;27;39;12

Right, and what a human could do now?

00;27;39;12 - 00;27;41;26

First,

principles would dictate that there is a

00;27;41;26 - 00;27;42;29

I mean, we know there's a difference.

00;27;42;29 - 00;27;45;28

We know the hallucination makes errors

that humans would never make.

00;27;46;09 - 00;27;50;18

Ultimately, the thing that matters is the

performance as a particular task at hand.

00;27;50;18 - 00;27;52;02

So for any given project,

00;27;53;09 - 00;27;55;20

really, the name of the game is mundane.

00;27;55;20 - 00;27;58;28

You just want to put metrics,

you want to measure how good it is

00;27;59;03 - 00;28;02;19

at the exact text, task how often it

00;28;02;19 - 00;28;06;00

makes, unacceptable error at that task.

00;28;06;00 - 00;28;07;28

Right.

That's the thing that really matters.

00;28;07;28 - 00;28;09;27

But if you want to get

a little more philosophical,

00;28;09;27 - 00;28;13;05

kind of trying to define the difference

between a human and a large language

00;28;13;05 - 00;28;16;05

model, large language

models have only learned,

00;28;16;18 - 00;28;20;02

or have mostly learned from text,

like all the written text

00;28;20;02 - 00;28;23;02

and all the books

and the whole web and everything.

00;28;23;20 - 00;28;27;19

How much can you reverse engineer

about human cognition

00;28;28;03 - 00;28;31;19

by learning

from all those trillions of examples?

00;28;31;19 - 00;28;33;25

And the example is

I've written this much so far.

00;28;33;25 - 00;28;35;09

What should I write as the next word?

00;28;35;09 - 00;28;37;29

Okay, now I've written this much so far,

which I write the next way.

00;28;37;29 - 00;28;42;10

Those make the learning cases right, just

the same as in predictive AI projects.

00;28;42;13 - 00;28;46;04

Hey, this customer had done all this

and then they canceled their subscription,

00;28;46;04 - 00;28;47;21

right? Either way, right.

00;28;47;21 - 00;28;52;00

You've got many cases in the case

of predictive may have 100,000 cases.

00;28;52;14 - 00;28;55;22

In the case of generative AI,

you're learning from trillions.

00;28;55;22 - 00;28;57;09

But either way you've got these cases.

00;28;57;09 - 00;28;58;26

You're learning from them.

00;28;58;26 - 00;29;03;09

How much would that process do over

all the human text ever created?

00;29;03;09 - 00;29;07;10

Essentially, how much can that

reverse engineer human cognition?

00;29;07;10 - 00;29;09;08

Well, apparently quite a bit.

00;29;09;08 - 00;29;11;05

But we're going to

find that there's a ceiling.

00;29;13;29 - 00;29;14;07

Yeah.

00;29;14;07 - 00;29;17;07

And I guess the question is,

is where that ceiling is.

00;29;17;07 - 00;29;21;08

And I, I mean, your guess is,

is probably as good as mine for that,

00;29;21;08 - 00;29;24;01

but I think, I mean, you mentioned

this notion of diminishing returns.

00;29;24;01 - 00;29;24;08

Right?

00;29;24;08 - 00;29;27;11

So I maybe I'll,

maybe I'll ask it that way

00;29;28;15 - 00;29;31;15

as we progress from here on out,

do you see us

00;29;31;28 - 00;29;35;08

accelerating the progress there, or is it,

you know, going to be sort

00;29;35;08 - 00;29;39;05

of logarithmic and it's going to flatten

as we get as this thing gets better?

00;29;39;10 - 00;29;39;16

Yeah.

00;29;39;16 - 00;29;42;19

I think, there's lots of signs

that it already is doing that,

00;29;43;12 - 00;29;44;24

that there is diminishing returns.

00;29;44;24 - 00;29;47;08

We've sort of exhausted

those learning examples.

00;29;47;08 - 00;29;49;05

There's other,

00;29;49;05 - 00;29;51;28

feedback that we gather with,

00;29;51;28 - 00;29;55;13

with reinforcement learning,

sort of layered on top of that.

00;29;56;23 - 00;29;57;20

And there's another

00;29;57;20 - 00;30;02;16

force to be reckoned with, though,

when we also consider just to what degree

00;30;02;24 - 00;30;07;18

the world can adopt this as autonomous

or as the sort of manifestation of

00;30;07;18 - 00;30;11;14

what's the spirit of the word agent

or the spirit of the word

00;30;11;19 - 00;30;14;19

artificial intelligence,

which is often encompassed

00;30;14;24 - 00;30;18;07

by artificial general intelligence,

and that other force to be reckoned with

00;30;18;26 - 00;30;21;26

is that, we,

00;30;22;26 - 00;30;26;01

we won't accept errors as easily

from an artifact.

00;30;26;01 - 00;30;26;15

There's not

00;30;26;15 - 00;30;30;07

there's no there's no getting around

the fact that this thing is artificial.

00;30;30;07 - 00;30;35;25

It's it's it does not have personhood

and it does not have accountability.

00;30;36;00 - 00;30;36;10

Right.

00;30;36;10 - 00;30;37;23

So, you know,

00;30;37;23 - 00;30;42;22

whether it's for a self-driving car

or for any enterprise system, right.

00;30;42;22 - 00;30;46;21

There's estimations that range from

we could accept,

00;30;47;03 - 00;30;52;15

you know, 1%, maybe up to 10% of the error

rate of a human.

00;30;52;27 - 00;30;55;08

But it's got to be a lot

better than a human

00;30;55;08 - 00;30;58;21

if it's going to be fully

autonomous on a high stakes task.

00;31;01;00 - 00;31;01;27

Yeah, that makes sense.

00;31;01;27 - 00;31;02;25

It does.

00;31;02;25 - 00;31;05;23

It does feel like it's

an order of magnitude thing.

00;31;05;23 - 00;31;07;26

It's not just like incremental.

00;31;07;26 - 00;31;10;01

Maybe two orders. Yeah.

00;31;10;01 - 00;31;10;18

Yeah, yeah.

00;31;10;18 - 00;31;13;02

One to 1 to 2 orders of magnitude,

let's call it.

00;31;13;02 - 00;31;16;29

So I mean, let me ask you

maybe a zoomed out question here,

00;31;17;06 - 00;31;20;02

but we talked about untapped potential.

00;31;20;02 - 00;31;21;05

We talked about the fact

00;31;21;05 - 00;31;26;05

that organizations are, you know,

frequently confused about predictive

00;31;26;05 - 00;31;31;06

AI versus generative AI,

that the investment mix is probably wrong.

00;31;31;06 - 00;31;35;16

There's, you know, that this whole pilot

to production gap that we're seeing.

00;31;35;16 - 00;31;38;16

So to come back to the big question,

like what

00;31;38;25 - 00;31;41;17

fundamentally is it

that businesses are getting wrong here?

00;31;41;17 - 00;31;44;27

Like what are the big rocks

that are preventing

00;31;45;11 - 00;31;48;09

a lot more value from being created?

00;31;48;09 - 00;31;52;18

I mean, there's a real amalgam of missteps

that are somewhat common.

00;31;52;29 - 00;31;54;13

And it depends on the organization.

00;31;54;13 - 00;31;57;15

But what I mean, in general,

when we talk about hype and overpromising,

00;31;57;15 - 00;32;00;06

we're

also sort of alluding to a solution ism.

00;32;00;06 - 00;32;01;21

So what's your AI strategy?

00;32;01;21 - 00;32;04;21

And there's a famous quote that's like

saying what's your Excel strategy.

00;32;04;26 - 00;32;05;03

Right.

00;32;05;03 - 00;32;09;12

We should be trying to improve

efficiencies and effectiveness and solve

00;32;09;12 - 00;32;10;16

specific problems,

00;32;10;16 - 00;32;15;03

rather than being treating

a particular technology as a holy grail.

00;32;15;03 - 00;32;15;17

Right.

00;32;15;17 - 00;32;21;12

It's as awesome and amazing and

and it has as it is and how and however

00;32;21;14 - 00;32;25;13

big the magnitude of that first

Wow might become.

00;32;25;21 - 00;32;27;19

It's still a tool, right? We live in a

00;32;28;19 - 00;32;30;23

world of humans run by humans.

00;32;30;23 - 00;32;32;22

For humans, that's not changing.

00;32;32;22 - 00;32;36;26

These are still tools that that concept is

00;32;38;05 - 00;32;41;05

compromise a little by the word agent,

I think.

00;32;41;12 - 00;32;44;25

So there's, there's it's

it's it's a strange thing though, right?

00;32;44;25 - 00;32;47;29

I mean, there's lots of very legit

senior techies

00;32;47;29 - 00;32;50;29

who know exactly what they're

talking about who love the word agent.

00;32;51;26 - 00;32;55;06

But I would say they probably tend to

00;32;56;08 - 00;32;58;26

not fully

put their feet in the shoes of a less

00;32;58;26 - 00;33;02;07

technical executive who then takes

the word agent and runs with it.

00;33;02;07 - 00;33;05;07

Right. Which much to the,

00;33;05;25 - 00;33;08;08

much to the pleasure of, of,

00;33;08;08 - 00;33;12;13

you know, the large foundation

model companies. So.

00;33;15;29 - 00;33;18;03

Where where were we?

00;33;18;03 - 00;33;20;23

We're talking about what businesses

get wrong and the major mistake.

00;33;20;23 - 00;33;21;12

Yeah. Okay.

00;33;21;12 - 00;33;21;28

So yeah.

00;33;21;28 - 00;33;24;09

So the so solution ism. Right.

00;33;24;09 - 00;33;25;29

We got to solve problems. Right.

00;33;25;29 - 00;33;26;25

We got to improve

00;33;26;25 - 00;33;30;13

various concrete specific operations

or introduce a new operation that serves

00;33;30;15 - 00;33;32;08

a specific need.

00;33;32;08 - 00;33;33;24

And that's feasible.

00;33;33;24 - 00;33;37;10

And that doesn't just sort of

get people euphoric

00;33;37;10 - 00;33;38;22

because there's an incredible demo

00;33;38;22 - 00;33;42;16

that that appears to work extremely

well on some handful of examples.

00;33;43;19 - 00;33;44;22

That's not meaningless, but

00;33;44;22 - 00;33;47;22

it's also not necessarily meaningful

in terms of value.

00;33;48;24 - 00;33;51;08

So, you know,

00;33;51;08 - 00;33;54;26

I mean, the, the euphoria with hype

00;33;55;03 - 00;33;58;23

that goes with any kind of tech hype,

there's a euphoria, right?

00;33;58;23 - 00;34;02;03

I mean, and there's a fear

that's the flip side of the same thing.

00;34;02;03 - 00;34;02;11

Right.

00;34;02;11 - 00;34;05;26

And the the antidote to euphoria or, or

00;34;06;17 - 00;34;09;13

existential fear is sobriety.

00;34;09;13 - 00;34;12;11

But sobriety is not boring, man.

00;34;12;11 - 00;34;15;08

It's not boring.

We have to learn how to geek out.

00;34;15;08 - 00;34;18;19

I gave a commencement speech

to my alma mater, Brandeis.

00;34;19;13 - 00;34;21;28

In one of the grad schools there,

00;34;21;28 - 00;34;24;03

like, a dozen years ago.

00;34;24;03 - 00;34;26;19

And that was my theme. I'm like, geek out.

00;34;26;19 - 00;34;27;27

Like, get it, get into it.

00;34;27;27 - 00;34;31;08

My dad said that he,

got through medical school

00;34;31;13 - 00;34;34;29

by way of getting really excited

and interested in the material

00;34;35;00 - 00;34;36;25

because it was grueling

how much he had to learn.

00;34;38;00 - 00;34;39;11

There's an analogy there, right?

00;34;39;11 - 00;34;42;11

There's something really, really cool

about learning from data

00;34;42;23 - 00;34;46;02

in a way that does

well on previously unseen cases.

00;34;46;02 - 00;34;46;08

Right?

00;34;46;08 - 00;34;51;12

You've drawn generalizations automatically

that hold in general, like that's

00;34;51;12 - 00;34;55;03

just like a tech accomplishment

that first Wow can get you pretty far.

00;34;55;12 - 00;34;59;25

And maybe that can be enough to fill

that human need

00;34;59;25 - 00;35;03;14

we apparently have for getting sort

of really excited and euphoric about it

00;35;03;19 - 00;35;07;06

without also sort of feeding, unrealistic

00;35;07;11 - 00;35;10;23

or sort of sci fi tropes

and all this kind of stuff.

00;35;10;23 - 00;35;14;22

So I'd say that that, broadly

speaking, is the error

00;35;14;22 - 00;35;17;22

that the world is making right now.

00;35;18;26 - 00;35;23;02

You you used a model there

that I want to talk about briefly,

00;35;23;02 - 00;35;26;14

because you talked about having

like a senior tech

00;35;26;14 - 00;35;29;14

architect or, you know, a tech guy or gal

00;35;29;14 - 00;35;33;19

then talking to a much less tech

savvy executive or business leader.

00;35;33;19 - 00;35;35;04

And so I want to use this sort of,

00;35;35;04 - 00;35;38;03

you know, dichotomy of technology

leader and business leader.

00;35;38;03 - 00;35;38;11

Yeah.

00;35;38;11 - 00;35;42;12

Because when you say geek out, I'm curious

if you're saying that to the tech

00;35;42;12 - 00;35;45;24

leader, to the business leader,

both with the same message,

00;35;46;01 - 00;35;49;02

both with different messages, or,

you know,

00;35;49;08 - 00;35;52;19

should we actually, like,

just explode that entire dichotomy?

00;35;53;03 - 00;35;56;05

My favorite pop scientist

is Neil deGrasse Tyson.

00;35;57;24 - 00;36;01;08

Because he speaks with such passion

about stuff

00;36;01;08 - 00;36;04;08

that I'm definitely not an expert

in, and I geek out on it.

00;36;04;11 - 00;36;07;29

And I and, you know, the way I wrote

both my books, Predictive Analytics

00;36;07;29 - 00;36;12;21

and the AI playbook,

which are both focused on predictive AI,

00;36;13;25 - 00;36;16;22

is to make it accessible

and sort of that pop science mode,

00;36;16;22 - 00;36;20;24

anecdotally driven right in my uncle

should be able to understand it.

00;36;22;23 - 00;36;24;04

And everyone should be excited about it.

00;36;24;04 - 00;36;27;11

I mean, if people get excited

about astronomy and,

00;36;28;13 - 00;36;32;05

you know, about certain kinds of pop math

and other kinds of science,

00;36;32;10 - 00;36;35;07

they should be at least as excited

about what I think is the coolest thing

00;36;35;07 - 00;36;39;17

in technology or any Stem, which is

machine learning at the core of it.

00;36;40;07 - 00;36;41;27

There's no reason not to get excited.

00;36;41;27 - 00;36;44;18

And in fact,

there is a reason to get excited.

00;36;44;18 - 00;36;46;05

I think that that sort of speaks

00;36;46;05 - 00;36;49;25

to the main reason predictive AI projects

so often fail to deploy,

00;36;50;07 - 00;36;54;02

because there is this semi technical

understanding that the business side,

00;36;54;08 - 00;36;57;10

the people in charge of the operation,

is meant to be improved by that

00;36;57;11 - 00;37;01;07

model's predictions, need to get to

that semi technical understanding.

00;37;01;07 - 00;37;03;14

It's not the rocket science,

it's accessible.

00;37;03;14 - 00;37;04;29

It is accessible.

00;37;04;29 - 00;37;07;07

It's extremely interesting.

00;37;07;07 - 00;37;11;02

And you only need semi technical

for any field to geek out on it.

00;37;12;03 - 00;37;12;29

So that,

00;37;12;29 - 00;37;15;28

you know, maybe geeking out

is the ultimate antidote to hype.

00;37;18;06 - 00;37;20;27

The that phrase semi technical is one

00;37;20;27 - 00;37;24;03

I've heard you use before,

and that you're advocating for.

00;37;24;03 - 00;37;24;26

And I'm curious.

00;37;24;26 - 00;37;29;09

I'd like to hear you defend that a little

bit more in terms of the right level.

00;37;29;09 - 00;37;32;24

And I'll make just briefly

the counterargument which is is semi

00;37;32;24 - 00;37;37;13

technical, like just un technical enough

to be dangerous, you know, and I say that

00;37;37;23 - 00;37;41;24

having a background where I've worked

with devs, I've worked with designers and

00;37;41;24 - 00;37;47;11

I sometimes worry like it, can it create

a false confidence that like you know

00;37;47;11 - 00;37;52;06

more about something that you than you do

that leads you down a wrong path?

00;37;52;06 - 00;37;55;05

Or am I just like

not defining it in your proper terms?

00;37;55;16 - 00;37;59;09

No, I say that's a misconception that

often kind of exacerbates the problem.

00;37;59;09 - 00;38;02;11

I don't try to explain this executive.

They're not going to get it anyway.

00;38;02;11 - 00;38;06;20

Therefore, in a way it's

there's the content is censored, but,

00;38;07;16 - 00;38;11;05

understanding and getting that under

that concrete understanding on a semi

00;38;11;05 - 00;38;13;04

technical level

doesn't mean you're issuing

00;38;13;04 - 00;38;16;18

the need for a technical expert

to also be involved.

00;38;16;27 - 00;38;18;28

So think of it as driving, right?

00;38;18;28 - 00;38;21;28

I don't know where the spark plugs are

in my car.

00;38;22;05 - 00;38;25;26

I have open the hood of my car to show

my kids and been like, isn't this amazing?

00;38;25;26 - 00;38;26;29

Look at this engine.

00;38;26;29 - 00;38;28;26

It's got all these little things right.

00;38;29;27 - 00;38;32;15

But I'm

an expert when it comes to driving.

00;38;32;15 - 00;38;37;03

I know momentum, friction, rules of the

road, mutual expectations of drivers.

00;38;37;12 - 00;38;40;11

You know, the way

the car operates, obviously.

00;38;41;03 - 00;38;44;14

So I'm an expert,

and that's the in that way,

00;38;44;14 - 00;38;47;14

that's the only way I can navigate

the streets to navigate the,

00;38;48;08 - 00;38;51;27

a, an enterprise machine learning projects

successfully through to deployment.

00;38;51;27 - 00;38;55;20

Everyone involves needs a basic

that kind of that level

00;38;55;20 - 00;38;58;20

of of semi technical understanding.

00;38;59;18 - 00;39;02;21

And in the case of predictive AI,

it just comes down to three things.

00;39;02;21 - 00;39;05;06

What's predicted

how well and what's done about it.

00;39;05;06 - 00;39;07;25

So what's predicted

and what's done about it.

00;39;07;25 - 00;39;12;18

That pair defines a predictive

I use case or project.

00;39;12;20 - 00;39;15;09

Right.

So who's going to respond market to them.

00;39;15;09 - 00;39;17;08

Which transaction

is most likely to be fraudulent.

00;39;18;12 - 00;39;21;04

Audit or block it.

00;39;21;04 - 00;39;22;28

Where's their oil drill there.

00;39;22;28 - 00;39;24;20

Which Sal is going to run out of battery.

00;39;24;20 - 00;39;27;20

Let's go check the satellite

and replace the battery.

00;39;28;09 - 00;39;29;08

Right.

00;39;29;08 - 00;39;32;08

As you can imagine,

there's a million such pairs.

00;39;33;00 - 00;39;35;12

Maybe not literally,

but there's an extremely long

00;39;35;12 - 00;39;38;00

tail of viable enterprise

uses of machine learning.

00;39;38;00 - 00;39;41;10

But marketing, risk management,

00;39;42;08 - 00;39;45;08

fraud detection,

those are sort of some of the biggest ones

00;39;46;02 - 00;39;49;03

and most common ones,

and that most large organizations do.

00;39;49;03 - 00;39;51;22

All three of those,

00;39;51;22 - 00;39;54;00

the other main marketing, one is churn

modeling

00;39;54;00 - 00;39;57;02

is predicting who's

going to a trait cancel, defect, churn.

00;39;57;02 - 00;40;00;03

Those are all synonyms

in order to target intervention.

00;40;00;03 - 00;40;00;10

Right.

00;40;00;10 - 00;40;03;29

You can't give a discount meant

to retain to 100% your customer base.

00;40;03;29 - 00;40;07;01

The only recourse is predicting putting

odds on those most likely to leave.

00;40;07;01 - 00;40;09;22

Right?

So it's that very kind of practical stuff.

00;40;09;22 - 00;40;11;28

So what's predicted

and what's done about it

00;40;11;28 - 00;40;15;23

that defines the project in terms of what

where you're going to try

00;40;15;23 - 00;40;19;21

to go with deployment

and then how well does it predict?

00;40;20;08 - 00;40;22;24

You know,

people often talk in terms of accuracy.

00;40;22;24 - 00;40;25;01

That's almost always the wrong metric.

00;40;26;07 - 00;40;28;23

That data scientists mostly,

00;40;28;23 - 00;40;32;12

almost universally make the same mistake,

which is they only evaluate how well it

00;40;32;12 - 00;40;33;19

predicts in terms of arcane

00;40;33;19 - 00;40;37;08

technical metrics like precision

recall area under the curve.

00;40;37;27 - 00;40;38;26

These are very common.

00;40;38;26 - 00;40;43;14

All the all the stars stations and,

and and, data scientists know those,

00;40;43;26 - 00;40;46;08

but they need to also translate

model performance

00;40;46;08 - 00;40;50;01

into business metrics

like monetary like money.

00;40;50;01 - 00;40;50;10

Right.

00;40;50;10 - 00;40;52;15

Like how much profit would it make

00;40;52;15 - 00;40;55;18

or how much money would we save

if we use this fraud detection model

00;40;55;22 - 00;40;59;07

to decide which transactions to block

or at least hold,

00;41;00;16 - 00;41;01;09

that kind of thing.

00;41;01;09 - 00;41;04;01

So what's predicted? How well,

what's done about it?

00;41;04;01 - 00;41;05;17

It's not rocket science.

00;41;05;17 - 00;41;08;24

It totally matters

to anybody on the business side running,

00;41;08;24 - 00;41;12;03

who's in charge of these large scale

operations that could be improved.

00;41;12;23 - 00;41;14;29

And it's cool. It's interesting. Right.

00;41;14;29 - 00;41;17;29

There's a lot to geek out on there,

both in terms of

00;41;18;06 - 00;41;21;06

the core technology

or leveraging learning from data

00;41;21;15 - 00;41;24;00

and what it means to say, like,

hey, what's the a?

00;41;24;00 - 00;41;25;20

It's it's it's just arithmetic.

00;41;25;20 - 00;41;27;15

But it's very particular arithmetic,

00;41;27;15 - 00;41;30;16

not rocket science as far as like, well,

what it mean if I have,

00;41;31;00 - 00;41;35;12

a million transactions a week

and I'm going to take the top point,

00;41;35;14 - 00;41;39;18

2% of them and temporarily block them

and have a human audit in,

00;41;39;27 - 00;41;42;15

you know, what would the let's run

the numbers, let's crunch

00;41;42;15 - 00;41;45;07

the numbers in terms of the monetary

advantage of doing that.

00;41;46;17 - 00;41;47;26

It's an exercise very

00;41;47;26 - 00;41;51;06

rarely done in a sort of cohesive way.

00;41;51;23 - 00;41;55;18

And that's why most of those projects

that it's theatricality, right?

00;41;56;01 - 00;41;56;20

The first.

00;41;56;20 - 00;41;59;11

Wow, the coolness

of learning from data to predict

00;41;59;11 - 00;42;02;13

and look how well it predicts

in terms of these technical metrics.

00;42;02;14 - 00;42;03;19

Right.

00;42;03;19 - 00;42;07;13

Has everybody consumes like we fetishize

the core technology of machine learning.

00;42;07;13 - 00;42;09;16

And it's been like that for a long time.

00;42;09;16 - 00;42;10;15

Let's take a step back

00;42;10;15 - 00;42;13;16

and be like, well, that's cool,

but let's not stop geeking out there.

00;42;13;16 - 00;42;17;15

Let's be a little more concrete

and and talking to the very accessible

00;42;17;15 - 00;42;20;15

business terms of

what would it mean to deploy it?

00;42;22;19 - 00;42;24;22

I want to I want to scratch at something

00;42;24;22 - 00;42;26;23

because I think I agree

with you, but there's some

00;42;27;29 - 00;42;29;16

you know, there's some narratives here

00;42;29;16 - 00;42;33;21

that I wanted debunker

that I want to potentially argue about.

00;42;33;21 - 00;42;36;23

So one of them is we've seen that

00;42;36;23 - 00;42;40;12

I mostly generative AI I I'll say

00;42;41;05 - 00;42;45;16

has actually there's an argument

that it's actually going to destroy

00;42;45;16 - 00;42;49;14

a large number of technical jobs

that it started coming for developers,

00;42;49;14 - 00;42;53;02

for technical folks, and it's going

to take more and more of those jobs away.

00;42;53;10 - 00;42;56;04

And I think that there is an underlying

00;42;56;04 - 00;43;00;15

belief there

that with this technology in hand,

00;43;00;25 - 00;43;04;19

semi technical to non-technical

is actually technical enough.

00;43;04;24 - 00;43;08;17

You don't need someone fully technical

because just like you don't need

00;43;08;17 - 00;43;12;04

a human sparkplug, you can be the human

driving the spark plug

00;43;12;05 - 00;43;15;18

that you know, the spark

plug is the tool, the AI is the tool.

00;43;16;05 - 00;43;20;08

And that makes me really, really nervous

because

00;43;20;25 - 00;43;27;04

that the worst uses of generative

AI I've seen in enterprises

00;43;27;14 - 00;43;31;09

is when you have someone with limited

technical ability

00;43;31;13 - 00;43;34;24

saying, oh, I've built my agent

and it can do everything.

00;43;35;07 - 00;43;38;17

They don't have someone acting

as sort of a technical harness,

00;43;38;26 - 00;43;40;02

and you know,

00;43;40;02 - 00;43;43;02

they don't have someone who understands,

you know, to your point, predictive AI,

00;43;43;10 - 00;43;46;17

and they end up doing something

that's like wildly off the mark

00;43;46;23 - 00;43;50;24

or in some way completely tech

technically falls flat on its face,

00;43;51;01 - 00;43;51;23

and there's just

00;43;51;23 - 00;43;55;11

no way for them to know that

until they try and scale the damn thing.

00;43;55;22 - 00;43;57;20

So do you buy that argument?

00;43;57;20 - 00;44;02;13

Like, do you believe that we're going to

see these tech jobs get completely eroded?

00;44;02;19 - 00;44;06;04

Or do you think that, as you said earlier,

that like

00;44;06;04 - 00;44;10;00

semi technical is still not technical

enough to do the full job?

00;44;11;26 - 00;44;12;11

Yeah.

00;44;12;11 - 00;44;16;08

No, I don't think that we're going

to have a job apocalypse in any sector.

00;44;16;24 - 00;44;19;24

I think that that's part

and parcel to the hype.

00;44;19;24 - 00;44;22;19

But what you just said is

00;44;22;19 - 00;44;25;17

to, to try to address your question,

00;44;25;17 - 00;44;27;10

it sort of sounds to me

like you're describing.

00;44;27;10 - 00;44;30;09

Hey, look, there's a problem

because people are are being overly

00;44;30;09 - 00;44;34;07

reliant on, on a language model

and they're like, hey, I can handle this.

00;44;34;07 - 00;44;38;21

And then they, they therefore either

don't hire somebody or they fire somebody.

00;44;38;21 - 00;44;43;25

And in the end, it's a junior cheaper

person who's sort of working with

00;44;43;25 - 00;44;47;12

the language model because the expertise

is taken care of by the model.

00;44;47;22 - 00;44;51;01

And you're sort of that sounds to me

like it's the kind of thing where

00;44;51;01 - 00;44;54;17

then people start to see things go wrong,

and then they backpedal

00;44;54;29 - 00;44;58;28

and it, it, it's

naturally going to avert the job.

00;44;59;15 - 00;45;00;20

Job apocalypse. Right.

00;45;00;20 - 00;45;05;10

Whereas the thing that would cause

the job apocalypse is if the technology

00;45;05;13 - 00;45;09;00

is so darn good that we just don't need

the people anymore.

00;45;09;26 - 00;45;13;10

But doesn't sound like

you think that's going to happen.

00;45;13;11 - 00;45;14;13

Either's. All right.

00;45;15;16 - 00;45;17;18

No, I guess

00;45;17;18 - 00;45;20;28

I think we're of the same mind

that I don't think the technology can do

00;45;20;28 - 00;45;25;14

these things

that maybe some senior leaders boards,

00;45;26;10 - 00;45;29;10

you know, creators of the technology

are saying that it can do.

00;45;29;16 - 00;45;34;08

But we might have to find that out

the hard way by trying to do it

00;45;34;13 - 00;45;38;22

by firing swathes

of the wrong people, by saying, oh, shit.

00;45;38;22 - 00;45;42;26

Like, I'm just I'm afraid that

we're drowning in a tidal wave of stupid

00;45;42;29 - 00;45;44;05

right now.

00;45;44;05 - 00;45;48;02

We're we're learning the hard way

the limitations of this technology.

00;45;48;06 - 00;45;48;27

Yeah, yeah, yeah.

00;45;48;27 - 00;45;52;29

But I don't think we're necessarily

learning in a terribly hard way.

00;45;53;07 - 00;45;57;21

Like there may be okay moment

that the hard learned lessons may burn.

00;45;57;27 - 00;45;58;20

Burn your fingers,

00;45;58;20 - 00;46;01;23

but you're going to pull your hand back

pretty quickly, like it's not going to be.

00;46;02;01 - 00;46;04;29

It's sort of a blip on the radar,

00;46;04;29 - 00;46;07;28

especially in comparison

to what many are still believing,

00;46;07;28 - 00;46;10;28

which is that this thing is going to be

so darn good that we're really,

00;46;11;02 - 00;46;15;13

you know, there's going to be a too

dramatic of an economic shift too quickly.

00;46;15;13 - 00;46;19;23

And then that's going to cause mass

unemployment and all that kind of stuff.

00;46;19;26 - 00;46;20;25

And I don't really

00;46;21;25 - 00;46;22;23

so I don't

00;46;22;23 - 00;46;27;10

I don't tend to caution about, hey,

don't automate too much, don't you know,

00;46;27;15 - 00;46;30;15

or don't hurt anybody's feelings by,

00;46;30;19 - 00;46;33;19

owning up to the pursuit of automation.

00;46;33;24 - 00;46;36;07

I mean, that's why we make machines

just automate, right?

00;46;36;07 - 00;46;38;02

Any kind of machine that to to something.

00;46;38;02 - 00;46;41;02

Otherwise

a person or an animal would need to do.

00;46;41;04 - 00;46;44;18

And I don't, I don't,

I don't walk on eggshells like, oh,

00;46;44;18 - 00;46;47;14

we don't want people to feel like

we're just trying to take over their jobs

00;46;47;14 - 00;46;49;25

because I don't think we can like,

I don't I'm

00;46;49;25 - 00;46;52;25

not worried that things are going to go

terribly in that direction now.

00;46;52;27 - 00;46;55;18

It's not

there are going to be economic shifts.

00;46;55;18 - 00;46;57;12

There's always going to be more

automation.

00;46;57;12 - 00;46;59;08

There's always going to be improved

efficiencies.

00;46;59;08 - 00;47;00;14

But in general.

00;47;00;14 - 00;47;02;28

So my impression is that over the last,

00;47;04;09 - 00;47;05;01

year, we

00;47;05;01 - 00;47;08;01

kind of went through a big press cycle

00;47;08;04 - 00;47;12;15

and big media cycle where it was like job

apocalypse, job apocalypse.

00;47;12;15 - 00;47;17;06

And then pretty recently, all of a sudden,

the press is, super backpedaled

00;47;17;06 - 00;47;21;04

and found the experts who's like,

come on now, or the antidote or,

00;47;21;04 - 00;47;25;14

excuse me, the anecdotes where it's like,

well, we maybe went too far,

00;47;25;14 - 00;47;28;01

and then we had to backpedal,

these enterprises.

00;47;28;01 - 00;47;30;14

So that's sort of good news

as far as I'm concerned,

00;47;30;14 - 00;47;34;21

that that type of hype

has has simmered down greatly.

00;47;34;21 - 00;47;38;04

I'm sure there's still a lot of people

that still kind of buy it.

00;47;38;14 - 00;47;40;11

I think that the ability,

00;47;40;11 - 00;47;44;10

the vibe coding thing is definitely

a killer, is a killer.

00;47;44;10 - 00;47;46;29

App is also a subjective term, right?

00;47;46;29 - 00;47;50;18

So there's a way in

which we're still waiting for

00;47;50;18 - 00;47;53;23

what's when's the big killer

app of large language models.

00;47;53;23 - 00;47;56;28

You know, if it it's certainly an amazing

00;47;56;28 - 00;48;00;07

thought partner and way to sort of

get information

00;48;00;22 - 00;48;02;27

a little tricky

because you don't know when it's wrong

00;48;02;27 - 00;48;05;06

unless you happen to be asking you

about something you already know.

00;48;05;06 - 00;48;06;25

But you then why are you doing that?

00;48;08;13 - 00;48;08;22

But it

00;48;08;22 - 00;48;11;23

is, you know, but in any kind of use

like that, where you're

00;48;11;23 - 00;48;15;22

interacting directly with a chat bot,

you are the human in the loop.

00;48;15;22 - 00;48;20;10

Whereas the the hype is about automation

where there is no human in the loop.

00;48;20;10 - 00;48;20;23

Right.

00;48;20;23 - 00;48;23;21

So that level of killer app, well,

when's it going to be?

00;48;23;21 - 00;48;29;07

But if if you're an engineer and a coder,

depending on the project,

00;48;29;21 - 00;48;33;21

the, automatic coding

is definitely a killer app.

00;48;33;21 - 00;48;36;06

I don't it doesn't get rid of engineers.

00;48;36;06 - 00;48;37;07

It empowers them.

00;48;37;07 - 00;48;41;01

That's the general

even I know that you had.

00;48;41;17 - 00;48;42;11

Who did you have?

00;48;42;11 - 00;48;45;05

You had Laurence Moroney

saying stuff like that.

00;48;45;05 - 00;48;46;23

A lot of people have been saying stuff

like that.

00;48;46;23 - 00;48;50;16

You know, there's certain things

that it really removes the drudgery.

00;48;50;16 - 00;48;51;14

But for the most part,

00;48;51;14 - 00;48;54;26

you still need engineering

and architecture expertise and humans

00;48;54;26 - 00;48;58;04

and understanding the the business problem

being solved and all that kind of stuff.

00;48;58;04 - 00;49;01;11

So even within that realm

where there's something of a killer app,

00;49;01;26 - 00;49;03;09

it's not a job apocalypse.

00;49;06;02 - 00;49;08;19

I, I think that's fair.

00;49;08;19 - 00;49;12;00

And I appreciate the clarification

and I, I'm glad we could revisit

00;49;12;00 - 00;49;16;07

some of the talk around autonomy

and the difference between,

00;49;16;25 - 00;49;20;03

you know, as we said earlier,

partial or 80% autonomy

00;49;20;03 - 00;49;23;29

versus full autonomy and the implications

for, for business models,

00;49;24;12 - 00;49;27;14

business models and for workforces.

00;49;29;07 - 00;49;30;28

I want to come back

to like the implementation

00;49;30;28 - 00;49;33;19

piece here and getting value.

00;49;33;19 - 00;49;36;19

And you talked about,

you know, something that I,

00;49;36;22 - 00;49;40;16

I've had lots of conversations about,

which is starting with some sort

00;49;40;16 - 00;49;44;11

of business problem and business goal

and then figuring out how you can use

00;49;44;11 - 00;49;48;21

AI as a tool versus having,

you know, a dedicated AI strategy.

00;49;49;15 - 00;49;53;08

I want to press on that

in terms of like, then what?

00;49;53;15 - 00;49;55;01

And I'll tell you where I'm

going with this.

00;49;55;01 - 00;49;59;12

So saying, okay, we want to,

you know, create a new product with AI.

00;49;59;12 - 00;50;01;05

We want to cut costs with AI.

00;50;01;05 - 00;50;04;16

Like that's pretty easy

to do in a non-technical way.

00;50;05;01 - 00;50;08;13

One of the boots on the ground

challenges I'm

00;50;08;13 - 00;50;12;11

seeing in a lot of the organizations

I'm speaking with is this sort of,

00;50;14;03 - 00;50;14;15

you know,

00;50;14;15 - 00;50;17;19

this is not glamorous,

but almost like a turf war between it

00;50;18;07 - 00;50;21;07

and, business leadership

when it comes to AI.

00;50;22;17 - 00;50;26;10

And a lot of traditional enterprise

technology leaders saying, hey,

00;50;27;00 - 00;50;29;13

you know, we need to govern this,

we need to control this,

00;50;29;13 - 00;50;31;04

we need to figure out how to use it.

00;50;31;04 - 00;50;33;29

And a lot of business leaders,

maybe they're semi technical,

00;50;33;29 - 00;50;36;11

maybe they're less than that saying,

you know what,

00;50;36;11 - 00;50;38;05

we can kind of do this ourselves. Thanks.

00;50;38;05 - 00;50;41;12

This is too important for you

or we don't need you anymore

00;50;41;12 - 00;50;45;25

because we now have these tools

that are automating your part of the job.

00;50;46;02 - 00;50;49;03

So assuming you've done

that first step, you've

00;50;49;03 - 00;50;52;03

said, okay, here are some of the business

challenges we want.

00;50;52;06 - 00;50;53;28

How do you own this thing?

00;50;53;28 - 00;50;55;05

How do you build accountability.

00;50;55;05 - 00;50;58;12

How do you actually get to that solution

layer

00;50;58;17 - 00;51;01;18

from like, I don't know, like sort of

an operating model perspective?

00;51;01;27 - 00;51;03;27

Well, that's all the time we have today.

00;51;03;27 - 00;51;07;02

Now, I mean, you've had such you've asked

a really hard question, right?

00;51;09;17 - 00;51;11;03

You know,

00;51;11;03 - 00;51;13;07

it really depends on the organization.

00;51;13;07 - 00;51;17;11

And, there's that that's a, there's a,

there's an art to,

00;51;17;12 - 00;51;21;06

to addressing that issue

as organs, as enterprises do evolve.

00;51;21;06 - 00;51;25;11

I wish I had one formula,

but I sort of give a couple examples.

00;51;25;11 - 00;51;29;16

So in the case of predictive AI,

it's often the data scientist

00;51;29;26 - 00;51;32;19

who's familiar

with what a predictive AI project

00;51;32;19 - 00;51;35;21

could be, and therefore what its business

value would be.

00;51;35;27 - 00;51;40;02

At the same time,

the data scientist is is often,

00;51;41;14 - 00;51;44;29

tragically unaware of enough business

00;51;44;29 - 00;51;48;07

constraints and business considerations

to get the thing deployed.

00;51;48;07 - 00;51;51;07

But they're necessary

to get it off the ground

00;51;51;07 - 00;51;55;17

because that data scientist knows, hey,

we could predict

00;51;55;29 - 00;51;59;24

maybe this well, you know, which

transactions are fraudulent or whatever.

00;51;59;24 - 00;52;02;08

The thing is that you're trying to fix.

00;52;02;08 - 00;52;04;08

You know which insurance

claims are fraudulent, whatever.

00;52;04;08 - 00;52;07;22

So we could we have this

we have roughly this amount of data,

00;52;07;22 - 00;52;10;26

this amount of training data

and historical data from which to learn.

00;52;10;26 - 00;52;12;21

And maybe we would end up with a model

00;52;12;21 - 00;52;15;06

this well, and it could

potentially be used that well.

00;52;15;06 - 00;52;18;00

So in that way it's bottom up. Right.

00;52;18;00 - 00;52;23;00

Or it's kind of it's, it's a business

need, but it's sort of being introduced.

00;52;24;03 - 00;52;27;03

From the tech side.

00;52;27;18 - 00;52;30;23

I would say that, you know, ideally it's,

00;52;31;02 - 00;52;34;19

it's the business side

that decides what the business needs are,

00;52;34;19 - 00;52;38;12

but it's just practically speaking, it's

not going to always work that way.

00;52;39;27 - 00;52;41;14

The other thing is that you know

this. How?

00;52;41;14 - 00;52;43;25

Well, what's our AI strategy?

What's our AI budget?

00;52;43;25 - 00;52;50;11

Which, you know, we need to use AI more,

which tends to be solution ism, right.

00;52;50;11 - 00;52;52;01

And sort of to the to to a hammer.

00;52;52;01 - 00;52;52;29

The whole world's a nail.

00;52;52;29 - 00;52;56;29

And so it's going to be come a hammer

party for.

00;52;56;29 - 00;52;59;28

No, you know, for no good reason.

00;52;59;28 - 00;53;04;25

On the other hand, there is value to

the idea of a general purpose tool, right.

00;53;04;25 - 00;53;06;06

Like a computer. Right.

00;53;06;06 - 00;53;09;27

And it's worth, like making sure there are

enough computers in the building.

00;53;09;27 - 00;53;10;13

Right.

00;53;10;13 - 00;53;14;24

So the thing is, is they, on some level,

you have to be aware

00;53;14;24 - 00;53;18;19

of a bunch of representative

potential use cases, right?

00;53;18;19 - 00;53;22;21

Value propositions, problems

that could be solved or efficiencies

00;53;22;21 - 00;53;24;05

that could be gained, whatever it is.

00;53;26;17 - 00;53;29;14

So that's my hand waving, man.

00;53;29;14 - 00;53;33;15

I mean, like, it's going to be different

for every organization.

00;53;33;28 - 00;53;35;29

And there's no

00;53;35;29 - 00;53;39;07

simple, one size fits all solution.

00;53;39;07 - 00;53;41;06

I don't think.

00;53;41;06 - 00;53;43;22

Just, just quick detour

because you said something interesting,

00;53;43;22 - 00;53;46;22

comparing it to computers

and a general purpose tool.

00;53;47;07 - 00;53;51;10

Does the productivity paradox around

I like, does does that startle you?

00;53;51;10 - 00;53;52;25

Does that concern you?

00;53;52;25 - 00;53;57;03

Like, do you think that we're just

measuring the wrong things,

00;53;57;03 - 00;54;00;05

or do you think we're really not getting

a lot of value out of it?

00;54;00;14 - 00;54;02;11

I think we're not really getting

a lot of value of it.

00;54;02;11 - 00;54;05;11

Yeah, and I don't I think that.

00;54;07;03 - 00;54;08;18

Look, this is just a gut feeling.

00;54;08;18 - 00;54;10;29

It'd be very hard

to put real numbers on this,

00;54;10;29 - 00;54;15;14

but I think we might get sort of 5%

of the value that everyone's

00;54;16;20 - 00;54;17;15

making noise

00;54;17;15 - 00;54;20;17

that sort of is being implied

or that is very concretely,

00;54;21;16 - 00;54;24;10

represented by certain valuations.

00;54;24;10 - 00;54;27;10

Now, if it turns out to be five,

that's not

00;54;27;12 - 00;54;30;12

that's going to be disappointing and it's

going to cause some economic pains.

00;54;30;12 - 00;54;33;01

And I don't know how we

I don't know economically,

00;54;33;01 - 00;54;36;23

I don't know how we get there safely if,

if sort of things settle into that.

00;54;38;06 - 00;54;40;18

On the other, on the other side, it's

kind of optimistic.

00;54;40;18 - 00;54;44;08

I mean, 5% of, of these mammoth values

would be amazing.

00;54;46;05 - 00;54;47;09

There's no question that this,

00;54;47;09 - 00;54;50;09

that large language models are valuable.

00;54;50;17 - 00;54;55;05

I just think that that story of autonomy

and artificial general intelligence,

00;54;55;22 - 00;54;57;01

right, is out of,

00;54;57;01 - 00;55;00;25

science fiction playbook,

and it's so tantalizing and it sells so.

00;55;00;25 - 00;55;03;24

Well. We are stuck in,

00;55;05;22 - 00;55;07;28

a hyperbolic

00;55;07;28 - 00;55;10;28

chamber.

00;55;11;23 - 00;55;14;14

Well, and for me,

00;55;14;14 - 00;55;18;01

as I think about this practically,

there's sort of two points of comparison.

00;55;18;01 - 00;55;21;01

There's saying, okay,

if you're at 5% right now,

00;55;21;09 - 00;55;24;19

you know,

can I get to 100% of what I'm hearing?

00;55;24;26 - 00;55;27;21

Or is that just like a stupid way

to frame it?

00;55;27;21 - 00;55;28;22

And like you should, what

00;55;28;22 - 00;55;32;22

you should be thinking about is like,

what percent can I reasonably get to?

00;55;32;23 - 00;55;35;14

What is my best competitor

getting to? What?

00;55;35;14 - 00;55;38;14

Like what is best of breed

actually look like?

00;55;38;14 - 00;55;41;18

Like what is the gap

between where most people are right now

00;55;41;29 - 00;55;44;28

and where they could be

if they just did this

00;55;44;28 - 00;55;47;27

a little bit more thoughtfully?

00;55;48;19 - 00;55;49;01

Yeah.

00;55;49;01 - 00;55;51;04

And the other thing is that it

depends on time frame.

00;55;51;04 - 00;55;55;16

So everything I just said, you know,

I consider a 5 to 7 years from now,

00;55;55;24 - 00;55;58;16

but if we go out more ten, 20 years, there

00;55;58;16 - 00;56;02;01

could be entirely new technologies

that nobody's conceived of.

00;56;04;13 - 00;56;07;13

You know, I always thought machine

learning was so exciting.

00;56;07;20 - 00;56;09;20

Say it because,

00;56;09;20 - 00;56;12;07

the sky's the limit in a certain way.

00;56;12;07 - 00;56;15;07

So long as there's data

from which to learn.

00;56;15;15 - 00;56;18;27

And supervised machine

learning is the kind of machine learning

00;56;18;27 - 00;56;22;25

where you have labeled examples,

they don't necessarily

00;56;22;25 - 00;56;26;04

have to be manually labeled like

this is a picture of a cat as a picture

00;56;26;04 - 00;56;31;00

a dog with with, typical enterprise

machine learning history speaks.

00;56;31;00 - 00;56;34;00

So, you know which customers canceled,

you know, which transactions

00;56;34;00 - 00;56;35;09

turned out to be fraudulent.

00;56;35;09 - 00;56;39;07

There's there's not an additional

there's no bottlenecks.

00;56;39;07 - 00;56;42;07

There's not an additional effort

needed for those labels.

00;56;42;19 - 00;56;46;08

And in the case of large language models,

the labels are what was the next word

00;56;46;08 - 00;56;47;12

written in this document?

00;56;47;12 - 00;56;48;04

You know, we

00;56;48;04 - 00;56;51;25

we wrote these ten and half pages

and then this word was the next one

00;56;51;25 - 00;56;52;28

that come right next.

00;56;52;28 - 00;56;57;13

So let's try to get the neural network

to predict that word right.

00;56;57;13 - 00;57;00;12

Not necessarily will

because there's a random element.

00;57;00;12 - 00;57;03;02

Right. And it's not it's not perfect.

00;57;03;02 - 00;57;04;06

Perfection is not the goal.

00;57;04;06 - 00;57;05;27

But if you sort of veer in that direction

00;57;05;27 - 00;57;07;25

and that's what the training process does.

00;57;07;25 - 00;57;10;18

Now let's take all the words up to

and including that word.

00;57;10;18 - 00;57;12;14

What was the next word after that? Right.

00;57;13;13 - 00;57;15;09

Those are labeled examples.

00;57;15;09 - 00;57;19;19

Again, extremely large number of them

because we didn't have to manually

00;57;19;19 - 00;57;20;06

label them.

00;57;20;06 - 00;57;24;00

Or that is to say, we didn't have to make

any extra effort labeling them.

00;57;24;00 - 00;57;25;05

We it was organic.

00;57;25;05 - 00;57;29;01

We did label

all the humans, wrote all these documents.

00;57;29;01 - 00;57;29;08

Right.

00;57;29;08 - 00;57;32;14

And that turns out to be great amount

of training data.

00;57;33;22 - 00;57;37;18

So at this point,

we've tapped that unprecedented resource,

00;57;37;18 - 00;57;39;07

that incredible amount.

00;57;39;07 - 00;57;43;12

Now there's other things, there's video,

there's other kinds of media and content

00;57;43;12 - 00;57;45;11

where it's like,

well, here's the frame of the video.

00;57;45;11 - 00;57;46;15

And then this was the next frame

00;57;46;15 - 00;57;50;24

that came after it, or this object

moved in this direction in this video.

00;57;50;24 - 00;57;51;00

Right.

00;57;51;00 - 00;57;55;25

So there's there's still a lot more,

but it's not literally limitless.

00;57;57;09 - 00;57;59;05

And so there is

00;57;59;05 - 00;58;02;07

to feel for things to become

00;58;02;27 - 00;58;05;14

to, for things to approach this sort of,

00;58;05;14 - 00;58;09;05

almost deity level

promise of supreme autonomy.

00;58;09;14 - 00;58;12;19

And, you know, a human in a box, right?

00;58;12;19 - 00;58;15;08

An artificial human,

all that kind of stuff.

00;58;15;08 - 00;58;20;24

It's going to need something very

different to then this sort of in machine

00;58;20;24 - 00;58;25;12

learning is the most important general

purpose technology, of the century.

00;58;25;12 - 00;58;27;14

That's what Harvard Business Review said.

00;58;27;14 - 00;58;29;08

And that always resounded with me.

00;58;29;08 - 00;58;32;08

But it depends on the learning examples,

the data

00;58;32;27 - 00;58;35;06

and and it's incredible

00;58;35;06 - 00;58;38;06

what it can do with this

amazing amount of data that we have.

00;58;38;09 - 00;58;39;10

But there's limits.

00;58;39;10 - 00;58;41;03

And what else could we do?

00;58;41;03 - 00;58;44;03

What other technology,

what other approach?

00;58;45;12 - 00;58;49;00

You know, there's replicating

each neuron in the brain, but that is like

00;58;50;10 - 00;58;52;07

thousands of years in the future

or whatever.

00;58;52;07 - 00;58;54;08

I mean, that's

just not on the table right now.

00;58;54;08 - 00;58;57;08

So if for things to reach

00;58;57;19 - 00;59;02;04

the overall level

of promising, of supreme autonomy,

00;59;02;23 - 00;59;06;10

something unprecedented

and unexpected would have to happen.

00;59;08;07 - 00;59;11;23

I think we both agree with that,

that there's like a 20 year gap

00;59;11;23 - 00;59;15;02

between what people are talking about

and what people are doing right now, like,

00;59;15;03 - 00;59;16;12

like order of magnitude.

00;59;16;12 - 00;59;16;27

I'm not,

00;59;16;27 - 00;59;20;11

you know, trying to be mathematical

about it with, like 5% to 100%.

00;59;21;20 - 00;59;24;22

But if we talk about the here

and now and I'm thinking about this

00;59;24;22 - 00;59;26;24

in relationship

to like the AI playbook, right?

00;59;26;24 - 00;59;31;18

This, this, you know, sort of handbook

you've written for people doing this, the,

00;59;31;29 - 00;59;35;24

the difference between I'm

sort of bumbling my way through

00;59;35;24 - 00;59;40;11

trying to get value out of this,

and I'm actually doing this properly.

00;59;40;19 - 00;59;44;07

Like how much can you unlock there?

00;59;44;07 - 00;59;48;21

Like, like how much better can you get

I guess, and like

00;59;48;21 - 00;59;53;01

how much value is still on the table right

now, practically speaking.

00;59;53;19 - 00;59;56;28

And, you know, I guess

how do you best unlock that?

00;59;57;24 - 01;00;01;20

Well, medium to large organizations

have millions of dollars

01;00;01;20 - 01;00;04;20

per quarter per unperformed

01;00;05;08 - 01;00;08;08

or under executed potential project.

01;00;08;21 - 01;00;09;01

Right.

01;00;09;01 - 01;00;12;28

In general, I mean, there's so many ways

that it can improve efficiencies.

01;00;12;28 - 01;00;13;06

I mean,

01;00;14;12 - 01;00;17;14

the way we're treated and served by

or the thing that makes the world

01;00;17;18 - 01;00;20;19

go round is how we're treated

and served by organizations, right?

01;00;20;19 - 01;00;22;20

Who they contact, mail,

01;00;22;20 - 01;00;26;07

test, diagnose, warn, investigate,

incarcerate, set up on a date or medicate.

01;00;26;07 - 01;00;26;14

Right.

01;00;26;14 - 01;00;31;13

And all these ways that we're treated

are gamble little gambles.

01;00;31;13 - 01;00;31;23

Right.

01;00;31;23 - 01;00;35;10

And you can play those gambles

so much better with probability.

01;00;35;10 - 01;00;36;18

We don't have a magic crystal ball.

01;00;36;18 - 01;00;38;16

The next best thing is probability.

01;00;38;16 - 01;00;39;14

Sounds boring.

01;00;39;14 - 01;00;41;19

We try not to use the word probability,

01;00;41;19 - 01;00;43;25

but look, it's

just a number between 0 and 100 of.

01;00;43;25 - 01;00;47;01

What's the likely outcome for this

particular case or this individual.

01;00;47;23 - 01;00;49;18

And that turns out to be really valuable.

01;00;49;18 - 01;00;53;24

If only we could, execute on that concept

and actually get these things deployed

01;00;53;24 - 01;00;56;24

systematically,

make things work more effectively.

01;00;56;25 - 01;00;58;29

It's a win win in general, right?

01;00;58;29 - 01;01;01;23

The you can argue capitalistic

01;01;01;23 - 01;01;04;23

that the organization stands

to improve more.

01;01;04;23 - 01;01;08;29

But in general, if these are efficiencies

that really largely also help

01;01;08;29 - 01;01;12;11

the consumer for a lot of these projects,

you're going to have less fraud.

01;01;12;25 - 01;01;15;14

Health care could be improved,

you can have less spam.

01;01;15;14 - 01;01;18;16

There's a you know,

this efficiency is is a good thing.

01;01;21;21 - 01;01;23;27

And right.

01;01;23;27 - 01;01;26;26

The the thing that so the AI playbook,

01;01;27;03 - 01;01;30;03

my book introduces this,

01;01;30;05 - 01;01;32;24

paradigm

framework playbook call that I call

01;01;32;24 - 01;01;34;29

bismol bismol.

01;01;34;29 - 01;01;37;12

And that's the website for the book

because

01;01;37;12 - 01;01;39;02

it's about to come out in paperback.

01;01;39;02 - 01;01;40;23

You can get free audio books about busy.

01;01;40;23 - 01;01;43;03

MLB.com.

01;01;43;03 - 01;01;45;06

And the six step practice there

01;01;45;06 - 01;01;48;23

basically introduces

three pre-production steps that have

01;01;49;11 - 01;01;54;20

an, an gender's, a, a deep collaboration

between the both the biz and tech,

01;01;54;20 - 01;01;58;09

the stakeholders and the data scientists

or the data professionals.

01;01;58;19 - 01;01;58;28

Right.

01;01;58;28 - 01;02;02;06

And it introduces sort of three

pre-production ones, which have to do

01;02;02;06 - 01;02;06;12

with defining in a very concrete,

practical way what's predicted,

01;02;06;12 - 01;02;09;12

how well and what's done about it

that we were talking about earlier.

01;02;09;14 - 01;02;11;26

Then the other three steps

are the same technical steps

01;02;11;26 - 01;02;14;29

that have always existed since the 60s,

when the stuff was first years,

01;02;14;29 - 01;02;16;24

which is like prepare

the data, learn from it.

01;02;16;24 - 01;02;19;25

That's the that's the machine learning

part of the machine learning project.

01;02;20;00 - 01;02;23;04

And notice that that penultimate steps,

that five out of six is

01;02;23;10 - 01;02;26;14

you don't even get to actual machine

learning of a machine learning project

01;02;26;14 - 01;02;29;21

till step five, because you've done

everything else properly ahead of time.

01;02;29;27 - 01;02;33;24

You're not just doing

awesome, cool technology for its own sake.

01;02;34;04 - 01;02;36;24

It's okay to enjoy the first. Wow.

01;02;36;24 - 01;02;39;01

But unless you're going to plan

to get to the second while it's

01;02;39;01 - 01;02;42;25

just a hobby, and then the last step

is to actual deployment, that's step six.

01;02;44;17 - 01;02;45;23

It's well, it's interesting to me.

01;02;45;23 - 01;02;49;19

And you kind of touched on it that the,

the model you're using is like,

01;02;49;26 - 01;02;53;03

so front loaded with

you need to do all this thinking.

01;02;53;03 - 01;02;55;13

You need to get all this stuff right

01;02;55;13 - 01;02;58;19

before you get into the guts of it

to make it successful.

01;02;58;25 - 01;03;00;03

Is that fair?

01;03;00;03 - 01;03;02;01

For sure. It's just about positioning it.

01;03;02;01 - 01;03;04;03

It's just very practical.

01;03;04;03 - 01;03;08;23

You know, it's it's it's like,

okay, well, we have this fraud team

01;03;09;07 - 01;03;12;18

and, you know, they're looking

for a needle in a haystack.

01;03;12;18 - 01;03;16;20

If we could make the haystack smaller,

what will the numbers tell us?

01;03;17;06 - 01;03;21;28

And if so, well, then should we just give

a smaller haystack to a smaller team?

01;03;22;06 - 01;03;24;12

Well, that's a business decision, right?

01;03;24;12 - 01;03;29;22

So that's that's part of the recognition

of actually making this stuff valuable

01;03;29;22 - 01;03;34;07

and making it, viable to operationalize

is that there's just like these

01;03;34;07 - 01;03;37;26

very practical questions that are not

going to be answered by technology.

01;03;38;02 - 01;03;42;17

So if I learn from the data

to put odds on each transaction,

01;03;43;01 - 01;03;46;20

flagging the ones that are most likely

to be fraudulent, great.

01;03;46;28 - 01;03;51;03

But that process and the evaluation of how

well it was,

01;03;51;03 - 01;03;55;11

the predictive performance,

that's interesting, but it doesn't direct.

01;03;55;17 - 01;03;59;16

It doesn't ordain

exactly how you should use it

01;03;59;22 - 01;04;03;04

or whether you should shrink your fraud

team, or maybe even grow your fraud team.

01;04;03;04 - 01;04;04;21

You can capture even more fraud.

01;04;04;21 - 01;04;08;16

Those numbers

and the trade offs are informed

01;04;08;16 - 01;04;11;21

by the technology

and the performance metrics,

01;04;11;26 - 01;04;15;04

and they're just a matter

of very particular arithmetic.

01;04;15;13 - 01;04;18;13

But they ultimately come down to business

01;04;18;25 - 01;04;21;25

choices that are going to be informed

by the technology.

01;04;22;00 - 01;04;26;17

And, and that's, that's where that's

where you need this tight collaboration

01;04;26;23 - 01;04;29;23

so that you can not only make

a, predictive model,

01;04;29;23 - 01;04;32;23

a machine learning model

that's viable and potentially valuable.

01;04;33;00 - 01;04;37;10

You can make a very nice, transparent

to informed decision

01;04;37;10 - 01;04;40;06

about whether to use

it and exactly how to use it.

01;04;42;16 - 01;04;45;04

You've

got a quote that I found interesting

01;04;45;04 - 01;04;48;22

and I feel like probably comes into play

in, in some way here, which is that,

01;04;50;09 - 01;04;52;21

and I may be paraphrasing, but

01;04;52;21 - 01;04;55;17

this is more of a consulting gig

and less of a technical

01;04;55;17 - 01;04;59;04

install

to actually get value out of this stuff.

01;04;59;04 - 01;05;01;25

Can you can you unpack that a bit?

01;05;01;25 - 01;05;02;04

Yeah.

01;05;02;04 - 01;05;05;09

And that quote was pertain to

to think about it for generative.

01;05;05;09 - 01;05;08;09

But that that quote was meant

for predictive projects because what

01;05;08;10 - 01;05;12;15

they're

they're about the fundamental execution

01;05;12;15 - 01;05;17;11

of your existing largest scale operations,

the things that makes your organization

01;05;17;11 - 01;05;20;27

profitable,

the things that make it provide value.

01;05;22;13 - 01;05;26;25

So it's not like tech, like, hey,

let's just put a new database solution

01;05;26;25 - 01;05;29;06

or a new server

and you just sort of swap it in

01;05;29;06 - 01;05;32;04

and everything still operates

the same. That's a tech install.

01;05;32;04 - 01;05;36;10

No, these projects are all necessarily a

a business consulting gig.

01;05;36;12 - 01;05;40;10

So another way to think of it is they're

referred to as a machine learning project.

01;05;40;18 - 01;05;42;09

But that's kind of a misnomer.

01;05;42;09 - 01;05;45;19

They should be reconceived

as a business, operations

01;05;45;19 - 01;05;48;21

improvement project that happens

to necessarily use machine learning.

01;05;50;11 - 01;05;53;10

I like that, that that makes more sense

to me.

01;05;53;10 - 01;05;56;19

And then, just

just as we sort of move into,

01;05;56;22 - 01;05;58;07

you know, I guess kind of the final phase

01;05;58;07 - 01;06;01;19

of our discussion, coming back to the

the AI playbook, you've,

01;06;02;17 - 01;06;05;17

you've released

a new version of this recently,

01;06;05;24 - 01;06;10;21

and I want to talk about why

and I guess to to frame that up

01;06;10;21 - 01;06;14;18

a little bit differently, like,

what's new, what's different now

01;06;14;23 - 01;06;17;25

when it comes to, having an AI playbook

than a few years ago?

01;06;17;27 - 01;06;20;00

Well, the

I play books about predictive AI,

01;06;20;00 - 01;06;21;27

and fundamentally not

that much has changed.

01;06;21;27 - 01;06;25;13

In fact, the new version is paperback,

which means it's cheaper.

01;06;25;13 - 01;06;28;16

There is one other difference,

which is that we wrote a,

01;06;29;17 - 01;06;31;00

I wrote a new preface.

01;06;31;00 - 01;06;34;09

So it's the paperback edition

with a new preface by the author.

01;06;34;09 - 01;06;37;04

And that preface compares generative

AI to predictive AI.

01;06;37;04 - 01;06;39;18

So the biggest difference

from when the book first came out

01;06;39;18 - 01;06;43;27

less than three years ago is and later,

right within the book, I call it

01;06;43;27 - 01;06;45;28

machine learning

project, a machine learning project.

01;06;47;14 - 01;06;51;16

So let's go back in

time to when I first became an independent

01;06;51;16 - 01;06;55;29

consultant in, 2000 and, three,

and I walked

01;06;55;29 - 01;06;59;11

around the streets of San Francisco

and I was like, hey, you want to hire me?

01;06;59;11 - 01;07;00;23

I'm a machine learning consultant.

01;07;00;23 - 01;07;02;19

And everyone was like, you're a what?

01;07;02;19 - 01;07;05;04

And everyone was calling it, data mining.

01;07;05;04 - 01;07;08;04

And I was like, well,

that's not a great term for it.

01;07;08;17 - 01;07;11;01

So I latched on to predictive analytics,

which was at

01;07;11;01 - 01;07;13;18

the time was just sort of a new term.

01;07;13;18 - 01;07;15;25

It was just starting to be up and coming.

01;07;15;25 - 01;07;18;21

And that's the name of my first book,

the first book, sort of how it works

01;07;18;21 - 01;07;19;19

and how it provides value.

01;07;19;19 - 01;07;21;26

And the second is how to capitalize on it.

01;07;21;26 - 01;07;23;21

But they're both about predictive AI.

01;07;25;18 - 01;07;26;11

So since

01;07;26;11 - 01;07;30;18

then, the term machine learning has

very much become in vogue, and AI has gone

01;07;30;18 - 01;07;33;18

through all sorts of resurgences,

especially recently.

01;07;34;01 - 01;07;37;01

AI is nothing

if not a subjective term, but,

01;07;37;19 - 01;07;40;24

machine learning is a well-defined term,

and so is predictive analytics.

01;07;40;24 - 01;07;41;15

Predictive analytics is

01;07;41;15 - 01;07;44;17

the use is the enterprise use machine

learning for those types of projects?

01;07;44;17 - 01;07;47;20

Essentially,

the idea of learning from data to predict

01;07;47;20 - 01;07;50;22

like that's pretty specific concrete

type of technology.

01;07;51;18 - 01;07;55;28

So now with generative AI,

excuse me, with generative AI,

01;07;55;28 - 01;07;59;14

I'd say that predictive analytics is now

01;07;59;28 - 01;08;03;15

perhaps more commonly

referred to as predictive AI.

01;08;03;26 - 01;08;06;07

But a rose by any other name.

01;08;06;07 - 01;08;09;15

In the time that I wrote this book,

I called the Machine Learning Project.

01;08;09;15 - 01;08;13;08

So now the new preface says, hey,

at the very end, there's little italics.

01;08;13;08 - 01;08;15;11

No, this is hey,

look for the rest of this book.

01;08;15;11 - 01;08;17;07

It says ML project.

01;08;17;07 - 01;08;19;18

But now every time you see that,

just think predictive AI.

01;08;21;12 - 01;08;22;15

So the terms change.

01;08;22;15 - 01;08;26;15

But the idea of learning from data

to predict and using those probabilities

01;08;26;15 - 01;08;29;19

to improve existing large scale

operations, that's never going to change.

01;08;29;24 - 01;08;32;05

We're never going to have a magic crystal

ball.

01;08;32;05 - 01;08;35;22

The essence of the discipline

of those projects is immutable.

01;08;36;00 - 01;08;39;00

It doesn't matter

how complex and sophisticated

01;08;39;09 - 01;08;42;28

the the tech part of the project

will change, right?

01;08;42;28 - 01;08;45;10

The way

in which we leverage large language models

01;08;45;10 - 01;08;48;16

in all sorts of varied ways

to help improve the predictive project

01;08;48;22 - 01;08;51;17

and improve the execution of the project

will change.

01;08;51;17 - 01;08;52;26

But the concepts,

01;08;52;26 - 01;08;56;10

the fact that there's diminishing returns

on how well we can predict,

01;08;56;18 - 01;08;59;14

but that there's great value in predicting

better than guessing,

01;08;59;14 - 01;09;02;27

and how do we make decisions

about how to actually operationalize that,

01;09;03;10 - 01;09;04;19

those fundamentals?

01;09;05;20 - 01;09;08;20

I don't think they're going to change.

01;09;08;24 - 01;09;10;04

I think that's well said.

01;09;10;04 - 01;09;14;27

And it gives it I mean, kind of,

I think well-earned timelessness

01;09;15;12 - 01;09;18;12

about it as well, that

this has been valuable for a long time.

01;09;18;19 - 01;09;21;03

It has a lot of runway ahead of it.

01;09;21;03 - 01;09;24;11

And, you know, untapped

value is I think you said earlier.

01;09;24;21 - 01;09;26;22

Yeah for sure.

01;09;26;22 - 01;09;29;22

So maybe to, to kind of put a bow on this.

01;09;30;16 - 01;09;33;21

You know what,

what's kind of your parting advice

01;09;33;28 - 01;09;38;00

for technology

leaders and business leaders as they,

01;09;38;01 - 01;09;42;00

you know, kind of think about what

they're going to do next to, to create

01;09;42;00 - 01;09;45;20

more value either broadly in the AI space

or maybe specifically

01;09;46;01 - 01;09;48;20

with predictive AI.

01;09;48;20 - 01;09;51;20

Oh, just that value, right.

01;09;51;22 - 01;09;54;22

Get excited about value stories

about value,

01;09;54;25 - 01;09;57;29

not about buzzwords and how many buzzwords

you can jump jam in.

01;09;57;29 - 01;09;59;29

They're not trends.

01;09;59;29 - 01;10;03;29

And I think with that focus,

you'll naturally start to say, hey, look,

01;10;04;22 - 01;10;07;10

probably about half the time, if not more,

01;10;07;10 - 01;10;10;22

that's the lowest

hanging fruit is a predictive AI project,

01;10;11;29 - 01;10;12;16

you know,

01;10;12;16 - 01;10;15;16

get things to work better,

create more value for

01;10;15;27 - 01;10;19;21

for your customers

and for your state, your shareholders.

01;10;19;21 - 01;10;20;25

Right.

01;10;20;25 - 01;10;23;25

Make things work better. And,

01;10;25;05 - 01;10;27;19

that's, that's going to be the antidote

01;10;27;19 - 01;10;31;16

to fetishizing technology, to solution ism

01;10;31;29 - 01;10;34;29

and to science fiction narratives.

01;10;36;28 - 01;10;39;13

I think that is extremely well said.

01;10;39;13 - 01;10;39;23

Eric.

01;10;39;23 - 01;10;42;13

I wanted to say a big

thank you for coming on the program today.

01;10;42;13 - 01;10;43;13

Lots to think about.

01;10;43;13 - 01;10;43;22

Lots.

01;10;43;22 - 01;10;46;21

I'm still digesting

and I really appreciate your insights.

01;10;46;21 - 01;10;49;20

My pleasure. Thank you. Geoff.

01;10;49;27 - 01;10;54;01

Most viewers don't know this, but Digital

Disruption is developed by Infotech

01;10;54;01 - 01;10;58;05

Research Group, a leading advisor

to technology leaders around the world.

01;10;58;13 - 01;11;00;23

If that's not you, you don't need to care.

01;11;00;23 - 01;11;03;01

So skip ahead and enjoy our content.

01;11;03;01 - 01;11;05;23

But if you are a technology

leader, Infotech helps

01;11;05;23 - 01;11;09;21

IT teams get projects done faster, better,

and at a lower cost.

01;11;10;07 - 01;11;14;06

Infotech provides unlimited access

to practical tools and expert guidance.

01;11;14;06 - 01;11;18;11

You need to execute at a fraction

of the cost of traditional consulting,

01;11;18;22 - 01;11;20;03

no matter the project.

01;11;20;03 - 01;11;24;28

From AI strategy to cybersecurity

to vendor negotiation, Infotech has you

01;11;24;28 - 01;11;25;22

covered.

01;11;25;22 - 01;11;28;29

Check it out at the link below,

and don't forget to like and subscribe!