230: The Data-Driven Way to Make Decisions (Parenting, Health, Career): Emily Oster

Speaker: 00:00:22

That's Emily Oster, a Harvard

trained economist who is the expert

2

: 00:00:26

on making data-driven decisions

when there's not always good data.

3

: 00:00:30

And I'm a huge fan of her data-driven

parenting books, and they've helped me

4

: 00:00:34

raise my own kids and parent them in

a way that I feel really comfortable.

5

: 00:00:37

Today, she'll give us the key, the method

to actually making good decisions even

6

: 00:00:42

when there's poor data, there's not

data, or we're in a lot of uncertainty

7

: 00:00:46

and there's a lot of unknowns.

8

: 00:00:47

By the end, you'll have a great framework

for making great decisions like a data

9

: 00:00:51

analyst, even if you're not one already.

10

: 00:00:53

So let's go ahead and get into it

11

: 00:01:00

Emily Oster is the founder and

CEO of ParentData, a professor of

12

: 00:01:04

economics at Brown, and a three-times

New York Times bestselling author.

13

: 00:01:08

Emily, welcome to the Data Career Podcast.

14

: 00:01:10

Thank you for having me.

15

: 00:01:11

Super excited to have you.

16

: 00:01:12

I am a big fan.

17

: 00:01:13

I have the, the books right here.

18

: 00:01:15

If you guys haven't checked out- Amazing

19

: 00:01:16

Emily's books before,

definitely, um, check them out.

20

: 00:01:20

They are amazing.

21

: 00:01:21

Um, but we live in a really

interesting time, Emily.

22

: 00:01:24

Uh, there's, like, so much infor-

misinformation going around, um,

23

: 00:01:28

in parenting and in everything

in politics and finance.

24

: 00:01:31

Um, you know, everyone's trying

to tell you how you should parent

25

: 00:01:34

your kids and what decisions

you should make for your kids.

26

: 00:01:37

Uh, so my question t- for you,

is it possible to make good life

27

: 00:01:41

decisions in a world where we're

constantly bombarded by different

28

: 00:01:45

opinions and different data sets?

29

: 00:01:47

I believe yes, uh, but I think it

requires us to think about the structure

30

: 00:01:54

of our decisions rather than just

ask the question what the data says.

31

: 00:01:59

So a lot of times people will come to

me and they'll be like, "Okay, well,

32

: 00:02:01

just tell me what the data says."

33

: 00:02:03

It's like, that's not always that helpful

a question, and if your approach to

34

: 00:02:08

decision-making is to just, like, see

the last piece of data and, like, make

35

: 00:02:12

a decision based on that, you aren't

necessarily gonna make good decisions,

36

: 00:02:16

and I think part of what makes our current

information environment so challenging

37

: 00:02:20

is that people are constantly getting

bombarded with data, and every time they

38

: 00:02:25

see a new piece of data they're, like,

not necessarily ready to incorporate

39

: 00:02:29

it into their decisions in a smart way.

40

: 00:02:32

So I think the answer is yes, we

need data, and we can make good

41

: 00:02:36

decisions, but we have to have the

decision-making sort of lead the data.

42

: 00:02:40

So I would tell people, like, you

need to wait until you're ready to

43

: 00:02:43

make a decision, and then think about

what your choices are, structure the

44

: 00:02:47

decision, and then you get the data

that you need to make the decision and

45

: 00:02:52

then make the decision based on that.

46

: 00:02:55

But I think it's, it's too hard to Only

use data, I guess if that makes sense.

47

: 00:03:02

Yeah, for sure.

48

: 00:03:03

I think, I think it's also interesting

when we're talking about data to maybe

49

: 00:03:07

specify what we're talking about.

50

: 00:03:08

'Cause, you know, some of the topics that,

that you take on, um, like for instance

51

: 00:03:12

Encryptshe, is, you know, is breast best?

52

: 00:03:14

Like, is it actually good to…

53

: 00:03:16

Is it better to breastfeed your,

your baby, or is bottle feeding okay?

54

: 00:03:20

Um, another, you know, one of the other

things you tackle is vaccinations.

55

: 00:03:23

Do vaccinations, you know, cause autism?

56

: 00:03:27

And I think a lot of people, maybe for

people who are listening to this, they're

57

: 00:03:30

data nerds and, you know, they're able

to, you know, maybe go out there and

58

: 00:03:34

try to find some data on, you know,

autism rates and vaccination rates, and

59

: 00:03:38

maybe, you know, put together some sort

of a statistical analysis to do so.

60

: 00:03:42

But I think a lot of people are getting

their data from, like, Instagram posts.

61

: 00:03:47

Yeah.

62

: 00:03:47

Um, so, like, how do you try to navigate

the, the world where it's, like, a lot,

63

: 00:03:53

where a lot of data is presented to us

in, like, an Instagram post or something

64

: 00:03:57

that's, that's maybe not very structural

and, and hard to interpret in the moment?

65

: 00:04:02

Yeah, I hate data from Instagram posts

because it's always like, "Here's a

66

: 00:04:07

study that shows blah, blah, blah, blah."

67

: 00:04:10

And it's like, well, what, like,

is it the only study of this topic?

68

: 00:04:14

Is it the biggest study of the topic?

69

: 00:04:15

Is it the best study of this topic?

70

: 00:04:17

Is it some random thing from 1987

that you pulled out of, like, the

71

: 00:04:20

journal of, like, made-up results?

72

: 00:04:22

Which is usually the answer.

73

: 00:04:23

And so i- Er, data is it's like I get so

frustrated because I think we, we really

74

: 00:04:31

need to prioritize the best data, but part

of what is very challenging for people

75

: 00:04:34

is it can be hard to know what that is.

76

: 00:04:36

And there is a fair amount of

training that goes into the

77

: 00:04:40

question of like, is this good data?

78

: 00:04:42

Is this less, less good data?

79

: 00:04:45

Um, so I guess I would say it

is never a good idea to make a

80

: 00:04:49

decision about something based

on a single Instagram post.

81

: 00:04:53

If you are in a position to need to

know whether some relationship is

82

: 00:04:57

true, are vaccines causing autism,

for example, you need to step way back

83

: 00:05:01

out of Instagram or out of TikTok or

whatever it is and figure out what are

84

: 00:05:07

some sources you can go to that are

gonna give you a better, more nuanced,

85

: 00:05:11

more thoughtful answer to that question.

86

: 00:05:13

And there are a few things people can

look for in, you know, what makes a

87

: 00:05:16

good data set, things like, is it big?

88

: 00:05:19

Is it likely to be randomized?

89

: 00:05:20

You know, things like that, and

that's, that's kind of the core.

90

: 00:05:23

But, you know, a single study

says and somebody puts it in a

91

: 00:05:27

carousel on Instagram, that's a

crappy way to learn about data.

92

: 00:05:31

That- that's…

93

: 00:05:32

I mean, that's unfortunate.

94

: 00:05:33

I wish we could always just like

trust what we, what we see online.

95

: 00:05:36

Um, but obviously we, we can't.

96

: 00:05:39

Cannot.

97

: 00:05:39

That's one of the things that I think,

um, you do a really good job in, in

98

: 00:05:42

Crib Sheet especially of like, you

know, we're, we're debating, we're

99

: 00:05:46

debating something like, uh, should

we like co-sleep with our babies or

100

: 00:05:50

should we sleep train our babies?

101

: 00:05:51

Um, you know, you pull up like

all these different studies

102

: 00:05:55

that have been done on that.

103

: 00:05:56

And some of the studies like that, maybe

let's just say for example, that say,

104

: 00:06:00

oh, you know, you know, co-sleeping is

like really good for your baby actually.

105

: 00:06:04

Um, you might throw…

106

: 00:06:06

Correct me if I'm wrong, but like

you might like throw that study out

107

: 00:06:09

the window and kind of ignore the

results because maybe it's not a

108

: 00:06:13

large sample size or maybe it's not

a diverse sample size or, or maybe

109

: 00:06:17

like the actual experiment was wrong.

110

: 00:06:20

So even though the results say something,

like you're not necessarily one to

111

: 00:06:23

just trust, uh, the results kind of

randomly from a, uh, an experiment.

112

: 00:06:27

Is that kind of correct?

113

: 00:06:28

Is that kind of your way of thinking?

114

: 00:06:30

Yeah, definitely.

115

: 00:06:30

I think that a lot of what distinguishes

the way that I approach sort of large

116

: 00:06:34

corpuses of data from the way that you

would and it sort of, um, that other

117

: 00:06:39

people would perhaps, is that I am much

more willing to say, okay, let me find the

118

: 00:06:42

best studies here and base our conclusions

on the best studies rather than just like

119

: 00:06:46

every study should get their, their voice.

120

: 00:06:48

Like some studies don't deserve a voice.

121

: 00:06:50

Um, and I think that is especially

true when we're outside of

122

: 00:06:54

the randomization space.

123

: 00:06:55

So in the parenting, like health, et

cetera space, there is a huge amount of

124

: 00:07:00

what we're told that is like we're just

comparing people who do one thing to

125

: 00:07:03

people who do another thing, and those

people are really different on like a

126

: 00:07:07

billion dimensions, and we're attributing

it to the one topic that we're studying.

127

: 00:07:12

And that kind of evidence I will

almost always say like, just forget

128

: 00:07:17

it Like just put it in the trash.

129

: 00:07:19

And that is actually a place where I

really differ from a lot of even, you

130

: 00:07:24

know, people who I think have a lot of

training and are in, you know, serious

131

: 00:07:28

like professors because there are people

who will tell you, "Well, okay, but

132

: 00:07:32

we have so many studies of something.

133

: 00:07:34

Like there are so many observations."

134

: 00:07:37

But if it's not causal, it doesn't

matter how many observations you have.

135

: 00:07:42

And so I'm, I'm very interested in how

we can have good data, really excellent

136

: 00:07:47

data that lets us make causal statements.

137

: 00:07:49

And if we have a study that isn't

gonna let us make causal statements

138

: 00:07:53

about something we wanna make causal

statements about, I just think

139

: 00:07:57

we should throw it in the trash.

140

: 00:07:58

That's all.

141

: 00:07:59

A- a- and, and I think that's important

for people to realize because, um, I

142

: 00:08:03

think there's some group out, out there

who don't really take any scientific

143

: 00:08:06

studies and, like, white papers.

144

: 00:08:09

Like, they don't ever look at that.

145

: 00:08:10

Like, they're only in the

Instagram world, you know?

146

: 00:08:12

Uh, maybe, maybe looking

at aggregations of things.

147

: 00:08:15

And there's some people who, who

maybe are a lot more, like, prone to,

148

: 00:08:18

like, "Oh, I really trust science,

um, a- and studies," but it's always

149

: 00:08:22

important to look into those studies.

150

: 00:08:23

Um, I'm curious, like, there's not

always a study for, for everything.

151

: 00:08:28

Right.

152

: 00:08:28

Um, so, like, what do we do when

we need to make an im- an important

153

: 00:08:33

decision, we wanna be data-driven

in our approach, but, like, there

154

: 00:08:37

isn't good data or there's no data?

155

: 00:08:39

What do we do then?

156

: 00:08:41

Yeah.

157

: 00:08:41

So I think the f- first, that's

very hard, and we have to first…

158

: 00:08:45

I think first it's like there's a radical

acceptance of just saying, like, "Hey,

159

: 00:08:48

I'm gonna have to make a choice here.

160

: 00:08:50

There is no option to, like,

wait until the data is better.

161

: 00:08:53

There just, I have to move

forward with one thing."

162

: 00:08:56

And if we paralyze ourselves w- with

the view that, like, we can't make

163

: 00:09:01

decisions until there are better data,

like, the decision will be made for

164

: 00:09:03

you in some direction by you waiting.

165

: 00:09:06

So just to recognize, like, sometimes

you'll have to make decisions

166

: 00:09:08

under uncertainty, and that is

unfortunate, but it is the way it is.

167

: 00:09:12

I think the second thing I would

tell people is in almost all of those

168

: 00:09:15

settings, it's not important, right?

169

: 00:09:19

So if something were really…

170

: 00:09:21

It's not uniformly true, but if

something is really important,

171

: 00:09:24

and I talk a lot about parenting,

but, like, really important in

172

: 00:09:27

parenting, like really, really

matters, you will see it in the data.

173

: 00:09:30

Like, the things that we know really

matter, like poverty, whether your

174

: 00:09:35

kid has a stable place to sleep,

whether they have enough to eat, those

175

: 00:09:38

things really show up in the data.

176

: 00:09:40

The correlations are really, really big.

177

: 00:09:42

We have good causal evidence.

178

: 00:09:44

The kinds of questions where people say,

"Oh, I wish I had better data on this.

179

: 00:09:47

You know, is it better to enroll

my kid in travel soccer or in,

180

: 00:09:51

you know, travel lacrosse?"

181

: 00:09:52

Or where, like, there's no data

on that, but you know what?

182

: 00:09:54

It's not important.

183

: 00:09:55

And so I think just, like, dialing

down and asking ourselves, "How

184

: 00:09:59

likely is it that this thing matters?

185

: 00:10:01

Given everything else about my family,

like, is this likely to be an important

186

: 00:10:05

decision for my kids' outcomes?"

187

: 00:10:07

And most of the time you're gonna say no.

188

: 00:10:09

And then that actually makes the

decision-making much easier because

189

: 00:10:12

it allows you to focus on all the

things that do matter for good

190

: 00:10:15

decision-making, like how will

this logistically affect my family?

191

: 00:10:19

How much does it cost?

192

: 00:10:20

Whatever are the things that really

should go into that decision.

193

: 00:10:22

So I think just reminding yourself you

gotta make decisions when it's uncertain,

194

: 00:10:26

and a lot of things are not important.

195

: 00:10:27

And it's okay to say, like,

"This probably isn't important.

196

: 00:10:30

Maybe it matters a little bit in

one direction or another, but on

197

: 00:10:32

the whole, it's not the thing that's

gonna break or break my kid," which

198

: 00:10:36

almost there is nothing like that.

199

: 00:10:38

I think that's such an interesting

approach, and, uh, I'm just thinking, you

200

: 00:10:42

know, about the people who are listening.

201

: 00:10:43

Uh, like so many of us are data

engineers, data analysts, data

202

: 00:10:46

scientists, and it's like our whole life

is, you know, helping businesses make

203

: 00:10:50

better decisions with, with the data.

204

: 00:10:52

Um, and so I think it's hard for me, like

as a data nerd, to be like, oh, you know,

205

: 00:10:57

sometimes the, the data doesn't matter.

206

: 00:10:59

Like, or, or even maybe, maybe the

choice, um, doesn't really matter.

207

: 00:11:03

Um, but, but we're- I, I would…

208

: 00:11:04

So let me tell you, I think this actually,

there's such a strong, uh, like data

209

: 00:11:08

analyst parallel here that I would make

for people, which is like, you know, the

210

: 00:11:12

difference between like great success

and not great success in your business

211

: 00:11:17

is like did you launch the right product?

212

: 00:11:20

You know, did you…

213

: 00:11:20

Like there's some big strategic

decision that is happening, you know,

214

: 00:11:23

usually above the heads of everyone

and like where somebody at the top

215

: 00:11:26

is making a big strategic play in one

direction or another, and that's gonna

216

: 00:11:30

determine like whether the business

is successful or not successful.

217

: 00:11:34

The job of the data analyst, and I

do this like for my own business, is

218

: 00:11:39

to be like in the weeds and be like

can I get 1% more if I like send the

219

: 00:11:45

email in this way or this other way?

220

: 00:11:46

Or like can I optimize

the pricing in this way?

221

: 00:11:48

Like let me do an AB test on

this, that, and the other thing.

222

: 00:11:51

And like those things are really important

for your business, but they're not

223

: 00:11:55

important like the big strategic question.

224

: 00:11:57

In the background, they're kind

of optimizations on the margin.

225

: 00:12:00

These kind of choices that we sometimes

get obsessed with with our kid, they're

226

: 00:12:03

optimizations on the margin, and that

margin for parenting is really small,

227

: 00:12:07

and there's so much noise And so

thinking about like, if only I could

228

: 00:12:11

optimize this tiny thing, it's like,

well actually that's not important.

229

: 00:12:15

Like, that's one tiny AB test in

one place and like if you get it

230

: 00:12:19

right, you, you don't get it right in

parenting, it's, it's a lotta noise.

231

: 00:12:22

So I don't know.

232

: 00:12:23

That's, that's often how I

think about the parenting piece.

233

: 00:12:26

I mean, you bring up a good point because

it's like if, should we launch product

234

: 00:12:29

A or product B, and let's say we choose,

you know, product A, but actually product

235

: 00:12:33

B was the right choice, but there was

just not data to make that decision.

236

: 00:12:36

You know, what we're optimizing, let's

just say like a subject line on an email,

237

: 00:12:40

and you know, you AB test it and you

find, oh, we get a lot more conversions

238

: 00:12:43

with, you know, this, this subject line.

239

: 00:12:45

It's like well, the bigger decision

was really we shoulda gone product B.

240

: 00:12:48

Like, the, the amount of money

you can make on the- Totally

241

: 00:12:51

you know, having the right subject

line really is probably dwarfed by

242

: 00:12:53

actually launching the right product.

243

: 00:12:55

Which that's, that's an

important thing to realize.

244

: 00:12:57

I think it's important for people

in, in, you know, in their p-

245

: 00:13:00

careers to realize that as well.

246

: 00:13:01

That it's like we're, we can analyze

data, um, but we're, we're always trying

247

: 00:13:07

to do so for business purposes and,

you know, m- move the business forward.

248

: 00:13:12

So it, whether that's for our, our kids…

249

: 00:13:15

And, and even like you said, like

travel lacrosse versus travel soccer.

250

: 00:13:18

Let's say that there was data on that.

251

: 00:13:20

It's like, well who's to say that

your kid is like the average-

252

: 00:13:22

Totally … you know, it, it- Soccer

kid … doesn't take into account.

253

: 00:13:25

Yeah.

254

: 00:13:25

Yeah.

255

: 00:13:26

Yeah.

256

: 00:13:26

It's like- No, it-

257

: 00:13:27

you don't have data on your kid.

258

: 00:13:28

Totally.

259

: 00:13:28

And I think in our, in our parenting

decisions, of course like everything

260

: 00:13:31

in our family is so much more

complicated than like the subject line

261

: 00:13:34

of the email that you're optimizing.

262

: 00:13:36

And so every one of these decisions has

some other decisions associated with it

263

: 00:13:40

which may actually be more important.

264

: 00:13:41

So we can get very like laser

focused with our kids on

265

: 00:13:45

thinking, you know, well let me…

266

: 00:13:47

What is the right activity or choice or

school to like optimize, you know, some

267

: 00:13:52

outcome metric that we have attached

to our kid, without stepping back and

268

: 00:13:56

saying, you know, really what we're

trying to optimize is like that our kid

269

: 00:13:59

is a, you know, happy, productive adult

who likes us and comes home for meals.

270

: 00:14:04

And like that's actually a much broader

optimization than like, you know,

271

: 00:14:07

are we gonna Achieve Junior Olympic

status or whatever, which you won't.

272

: 00:14:13

One of the things I, I really like, uh,

along these lines and in the, in the book

273

: 00:14:16

is, you know, you mentioned that parenting

you're gonna have a million decisions,

274

: 00:14:20

and there's no way to guarantee that

you're gonna make the right decision.

275

: 00:14:22

In fact, you're probably gonna make

the wrong decision quite often.

276

: 00:14:26

Um, and instead of optimizing for making

correct decisions, you talk about,

277

: 00:14:30

um, making decisions through, like a

framework and, and having education

278

: 00:14:35

around the decision that you are making.

279

: 00:14:37

Can you walk wa- about the difference

between, like, making right decisions

280

: 00:14:41

and making the decision the right way?

281

: 00:14:42

Yeah, absolutely.

282

: 00:14:43

So, think the most important distinction

is one of those things you can

283

: 00:14:46

do, and the other one you cannot.

284

: 00:14:48

So you can never guarantee that

you will make the right decision.

285

: 00:14:51

It's just, like, not,

that's not available to us.

286

: 00:14:54

But we can say ex ante that we approach

the decision the right way, and so I

287

: 00:14:58

talk about, you know, being structured

in how we make our choices, and starting

288

: 00:15:03

by really outlining what our choices are.

289

: 00:15:06

So being clear on the

choice that you're making.

290

: 00:15:09

People often will ask me, you know,

"Well, should I do this or not?"

291

: 00:15:13

You know, "Should I send my

kid to this school or not?"

292

: 00:15:16

It's like, well, or not is not an

available schooling option, so, like, you

293

: 00:15:19

better tell me what's on the other side of

that, because you're never gonna be able

294

: 00:15:23

to compare something to, like, the vast

array of other things on the planet Earth.

295

: 00:15:27

So you gotta think about

what your choice is.

296

: 00:15:29

You've gotta collect the

information that you need.

297

: 00:15:32

You need to then actually force yourself

to make a decision, and I think that's

298

: 00:15:35

the hardest part, because in a world in

which we want to make the right decision,

299

: 00:15:40

we can get, like, paralyzed by the

realization that we cannot guarantee that.

300

: 00:15:44

And so in order to work ourselves

past that, we often have to really

301

: 00:15:48

put in place, like, okay, I am

gonna sit down and make a decision.

302

: 00:15:52

Like, I, this is the date.

303

: 00:15:53

I'm putting it in my calendar.

304

: 00:15:55

I'm scheduling it.

305

: 00:15:55

Like, this is the time we're gonna

make whatever is this choice.

306

: 00:15:58

Because otherwise you can just,

like, spiral and spiral and spiral

307

: 00:16:02

forever and never actually make any

choices, and then usually the world

308

: 00:16:06

will choose for you in some, in

some way if you don't do anything.

309

: 00:16:10

Um, and I, I think the, the value of

having this kind of structure to a

310

: 00:16:14

decision process is that on the other

end of it, you can be confident, again,

311

: 00:16:20

not that you made the right choice, but

that you made the choice the right way.

312

: 00:16:24

I think that's quite

protective for people.

313

: 00:16:27

I will say one other thing,

which is I tell people, after

314

: 00:16:31

you have made this choice, you

should make a plan to revisit it.

315

: 00:16:35

You know, there are some choices

we can never revisit, you know?

316

: 00:16:39

W- should I or should I

not have a second child?

317

: 00:16:41

We- well, once you choose that,

you've pretty much, that's,

318

: 00:16:44

you're pretty much committed.

319

: 00:16:44

We're not revisiting that.

320

: 00:16:46

But many of the choices we

make you can revisit, right?

321

: 00:16:48

Like, I sent my kid to this school,

but I can choose another school.

322

: 00:16:51

I sent them to this activity, but I

could choose another activity, like,

323

: 00:16:54

later, or we could not do this activity.

324

: 00:16:56

And I think we owe it to ourselves to

plan this kind of revisiting of the

325

: 00:17:00

choices that we make, because if we

don't, then we will never revisit them.

326

: 00:17:03

And if we plan to revisit them,

we're much more likely to do so.

327

: 00:17:07

So much good to unpack there.

328

: 00:17:08

Um, one, one thing I wanna zero in on

is, like, you, you mentioned if you're

329

: 00:17:13

deciding between A and B, and you're

like, "I don't have enough data to do

330

: 00:17:16

that," um, sometimes we don't make the

choice, and that itself is a choice, is

331

: 00:17:21

what it sounded like you were saying.

332

: 00:17:22

And oftentimes it's w- the worst

choice of the three, it seems like.

333

: 00:17:25

Like, it's even worse than deciding wrong.

334

: 00:17:28

Yeah, I think it, it sort of like, it,

it, it robs you of the opportunity to

335

: 00:17:32

improve the choices you have, right?

336

: 00:17:34

So this is like, decisions are

particularly hard if neither

337

: 00:17:36

choice is really something we want.

338

: 00:17:38

If we're sort of like, "Well, both of

these choices are kind of poor," we're,

339

: 00:17:42

we're really reluctant to make them

because there's something very aversive

340

: 00:17:44

about choosing something you don't want.

341

: 00:17:46

But if you don't on purpose choose,

you will end up in one of the, in one

342

: 00:17:52

of the s- the branches, typically.

343

: 00:17:55

And if you haven't chosen it,

you won't have had an opportunity

344

: 00:18:00

to make it as good as possible.

345

: 00:18:01

Like, among the bad options, how can

I make this option the least bad?

346

: 00:18:05

And that's, uh, that's missed out

on if you just decide to ignore

347

: 00:18:09

the problem and forget about it.

348

: 00:18:11

Uh, that's a good lesson for me to learn,

because I am the king of doing that,

349

: 00:18:14

and, uh, that stings a lot of the time.

350

: 00:18:16

So, uh- … I'm gonna try

to do better with that.

351

: 00:18:19

Um, I'm curious, like, so I'm used

to data, quote, unquote, as, like, a

352

: 00:18:25

table in Excel or, like, in, you know,

in a SQL database type of a thing.

353

: 00:18:29

Um, but like you said, we don't always

have data to, to make decisions.

354

: 00:18:35

One of the things that you talk

about in your other book, The

355

: 00:18:37

Family Firm, is, like, other

ways that you could potentially

356

: 00:18:40

get data that aren't necessarily

from, uh, like a, a spreadsheet.

357

: 00:18:45

And, you know, maybe they're not as

high integrity as, as spreadsheet data.

358

: 00:18:48

Um, but they're still

valuable in making decisions.

359

: 00:18:51

Um, so, like, one of the things you

mention about is, you know, information

360

: 00:18:54

about your personal circumstances.

361

: 00:18:56

And, and you mentioned earlier,

like, you know, if you're making

362

: 00:18:59

decision A or B, like, which one

are you actually going to like more?

363

: 00:19:02

Even, even if one's more optimal

than the other, like, which

364

: 00:19:04

one are you going to like?

365

: 00:19:05

And the other thing that you mentioned

about is talking to, to others.

366

: 00:19:08

I just am curious to hear your thoughts

on, like, when you're getting data

367

: 00:19:11

that isn't necessarily, like, tabular

data, really high quality data, um,

368

: 00:19:17

it's, it's more, like, around you

data, what, what are you looking for?

369

: 00:19:20

What signals are you looking for, and

how can you know if you can trust it?

370

: 00:19:24

Yeah, so I think really here I'm talking

about getting i- e- information, I think.

371

: 00:19:29

Look, data is just pieces of information.

372

: 00:19:31

So I'm really telling you, like get

some data on how you feel about stuff.

373

: 00:19:34

And I think for many people who are

kind of like us, actually maybe you

374

: 00:19:38

wanna put that in a spreadsheet.

375

: 00:19:39

Like I'm not averse to the idea that

like you should collect data on your

376

: 00:19:43

preferences and logistics and constraints

in the same way that you would imagine

377

: 00:19:47

collecting data on, you know, k- test

score outcomes or, or whatever it is.

378

: 00:19:52

But I think a big piece of this

is, is kind of th- interrogating,

379

: 00:19:57

like if I make this choice, what,

what are the actual implications?

380

: 00:20:03

You know, both how much am

I gonna like my day-to-day?

381

: 00:20:05

Really think about it.

382

: 00:20:06

Like if I get up and I face, you know

… Le- let me put a concrete example in it.

383

: 00:20:10

So a lot of people talk, talk to me

about like choosing between preschools.

384

: 00:20:14

Like I have this preschool and

it's right close to my house, but

385

: 00:20:17

it, you know, isn't very fancy.

386

: 00:20:20

N- only half the teachers have master's

degrees or whatever, or there's this

387

: 00:20:23

preschool that's like 40 minutes away

and it's like super fancy, right?

388

: 00:20:27

It's like how do I think about that?

389

: 00:20:28

Okay, and so one piece of that

data is, you know, how much do

390

: 00:20:30

we know about differences in

preschool outcomes or whatever.

391

: 00:20:33

And another piece of that data is

like how do I feel about commuting

392

: 00:20:36

40 minutes each way with my kid?

393

: 00:20:38

You know, what's that gonna do to my day?

394

: 00:20:40

Like what's my day … That's

a piece of data.

395

: 00:20:42

Like what is my day gonna look like?

396

: 00:20:43

How are we gonna manage this logistically?

397

: 00:20:45

How am I gonna feel about, am I gonna

fight with my partner about this?

398

: 00:20:48

Like are we gonna argue every day

about who is gonna take their kids?

399

: 00:20:52

That's a piece of data.

400

: 00:20:53

You know, so there's a bunch of

stuff in there that you want to put

401

: 00:20:56

in your decision making, even if it

isn't numbers in a spreadsheet, but

402

: 00:21:01

it is information in a document.

403

: 00:21:04

Uh, and another piece of that is

asking other people, you know,

404

: 00:21:06

what do they think about it?

405

: 00:21:07

How did they experience this?

406

: 00:21:08

But like-

407

: 00:21:11

It's…

408

: 00:21:12

For people who love data and evidence,

there's an aversion to the idea of, like,

409

: 00:21:18

going with your gut, and people will talk

about this as like those are two choices.

410

: 00:21:22

You could go with the data,

or you could go with your gut.

411

: 00:21:24

My view is like the data's not bossy.

412

: 00:21:27

It's not gonna tell you what to do.

413

: 00:21:28

It's only an input to decision-making,

which also needs your preferences.

414

: 00:21:32

People say, "I'm going with my gut."

415

: 00:21:34

What they really mean is,

"This is the thing I want."

416

: 00:21:36

Okay.

417

: 00:21:36

But you can actually bring those things

together, and that's gonna be better

418

: 00:21:39

than your data or your gut alone.

419

: 00:21:43

I'm, I'm laughing because I'm literally…

420

: 00:21:46

Like, we're putting, uh, our,

our daughter in preschool.

421

: 00:21:48

Literally went through that exact…

422

: 00:21:49

Do we go to the s- the preschool we think

is, like, better, quote unquote, you know,

423

: 00:21:54

but it's further away, or do we just kinda

do one that, that's close by and near?

424

: 00:21:57

So, uh, I'm laughing 'cause I went

through that, that exact analysis- Yeah

425

: 00:22:00

recently.

426

: 00:22:01

I will tell you, we went to the, we…

427

: 00:22:02

The better one that's further away.

428

: 00:22:03

It's not 40 minutes away.

429

: 00:22:04

If it was 40 minutes away, I don't

think I would be able to do that.

430

: 00:22:06

But now that we're in the process, and,

like, my wife is the one who takes her

431

: 00:22:10

the majority of the time, it's like,

"Oh, we don't really like this commute.”

432

: 00:22:13

So, you know, next year or next

kid or next semester, I don't know

433

: 00:22:17

if we're gonna do this or not.

434

: 00:22:18

Um, so sometimes you make a decision

the best you can, and you make- Yeah

435

: 00:22:22

… maybe a wrong decision, but it's just

another data point where it's like, "Oh,

436

: 00:22:25

actually, I don't like being in the car

for-" Yeah "… X amount of minutes a

437

: 00:22:27

day" We learned this- And we're done

438

: 00:22:29

totally, and then I think that's

why you need the, the other step

439

: 00:22:31

because if you don't plan to revisit

that choice, then you, you're not

440

: 00:22:36

gonna do it 'cause you're not gonna

wanna admit that you made a mistake.

441

: 00:22:39

And it's like it's not

that you made a mistake.

442

: 00:22:40

You did a trial, and, like, maybe

that trial turns out like, well,

443

: 00:22:44

we learned something that we wanna

do something differently next time.

444

: 00:22:49

A- and I think it's important that not

only we do this with, with preschool, but

445

: 00:22:52

when I worked for ExxonMobil, I was a data

scientist there, and one of my jobs was

446

: 00:22:56

to make machine learning algorithms to

predict how much gasoline we should buy

447

: 00:22:59

in each, you know, one of the thousands of

different stores we have across America.

448

: 00:23:03

And I, I built, you know, a machine

learning algorithm that, that was

449

: 00:23:06

the most accurate we could make it,

um, to predict, you know, gasoline.

450

: 00:23:10

But it wasn't like, hey, the, the

gasoline prediction that my machine

451

: 00:23:14

learning model puts out is what we order.

452

: 00:23:17

That, that number goes to, uh, a trader,

a buyer of gasoline, and they can

453

: 00:23:21

totally ignore my number, and they can,

you know, use my number as, as aid.

454

: 00:23:25

So it's so interesting that, like, you

know, these preschool decisions are

455

: 00:23:29

kind of the same framework that, you

know, multi-million, billion, you know,

456

: 00:23:33

in- industry decisions are making.

457

: 00:23:34

I'm sure you've seen that kind

of with, with your research and,

458

: 00:23:36

and, and your analysis as well.

459

: 00:23:38

Totally.

460

: 00:23:38

And I think we under even…

461

: 00:23:40

It's, it's sort of interesting always for

me to watch people who are so good at this

462

: 00:23:43

approach to their job, who are so, like…

463

: 00:23:46

And then I'm like, "Well, can't you…"

464

: 00:23:47

Like, you should just be

porting that into your life.

465

: 00:23:50

Like, it's the same thing.

466

: 00:23:51

Like whe- you know, when you are running,

when you are married to someone and you

467

: 00:23:55

have, or you like have a partner and

you have kids together and you have two

468

: 00:23:59

jobs and whatever, like you're running

a small to medium sized enterprise.

469

: 00:24:02

And there actually are like quite a

lot of tools from how companies do

470

: 00:24:06

that, that I think will make this

easier for, for people, even though

471

: 00:24:10

it's like a really weird thing to say.

472

: 00:24:11

And when you tell people that they're

like, you know, they're like, "Well,

473

: 00:24:14

do you boss your husband around?"

474

: 00:24:15

It's like, "Of course I boss him around,"

but that's not because of the, it's

475

: 00:24:18

not because of this enterprise idea.

476

: 00:24:20

That's awesome.

477

: 00:24:21

And I love that.

478

: 00:24:22

And that's, you know, one of the, the

things you talk about in, in The Family

479

: 00:24:24

Firm is like, we should, we should really

kind of organize our family, uh, like

480

: 00:24:29

we r- organize teams and, and businesses

in a fun way, not like in a boring way.

481

: 00:24:33

Yeah.

482

: 00:24:33

But like- Fun way … we

should have objectives.

483

: 00:24:34

We should have goals.

484

: 00:24:35

Like, we should, you know, make decisions

that are for the optimal happiness

485

: 00:24:40

and health o- of our family, which

I think I've, I've been reading it.

486

: 00:24:43

I've really been, um, enjoying that.

487

: 00:24:46

Um, I'm curious, so you know,

we talked about like making

488

: 00:24:49

data-driven decisions as, as parents.

489

: 00:24:51

I'm curious, like where else in our

personal lives, um, that we can make

490

: 00:24:55

data-driven decisions and kind of

adopt the approach that you've taken

491

: 00:24:59

in, you know, in breastfeeding and in,

you know, sh- what kids, where, where

492

: 00:25:03

should we send our kids to school?

493

: 00:25:04

What other places in our life can we

live data-drivenly other than parenting?

494

: 00:25:09

Yeah, I mean, I think the other obvious

one that people like is health, um,

495

: 00:25:12

is sort of like there are a lot of

health decisions that you have to make.

496

: 00:25:15

Uh, and there's a lot of data on those.

497

: 00:25:18

Uh, and I think it has many of the

same issues that parenting has.

498

: 00:25:22

You know, there's some of the

data is better than others.

499

: 00:25:26

Um, and You know, we, you have

to make decisions that take

500

: 00:25:30

into account your constraints.

501

: 00:25:32

Um, and I actually think this is

a place for me where we are…

502

: 00:25:36

A mu- much of the discourse

misses the idea of constraints.

503

: 00:25:39

We're like really good in the health

space about talking about like how

504

: 00:25:42

to optimize, like, you know, let's

track every da, da, da, da, da.

505

: 00:25:45

Like, you know, how do you like get

all of your n- numbers to be exactly

506

: 00:25:49

optimal in these various ways?

507

: 00:25:50

But without helping people sort of see

like, okay, well, you probably don't

508

: 00:25:55

have 17 hours a day to like fully

optimize a 27-step life protocol.

509

: 00:26:03

Uh, and so how do we incorporate the

data with the constraints and ask,

510

: 00:26:08

you know, what are, what are the

most effective things, uh, to do?

511

: 00:26:12

And I think the other place is just how,

in like sort of general, like how do we

512

: 00:26:15

choose our jobs and how do we, you know,

operate our like professional lives?

513

: 00:26:20

Um, but I think health is the, health

is the other obvious space for me.

514

: 00:26:25

Health is a really interesting one,

um, because obviously, like, it-

515

: 00:26:29

it's … If you don't have health,

you have nothing in your life, right?

516

: 00:26:31

Right.

517

: 00:26:31

'Cause, like, I think we all have

known someone that's, that's lost

518

: 00:26:34

health, and we just see, you know,

how much of a detriment to life that

519

: 00:26:37

is and how, how difficult things are.

520

: 00:26:39

So that makes sense to optimize it.

521

: 00:26:40

Um, just, like, a, a concrete example

of that is, and I know you, you talk

522

: 00:26:45

about, um, this in, in your books.

523

: 00:26:47

Um, but I got diagnosed with ADHD last

year, and I had a, I had a decision.

524

: 00:26:52

It's like, do I try, you know, after

trying, you know, six months of

525

: 00:26:56

non-medication ways to, to, to solve the

problem, like, do I try medication or not?

526

: 00:27:00

Yeah.

527

: 00:27:00

Um, and you know, one of the things

that I do is, like, I pretty much

528

: 00:27:03

constantly wear, uh, an Apple Watch.

529

: 00:27:06

And so I've been taking, you know, ADHD

medicine for, for almost nine months now.

530

: 00:27:10

And one thing I've seen is my

resting heart rate has, has risen,

531

: 00:27:13

like, five beats per minute.

532

: 00:27:14

Yeah.

533

: 00:27:15

And it's like, okay.

534

: 00:27:16

Um, that's, like, a side

effect of taking ADHD medicine.

535

: 00:27:18

It's like, do I, do I like that?

536

: 00:27:21

Do … Is, is the health risk that with

my heart worth the effects of m- you

537

: 00:27:26

know, maybe me getting more work done

or maybe me being a more patient parent?

538

: 00:27:29

Um, those types of things.

539

: 00:27:30

And it's hard because it's like there's

not really all that data out there

540

: 00:27:34

that can support … Or, or maybe

there is a bunch of data out there,

541

: 00:27:37

but it's like which one do I trust, and

how do I apply it to my personal life

542

: 00:27:40

and my business and my family life?

543

: 00:27:42

Um, so I guess making decisions

under uncertainty with

544

: 00:27:45

health definitely makes- Yeah

545

: 00:27:47

makes a lot of sense.

546

: 00:27:47

Yeah, and un- under uncertainty, and

they're really also under constraints.

547

: 00:27:51

Like, you're, like, really what

you're describing is a constraint,

548

: 00:27:53

which is like you, like, if you do

this one thing, it has this effect.

549

: 00:27:58

But y- like, you're … It's trade-off,

uh, and we're not that good at trade-offs.

550

: 00:28:02

And a lot of the health messaging

in particular sort of doesn't like

551

: 00:28:05

to acknowledge the existence of

trade-offs, so they're just like,

552

: 00:28:08

"Do all 4,000 of these things."

553

: 00:28:10

And it's like, okay, but I, I, I can't do

that, or it's, like, literally impossible

554

: 00:28:14

to do two of these things at the same

time, and so now I have to pick one.

555

: 00:28:18

And, you know, people ask me,

like, "Should I sleep or exercise?"

556

: 00:28:21

Like, I only have 30 minutes.

557

: 00:28:22

I have to pick sleep or exercise.

558

: 00:28:24

Like, which thing is better?

559

: 00:28:25

And, uh, that's, that's a hard question.

560

: 00:28:28

What's the answer?

561

: 00:28:29

Hmm, kind of, kind of depends how much

sleep you're getting, but probably sleep.

562

: 00:28:34

Yeah.

563

: 00:28:34

I, and ob- obviously there's so

many factors that, you know, that

564

: 00:28:37

go into your life, and it's hard

to, to make a blanket statement.

565

: 00:28:40

It's funny that you're, you're

mentioning this, 'cause my other job

566

: 00:28:42

at ExxonMobil, my other problem that

I solved at ExxonMobil, uh, and I

567

: 00:28:46

didn't solve this problem on my own.

568

: 00:28:47

We worked as a really big team to do this.

569

: 00:28:49

But we, we made mathematical models

of the entire refinery, um, which

570

: 00:28:54

was, like, 140,000 equations.

571

: 00:28:57

Um, and we were trying to optimize, you

know, how much money the refinery can

572

: 00:29:01

make with all these different constraints.

573

: 00:29:03

Like, we can't…

574

: 00:29:04

We need to make sure that our, our

pollution's abov- uh, below this level.

575

: 00:29:07

You know, our, our tower

can only take this many-

576

: 00:29:09

Yeah … barrels of crude every day.

577

: 00:29:11

And that was a really

hard problem to solve.

578

: 00:29:12

Yeah.

579

: 00:29:13

Um, and we knew, we knew

all the, the math behind it.

580

: 00:29:17

It's like you can't really do that in

your life, 'cause, like, you can't really

581

: 00:29:20

model life outcomes, I don't think.

582

: 00:29:22

Nope.

583

: 00:29:22

Um, maybe you can.

584

: 00:29:23

I don't know.

585

: 00:29:23

No, and you have…

586

: 00:29:24

And, and of course, then when people try

to, they come up with, like, crazy things.

587

: 00:29:28

Somebody sent me a paper the other

day which, in which these people,

588

: 00:29:32

like, tried to assign a number of

minutes of life to, like, each food.

589

: 00:29:38

So, like, if you have one

Diet Coke, it costs you, like,

590

: 00:29:41

this many minutes of life.

591

: 00:29:42

But it's like, that's a cra- like,

first of all, that's bananas.

592

: 00:29:45

Like, you definitely can't do that.

593

: 00:29:47

It's all of the data is from correlation.

594

: 00:29:49

It's not causal, whatever.

595

: 00:29:50

But it also just, like,

didn't make any sense.

596

: 00:29:51

It was like a Diet Coke costs you 12

minutes, but, like, a peanut butter

597

: 00:29:55

sandwich gains you, like, 33 minutes.

598

: 00:29:56

And it's like, okay, well, if

I eat them together, can I get

599

: 00:29:59

fif- like how does this work?

600

: 00:30:01

But it was so, like, so much in the

space of people just want an answer.

601

: 00:30:05

They wanna know, like,

okay, how much is…

602

: 00:30:07

Like, what's the cost of this Diet Coke?

603

: 00:30:09

And the answer is, like- We don't, you

know, we don't have it, probably zero.

604

: 00:30:13

Uh, or have it with a peanut butter

sandwich, and then I get to negative 17.

605

: 00:30:18

That's awesome.

606

: 00:30:18

I love that.

607

: 00:30:19

That's, that's very cool.

608

: 00:30:20

Um, okay.

609

: 00:30:21

I saw something really cool that you,

um, posted on your Twitter recently

610

: 00:30:25

and your Substack, and we'll make

sure to have a link to your social

611

: 00:30:27

in the description down below.

612

: 00:30:29

Um, but it was a really cool analysis

you've done recently on New York education

613

: 00:30:33

data and kind of like their testing data.

614

: 00:30:35

Um, and one of the things you

actually published that caught

615

: 00:30:38

my eye was a Claude artifact.

616

: 00:30:40

Yeah.

617

: 00:30:40

So for those who are unfamiliar

with Claude, it's basically

618

: 00:30:42

like ChatGPT, but it's from a

different company called Anthropic.

619

: 00:30:45

Um, I really like it for doing things

like data analysis, and it creates

620

: 00:30:48

these things called artifacts, which are

basically, you can think as like a, a, a

621

: 00:30:52

published something, a URL that goes to

some sort of a page that has text on it.

622

: 00:30:56

And in your case, you were analyzing

data, so it had text and graphs and

623

: 00:30:59

different analyses and like that.

624

: 00:31:01

Um, and I wanna talk about the

New York- Yeah … education

625

: 00:31:03

study, uh, study that you did.

626

: 00:31:04

But first, I want to kind of walk me

through your, like, data pipeline.

627

: 00:31:08

Like, how, how did this, like, come to be?

628

: 00:31:11

Like, where are you getting your data?

629

: 00:31:13

How are you analyzing it?

630

: 00:31:15

How are you publishing it?

631

: 00:31:16

I was really curious about that, if

you don't mind sharing maybe, like,

632

: 00:31:18

a high, high, uh, view of that.

633

: 00:31:20

Sure, yeah.

634

: 00:31:21

So in, in that case, actually, the

key to that entire analysis is one of

635

: 00:31:26

the projects I d- I do is something

called the Education Data Center,

636

: 00:31:29

uh, which is a, a project where we,

uh, try to clean and organize all

637

: 00:31:34

the state-level test score data.

638

: 00:31:36

So if your kids are in, you know,

public school in grades three through

639

: 00:31:38

eight in the US, they will take,

uh, math and ELA tests every year.

640

: 00:31:41

That's like an important

part of accountability.

641

: 00:31:44

Uh, but the state's data

is, like, a hot mess.

642

: 00:31:47

Like, every state is issuing

it in a different thing.

643

: 00:31:49

If you wanna have the data from Montana,

it's 3,000 separate spreadsheets.

644

: 00:31:52

Like blah, blah, blah, blah.

645

: 00:31:54

And so one of, one thing I really

care about data transparency.

646

: 00:31:58

Uh, and so in this project we,

like, download all of this stuff,

647

: 00:32:02

and we have, so we have like a, a

website where you can sort of get

648

: 00:32:05

all the microdata for, for this.

649

: 00:32:07

And I mention that because that's

a sort of core, like, backend

650

: 00:32:10

pipeline for a project like this.

651

: 00:32:13

On that particular project, there's

a, this thing that happened in New

652

: 00:32:16

York with the test scores, which we

can talk more about, but where, like,

653

: 00:32:19

basically somebody called me, some

reporter called, and they were like,

654

: 00:32:22

"Here are the, you know, here are the

test scores that are gonna come out.

655

: 00:32:24

Like, what do you think?"

656

: 00:32:25

And I looked at them, and I

was just like, "They're wrong.

657

: 00:32:28

Like, I don't like, I don't know

what to tell you, but, like,

658

: 00:32:30

data doesn't look like that.

659

: 00:32:31

Like, somebody made a

mistake probably last year."

660

: 00:32:34

And then I got really, like- exercise.

661

: 00:32:39

I, like, I really love the, the

piece of data where you try to,

662

: 00:32:43

like, learn what's going on.

663

: 00:32:46

Just like, I just wanted

to know what's going on.

664

: 00:32:48

Like, I couldn't, like, let it go.

665

: 00:32:49

And so then my data pipeline is, you know,

in the, in the back end, I'm basically

666

: 00:32:52

using Claude with an API pulling down

this, this raw data and kind of writing

667

: 00:32:58

code in Python to, like, figure out,

try to figure out what's going on.

668

: 00:33:01

And this is a place where the, these AI

tools and sort of Claude in particular

669

: 00:33:06

has really changed how quickly I could do

something like this because, you know, on

670

: 00:33:10

the back end, I could have written this

code in Stata on my own and so on, but I

671

: 00:33:16

probably would not have had the bandwidth

to do it without a kind of LLM tool.

672

: 00:33:23

Uh.

673

: 00:33:23

It was awesome, first off.

674

: 00:33:25

Uh, it was super cool.

675

: 00:33:26

It was so fun.

676

: 00:33:26

It was like, it was so cool.

677

: 00:33:28

Yeah, so basically what, what I'm

hearing is like you, you obviously

678

: 00:33:31

know how to do this analysis.

679

: 00:33:33

You could obviously do it from scratch.

680

: 00:33:35

And I love the idea of from scratch.

681

: 00:33:36

It's like none of us are actually

doing this by hand and paper.

682

: 00:33:38

Right.

683

: 00:33:38

Like, that's probably from scratch.

684

: 00:33:39

No.

685

: 00:33:39

So it's like, oh, like-

My dad was an economist.

686

: 00:33:41

He used to like, I think, do this by…

687

: 00:33:42

They would like multiply the

matrices, but that was a while ago.

688

: 00:33:45

See, but that, that's kind of

my point, is it's like, oh, and

689

: 00:33:48

then R came out, and then Python,

or and then, then Stata- Right

690

: 00:33:50

or whatever.

691

: 00:33:50

Uh, you know, and it's like now we just

have Claude and ChatGPT, which I just see

692

: 00:33:54

as like a new tool, like you said, that

enables this- Yeah … type of analysis.

693

: 00:33:57

Um, and it's like I don't think we

could've taken a random Joe off the

694

: 00:34:00

street and, you know, had them create

this analysis that you created.

695

: 00:34:03

I don't think this analysis, the

AI could've created on its own.

696

: 00:34:06

Um, so it's cool to get like

a little bit of glimpse on, on

697

: 00:34:08

how you're using AI to do that.

698

: 00:34:09

Um, so that, that's very cool.

699

: 00:34:11

Um, I do wanna get into like the,

the details of, of, of what happened

700

: 00:34:15

in this, in this, I wanna call

it a study, but it's not a study.

701

: 00:34:17

These test results.

702

: 00:34:18

So basically, um, if I'm understanding,

uh, correctly, the test results…

703

: 00:34:22

L- l- let's make it as simple as possible.

704

: 00:34:24

The test results were around a certain

level, and then the next year they jumped

705

: 00:34:27

up like 10%, from like 43 to like 52%.

706

: 00:34:31

And maybe we'll pop up the, the

graph on the screen that your

707

: 00:34:33

Claude created to, to show people.

708

: 00:34:35

Um, and then they've fallen back down

to normal levels- Yeah … this year.

709

: 00:34:39

Yeah.

710

: 00:34:39

And so what you're arguing, if I'm not

mistaken, is basically something happened

711

: 00:34:43

in that middle year where it's like,

no, we didn't see improvements of 9%.

712

: 00:34:47

Like, something weird happened.

713

: 00:34:48

Like, there was some error in the testing

or some error in the analysis- Yeah

714

: 00:34:52

where it's like…

715

: 00:34:53

And this is important because it

looks like the state's doing a great

716

: 00:34:56

job, we're really improving, when

in reality it didn't improve at

717

: 00:34:59

all, and that's, that's really big

implications on like funding and money.

718

: 00:35:03

Is that correct?

719

: 00:35:04

Yeah, absolutely.

720

: 00:35:05

I think the, so, so two

things I would add to that.

721

: 00:35:07

So one is it, it really can't

be that they s- the scores went

722

: 00:35:12

up this much and down this much.

723

: 00:35:13

Like, this is a place where we

have so much data on how much

724

: 00:35:17

test scores like this vary.

725

: 00:35:19

We know so much about

just what is going on.

726

: 00:35:22

And, and the numbers here would imply

that like the tip, the, the sort of

727

: 00:35:25

across the entire state in basically every

school across every demographic group,

728

: 00:35:29

like fourth graders in one year learned

two-thirds of a year more, and then in the

729

: 00:35:33

next year they lost all of that and more.

730

: 00:35:36

Like, it's just like this

isn't, you know, it's, it…

731

: 00:35:38

No.

732

: 00:35:39

This is not right.

733

: 00:35:39

And I think that's a piece where

probably the person part of this is

734

: 00:35:44

really like I, like I have so much

experience with this, I can just look

735

: 00:35:48

at that and be like, "That's wrong."

736

: 00:35:49

And now I can go into, you know, some

LLM and be like, "Okay, I'm sure this,

737

: 00:35:52

like I'm pretty sure this is wrong.

738

: 00:35:53

Like, let's try to, try to understand it."

739

: 00:35:55

Um, and okay, so that's the first piece.

740

: 00:35:59

And then, yeah, the question is what,

uh, like what, what happened behind this?

741

: 00:36:04

And it really does matter because as

you say, you know, funding decisions

742

: 00:36:08

are made on these, on these numbers.

743

: 00:36:09

And for example, New York allocated,

you know, some three-year funding grants

744

: 00:36:14

of $250,000 a year across schools-

based on these flawed test scores,

745

: 00:36:20

which like basically made more middle

schools get this and some elementary

746

: 00:36:25

schools not get these like large grants.

747

: 00:36:26

There's like a lot of money behind,

millions and millions of dollars

748

: 00:36:30

behind these scores, and some

of them are, in my view, wrong.

749

: 00:36:35

That's, uh, crazy, and thank you

for, um, you know, organizing this

750

: 00:36:39

and trying to suss out these things.

751

: 00:36:41

That's, that's really important, I think.

752

: 00:36:43

Yeah.

753

: 00:36:43

I w- I mean, look, it was…

754

: 00:36:44

It- I think it is important, but it was

also very interesting and fun because

755

: 00:36:47

it required really, like, getting

into, you know, like, well, what…

756

: 00:36:51

Like, what did you do wrong?

757

: 00:36:52

Like, what exactly?

758

: 00:36:53

And that's, and that's where I think

the, the ability to move quickly with

759

: 00:36:57

these LLMs and the ability to have

an LLM read, like, you know, like,

760

: 00:37:00

600 pages of technical documentation

and be like, "Okay, you know, here,

761

: 00:37:04

like, let's kind of problem solve.

762

: 00:37:05

Like, where could possibly

this have, have fallen apart?"

763

: 00:37:08

And I got much further than I think

I would have been able to alone.

764

: 00:37:13

I still need…

765

: 00:37:14

I, I'm not done.

766

: 00:37:15

I mean, I'm, I'm done, but I'm…

767

: 00:37:17

We're not done.

768

: 00:37:18

But I think that somebody is

gonna figure out what actually ha-

769

: 00:37:20

happened, and hopefully fix it.

770

: 00:37:23

Super cool.

771

: 00:37:23

I hope.

772

: 00:37:23

We'll, we'll include a link to

your, um, Substack, 'cause I know

773

: 00:37:26

you have, like, a whole Substack

dedicated to the education stuff- Yes

774

: 00:37:29

um, and that analysis as well.

775

: 00:37:30

And I actually wanna talk about the

education data, 'cause one of the things

776

: 00:37:33

I do is I run a, I run a data boot

camp, um, where I try to help people

777

: 00:37:36

learn how to become data analysts.

778

: 00:37:37

And one of the projects we do, the

second project that we do, 'cause I'm

779

: 00:37:40

really, like, hands-on, project-based,

is we actually analyze, uh, the data

780

: 00:37:44

from Massachusetts and the- Nice

781

: 00:37:45

the, the results that they get from that.

782

: 00:37:47

And so, one, I'm familiar with how messy

and how many Excel spreadsheets and how-

783

: 00:37:51

Yeah … how hard it is to, like, join

that data and- Massachusetts is actually

784

: 00:37:53

very good relative to the average state.

785

: 00:37:54

It's not bad.

786

: 00:37:55

One of the reasons I chose it,

'cause it's the second project.

787

: 00:37:57

We don't wanna get too hard

and, like, actually joining

788

: 00:37:59

and bajillion different things.

789

: 00:38:00

Um, but first off, I was, like, super

stoked to see, like, oh, lookit, this

790

: 00:38:04

is, like, a really viable project that

people are doing similar things in real

791

: 00:38:07

life- Yeah … 'cause we try to create

a dashboard based off of, like, what's

792

: 00:38:09

happening in the schools and who's

doing well and who's not doing well.

793

: 00:38:12

So I was stoked to see that, that.

794

: 00:38:14

And the second thing, like, no, no

pressure obviously, but one of the

795

: 00:38:17

things we do is we have a team of data

analysts on our, uh, you know, on our

796

: 00:38:20

program who are always happy to analyze

data, especially for good causes.

797

: 00:38:23

So if there's ever, you know, another

school that's cheating or doing

798

: 00:38:26

something wrong and you don't have

the bandwidth, we're happy to, to,

799

: 00:38:29

to, to do an internship project

and, and- Oh, that'd be awesome

800

: 00:38:32

analyze that data and try to give

you- All right … our results.

801

: 00:38:34

That would be very fun.

802

: 00:38:34

Yeah.

803

: 00:38:35

I think we- Yeah … we are, in that

project, we are so focused on the,

804

: 00:38:39

like, just getting the data together,

and I think that sometimes we…

805

: 00:38:42

Like, there isn't bandwidth to do the,

like, okay, can we really understand

806

: 00:38:46

why these things changed in the way

they, they did, other than mistakes.

807

: 00:38:49

Okay.

808

: 00:38:49

Well, I'm, I'm serious.

809

: 00:38:50

Maybe we'll talk offline

on, on how to do that.

810

: 00:38:52

Yeah.

811

: 00:38:52

'Cause I have so many people

that would voluntarily do some

812

: 00:38:55

pretty interesting analysis.

813

: 00:38:56

Um, okay, that was awesome.

814

: 00:38:58

Okay.

815

: 00:38:59

Um, the, the last thing I wanna

do with you is play a game.

816

: 00:39:02

And, uh, you know, you are the

queen at, like, taking a complex

817

: 00:39:06

problem and being like, "Oh, you

know, this is what the data says.

818

: 00:39:10

You know, this is, this is, like,

whether the data's good or not, this is

819

: 00:39:14

what maybe you should do in your life."

820

: 00:39:15

And- Okay … uh, you're,

you're very good at nuance.

821

: 00:39:17

But I wanna ask you a few rapid

fire myths, and you tell me in one

822

: 00:39:21

sentence, uh, whether it's true or not.

823

: 00:39:24

Does that sound good?

824

: 00:39:24

Okay.

825

: 00:39:25

Yeah.

826

: 00:39:25

Great.

827

: 00:39:26

Okay.

828

: 00:39:26

Number one, uh, is

Tylenol safe in pregnancy?

829

: 00:39:29

Yes.

830

: 00:39:31

Tylenol is safe in pregnancy.

831

: 00:39:33

Okay.

832

: 00:39:33

Perfect.

833

: 00:39:33

That's easy.

834

: 00:39:34

Number two, is breast best?

835

: 00:39:37

Breastfeeding has some early life

benefits, but many of the benefits that

836

: 00:39:43

you are sold on, like IQ and obesity and

so on, are not supported in the best data.

837

: 00:39:50

Okay.

838

: 00:39:51

Number three, are phone

bans in school a good idea?

839

: 00:39:55

Yes, but not because they're going

to dramatically change a lot of test

840

: 00:40:00

scores, but because they are good for

kids' interactions with each other.

841

: 00:40:05

Awesome.

842

: 00:40:05

Number four, do cell phones cause cancer?

843

: 00:40:08

No.

844

: 00:40:09

And you know how we would know?

845

: 00:40:11

If brain cancer had gone up a lot

over time instead of actually what has

846

: 00:40:15

happened, which is that it's gone down.

847

: 00:40:17

That's good news.

848

: 00:40:18

All right.

849

: 00:40:18

Uh, is red meat bad for your health?

850

: 00:40:22

No.

851

: 00:40:24

Do vaccines increase autism odds?

852

: 00:40:27

No, they do not.

853

: 00:40:29

Um, should you take creatine?

854

: 00:40:33

Yes, if you are strength training.

855

: 00:40:35

There is no point if you

are a sedentary person.

856

: 00:40:39

Okay.

857

: 00:40:40

And last one, can you

get Botox while pregnant?

858

: 00:40:44

You can, but no one's gonna do it for you.

859

: 00:40:47

Okay.

860

: 00:40:48

There you go.

861

: 00:40:49

Uh, well, if you guys want the more

nuanced, data-driven, longer answers

862

: 00:40:53

to all these questions, you'll find

a bunch of in-depth articles, uh,

863

: 00:40:57

on Emily's website, parentdata.org.

864

: 00:40:59

It's actually one that I, I use pretty

often when I have a question in my

865

: 00:41:02

life, especially when parenting.

866

: 00:41:03

Mm-hmm.

867

: 00:41:03

Um, I subscribe to Emily's newsletter.

868

: 00:41:05

We'll have links to those down below.

869

: 00:41:06

I find them incredibly

helpful, uh, with parenting.

870

: 00:41:09

But even if you don't have kids, I

think you'll find Emily's style of

871

: 00:41:12

taking data and life information and

making good decisions or at least

872

: 00:41:17

having good frameworks for making,

uh, good decisions really helpful.

873

: 00:41:20

So we'll have a bunch of Emily's

links in the description down below.

874

: 00:41:23

And, and once again, check out Emily's

books, The Family Firm and Crib Sheet.

875

: 00:41:26

And there's another one that's Expecting

Better, is that what it's called?

876

: 00:41:28

Yeah, Expecting Better.

877

: 00:41:29

Okay.

878

: 00:41:29

That's the OG.

879

: 00:41:30

It's about pregnancy.

880

: 00:41:31

I, I was too late to have that one.

881

: 00:41:32

I don't have that one, but, uh, maybe

next kid, we'll, we'll get that one.

882

: 00:41:35

Um- Next kid, next kid.

883

: 00:41:36

E- Emily- Thanks … thanks so much

for coming on the Data Career Podcast.

884

: 00:41:38

Thanks for having me