Transcript
Intro
0:00 · Where does management come from? What do managers do in the future if we don't train entry- level employees? So, if we stop hiring as many entry- level staff, then we decimate the future managers and leaders of the company.
0:14 · [music] Welcome to AI Answers, a special Q&A series from the artificial intelligence show. I'm Paul Ritzer, founder and CEO of Smarter X and Marketing [music] AI Institute. Every time we host our live virtual events and online classes, we get dozens of great questions from business [music] leaders and practitioners who are navigating this fastmoving world of AI. But we never have enough time to [music] get to all of them. So we created the AI answers series to address more of these questions and share real time insights into [music] the topics and challenges professionals like you are facing.
0:47 · Whether you're just starting your AI journey or already putting it to [music] work in your organization, these are the practical insights, use cases, and strategies you need to grow [music] smarter. Let's explore AI together.
1:00 · [music] Welcome to episode 236 of the Artificial Intelligence Show. I'm your host, Paul Ritzer, along with my co-host, Kathy McFillips. Welcome, Kathy.
1:13 · Thank you.
1:14 · Kathy is our chief marketing officer at Smarter X. If you are a regular listener to the podcast, you have heard Kathy many times before. So, uh, Kathy is not taking Mike's job. Mike and I are still doing the weekly every Tuesday. And Mike is the host of the AI transformation series that we've started. We've done three of those now. And then Kathy and I do these special editions of what we call the AI answers episodes. So these episodes are based on questions that we receive during our monthly intro to AI and scaling AI classes.
1:45 · Uh as well as some questions that we get from virtual events. So these are all communitydriven questions and we just kind of go through there's usually I don't know Kathy what do we have 15 12 to 15 of these things.
1:57 · Yes we do.
1:58 · Um Kathy prepares a brief that I look at as we start recording. Um it's a wonderful brief but like I I just kind of dive in and we go. So um yeah. So today's 15 questions, they are based on the uh last two intro to AI and scaling classes that we did. The intro to AI was July 29th and uh scaling AI was July 9th. So these are both from late July.
2:23 · Um and yeah, so the the interesting thing and we'll kind of get into this as we go through these questions is the intro class I have been teaching since fall of 2021. So, we've had north of 60,000 people register for that series.
2:40 · And back in 2021, pre-Chat GPT, you can imagine we were getting some very foundational questions, just very basic um trying to understand what the technology was and where it was going and how to figure out use cases. And the intro questions have evolved. They have become much more advanced. So, even the people that are attending the intro class generally have really, really good questions. Um, and then the scaling AI we have now done for probably going on two years, Kathy, I would say we've been teaching that one just about.
3:11 · Yeah, we've done 18 of those. And so that's probably had 10 to 12,000 people, I would guess, go through that series.
3:17 · And that one teaches our fivestep framework for scaling AI in an organization. So those questions tend to be more organizationally driven, team, department driven. They're thinking bigger picture about how to move beyond just pilot projects and actually scale it in the organization. So, if you've never had a chance to attend either of those, we do continue to teach them. You can um go look in the show notes and there's links to sign up for the next ones. I think we have one coming up at the end of September, maybe. Does that sound right, Kathy?
3:46 · We have September 10th is our next scaling AI and September 30th is our next intro to AI.
3:52 · Okay. So, apparently I'm teaching a class next week. Um All right. So, [laughter] so September you can join us for both of these if you'd like. All right.
4:00 · Actually, no, it's it's it's on Friday, actually. It's on the 11th because you're not going to be in town on the 10th. So, we had to shift that date.
4:06 · Yes, we did.
4:07 · All right. So, um Okay. So, today's episode is brought to us by marketing AI month. This is a brand new thing and I am going to let Kathy um kind of tee this up for us and then we will get into the Q&A.
4:20 · Do you also do you think you ever need to say that I'm gonna People are worried I'm going to take Mike's job? I do because I think that these episodes with you Kathy are so popular and people might dip in and just want to hear you and I talk and they don't realize there's this broader show. So I don't know. I think like I never know when someone enters into the artificial intelligence show world and it might be their first experience with it is fair and I answers on like a YouTube clip. I don't know. So yeah, [laughter] I always try and set it up that way.
4:49 · I'm not taking Mike's job. I don't want Mike's job. We are all We are all good. I don't want Mike's job either.
4:55 · [laughter] So, thank you for being here. Uh, this episode is brought to you by brought to us by marketing AI month. We launched that Tuesday of this week. Um, so this month we are doing something really fun. We've got this amazing AI for marketing course series as part of AI Academy all month long. You can get it free. It's valued at $499.
5:17 · So, you'll get the series. It's five expertled sessions by Mike, practical tools, frameworks that we actually use as well and a professional certificate when you are done. So that's free through September 30th. You do not need to complete it by September 30th. You just need to enroll by that date. And to close out the month, which we're actually doing on October 1st, we are holding an AMA where similar to this format, I will be peppering Mike and Paul with all of your questions. Anyone that has registered for the or enrolled in the course is eligible to attend. So, if you only want the AMA, enroll anyways.
5:50 · But the course itself is amazing and I highly recommend it. I took it myself over the holidays and um kind of kicked off my year really strongly, so I was really excited about that. So, if you visit smarterx.ai/marketing, you can enroll right on that page.
6:05 · Anything to add on that, Paul?
6:07 · No, take advantage of it. It's our most popular course series. Um AI Academy has, I think, 22 professional certificate course series right now. So the idea here was like let's just accelerate marketing understanding and adoption because our experience with a lot of our AI Academy uh business account customers is that marketing is often the tip of the spear.
6:27 · It's the department that's leading on AI adoption, but then also an essential part of communication and strategy to bring other departments in and drive AI adoption and um transformation through other departments um being communicated through the marketing team. So marketing to us is a compounding value for the industry. the more marketers we can get to understand this stuff and be adept at using it in a responsible way, the faster we can, you know, spread that throughout organizations.
6:58 · So, if you're a listener and you're not a marketer, um, make sure to send that link to to your marketing peers within your organization because it again is open to everyone and um, it's a great value for everybody for sure. Which takes us to our first question. So, let's jump right in.
What are the best ways to use AI in marketing?
7:15 · Uh, number one, what are the best ways to use AI in marketing? I might turn this one back to you and say, "How are you using it, [laughter] Kathy?" Um, as our chief marketing officer, I'll I'll keep it simple and then if you have something to add, Kathy, I'd love to hear your perspective on this one. Um, I I just start with what what do you do?
7:31 · Like I would look at the tasks and the workflows that you are expected to perform every day. I would look at the goals that you're expected to achieve. And then I would go in and just ask your favorite AI assistant. You could use our jobs GPT tool if you want, but or you could just give this prompt to any of your favorite assistants and say, "I am a director of marketing. My job is this.
7:50 · Here's the KPIs I'm responsible for. Help me figure out like the top three to five ways that I could be using AI today. These are the tools I have access to. Here's the data I can connect to." Just give it the information and let it help you figure that out. But the answer is it's personal to you and your role in the company is how I kind of simplify that.
8:11 · Agreed. Um, I think I would add, you know, I'm using it a lot for strategy.
8:15 · We are doing, we are rolling things out on a regular basis. So, me getting a head start on getting all of the strategy out of my head onto paper. It's repurposing strategies that have worked in the past. It's building this knowledge base of all the things that we're doing. Um, it infrequently helps me write because I do enjoy writing, but it does give me a gut check on some things. I'm using some of our paid advertising to help me figure out new ways to message. um looking at data, looking at trends that I might not be able to surface with my non-analyst eyes.
8:47 · Um so one thing we've done on the marketing team is we have these documented processes for virtually everything that we're doing. So we're trying to on a quarterly basis or whatever cadence we decide running it through saying can we do something more can we do something more. So having the processes having the use cases documented and just because things are changing so rapidly updating that regularly has been really helpful for our team. Yeah, one example and we don't have to get into great detail on this, but the marketing a AI month that we just talked about, we decided like last Wednesday that we were doing this.
9:19 · Like it was a very spur-of-the- moment, hey, let's just give AI for marketing away.
9:25 · Like, let's just try and accelerate awareness and understanding in the marketing industry leading up to our make event to our big conference in October, which is for marketers and business leaders. And so Kathy, I don't even know how she did it all, but we launched that campaign three business days later. Like it was it was an entire thing. And I'm sure there was ways Kathy leveraged AI to help in the planning, the production of assets, things like that. But to launch a major campaign in the matter of three business days is is wild.
9:56 · Like if you think what could we have done historically? Now, I'm sure Kathy worked really long hours and it wasn't like she just hit a button and boom, here's your campaign. But even that, there's just no way you do that.
10:09 · We have a small team. Like, we we don't have 20 marketers where Kathy's just like, "All right, everybody go do your things." It's like, "No, we get our hands dirty. We're actually in this doing this stuff." So, that's an example of like a very real um relevant thing that I'm sure we couldn't have pulled off without the support of AI, right? and people I mean Jeremy Zimmer
10:29 · our director of marketing for academy he was obviously integral in all of this Mac in social media everyone and then I built this brief and like here's all of the things we could be doing with my knowledge with AI's knowledge and I told the team I'm like I know I'm missing something so please as humans go in and help me figure this out what am what are all the pieces parts that I'm missing but yes AI definitely gave me a very good road map to get it all done okay number two I have a team that is mostly made up of AI I beginners. They are curious and optimistic, but also time starved.
How do you carve out time for AI when the team is already slammed?
10:59 · The most common push back I hear is, "I believe that if I invest time in AI, it will eventually generate efficiencies in my day-to-day work, but how do I carve out the time for now?
11:10 · Sales targets and client deadlines won't pause while I do this. Are there any good tips to respond?"
11:16 · I would do a forcing function of a workshop. Um, it could be a one-hour workshop, it could be two hours, it could be half day, however you want to do it. But I would do an AI workshop where you teach them a framework. So we have a use case model we use. It's it's free. You can go I mean it's in our book. We teach it online. Like there's plenty of ways to go learn it and you can apply that framework to your own company. But basically, let's say it's let's just pick 90 minutes. We are all going to leave here in 90 minutes with one to two AI use cases that we can implement starting tomorrow morning.
11:48 · that's going to save us at least three hours a week. Like just just set some parameters because what it does is it forces you to look through your tasks and workflows that you do over you know any given week and say which are the ones that take more than 3 hours. Okay, now you've narrowed that down to like let's say there's five things on that list. Um which ones have a repetitive process that we could automate elements of? Which ones require creation of strategy documents or analysis of data?
12:14 · like you go look for the use cases that are a good fit and you just pick one and you build a project, you build an agent, like you just build something. So make the first like you know 20 minutes setting the stage, make the next 2530 minutes brainstorming where people are prioritizing lists. Make the last 30 minutes or so sharing what you've done or actually building the thing. Um so maybe you extend it to two hours and now you can actually have a build session with it.
12:40 · So I would you have to force it like you have to have something that says we aren't leaving here today until everyone has built a thing that's going to save them time and then if that works do it every other week or do it once a month and like then you just stack those things over time and all of a sudden everybody's got five agents or apps that are you know saving them 10 15 hours a week.
13:01 · Yeah.
13:01 · And I think showing them how, you know, I think yeah, the jam sessions we're doing where people on our team are opening up their computer, sharing it on the TV, we're all in the same room or online. We're watching them click the buttons and doing the things because that is daunting. People like, "Okay, I think I can do that. I probably have the brain power and the capacity to do it. I just don't even know how and I don't have time to really figure it out." So having someone show you how to do it is, I think, so valuable.
13:25 · Yeah.
13:25 · Yeah. And I'll I don't know again I haven't really looked at these questions so I don't know if I'm going to get into this and I'm not even actually 100% sure I want to share this yet but like I started a project this morning where I'm basically going through and trying to look big picture at where agents can be infused into my role. And as I was doing it I was like documenting journaling it like I normally would. Like okay I'm going in here I'm creating this. I'm going to test this agent. And and so I would go through and like this whole dialogue with myself.
13:52 · But then I realized, wait a second, maybe I'll just turn on screen recording and I'll just like record this in Zoom or Google or wherever and then eventually take that and turn it into like demos for the team. But then I'm like documenting all the steps I'm going through, all the visualizations I'm seeing as well. So all of that is helpful. But I think you're right, Kathy, like just you need to infuse whether they're workshops, build sessions, jam sessions, whatever you want to call them, those have to just become part of the routine.
14:20 · And then it gives people the the time and the permission to to do these kinds of things, right? The build sessions at MCON this year, not to plug MCON, but I will. Um I'm really excited about that because you bring your computer, you sit down, and 45 minutes you leave with something tangible. And it's just so helpful, even if it needs to be, you know, sused out after you get home. But just being there and watching someone else do it is pretty cool. Okay, number three. We are a small company. How should we think about using external consultants to help?
Should a small company hire outside consultants to build agents?
14:51 · The team is so busy doing day-to-day work that we are not making time to create agents and take us to the next level. We have lots of ideas but need the time, skills. Um what thoughts do you have on that?
15:03 · Yeah, I I mean there's a decent chance you can find consultants who come in and help you analyze your workflows. Um look for ways to drive innovation, look for ways to drive, you know, automate things, build some agents. [gasps] I um I think that that's a short-term fix. Honestly, I I just feel like this is going to be so critical to every organization's operational structure and the way they manage the company that you have to own it.
15:32 · Like so like I explained this a little bit on the podcast, but we're building Smarter X Labs within our organization. We're a small company. We have 19 employees. Um, I think of labs as like the R&D unit and their job is going to be to go to each team or department or business unit and work with them to say where are the barriers right now to achieving your goals?
15:55 · What are the workflows that you know take the longest or what are the major things that we should be doing as a company that we just don't have someone in that seat right now? Like we can't do it. Um and so they're going to basically function as um forward deployed engineers. So that's the concept we tal in the podcast where you basically have people who go in look for things to build, problems to solve and then they build them for them.
16:19 · But what I I believe the reason we have to internalize this is because we have to maintain that that IP like we have to maintain that knowhow within the organization because if we go in and we solve something with Kathy and her team on the marketing side, it may translate over to the sales side or the ops side.
16:37 · Like an agent we build here might just need a couple of tweaks and all of a sudden it's a sales agent too. And so because AI being infused into all these areas is so critical to what we're trying to build, I just see it as like absolutely fundamental skill we have to build internally. So if you're going to rely on consultants in the early going, I think that's fine. But I would I would be training or developing people who can function in that role moving forward.
17:05 · And it could be a, you know, a marketer who's just, you know, also pretty technical, like very comfortable in the data and the systems that's trained to become like an AI ops person. And it doesn't, you don't have to go hire software development people. You don't have to go hire AI engineers. They can be business people, you know, knowledge workers with domain expertise who are the just the most advanced users of the AI platforms you use. So, right, that's how I would think about it.
17:29 · And you know, you think about subject matter expertise, you know, being like you said, domain expertise, institutional knowledge, like a consultant can't help you with most of that stuff. So, or do you So, do you start building notebooks and knowledge bases now if you're going to start using someone on the outside?
17:48 · I mean, if you're if you're going to bring in a consultant, they're just going to come in initially and like document processes, audit workflows.
17:54 · They're going to look at what the eyes going. Again, all could really be valuable. Like if you truly don't have the time to do this and like the alternative is no one does it then yeah like hire somebody to at least get this moving but I don't know I just it's one of those things I think is so critical to every company that I would be very aggressively looking at what is that like defining what that role is and
18:16 · seeing do we have someone on our team who could this is a career opportunity for them like they would work really well we've seen this in companies Kathy you and I both know people who are in AI ops roles today who you 12 months ago, 18 months ago were were marketers or they were project managers or like whatever. So this to me is that whole creating seeing an opportunity for a career path for yourself defining it and then going and building it.
18:39 · And when we start this internally, I would like to be tribute and I would like you to tear apart my department and tell me all the things that I will swallow my pride and I can't wait to see what comes out of it.
18:50 · I think it is just like a totally collaborative thing. I think it's more of like you have a wish list like you have roles you'd love to fill and it I think this division is like okay Kathy what are the five roles you would put on your team today if you had the approval to hire them and they might look and say I think we can get you like 50% of the way there on three of these in like two weeks it's not like it's not going to be your FTE but like we could probably build something that'll at least give you that functionality that's how I'm thinking about it and then you kind of like go down from there I'm ready okay number four who should be
Who should own AI transformation in the enterprise?
19:20 · the point person for AI transformation in the enterprise. In some organizations, it's the CFO with project manager who understands the systems and knows where the data sets are hidden.
19:30 · Has this been your experience?
19:32 · I don't know that there's a right answer for this one yet. I do think it depends so much on the skill set of this this person. So I've I've seen one where it was like a chief legal officer who became like the head of transformation because that person was just also extremely qualified on the AI side and understood the business deeply. Um I've seen it being driven we had our first AI transformation spotlight that Mike did with good karma sports that is a CEO driven transformation.
20:03 · I mean Tai who was you know did the interview was telling he's a he's a key key driver of the whole thing but that is being very much led from the top by Craig at that organization and so that's a this is a CEO who got into it said this is the future of the company I'm all in and he is like right there every week in every meeting about AI strategy so in that case I don't know that they have a chief AI officer but it's probably Craig like he's the driver um I've definitely seen where the CIO does it if that CE CIO
20:34 · has deep business knowledge like if they're not just thinking about it from a technical perspective. Um we've definitely seen instances where like um Dan Slen who's speaking at MCON this year marketer CMO and Dan just sort of emerged into being an AI transformation leader at a major company because he just became the most adept at the technology. So I don't think that there is a right answer. I I really do think it is where is your organization at?
21:00 · Where is the CEO in the organization at?
21:02 · Because I am a believer that if the CEO is extremely AI forward, that is probably the person who should be at the top of the chain when it comes to spearheading the transformation. Um, obviously doesn't have the ability to be in every detail, but the CEO should be the driver. Um, that's probably going to be the most successful organizations is when it is a top three priority for the CEO and they are actively involved in everything. Yeah, it just opens up that permission and enablement when the CEO's in that role.
21:33 · Yeah, like I actually I this is totally behind the curtain stuff. Um I actually messaged our team today because I was working on something uh pretty significant for our organization. And so I just messaged I was like, "Hey, can you guys send me the excerpt from our AI policy regarding connectors because and Tracy and Elizabeth are going to laugh when they hear this now? Um because I'm I may be instituting an R&D exemption.
22:00 · Uh that's a term I just made up. I literally put in parenthesis to [laughter] them. Um and like approving things myself. Um so I'm not bypassing our approval channels in our AI policy, but I am I mean it kind of I'm accelerating them.
22:15 · And so within like 10 minutes I had I was copied on emails to our IT team and our lawyers [laughter] saying that you know basically our CEO is planning to accelerate approvals we'll call it on our AI policy.
22:29 · Now let's think about companies think about companies we know where the CEO has gone rogue. Paul let's not Yes. Oh no. Tracy did her job. She was like here's the 15 things that I'm basically like want to check before I like go back to Paul. But like, and they're all really smart things. It's like, yeah, I know. Like, I I'm not going to just like hit the buttons.
22:48 · But, um, yes, when the CEO is has an urgency like, hey, I'm doing a presentation on this in 7 days and I I need these connections in place right now, then things happen and you do it within the safest way possible. But you have to also like anam again, I don't want to like spend our whole time on this, but like Tracy's thing is, okay, cool. You turn that on.
23:12 · Does everyone have access? Do we have to like communicate with everybody not to use the access you just granted?
23:18 · So while I want to move fast, I I that is why I reached out to her first and said, I'm going to do this within our guard rails. We have to set that example, but we also have to find a balance here because I have no choice.
23:30 · Like I I have to do this in the next seven days. So how do we do this in in the the most um responsible way possible related to our guidelines and policies?
23:41 · Yeah. So you may have you may have some access in your chat GPT account later today that doesn't look familiar.
Are there generic agents everyone can use, or do you build your own?
23:49 · Okay. Number five. Are there generic agents everyone can use or for agents to be beneficial for us? We need to build them on our own.
23:57 · It depends on what platform you're in.
23:59 · So this is actually the initiative I'm I'm referring to. I'm I'm working deeply on integration of agents into smarter X and I had this I I haven't talked about this internally. So again, I was it's not even a joke. Like people think I'm like making this up maybe on the podcast. I'm like, "Yeah, no one on our team knows." Literally, no one on our team knows about some of the things I'm working on and they hear about it on the podcast.
24:18 · So, a couple weeks ago, I I started a personal initiative um to deeply infuse agents by the end of this calendar year and in essence figure out what that looks like moving forward, all the connections, how we're going to structure them through the SmartX Labs division, um things like that. And so, um it I have a forcing function that I have a presentation next week related to in Agentic Enterprises. And then I'm teaching an AI executive workshop at Makeon on October 13th where I need to actually present a model for this.
24:51 · So I'm deep in the agent world at the moment. And what I'll say is like an example here would be if you're in chat GPT, they have standard agent templates that you could go in and just start with those. Like one that pops like a marketing strategy agent. They've pre-built the skills for it. They've built the system instructions for it.
25:14 · All you do is go in and give it a knowledge base or connect it to like HubSpot or whatever it is you want to connect it to if you're allowed to. Um, so I guess the short answer is yes. Like there are generic agents if you're in Copilot or you're if if you're in Google um chat GPT Claude, they've pre-built some agents that are more universal.
25:32 · But most likely you're going to be building things that are specific to your role or to your organization because for them to be really valuable, they're going to need um a knowledge base that is personalized to you or your company and they're going to need connections to data sources that your administrators are going to have to approve.
25:56 · Got it. Okay, number six. I love this question. Can you set up checks within an agent that require it to pause for human approval before taking specific actions?
Can you build human approval checkpoints into an agent?
26:07 · I wouldn't build an agent that you couldn't do that with. Um, so yeah, it just depends on how technical you are. Um, like where you're building these agents. But again, if you're working, I I'm going to assume most people that listen to our podcast are probably not like advanced codeex users, um, cursor users. Like, they're probably not the Now, I might be wrong. Again, we don't really, it's hard to know your podcast audience.
26:35 · It's just all these people. It's quite opaque. Um, so I'm assuming that most of our listeners are more uh marketers, sales professionals, business leaders um that are not deeply technical coders, software developers. So we're we I'm just going to use the collective we here are likely we're Salesforce users and we're going to be trying to figure out how to use the Salesforce agents that are available to us. We're going to be building co-pilots in Microsoft. We're going to be building um chat GPT agents.
27:06 · there's instructions within there that where you're going to say this is the checkpoint where you're going to do or you're going to give it read readon access where it can't actually go and send something or do something. So the vast majority of us are likely only working with and building agents where humans are in the loop or in the lead and yes you should know when an agent would be taking an action and you should be the one that giving it permission to do that. In lowrisk scenarios, you may give it permission to do the actions without approval.
27:37 · Like example there, send Kathy a um a summary report every Sunday night of marketing performance data the week prior and it goes to HubSpot. It gets the data, it pulls it, it summarizes it, it sends it to her. So, it's going to take the action because it's just between me and Kathy. There's no risk of it sending to our customer base. That that kind of thing.
27:59 · Okay, cool. Number seven, for organizations with many data sets containing personal and sensitive information, a complex enterprise architecture, legacy systems, shared shared services across very different departments, and pockets of AI forward teams.
How do you enable agents on sensitive data without adding risk?
28:14 · How can agents be enabled for those teams without creating risk for the rest of the organization with great care? [laughter] that that is like when you're talking about these agents that are accessing sensitive information, let's say like a health care system, um you know, a law firm, a bank, wealth management company, like whatever, like there's just we all have sensitive information. We can even say smarter X, we have customer data, um that we always obviously want to be extremely sensitive to. Um so anonymizing data before you put it into these systems, things like that.
28:42 · Um you have to have it and legal involved like it's just you can design what you want to have the outcome be. You can build prototypes of agents. You can do all of these things, but when it's going to do things that present risk to an organization, especially if you're in a heavily regulated industry, you you have to know in advance the approval systems you have to go through. You have to deeply understand your AI policies. And that's the example I was giving earlier with Tracy. Like Tracy just taught an entire course and we'll get to some AI policy stuff.
29:11 · I see it highlighted below. um she just taught a whole AI Academy live course on our AI policy, which is a very um comprehensive document. And if you're going to be building agents, you have to be familiar with that. You have to know what you're allowed to do and not allowed to do, what data you're allowed to put into it, what connections you're allowed to make.
29:31 · Um, and so what we're trying to do as an organization is these sort of like I'm just going to make up a term here like horizontal approvals like okay if we want to connect HubSpot to chat GPT let's approve it once so that if Kathy or Tamara or Dia or anybody wants to do something between chat GPT and HubSpot they don't have to come ask for the connection permission every time it's already been connected and then it's approved for readonly access for like all these different uses.
30:00 · So you do something once and then it it it permeates across the whole organization with permissions. And so I think you'd be in a situation like that. Like if you're in this kind of enterprise where you're going to be accessing personal sense of information, doing these complex architectures, try and try and solve for as many of those with a one-time permission and training process as you can so that like everybody's not coming and trying to get approvals are basically the same thing, right?
How should a company fund its AI investments?
30:31 · Number eight, how should a company fund its AI investments over the next few years and where should I expect the new spending to come from?
30:37 · Oh boy. Uh, it's got to be dynamic. I don't know. Like this is a really hard question to answer. Um, like I'm going to just like I'm thinking out loud here. People is my first thing.
30:52 · So like when I'm thinking about Smarter X, if I I'll just take it to make it personal. So for our company over the next three years, where am I going to put AI investment dollars? Um, my first investment is the people to help me figure out the answer to that question. So the people who think 247 about our AI investments, that could be human capital, it could be token capital. It's like where's our spend going to go? What platforms are we going to invest in? But first, you have to have the right people to make those decisions. Um second is going to be my my platforms. So what are our token budgets?
31:25 · Where how are we going to optimize those token budgets? Um how do you even budget for token budgets?
31:32 · Yeah. Like how do you like teach people what that even means? and how do you manage them?
31:37 · Um, and then third would be probably more of an infrastructure thing like do I need to be buying Mac minis? Do we need to be buying GPUs? like are there are there business cases from a hardware perspective where we actually need to be like thinking differently about the company um to where we have access and
31:57 · do we need to be building I guess the fourth would be is there going to be an increasing argument that we should be training our own model at some point you know I think that a lot of companies are going to head in that direction where they're going to get these openw weight um more efficient models and they're just going to you know do reinforcement learning you know training on um to be adept at their specific industry and business case. So yeah, I guess that's four. And again, I've never actually thought deeply about this question at all.
32:25 · Um so that's I I I reserve the right to change my answer at some point, but it is hard for I met with the group this morning and they're on a three-year cycle where they're trying to plan out the next three years. Like how do you even do that? And as leadership, are you able to give your team permission to just as intelligently as you can make those decisions knowing that they're going to change?
32:49 · Yeah, I think three years is hard, but every company's doing every enterprise obviously thinks in three to five year road maps like they're all looking in that direction. I think you need to make decisions that are as um flexible as possible and as like evergreen as possible so you're not like locking I
33:10 · don't know like one of the ways I think about this one is you know if we go really really deep with chat GPT as an example and everything starts running through that and all the connections are for chat GPT and then something happens to open AI or like their models or whatever and it's down for 2 hours 2 days 2 weeks what happens to the company? There's no redundancies. There's no backup plan.
33:34 · And so that that's a, you know, betting on a single provider of intelligence isn't a super scalable thing probably. And so when you're looking at three years, a strategy be like we need intelligence redundancy.
33:51 · And that that scales that like becomes a strategy that lives over three years and actually starts to affect the decisions you make because we've set a priority around redundancies. Um so hold that thought because I do have another question later that I don't want you to answer right now.
34:07 · Okay. All right.
34:08 · So let's end it right. Let's end that right there.
34:10 · That's the end.
How do you budget and forecast total AI spend, including tokens?
34:11 · Okay.
34:11 · Number nine. Um kind of going into this a little bit. Beyond subscription costs, how are companies actually budgeting and forecasting total AI spend, including training and education time, setup and integration, internal token usage and token usage driven by external user or customers. Is anyone doing this well enough to have a real forecasting model or is everyone still finding out when the invoice arrives?
34:33 · Are there any resources you would suggest?
34:36 · The best I've heard, I don't even know if best is the right term here. A model I have heard is literally just sending or setting a cap each month like here's our here's our budget for the month.
34:47 · There are definitely more and more organizations trying to figure out the token optimization and management. There was just something from Google. I think it was on Monday I saw this where they're starting to offer that kind of capability within their systems to help you manage it. There's startups emerging that do token optimization and basically route you to the like the most efficient models. Um I saw something I think it was this week where OpenAI is now actually experimenting with outcomebased pricing.
35:18 · So part of this is we're [clears throat] developing systems for the way that AI is priced right now, but most of these labs and providers of the intelligence know that this is not a viable scalable system. like per token
35:34 · utility based pricing works sort of for developers but once you start getting it into departments like marketing sales operations HR that's a really really bizarre way to try and price stuff like if it's not seatbased um with unlimited usage or it's not outcomebased if like the outcome is very predictable and able to be attributed specifically to the model's work then it it gets it gets kind of tricky.
36:02 · So, um yeah, I don't I don't know like an awareness of how the tokens are being used, what the different models charge per token. Um it's very early. I I don't have a great answer for like go look at these three companies because they're doing it perfect. I don't know anybody who's actually solved this yet, including uh some of the the big tech companies that are responsible for building the models.
36:27 · I don't know that other than providing unlimited uses of tokens to their employees. But even Meta just changed like they've made changes to the models that they're using for this reason because they they couldn't manage the token budgets and they're a builder of their own models like Right.
36:44 · So, and if you do know somebody listener, reach out. We we'll have them on the transformations.
36:49 · Yeah, totally.
36:51 · Okay.
36:51 · The next three questions are from the academy live we had on Tuesday this week with Tracy, our COO, and then okay, Samantha, our legal counsel. So, these were I thought these just fit in well with some of these discussions. So, the first one is um what does a right-sized AI vendor approval process look like?
What does a right-sized AI vendor approval process look like?
37:09 · How long should a review take? What happens to employees while they're waiting? And how do you catch it when a vendor update introduces new integrations or changes how an approved tool handles data? And I know that you're not doing, you know, that ops is doing this, but as the leader, what thoughts do you have on this?
37:26 · I would start early. I mean, we've we've been in situations with some customers of ours where they got excited about instituting, you know, different AI technologies into the organization.
37:38 · They'd made the business case for it and then they go to procurement and they would be told there was like a multi-month wait time before procurement would even look at anything. So it starts with really just understanding your organization's procurement process and what kind of risk and security considerations need to be made as we start you know getting into the AI technology that might require access to certain data sets and um yeah I think it it really is about an understanding of the process and then from there it's going through um and building that in.
38:10 · So if if you know that it's going to be 3 months before you can get the system you want or the approval for the use case you want or the connection you're looking for, you need a plan B about what that um workflow looks like without those tools.
38:24 · So yeah, I I think again this one is very um subjective to the organization and what their approval processes look like and how they're defined within the AI policy. So I know in our case we have to submit now through form requests to the operations team when you want a new vendor if you want to make a connection to a data source and then I think the
38:48 · team says like within 48 hours or something they'll at least give a preliminary hey we're going to have to look deeper into this. this is going to requ kind of the example I gave with Tracy earlier like I put in a request I we'll call it um and then there's a you know within a few minutes there's an email to the legal team there's an email to the IT team because for the ops team to make their decision they need the the expert perspective from those two bodies to make the informed decision around the vendor or the connection that's being requested.
39:16 · So yeah, it's going to vary by company to company, but the best thing you can do is deeply understand that process before you get started. So you don't run into that obstacle when you're ready to go. For sure. And in one instance that we had internally, the request was put in. It took longer than normal because it was brand new, brand new tech, and we were trying to figure out, well, apps was trying to figure all of it out with legal. And there were two there were some redundancies because things were being done twice. one the old way and one trying to do it with a new tool.
39:47 · So there was a minute of double work but in the long run it's going to be so much better now that everything got approved the right way.
39:53 · Yep.
39:53 · And I will just say being on the other end of this you know we have our AI academy which is a technology platform and so we have to go through procurement especially for these bigger enterprises and you have to meet some very stringent standards that are only getting stricter as we go. And so Tracy can attest like I mean she spends a fair amount of her time getting through procurement systems and she could probably sit here all day telling you about all the new requirements over the last like 18 months as a result of AI.
40:25 · Yeah. There was one she did last week or two weeks ago that she was just like this is the wildest form I've ever filled out.
40:30 · Yeah.
40:31 · Necessary. Yeah.
What happens to your data if an AI vendor is acquired or goes under?
40:33 · Okay.
40:33 · Number 11. When reviewing an AI vendor's terms and data policies, what should companies look for regarding what happens to their data if the vendor is acquired, goes bankrupt, or sells its assets? How can organizations manage or mitigate that risk or sells the data for training purposes?
40:50 · Um, I I I'm like this one. I just honestly feel like my best answer is you you have to rely on your legal team. Like everybody's legal team's going to have different appetites for risk. they're going to look at it through different lenses. It is just simply not a decision
41:08 · that a like a traditional business leader um department leader you don't have the expertise and likely you don't have the permit in your company to make those decisions. you have to involve legal and whatever other channels are required and hopefully you know based on our previous question that's all clearly document documented in the I policy and if it's not it needs to get documented so that you can follow this these procedures and then you know when it's a non-starter so you know you're getting
41:39 · really excited about this new AI tool you go in and pull up their terms of use throw it into Chad GPT and say tell me how they handle our data and immediately it pops up with like three red flags for you. Another way you could do it is um take your AI policy related to vendor data usage, drop that into chat GPT thread or Gemini or wherever and say, "Hey, here's our AI policy or build a project that has that AI policy baked into it. Um I'm going to give you a contract for a new vendor.
42:08 · Can you tell me how this relates to our terms of use requirements and uh flag for me what I should highlight for our attorneys?"
42:16 · Right? That's I do that kind of stuff all the time where I'm just like here's this doc, here's this doc, compare them and tell me what I should ask the attorney.
42:23 · For sure.
42:25 · And then, you know, just thinking about when they change the terms and conditions or when something I mean there's so many it's not like approve it once and it's done. It's like set up some normal cadence of just reviewing everything to make sure it's still Yeah. Yeah, like when Anthropic came out with Fable 5, they changed their data retention policy where they kept everything for 30 days and I a lot of people were up in arms about that.
42:47 · Right.
42:47 · Number 12. When should a company disclose externally that AI was used to create content or complete work? Is there a meaningful threshold or does it depend more on the nature of the work and its potential consequences?
When should you disclose that AI was used to create the work?
43:00 · This gets in again to AI policies. It everybody needs to make these decision decisions for themselves. I feel like it's a bit of a sliding scale where more and more it's just going to be kind of assumed and expected that AI played a role and I I think like tagging every single thing with like AI helped here or AI did this I don't think most people are going to care most of the time.
43:20 · Um so I think that there's some basic rules that at least I tried to live by and I I hope our company tries to live by is like when authenticity matters you should be doing the work. It should be human in the lead. Like it's okay if you're using AI to edit or you know verify ideas and things like that but um you know it should be you and you should stand behind that work. Um this came up on episode 234 35 of the podcast.
43:47 · We were talking about that one um author who used it and got called out on it and he's of course I use it. I use it all the time. Like what are you talking about? Um, I I I generally like to default to if you're using it, it can be something as simple as like co-authored with Claude or Claude supported in the editing and, you know, revisions of this document or Claude helped me with brainstorming, whatever.
44:11 · Like just disclose it because um it doesn't do any harm, but I don't think we're at the point we need to make like a huge deal of it anymore. Um, right. So yeah, again it's you got to make these decisions in your ad policies and then provide that guidance to your team because everybody's wondering. It's like I don't know if I'm supposed to tell somebody I didn't actually look at their email like my agent is now answering my emails for me. In that case, I would just say yes, you should disclose it if your agent is answering your emails for you.
44:39 · Um but yeah, I think it's a very personal decision and it needs to be decided in the policies.
44:46 · I was working on something last week and I can't even recall what it was but I kind of went into it blind not really knowing what I was going to be working like what I was going to be doing and I used AI a lot and I put [clears throat] that in the when I distributed it to the team I was like this is claw written with Kathy supporting it like I honestly I don't know so this is
45:07 · my first pass I'm learning and just so when people looking at this like this doesn't sound like her um that's so it's so so obvious when it's not you know but I just want people to know y that where I was on that that like one example and you could attest to this Kathy like sometimes [clears throat] we'll be working on a problem and I'll go into chat like here's the exact problem here's the context you need here's what I'm trying to think through build a brief for me like I just need to like think about this and it'll build a brief and I will
45:34 · take 10 minutes I'll read the brief I'll give it some thought and like this is really good I will send it to the team internally and say this is completely unedited but I have reviewed this and this is a really good starting point and then I turn it over to the team so they know I didn't do it but they know I've signed off on like this is a good direction come back to me when [snorts] you you all have made progress on this.
45:54 · Um so yeah I just I don't know I don't think there's like rigid rules around this. I feel like it's just evolving as our use of the models changes.
If AI does the entry-level work, where do future managers come from?
46:03 · Okay, number 13. Entry-level staff have historically learned their trade by implementing task based work assigned by their managers. if that work is now going to be assigned to AI, how do entry- level staff learn their trade and where do future managers come from?
46:18 · My theory is apprentice programs. I talked a little bit about this on the podcast um this week. Uh it was 2:35 I think it came up because there was that research that came out about uh apprenticeships were actually like the number one thing and it just that happens to be what I'm talking about for Meon. My opening keynote is the architect, the orchestrator and the apprentice. It's a theory and that it's very just a theory on what the future of work looks like and what the different roles we're going to need within organizations are.
46:45 · Um my belief is that we can actually accelerate learning and expertise through apprenticeships um with the support of like guided learning from AI and personalized learning. Um but I'm not sure yet. I have six weeks to figure out what that exactly that is. I I've been thinking about this model since March of this year. Um I I don't know.
47:12 · This has been my great debate where like where does management come from? What do managers do in the future if we don't train entry- level employees? So if we stop hiring as many entry- level staff, then we decimate the future managers and leaders of the company. And I don't know that again kind of like the the earlier question, I don't know of an organization that has this figured out.
47:34 · Like it's I've talked to lots of companies and there are lots of AI transformation leaders and HR leaders at big companies that are just starting to ask these questions themselves and and nobody seems to really know where to go with it yet.
47:48 · Yep. 41 days in case you're wondering till Mayon. No pressure.
47:53 · 41. I only have three talks to create in that time period. [laughter] Uh number 14. You describe the actual mechanics of an LLM practicing predicting the next word as a bit of a black box. Is it really? And if so, isn't that a little ominous?
Is an LLM really a black box, and is that ominous?
48:09 · It is really a black box. Um, there's a field of study called mechanistic interpretability where they try and understand why the models do what they do. [snorts] The LLM itself is making predictions based on proximity of of like words to each other within a graph.
48:26 · So, this is like a very oversimplified way to try and help people visualize this. Imagine all of human language as like this three-dimensional cube and within that cube is like every word in the human language and how close those words are to each other sort of shows a probability of them occurring next to each other.
48:47 · And so when chat GPT starts writing something, it's in essence going through this dimensional graph and it's like knowing that um you know Kathy sat on the chair and it knows that chair is a likely word connected to where Kathy would sit cuz people sit on chairs.
49:05 · Could be a bench, could be the table, could be all these things. But in essence, like imagine in that cube those words all appear near each other. Sat, chair, desk, whatever. Um, that's kind of like how it works. The way to think about the language model, the closest proximity to it is the human brain. And that would be like saying, "Well, do we really not know why Paul picked the words he just picked when he was explaining this?" Like, no, we actually don't.
49:30 · Like, you could put me under an MRI machine and it might see what neurons in my brain are firing when I'm talking, but it can't drill in and say, "Why did he choose the word he chose?"
49:42 · Um, it's likely just on my training and my experience as a human of the words that appear near each other in the English language. So that's kind of it. Like it Yeah, it's ominous. Yeah, it's weird. Like we don't really understand this intelligence that we've created.
50:00 · All we know is that we give it more and better data and we give it more chips to train on that it seems to just get smarter and more capable of simulating human behavior and language. And that's kind of it like that's the basis for the first scaling law of AI that has now stayed true for you know the last decade which is more data more computing power the bigger the model the better it seems to get. Um, there was a this sort of ominous thing that Ilia Sutska said back in the day at OpenAI, which was they they just want to learn.
50:31 · Like these models just want more information and they just get smarter.
Are we building businesses on rented land with LLMs?
50:38 · Okay, number 15. The nice thing about these questions is that I get to decide the question. So, I actually dropped in my own question. [laughter] Okay.
50:44 · My husband's been traveling for a week and I've had a lot of free time. So, this is these are the things I think about. Um, okay. When I worked with Joe Pulitzy at the Content Marketing Institute for years, he always emphasized the importance of building on land. You own your blog, your email list, your audience. We've seen we've seen um what can happen when people go all in on rented land like LinkedIn, Twitter, and other platform. And the algor algorithm changes, products, audiences um audience becomes harder to reach, the platform disappears.
51:14 · Now people are building products, audiences, and sometimes entire businesses on custom GPTs and other AI platforms. Are we watching the same story repeat itself with LLMs? And how should we think about the risk of building on rented land?
51:29 · That's a deep thought to have.
51:31 · I know a lot of free time. Not really free, not really free time actually, alone time.
51:36 · Yeah.
51:36 · I I think it kind of relates back to the thing I mentioned earlier about, you know, if you build on a single platform, you're sort of uh prisoner to what happens to that platform.
51:47 · I I look at this a lot now with like how transferable um skills are between agents and platforms. The system instructions that are used to build a GPT can very easily be copied and pasted into Claude and you can build the same capability over in Claude. [gasps] So I think that there's [clears throat] an inherent risk here.
52:07 · It's probably part of the reason why some organizations are now choosing to take openweight models and train their own models. It's just a more reliable system. The model lives on internal servers. You're not um you know captive to a cloud service going down and you no longer have access to it. So I do think that more organizations will start to build redundancies and backup plans in um at a very surface level. You can start by making sure everything is documented.
52:34 · We we've done this again internally at Smarter X where we're now at least from a governance perspective documenting when apps and agents are created by different people. Ideally, you include what the system instructions are, what the knowledge base they have access to is so we can go back and audit it. And so let's say like worst case scenario, you build everything around chat GPT. You've got 25 agents that are basically running the company. you've got 75 skills that those agents use to do what they do and then chat PT goes down for 5 days.
53:04 · You could worst case go grab the skills, recreate them in claude or you just mirror everything in claude.
53:14 · Like someone's job literally might just be maintaining a mirrored image of the chatbt instance in a second platform so that if something goes down you just flip a switch, everybody moves over to Claude and continues their work. I would imagine there are organizations that are actively doing what I just described. I haven't talked to them. It's just one of those things you can kind of make an assumption that someone is ahead of the game doing this because we are talking about entire companies running on this and now you have companies laying off thousands of people under the assumption that agents can do a lot of this stuff.
53:47 · Well, if those agents aren't available, work doesn't happen. So, there needs to be redundancies built.
53:52 · Yeah.
53:52 · And I was talking to Claire who um is our producer of the podcast and works on our um content. We were talking today about how Mike had actually done talked about this in one of his courses for academy about how to extract some of that from one tool to be able to upload it and put it in another tool or how to document it and what that process is.
54:13 · So it really is critical step of all of this.
54:16 · All right, that is number that is 15 questions.
54:19 · Those are good questions. I don't I feel like these are all different questions, too. Like I don't know that I've answered any of those. Like [laughter] maybe like the marketing use cases has probably been answered at different times, but it changes. But the rest of these are um relatively original. Really good.
54:31 · You don't you don't look at this beforehand, but Claire and I do.
54:34 · Yeah. So, she does a lot of she does a lot of has this been asked 20 times already? Let's not ask it again.
54:41 · So, yeah.
54:42 · Yeah. Good stuff.
54:44 · So, thank you, Paul. We you'll see a mic next Tuesday back on the podcast. If you are interested in learning more about uh marketing AI month or MCON, our big event coming up in October, please let us know. I'd love to help you get you to both of those.
54:57 · If you're registered, come to Cleveland.
54:58 · We've got Oh, do the pod because we have the pod 100 which gets you a VIP lunch with me and Mike. We're doing on that the final day of the October 15th a VIP lunch for anybody who's uses the pod 100 when they register. So yeah, mcon.ai get registered. We'd love to see you there.
55:13 · Pod 100. Thanks, Paul.
55:15 · Thank you.
55:17 · Thanks for [music] listening to AI Answers. To keep learning, visit smarterx.ai [music] where you'll find ondemand courses, upcoming classes, and practical resources to guide your AI journey. And if you've got a question for a future episode, we'd love to [music] hear it. That's it for now. Continue exploring and keep asking great questions about AI.