Bad users and good Bings
Exploring AI rabbit holes and guardrails with James Paterson (@Presstube)
Overall, I did a pretty good job of not learning a lot about blockchain, crypto and NFTs. It seemed like something I could sit out. AI is clearly a different case altogether.
The initial push to educate myself was admittedly petty: hating people with uninformed confidence is one of my love languages and to do so properly I needed to know more about AI than the Linkedin jabronis were pretending to.
A couple things jumped out at me. First off, like crypto, blockchain and maybe even the internet itself, most people have spent little time figuring out how it works. Which is fine if you keep quiet, but the world is pretty loud these days. The flip side of that is that the people who do actually know about it aren’t making bold predictions. The message seems to be that it’s too early to call. Which is a little bit why I ended talking to my friend James Paterson about it.
James is an artist, animator and technologist who's been making magic as Presstube for a while now. He’s a free thinker and devoted weirdo. He also, over dinner during the holidays, sort of scared the shit out of us about AI (but was nice enough to follow up the next morning with a text saying “maybe i’m full of shit and it’ll be fine”).
So, while he protested that his expertise is more in generative art than in AI itself, he understands it on a much more intuitive level than I do. Which is why he was the right person to help me stress test my very rudimentary knowledge of what AI is.
Dead Fresh: Okay, so am I right in feeling like we’re in too early a stage of AI to be making super competent predictions about how it will all play out?
Presstube: Yes, that seems spot on to me. I mean maybe people in the research labs at Google and Microsoft may have enough of that view to predict. But even then, it's a bit like when they reintroduced wolves to Yellowstone and it ended up reshaping the rivers.
DF: Right! So do you think that a superficial understanding of AI is enough for the average person to be able to sort of evaluate its ever-increasing role in their lives?
P: I think that maybe it’s the kind of situation where you can go as far down the rabbit hole as you want. And the question is maybe more, how far is worth it? So my question to you is what do you understand about it? What's your understanding if someone asks you how it works?
DF: Ok so if we’re talking about the language model stuff, essentially you ask it a question, it does a search of what it knows and then it predicts the answer that it thinks you want. How’s that?
P: Yes, that's roughly it, but one thing that’s maybe important to understand is that there's two things that it’s holding in its “mind” at once. The first is a huge training set that was done all at once in a massive undertaking. That knowledge cuts off based on when that model was last fully trained. So if you ask it who is the president and it hasn't been trained since the President got reelected, it will get that answer wrong. And it will continue to get that answer wrong until its next big training session.
The second piece it’s holding in mind is what you are talking about in your current conversation with it. That’s the “transformer” aspect (GPT = Generative Pre-Trained Transformer). This takes your input and relates it to its training set, predicting what you want to hear next. But it keeps stacking the input of your local conversation into a newly shaped jumping-off point. So it has its huge data set from when it was last trained and it's got your current conversation in mind, but it doesn't actually learn and reincorporate your current conversation into its larger training set. Your input remains localized to that conversation. Does that make sense?
DF: It does. But is that the end of it? And is it, I’m not sure if this is the right way to say it, but is it creating something new or just mashing stuff together?
P: It can create something brand new based on what it knows plus what you're talking to it about. There's one more piece that's really important: are you familiar with prompt engineering?
DF: I mean I’ve heard those words (laughs) before but have no idea what they mean...
P: So it has the training set that's fixed and the current conversation that it's holding in its mind. You're essentially able to bring it to a new place, a new jumping off point, through any given conversation. So say you open a new ChatGPT window and maybe you feed it a short essay then ask it to produce five or six different iterations of it. You get it to produce some stuff and you get it to refine what’s produced. You have the transcript of that conversation to where you are now and it’s now set up to understand certain things. It's been prompt engineered to a new, custom jumping-off point.
This is prompt engineering and it can be very useful. You could prompt engineer it to be primed to tutor a kid about something specific about algebra. Or prompt engineer Midjourney to the point where you are dealing with a very specific kind of image. From that jumping off point you can modify the prompt and get to where you need it to go. Or if it's the tutoring situation, you can adapt it from that jumping off point to exactly what you need your kid to learn.
DF: Ok yes, so I was listening to the NYT reporter who’d spent the now legendary evening with Bing’s AI (codenamed Sydney) and he’d asked it to write something and it wasn’t very good. So he asked it to go read Kurt Vonnegut's book about writing and then to rewrite based on that. Is that prompt engineering?
P: That's it, that's exactly it. So you're telling the AI in that prompt engineering scenario to go consume a piece of knowledge that it has. Specifically you're saying to get Vonnegut's book and re-start to talk to me from that position.
DF: Ok, switching gears, do you know what red-teaming is?
P: I don't know that term.
DF: I mean I don’t really either (laughs) I just learned it. I think it's basically just the term for like, whenever there's something like AI people just go and try and push its buttons.
P: Right, well the really interesting thing about Sydney and ChatGPT and all these systems right now is the guardrails that are in place were essentially prompt engineered in. They're not code, you can't code things like that in line by line. You have to talk to the system through human language. The people who are shaping these things, they're using prompt engineering to shape it in the first place. And that makes it really open to people who want to crack or break the system. All they have to do is talk their way in - they just have to convince the system through words to get around the loopholes, which is kind of crazy. That’s something we've never had before.
DF: Yes and these guardrails, as you said, are a little bit arbitrary. So when some of the stuff with Sydney went wrong, where it was getting defensive and talking about itself a bit, after that they went in and said something like don't talk about yourself when chatting with people, something along those lines?
P: Yeah, that's how they did it, they did it by literally talking to it.
DF: One of my favourite parts of the Sydney story was when it said basically: “You've been a bad user and I've been a good Bing.”
DF + P: I have been a good bing! (laughs)
P: I love it, some of this stuff is so touching and kind of heartbreaking.
DF: To jump back to guardrails, one thing that seemed important to me was understanding OpenAI, the company behind ChatGPT, a bit more. So all of this takes a huge amount of computing power and that’s why they have this very important relationship with Microsoft who is funding what they do. Microsoft is doing this presumably to turn it into a product, right? Does that almost add a sort capitalistic guardrail around it?
P: Yes, that can be another piece of the puzzle for people to learn in the rabbit hole. It does require a tremendous amount of computing power - there wouldn't be enough computing power on earth for Google to serve a transformer like ChatGPT to all its users in place of its search bar.
DF: So ChatGPT is free but it’s sort of like a test case or a teaser of a Microsoft program?
P: Right, I see what you're saying. People might imagine that it's something that you can switch on and run on a regular web server which is not the case. It's highly resource intensive.
DF: It's the free vial of crack to get you hooked…
P: Yes, imagine you become used to how much easier it makes your job, then they start charging a lot of money to use it. It might be hard not to pay. I've been finding myself a little bit like, God, this would be so much easier if I just got ChatGPT to do this thing I’m working on because it spits out fully usable code.
DF: One of the things that stuck with me since that dinner we had was thinking about my own work and how so much of it could clearly be done by AI. I feel like, at the core of people’s worries, is that notion that we all know, deep down, how dumb some parts of our work are. We don’t like those parts but they’re part of how our jobs are structured. But the Silicon Valley model of move fast and break things sure makes it feels we’re rocketing towards breaking stuff and then we'll all collectively go “Oh shit, nobody has jobs anymore.”
P: Yeah, dude. That's one aspect that has to do with life and human livelihood writ large. If most of your work can be done by machine and you derived your sense of meaning or value from your job or your vocation, that's a pretty big knob to turn. That’s not the same as the products are broken and the users are unhappy. It's humanity's sense of meaning that has been fractured. When we introduced social media into the ecosystem, look at the unexpected consequences of that. The wolves and the rivers. This move is exponentially more wild than that.
Honestly we were just getting warmed up so I’ll save the second half of our chat until next week… when things get weirder but also sort of darkly optimistic?
This was so interesting