For those of you that are youngsters out there seeing this article now, a little context is perhaps appropriate. I remember when this article came out, being a Wired subscriber at the time, as one did if they were in their 20's working at a "dotcom" and thinking this Internet thing was turning out pretty cool.
This article was viscerally and emotionally terrifying at the time. For whatever reason it seemed really real. I remember overhearing people talking about "grey goo" over cocktails at startup events -- the scenario here seemed real and imminent.
As usual, of course, as with most doomsday predictions, the actual future reality turned out to be equally terrifying and completely different.
I was 19 when this came out and this essay single-handedly changed my life and led me to Ray Kurzweil, Minsky, the Unabomber Manifesto, Naomi Klein, and then ultimately to spirituality through Hawking-trained-physicist-turned-cave-dwelling-monk Peter Russell's "Waking Up in Time."
The rest of my life quite literally flows from reading this essay over and over when it came out. Genuinely happy to see it here.
One of the things that keeps creeping into my head about sentient machines, the robot future etc etc is this: If they become sentient, and have extraordinary capabilities etc etc ...why on earth (no pun intended) would they stick around here? They certainly wouldn't be limited by the same things we biological beings are. They could just leave into the great expanse - poof, I'm outta here dad!
Furthermore, lets say, we become part of the machines (a la Greg Bear's EON / Way universe). The line of thinking and questioning starts to move towards resources. These incredible advances in machines, it seems would be accompanied by incredible advances in resource utilization. The notion of poor vs rich entities would be completely different, in relative terms, than what we think of them today.
Our insignificance relative to the machines might not matter in the slightest. fwiw - worth reading David Brin's uplift books - in those the machines seem to generally stay the heck away from bio life forms :)
This was a really big deal when I read it back then. It led me more seriously into robotics and ai and the discussion of the singularity and jobs. But Bill Joy's essay was not just waiting for the AI to become super-human he was aware that way before it would be dangerous to humans.
One of my favorite quotes from the article which really has informed a lot of my thinking later:
"Clarke continued: "Looking into my often cloudy crystal ball, I suspect that a total defense might indeed be possible in a century or so. But the technology involved would produce, as a by-product, weapons so terrible that no one would bother with anything as primitive as ballistic missiles."
I'd say the author's own work stands in opposition to his conclusion: vi is over 40 years old at this point and hasn't yet been rendered obsolete.
I've worked with Bill and even shared an office with him at Sun (although he was rarely there, and to this day probably just remembers me as that guy that drank a lot of Diet Dr. Pepper and argued about capabilities for the Java language :-) and his greatest strength and greatest weakness is that he can see too far ahead along a path. He would answer emails that I thought I had completely thought through with a one liner that would illuminate some fault in my logic. It was annoying and amazing at the same time.
In 1994 he was convinced that the kinds of "compromises" that James Gosling were putting into Java guaranteed it would be dead on arrival. He wasn't wrong, those choices would ultimately limit the language (and they have) but he completely missed the 20 years between then and now where Java would have a huge impact.
When this editorial came out I had moved on from Sun and was dealing the leading edge of what would be the dot.com implosion shockwave and now Bill was telling us it was all pointless, the world would probably die on its own desire to create cool new things. Well he wasn't saying it was pointless per se, he was saying we needed to confront the ethics of what we were doing now instead of in the middle of the crisis. And there is much to like about that, but recall that Facebook was created in a dorm room, not a laboratory like Bell Labs or Sun Labs. So there was no oversight, no 'adult supervision' of people who would ask, as Bill would have, what happens when ...?
So to understand Bill's essay in context I have to ask, "What would he have said to Mark Zuckerberg?" I don't doubt for a moment that had Mark confided in him his vision and his plans, that Bill would have foreseen the size and extent of its impact. Bill is a guy who made more money on Microsoft Stock than on Sun Stock because he sold the latter and bought the former, recognizing that at the end of the day Microsoft would have a larger impact. So what does he do? Does he convince Mark to throw it away? Does he say "You will be one of the richest people in the world but you'll have created a tool that nation states will use to undermine democracies around the world?" And how does Mark respond to that? Probably, "If not me, someone else will figure this out. Look at myspace.com, I'll take the money and figure out the rest after it becomes a problem."
The future doesn't need us, and neither does the present. It is the ultimate hubris of humans from the beginning of time that they are somehow "more special" than the rest of the machine that is the universe. When you read books like "The Vital Question" you might be struck that humans are just a 'step in the path' rather than the starting or ending point of that path. You can imagine self aware machines arguing over the notion that they evolved from meat.
The power of Bill Joy for me has always been his willingness to say something outrageous that was the logical extension of a path through the point of absurdity. And in that moment stretching the pre-conceptions of the people hearing him such that they were able to think of something new that previously they would not allow themselves to think it. I've felt it first hand and seen it in happen in others. The after the meeting discussion that goes "That was the craziest thing I think I've ever heard, but something that might not be crazy is if we did this ..."
Kaczynski has written two books since his incarceration: TECHNOLOGICAL SLAVERY (2010) and ANTI-TECH REVOLUTION: WHY AND HOW (2016)
His essential points, according to the preface to Technological Slavery, are:
1) Technological progress is carrying us to inevitable disaster. … 2) Only the collapse of modern technological civilization can avert disaster. … 3) The political left is technological society’s first line of defense against revolution. … 4) What is needed is a new revolutionary movement, dedicated to the elimination of technological society. …
I read Ray Kurzweil's The Age of Spiritual Machines when it came out and although it's low on specifics, the abstract concepts are solid. The main one IMHO being that once a chip can emulate the inputs and outputs of a neuron, then a human brain could slowly be replaced by silicon and the person wouldn't even be aware of when he or she became a machine.
We used to say that artificial general intelligence (AGI) was 20 or 30 years away. Now it's more like 10. The main thing stopping it is the utter lack of exploration in computer science of highly parallelized/concurrent processing. So hardware-wise that means we need a DSP with perhaps a million cores with local memory (like map reduce), this would be like a video card but with general-purpose cores that can run any traditional language instead of OpenCL/CUDA etc. The software side means that we need more experience with languages like MATLAB/Octave, Elixer, Erlang, Go etc so that we can build other tools like TensorFlow on top of them from first principles.
Given those two foundations, many of the methods in neural nets, genetic algorithms, etc become one-liners and kids could play around with them the way we might have written ray casters and fractal explorers when we were young. They'll quickly discover novel ways of solving problems, combining intelligent agents in various layers, automatically assigning hyperparameters, basically all the things we struggle with today. And not long after that, there will be a repo on GitHub where you can download a brute force AGI and see how fast it can do your homework.
This seems completely inevitable to me and I would have loved to have made a career of it but now there's no time. Any idea you can think of is between 2 weeks and 2 years away from being manifested by someone on the web. Which is why I think we're looking at 5 or 10 advances over the next decade that will make AGI all but certain (barring a global recession like the dot-bomb followed by an embrace of fear like the global war on terror that set tech advances back 10-15 years, but I digress).
This is a good thread for: http://marshallbrain.com/manna1.htm
Important reading for these topics.
"the robotic takeover did not start at MIT, NASA, Microsoft or Ford. It started at a Burger-G restaurant in Cary, NC"
> These engineered human beings may be happy in such a society, but they will most certainly not be free. They will have been reduced to the status of domestic animals.
Well, if the system is advanced enough that its actions become incomprehensible to humans, then it's a moot point. After all, nobody seems to go around questioning free will because of earthquakes.
The dangers seem to me to be from systems that aren't anywhere as sophisticated. Those little pyscho-dogs from Black Mirror are sufficiently scary. And they are much less likely to have read Sojourner Truth than a sentient cloud that keeps us as pets.
"Anti-Tech Revolution: Why and How is Kaczynski’s well-reasoned, cohesive composition about how revolutionary groups should approach our mercurial future….. I recommend that you read this compelling perspective on how we can frame our struggles in a technological society." -- The Tech, MIT's oldest and largest newspaper
I would be interested to hear people's opinions on Jini https://en.wikipedia.org/wiki/Jini (Apache River) and its influence.
I recall reading this too. It's apt to think now how gracious we've been afforded that software innovation has not kept up with hardware innovation. While computers have got relatively that much more powerful as he states, they are still performing maybe a little more than they were in the 90's. One day I think one of these dystopian realities will happen but I like to think we as humans are actually just really stupid, and that an army of super smart machines if they do come into existence will only happen once we actually stop killing each other.
I was interviewing for my first real job when this essay came out and they took the applicants to lunch. (The firm had a big name in consulting.)
The interviewer brought up this article and how scared he was an I told him he really should not worry about the AI part though maybe we should worry about the biological part.
He did not appreciate that.
It's weird that we could run a "Where were you when Bill Joy's article came out."
It's interesting in that he was right about when Moore's law would end. Maybe off by a couple years, but pretty dead on.
"The Machine Stops" E.M. Forster (1909)
For some perspective on the other side, check out http://idlewords.com/talks/superintelligence.htm.
I agree with the general thrust of much of this piece (and was shocked to read that he essentially classified these types of things as WMDs, just as I did for mass hacks of autonomous systems) but I will disagree with one main point.
I see no reason that humanity is worth saving over some other, better form of life. I'm not saying its ethical to mow down innocents to make way for the next generation, but some humans are pretty fucking awful. If our descendants are robots with the intelligence of Hawking and are full of kindness and grace then why should we insist that humanity exist forever? Even if we extinguished some of our evil impulses, like genocide and rape, what makes us better? My late aunt had Down syndrome but if I could invent a vaccine that stopped it from ever happening again I would even if I never harm someone with Down syndrome.
The best counterargument I can come up with is essentially the robots asking "What else is there to do?"
In some post-singularity paradise the robots can be heading in all directions at near the speed of light, but once they've figured out this universe's physics and ended armed conflict then it stands to reason that they'd end up creating or simulating places that did not have god-like creatures.
I think we might depend on the resource constraints and conflict to give our lives meaning. Kinda a riff on Eden's "knowledge of good and evil" a creature's intelligence and wisdom may depend on a universe of consequence.
For anyone stuck behind the paywall (PDF warning): https://www.cc.gatech.edu/computing/nano/documents/Joy%20-%2...