# OpenAI’s massive new NLP model

**URL:** <https://discourse.numenta.org/t/openai-s-massive-new-nlp-model/5449>\
**Category:** Lounge\
**Tags:** nlp, deep-learning, openai\
**Created:** [February 15, 2019, 3:26pm UTC](https://discourse.numenta.org/t/openai-s-massive-new-nlp-model/5449 "2019-02-15T15:26:54Z")\
**Posts on this page:** 14\
**Page:** 1

<div class="post-metadata">

**Author:** ![rhyolight](https://yyz2.discourse-cdn.com/flex030/user_avatar/discourse.numenta.org/rhyolight/32/3922_2.png) [@rhyolight](https://discourse.numenta.org/u/rhyolight)\
**Post date:** [February 15, 2019, 3:26pm UTC](https://discourse.numenta.org/t/openai-s-massive-new-nlp-model/5449/1 "2019-02-15T15:26:54Z")

</div>

This came out yesterday. I’m surprised no one is talking about it yet. It is really impressive.

> **[Better Language Models and Their Implications](https://blog.openai.com/better-language-models/)**
>
> We’ve trained a large-scale unsupervised language model which generates coherent paragraphs of text, achieves state-of-the-art performance on many language modeling benchmarks, and performs rudimentary reading comprehension, machine translation,...

Basically you create an NLP DL network that is really really big, then you show it the entire internet. Now by handing it a blurb of text, it will generate more text with the same sentiment, and the text is extremely convincing (it reads like human text). That’s because it sampled huge amounts of human text and is putting it together from millions of sources (the model has 7.5 _Billion_ parameters).

Here is thoughtful take from @Carlos_Perez:

> **[Is Massively Unsupervised Language Learning Dangerous?](https://medium.com/intuitionmachine/is-massively-unsupervised-language-learning-dangerous-5b98a307cef)**
>
> Ilya Stuskever, who leads OpenAI research, has a simple theory about how to reach AGI. He argues that all that is needed is massively…

This is going to feed the idea that to get to AGI we just need to create bigger deep learning systems. But of course we know this is still just a clever probabilistic model.

The repurcussions of this weak AI technology is still staggering. Our world is changing fast.

---

<div class="post-metadata">

**Author:** ![Bitking](https://yyz2.discourse-cdn.com/flex030/user_avatar/discourse.numenta.org/bitking/32/3099_2.png) [@Bitking](https://discourse.numenta.org/u/Bitking)\
**Post date:** [February 15, 2019, 3:40pm UTC](https://discourse.numenta.org/t/openai-s-massive-new-nlp-model/5449/2 "2019-02-15T15:40:39Z")

</div>

The music that will come out of such a system will be top-40 pop.  
For that matter - top-40 pop already follows a formula to the point where nobody will notice when the AI starts writing it.

Much the same could be said of romance novels.

---

<div class="post-metadata">

**Author:** ![Cairo](https://yyz2.discourse-cdn.com/flex030/user_avatar/discourse.numenta.org/cairo/32/4989_2.png) [@Cairo](https://discourse.numenta.org/u/Cairo)\
**Post date:** [February 16, 2019, 10:17am UTC](https://discourse.numenta.org/t/openai-s-massive-new-nlp-model/5449/3 "2019-02-16T10:17:16Z")

</div>

Your comment reminded me of this @Bitking

---

<div class="post-metadata">

**Author:** ![Cairo](https://yyz2.discourse-cdn.com/flex030/user_avatar/discourse.numenta.org/cairo/32/4989_2.png) [@Cairo](https://discourse.numenta.org/u/Cairo)\
**Post date:** [February 16, 2019, 10:24am UTC](https://discourse.numenta.org/t/openai-s-massive-new-nlp-model/5449/4 "2019-02-16T10:24:10Z")

</div>

This model was the first time I have actually encountered a GELU. I hadn’t heard of it before a couple hours ago. It’s apparently existed since 2016, though I haven’t heard people talk about it in any of the AI discussions.

Has some pretty impressive benefits over RELU, although the computation I would imagine is still beneficial with RELU.

**GAUSSIAN ERROR LINEAR UNITS (GELUS)**

**_Dan Hendrycks∗_**  
**_University of California, Berkeley_**

**_Kevin Gimpel_**  
**_Toyota Technological Institute at Chicago_**

> **[1606.08415.pdf](https://arxiv.org/pdf/1606.08415.pdf)**
>
> 2.62 MB

---

<div class="post-metadata">

**Author:** ![Bitking](https://yyz2.discourse-cdn.com/flex030/user_avatar/discourse.numenta.org/bitking/32/3099_2.png) [@Bitking](https://discourse.numenta.org/u/Bitking)\
**Post date:** [February 16, 2019, 2:41pm UTC](https://discourse.numenta.org/t/openai-s-massive-new-nlp-model/5449/5 "2019-02-16T14:41:58Z")

</div>



---

<div class="post-metadata">

**Author:** ![MaxLee](https://yyz2.discourse-cdn.com/flex030/user_avatar/discourse.numenta.org/maxlee/32/5546_2.png) [@MaxLee](https://discourse.numenta.org/u/MaxLee)\
**Post date:** [February 16, 2019, 4:56pm UTC](https://discourse.numenta.org/t/openai-s-massive-new-nlp-model/5449/6 "2019-02-16T16:56:14Z")

</div>

> [@Bitking](#):
>
> Much the same could be said of romance novels.

I specifically turned down a job from someone wanting me to create a system just for that. I guess a system like OpenAI’s could probably be given an initially generated outline, then just fill in most of the details. Afterwards, it would just be an editing job, rather than a writing one.

Here’s an example from Axios that I saw just this morning. It’s pretty darn convincing, minus the fact that it’s behind on the current state faculty at the White House.

[https://www.axios.com/ai-wrote-this-story-b71270f8-dffb-434d-a5a6-d7ef0d791b0a.html](https://www.axios.com/ai-wrote-this-story-b71270f8-dffb-434d-a5a6-d7ef0d791b0a.html)

---

<div class="post-metadata">

**Author:** ![acatovic](https://yyz2.discourse-cdn.com/flex030/user_avatar/discourse.numenta.org/acatovic/32/4553_2.png) [@acatovic](https://discourse.numenta.org/u/acatovic)\
**Post date:** [February 16, 2019, 6:11pm UTC](https://discourse.numenta.org/t/openai-s-massive-new-nlp-model/5449/7 "2019-02-16T18:11:01Z")

</div>

It actually made massive headlines everywhere, even in mainstream newspapers in Australia (Sydney Morning Herald) which is unheard of! I don’t know if that’s because of clever marketing (e.g. stating on the blog that they will not release the model due to “concerns about malicious applications of the technology”) or because the language model is pretty darn impressive.

---

<div class="post-metadata">

**Author:** ![acatovic](https://yyz2.discourse-cdn.com/flex030/user_avatar/discourse.numenta.org/acatovic/32/4553_2.png) [@acatovic](https://discourse.numenta.org/u/acatovic)\
**Post date:** [February 16, 2019, 6:12pm UTC](https://discourse.numenta.org/t/openai-s-massive-new-nlp-model/5449/8 "2019-02-16T18:12:38Z")

</div>

And here’s the real pickle Matt - you cannot be 100% certain if this text is now being written by a person or by an algorithm 😛

---

<div class="post-metadata">

**Author:** ![Falco](https://avatars.discourse-cdn.com/v4/letter/f/8797f3/32.png) [@Falco](https://discourse.numenta.org/u/Falco)\
**Post date:** [February 17, 2019, 12:17am UTC](https://discourse.numenta.org/t/openai-s-massive-new-nlp-model/5449/9 "2019-02-17T00:17:45Z")

</div>

The real pickle will be when the AI generated posts outnumber the real posts a billion to 1.

---

<div class="post-metadata">

**Author:** ![Bitking](https://yyz2.discourse-cdn.com/flex030/user_avatar/discourse.numenta.org/bitking/32/3099_2.png) [@Bitking](https://discourse.numenta.org/u/Bitking)\
**Post date:** [February 17, 2019, 1:28am UTC](https://discourse.numenta.org/t/openai-s-massive-new-nlp-model/5449/10 "2019-02-17T01:28:19Z")

</div>

You will need an AI to sort through them!

---

<div class="post-metadata">

**Author:** ![rhyolight](https://yyz2.discourse-cdn.com/flex030/user_avatar/discourse.numenta.org/rhyolight/32/3922_2.png) [@rhyolight](https://discourse.numenta.org/u/rhyolight)\
**Post date:** [February 17, 2019, 1:39am UTC](https://discourse.numenta.org/t/openai-s-massive-new-nlp-model/5449/11 "2019-02-17T01:39:40Z")

</div>

And the nature of the new cyber warfare arena is revealing itself. This is serious stuff, and things are happening very fast. These skills are super valuable today, and they are going to stay there unless the system fails or we really do create something that can start creating AIs itself.

Our primary job as humans is to continue building the future and survive whatever comes next. We must remain vigilant and continue thinking about how these systems will shape the future of our planet. We must continue to think about how we can steer our common fate in a direction that includes humanity.

---

<div class="post-metadata">

**Author:** ![rhyolight](https://yyz2.discourse-cdn.com/flex030/user_avatar/discourse.numenta.org/rhyolight/32/3922_2.png) [@rhyolight](https://discourse.numenta.org/u/rhyolight)\
**Post date:** [February 17, 2019, 2:00am UTC](https://discourse.numenta.org/t/openai-s-massive-new-nlp-model/5449/12 "2019-02-17T02:00:32Z")

</div>

Too dark?

---

<div class="post-metadata">

**Author:** ![Bitking](https://yyz2.discourse-cdn.com/flex030/user_avatar/discourse.numenta.org/bitking/32/3099_2.png) [@Bitking](https://discourse.numenta.org/u/Bitking)\
**Post date:** [February 17, 2019, 8:00am UTC](https://discourse.numenta.org/t/openai-s-massive-new-nlp-model/5449/13 "2019-02-17T08:00:58Z")

</div>

In today’s reading:

> **[AI is reinventing the way we invent](https://www.technologyreview.com/s/612898/ai-is-reinventing-the-way-we-invent/)**
>
> The biggest impact of artificial intelligence will be to help humans make discoveries we couldn’t make on our own.

---

<div class="post-metadata">

**Author:** ![Falco](https://avatars.discourse-cdn.com/v4/letter/f/8797f3/32.png) [@Falco](https://discourse.numenta.org/u/Falco)\
**Post date:** [February 17, 2019, 12:04pm UTC](https://discourse.numenta.org/t/openai-s-massive-new-nlp-model/5449/14 "2019-02-17T12:04:11Z")

</div>

> [@rhyolight](#):
>
> Too dark?

Cato wouldn’t have put it better.

–  
Furthermore I think the Numenta website should have a merchandise section offering the Numenta cup.