Hello!
Here will be a playground for the following scenario: I take a quote from a book, introduce it to the GPT-J pretrained model of EleutherAI, through the GPT-J-6B pretrained model library and let the AI model continue the quote with its own text completition.
The result – we’ll see how creative this AI model can become based on the given quote.
We will be able to see if and how, and especially how far it can go from the continuation we can find in the book or article that was originally introduced as a starting point.
Will there be any major surprises? Can we expect in the more or less near future to have more cursive sequels of a given text, in a manner close to that which the original author had?
The question “can an AI be creative?” he will not find the answer here, but we can play, we can look for more answers that can confirm or infirm this.
GPT-J-6B is a 6 billion parameter, autoregressive text generation model trained on The Pile.
Code can be found at https://github.com/kingoflolz/mesh-transformer-jax/#gpt-j-6b
The model consists of 28 layers with a model dimension of 4096, and a feedforward dimension of 16384. The model dimension is split into 16 heads, each with a dimension of 256. Rotary position encodings (RoPE) was applied to 64 dimensions of each head. The model is trained with a tokenization vocabulary of 50257, using the same set of BPEs as GPT-2/GPT-3.
Let’s see how it goes so, keep close.