Trained on 70 records (10 records on average across 7 subjects)
So I finally got a decent pipeline going with this gpt-neo finetuning. A lot of trial and error with the learning_rates.
Using gpt-neo tokenizer, I built a faiss index of quotes and provide this as context to the very minimally trained LLM (atm just trained on a a sample size of 20 across ~6 different datasets, this is continuing pre-training of gpt-neo which is based on the pile)
I will randomly parse out 7 quotes from top k 50 as context for the model to respond to, and these are some of the responses I got when asked ‘What meaning can be found in life?’
Here are some of the responses
Response: It’s not like you can do anything to it. If you love a piece of text, you should love yourself. Otherwise, it’s impossible to love the text of another. But you have to learn to understand what you are saying to do. The text is the voice that brings us together. A life of love, which is all that comes before. You should not, therefore, expect to be able to live that life for yourself but should, at the same time, be in awe of the life that you may be living.
Response: Life is a series of events that happen in a given time. Events are the result of the interaction of two or more individuals. The events are not the same. Life can only be defined as the process of making a new connection with another person. Response: There is one quality to life that is worth knowing. I love you
Response: What does a person do when they are faced with an obstacle or challenge from their pasts?
My synthesis of the responses:
so in this case, I guess it’s reflecting on the meaning of life is defined by how you respond to it. The meaning of life is experienced as the defining decisions made during trials and tribulations.
I was thinking about how GAN’s have been used to create real life looking images and thought… we have chatbot’s that sound realistic enough (chatGPT). Couldn’t GAN’s be used to develop a conversation system (chatbot for lack of better word) that was validated by a GAN in much the same way as GAN’s are used to generate realistic images? The engine would just keep correcting itself until it passed the GAN’s thresholds.