I was thinking about how GAN’s have been used to create real life looking images and thought… we have chatbot’s that sound realistic enough (chatGPT). Couldn’t GAN’s be used to develop a conversation system (chatbot for lack of better word) that was validated by a GAN in much the same way as GAN’s are used to generate realistic images? The engine would just keep correcting itself until it passed the GAN’s thresholds.
#turing
Modified to include a rolling mean of the past 4 quarters return prior to inputting it into a rolling 4 period correlation
I came up with an ingenious way to find stable time series correlations.
Using panda’s rolling function.
10 years, reduce to quarterly data.
corr_set = [[df.pct_change(1).rolling(4).corr()]]
then derive 2 pair combinations of corr_set.columns
iterate over those pairs (this can take a long time, this is where clustering comes in handy)
For a pair’s given set of correlations (for each set of 4 quarters) (i.e. rolling windows of 4 quarters means any given point is a measure of a year’s worth of correlations), which is a single list (1 value for each date). then find the median from this list.
If the median correlation is critical. Then you know half the dataset has a solid correlation identified. You also know this measure is the median correlation of returns within any given year period
I also derived the percent each 4 period set had positive and negative correlations which was always above 50% for only one side of either negative or positive (all correlations converged on positive correlations). So I simply display the median correlation found.
This is over a 10 year period.
Ideas have a life of their own
Archetypes have a life of their own
Thoughts are what exist in our conscious mind
But they are sourced from prima materia, the external world, aka the collective unconscious
In the collective unconscious, the loadstones of thought exist as archetypes.
We evolved to understand/recieve them as thoughts and instincts
Something like that.
Thoughts still do exist, but as a subproduct (in programming we would say higher level language) of evolution interacting with archetypes
The whole no sex before marriage bit in Abrahamism
Its a patriachial sociological engineered function.
Women’s sexuality [, specifically that which] empowers a woman (i.e. sex as a commodity) is viewed as a threat to the existing male dominated hierarchy.
Rather, carefully selected archetype role models of women are propagated in the texts such as Esther, Delilah, Mary Magdalena, Virgin Mary, etc.
All have specific archetypes they exhibit which in turn shape [unconsciously] the gender roles promoted for [abrahamic] adherants.
Income High and Low based on best model picked. Clusters are based on cutoffs of prediction variable.
I use a special transform that normalizes the variable
The model vars are all significant.
The clusters more or less represent upper and low 25%, but this is a special transform that scales to a percent, but it’s based on mean, but it’s not a cumulative distribution function. It’s a minmax that does a **cumulative** sum (have to sort original values, and then map back, easier to do in python vs excel) using both polar sides of z-scores to scale to 0 (i.e. min z-score) to 50 (i.e. 0 z score) to 100%. (i.e. max z score).
Infant Mort
White
Crime
Doctors
Traf Deaths
University
Unemployed
Income
Population
R-squared (uncentered): 0.907
After using this transform method. Different results become prominent for university. The transform I’m using is non linear (also ZCA’d). It’s akin to an area under the curve calculation which allows for smoother cutoffs.
You can clearly see where the distinctions are.
Edward Eddinger
Ego and Archetype
Christ as Paradigm of the Individuating Ego
I myself have been saying autonomy and dealing with the loss of a friend who I projected my anima (that I identified as Hypatia) onto. This is very consoling to read.
“The end of this passage makes clear the purpose of inciting discord. It is to achieve the solitary condition, the state of being an autonomous individual. This can be achieved only by a separation from unconscious identification with others. In the early stages, the separatio is experienced as painful strife and hostility. Parents and family are the most frequent objects of unconscious identification. Jesus singles out the father for special mention:
And call no man your father on earth, for you have one Father who is in heaven.
Parents have power over their grown children only because the latter continue to project the images of the archetypal parents to their personal parents. To call no man father means to withdraw all projections of the father archetype and discover it within…
…’…if a man will lose his ego for my sake, he will find the Self.’ [paraphrasing Jesus]
…
‘It is no easy matter to live a life that is modelled on Christ’s, but it is unspeakably harder to live one’s own life, as truly as Christ lived his.’ [Jung]
…
…blessed are those who are aware of their spiritual poverty and are humbly seeking what they need. Understood psychologically, the meaning would be: The ego which is aware of its own emptiness of spirit (life meaning) is in a fortunate position because it is now open to the unconscious and has the possibility of experiencing the archetypal psyche (the kingdom of heaven).
Blessed are those who mourn, for they shall be comforted. Mourning is caused by the loss of an object or person who was carrying an important projected value. In order to withdraw projections and assimilate their content into one’s own personality it is necessary to experience the loss of the projection as a prelude to rediscovering the content or value within. Therefore, mourners are fortunate because they are involved in a growth process. They will be comforted when the lost projected value has been recovered within the pysche.”
I figured out how to whiten data using ZCA and convert back to nominal values all by “hand” (when I say by hand I mean by matrix multiplication)
This was important for me because I can’t use whitened data in multiple regression unless I can convert back from it.
https://github.com/thistleknot/matrixMultiplicationRegression/blob/master/whitening.R
I think the reason I hypothetically upset J***, C******, and T** was in their eyes they are the gatekeepers to everyone’s hope, an opportunity at full time.
So if I came in there and said I was ok with getting laid off cuz I’d still get my masters, was an insult to their opportunity which they viewed as the best they could offer anyone (exposure to CICD and C suite). They wanted me to integrate there with my masters.
I truly believe the reason everyone acted the way they did by turning a blind eye to the affair was its expected in Abrahamic patriarchy culture to objectify women. Even S***** acknowledged that by turning a blind eye to my return (this I didnt know at the time) because she exhumes that lifestyle. However with me I did feel guilty and was violating the patriarchy’s unspoken rule of objectification is expected and simultaneously violated the republican rule “keep it yourself” to scandal.
“Sex and the Office” says women caught sleeping their way to the top are halted generally once outted.
I think it’s okay to have sex in the workplace but the way its discusses imo is from a patriarchial sanctioned “don’t ask don’t tell” POV.
Which creates this anxiety in people about it and it remains very much a power move.
Selected Response: “that is absolutely true. But desperate times man… I’ve certainly been presented with the opportunity to do so, but don’t want to subjugate myself to whims of some dude on a power trip.”**
At that point all hell broke loose coupled with the fact I didn’t care for the opportunity* and was creating scandal with talk of dominatrix, I think that’s when the uppers gave their blessing to let those angry at me have at it. They did what is always done in criminal cases. Try to get former friends to turn on each other, let them do the dirty work for you. To get them to show they are hungry for an opportunity and willing to get their hands dirty. That’s my best guess.
*I thought they misinterpreted me, that I didn’t want it. I did its just that I was truly mentally ill from seeing her and it was hurting me psychologically with insomnia and mania daily. But they didn’t care. It violated their world view which expected me to be a player about it. But that’s the thing. I’m a soft heart (I can be mean, though usually unintentionally by usually just walking away from a relationship (ghosting), ironically, but lately I’ve tried to be more mature about it and am on the other end of the spectrum where I can’t walk away). Idk I just can’t be mean to someone I loved (I can be passive aggressive and make things uncomfortable though). I can’t block it out. That to me is being disingenuous to oneself and I feel corporate life expects you to swallow your emotions because they wanted to prove to everyone that emotions won’t get you paid
It’s kind of like saying women want to be respected but then you see classic objectifiers fucking the women you admire at work… I had a white knight savior complex only to be friend zoned and discarded which taught me that the paradigm I was using does not get me to the desired end goal (nice guys finish last) which was at least continued friendship so I had the affair and fulfilled the expectations of the very thing I didn’t like. Patriarchy. It was very traumatic for me in a way
It’s a turn on to be objectified but people want to use it in certain contexts that benefit them (cost benefit analysis)
**My friend confirmed it is a power move and some women think they have to sleep their way to the top
So power is attraction
I got the distinct impression she was doing it for power because people would tell me she got around, plus the fact that she was giving me personal time, yet when I naively told her I had feelings for her (after hanging out 1:1 multiple times). She started to distance herself. Like I was breaking some unspoken rule. I think she then classed me as a nice “married” guy who wasn’t going to make a move and lost interest cuz I obviously wasn’t a power move nor a Don Draper to her anymore
I think I touched the 3rd rail. Whoever obliged her power moves.
Attraction is power.
Sensitivity: 97.4%
Found another single variable that accounts for 97.4% of my samples diagnostic’s
End State Renal Disease
Cut off was .43
I modified my algorithm to find the cutoffs from the training partition vs the cross validation test partitions. I was still trying to solve for specificity, but alas, it converges on sensitivity.
I’m not overly worried about it. I can always recode the response variable and converge on sensitivity.
Specificity
Okay… so I tried or sensitivity and it converged on specificity.
The only thing I can think of is changing to training cutoffs vs cross validation test partitions for cutoff was it.
I use a function optimalCutoff and a var optimizeFor = Zeros or Ones depending on a flag at the beginning.
I also check for specificity or sensitivity on confusionMatrix output based on this flipped flag.
Anyways… if I flip this flag, it does either sensitivity or specificity. So that is working. Why it’s inverted from the default parameters… still not sure
But this IS better that it ALWAYS converging on sensitivity.
The reason for the slightly different results each pass is due to imputed variables I suspect and my static spss dataset which is an output of just one imputed set? I use the same seed (poor programmming practice I know, but data science is supposed to converge on the same results regardless of randomization, aka cross validation). In this case not so much the factors, but the classification scores (confusion matrix results).
Note: “3rd Try” is my specificity model (I coded the 1’s and 0’s backwards and mistook it for the true sensitivity model I was looking for)
An even better model
Sensitivity: 99.3%
Cutoff: .345035
Optimizing for cutoffs
I do not understand why. But when I tell R to test for specificity, I converge on sensitivity
The Answer is
All metrics are derived from test partitions from cross validation (to include cutoffs). I’m hitting the ball all right.
This is my 3rd try at that Medical Data problem and the ask was to test for sensitivity.
This is my final result
I hit the ball out of the park.
Problem with this: I forgot my 1’s were incorrectly coded for the class of non interest.
Optimized for sensitivity using cross validation 🙂
Code is saved on my private github. It was a lot of trial and error, but I got it.
Using guide here: http://www.sthda.com/english/articles/36-classification-methods-essentials/150-stepwise-logistic-regression-essentials-in-r/
I initially tried cross validation using this hash matrix, but it’s still a WIP
Fundamental weakness: not optimized for sensitivity. Bugs in code. 2 level variables shouldn’t be factors.
Answer
SEX2
Liver.cirrhosis2
Cough2
SOB2
INR
BAL.cytology2
BAL.cytology3
EBUS.TBNA..no.of.biopsies.
Bronchoscopy.findings..EBL.vs.No.EBL..2
I finished a work sample challenge for medical data
I even imputed data!
Yeah, I consider myself a data scientist
70% accuracy
Fundamental weakness: didn’t use factors
#DataScience
Data is based on 2007 Statistical Abstract of the United States
I’ve thoroughly analyzed my “best” cross validated model and further pruned the model using backwards stepwise regression on the final dataset from the cross validated term algorithm and did model diagnostics and highlighted in green variables that had positive residuals marked out and red for negative residuals (for further analysis)
This is the type of work I was thinking about publishing
This model is the “Income” model. I’ve included quadratic terms and got an amazing MAPE of 2.65% and an Adjusted R^2 of .97
I’ve noticed the cross validation pruned the hierarchical dependency (crime). I’m not sure what to make of that atm but I trust it knowing the MAPE was cross validated. I know I can exclude INTERACTION hierarchy dependencies (unsure about quadratic), but I know Crime is also captured in the interactions, so technically maybe that’s why it’s showing significant.
I’ve included studentized residuals and mapped it to cook’s distance which gives a great view of outlier’s.
I would give my stamp of approval on this model and say it passes the 4 model assumptions
State’s of Note
* Alaska
* Arizona
* Connecticut
* Illinois
* Lousiana
* California
* Maryland