{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "* 请在环境变量中设置`DB_URI`指向数据库\n",
    "* 请在环境变量中设置`DATAYES_TOKEN`作为通联数据登陆凭证"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "%matplotlib inline\n",
    "import os\n",
    "from matplotlib import pyplot as plt\n",
    "import uqer\n",
    "import numpy as np\n",
    "import pandas as pd\n",
    "from uqer import DataAPI as api\n",
    "from alphamind.api import *\n",
    "from alphamind.data.neutralize import neutralize\n",
    "\n",
    "plt.style.use('ggplot')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "_ = uqer.Client(token=os.environ['DATAYES_TOKEN'])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "ref_date = '2017-06-23'\n",
    "factor = 'EPS'\n",
    "\n",
    "engine = SqlEngine(os.environ['DB_URI'])\n",
    "universe = Universe('zz800')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Algorithm Description\n",
    "--------------------------"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "猜测的 ``neutralize`` 残差$\\bar Res$计算公式：\n",
    "\n",
    "$$\\bar Res_{i,k} = \\bar f_{i,k} - \\sum_j \\beta_{j,k} \\times \\bar Ex_{i, j, k}$$\n",
    "\n",
    "其中：$k$为行业分类，$i$为该行业中第$i$只股票，$j$为第$j$个风险因子。$\\bar f$为因子序列，$\\bar Ex$为风险暴露矩阵。系数$\\beta_{j,k}$由OLS确定。\n",
    "\n",
    "下面的章节，我们分别比较三种``neutralize``的方法差别：\n",
    "\n",
    "* **UQER Neutralize**\n",
    "\n",
    "    使用优矿的SDK计算因子残差。\n",
    "\n",
    "\n",
    "* **Alpha-Mind Neutralize**\n",
    "\n",
    "    使用alpha-mind计算因子残差，alpha-mind可以由以下地址安装：\n",
    "    \n",
    "    ```\n",
    "    https://github.com/wegamekinglc/alpha-mind\n",
    "    ```\n",
    "\n",
    "* **Direct Weighted Least Square Fit Implementation**\n",
    "\n",
    "    直接使用scikit-learn的线性回归功能来计算因子残差。\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Raw Data\n",
    "---------------------------"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "codes = engine.fetch_codes(ref_date, universe)\n",
    "factor_data = engine.fetch_factor(ref_date, factor, codes)\n",
    "risk_cov, risk_expousre = engine.fetch_risk_model(ref_date, codes)\n",
    "total_data = pd.merge(factor_data, risk_expousre, on=['code']).dropna()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "total_data['ticker'] = total_data.code.apply(lambda x: '{0:06}'.format(x))\n",
    "total_data.set_index('ticker', inplace=True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "len(total_data)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# UQER Neutralize\n",
    "-----------------------"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "%%timeit\n",
    "neutralized_factor_uqer = uqer.neutralize(total_data[factor],\n",
    "                                          target_date=ref_date.replace('-', ''),\n",
    "                                          industry_type='short')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "neutralized_factor_uqer = uqer.neutralize(total_data[factor],\n",
    "                                          target_date=ref_date.replace('-', ''),\n",
    "                                          industry_type='short')\n",
    "df = pd.DataFrame(neutralized_factor_uqer, columns=['uqer'])\n",
    "df.head(10)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "len(neutralized_factor_uqer)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "risk_exposure_uqer = uqer.DataAPI.RMExposureDayGet(tradeDate=ref_date.replace('-', '')).set_index('ticker')\n",
    "targeted_secs = risk_exposure_uqer.loc[neutralized_factor_uqer.index]\n",
    "\n",
    "style_exposure = neutralized_factor_uqer.values @ targeted_secs[risk_styles].values\n",
    "industry_exposure = neutralized_factor_uqer.values @ targeted_secs[industry_styles].values\n",
    "\n",
    "exposure = pd.Series(np.concatenate([style_exposure, industry_exposure]), index=risk_styles+industry_styles)\n",
    "exposure"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Alpha-Mind Neutralize\n",
    "--------------------------"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "x = total_data[risk_styles + industry_styles].values\n",
    "y = total_data[factor].values"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "%%timeit\n",
    "neutralized_factor_alphamind = neutralize(x, y, weights=np.ones(len(y)))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "neutralized_factor_alphamind = neutralize(x, y, weights=np.ones(len(y)))\n",
    "alphamind_series = pd.Series(neutralized_factor_alphamind.flatten(), index=total_data.index)\n",
    "df['alpha-mind'] = alphamind_series\n",
    "df.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "len(alphamind_series)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# The Ticker Missing in UQER but Still in Alpha-Mind\n",
    "-----------------------------------"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "missed_codes = [c for c in alphamind_series.index if c not in neutralized_factor_uqer.index]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "total_data.loc[missed_codes]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Direct Weighted Least Square Fit Implementation\n",
    "------------------------"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import statsmodels.api as sm"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "mod = sm.WLS(y, x, weights=np.ones(len(y))).fit()\n",
    "lg_series = pd.Series(mod.resid, index=total_data.index)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "df['ols'] = lg_series"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Comparison\n",
    "------------------"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "df['uqer - ols'] = df['uqer'] - df['ols']\n",
    "df['alphamind - ols'] = df['alpha-mind'] - df['ols']"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "df[['uqer - ols', 'alphamind - ols']].plot(figsize=(14, 7), ylim=(-1e-4, 1e-4))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "df.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.5"
  },
  "varInspector": {
   "cols": {
    "lenName": 16,
    "lenType": 16,
    "lenVar": 40
   },
   "kernels_config": {
    "python": {
     "delete_cmd_postfix": "",
     "delete_cmd_prefix": "del ",
     "library": "var_list.py",
     "varRefreshCmd": "print(var_dic_list())"
    },
    "r": {
     "delete_cmd_postfix": ") ",
     "delete_cmd_prefix": "rm(",
     "library": "var_list.r",
     "varRefreshCmd": "cat(var_dic_list()) "
    }
   },
   "types_to_exclude": [
    "module",
    "function",
    "builtin_function_or_method",
    "instance",
    "_Feature"
   ],
   "window_display": false
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}
