{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "* 使用行业内的排序，进行因子测试；与回归版本，以及原始因子值版本进行比较。本部分参考自《QEPM》 p.p 117\n",
    "* 请在环境变量中设置`DB_URI`指向数据库"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 参数设定\n",
    "--------------"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "%matplotlib inline\n",
    "import os\n",
    "import pandas as pd\n",
    "import numpy as np\n",
    "from PyFin.api import *\n",
    "from alphamind.api import *\n",
    "\n",
    "factor = 'CFO2EV'\n",
    "universe = Universe('zz800')\n",
    "start_date = '2010-01-01'\n",
    "end_date = '2018-04-26'\n",
    "freq = '20b'\n",
    "category = 'sw_adj'\n",
    "level = 1\n",
    "horizon = map_freq(freq)\n",
    "\n",
    "engine = SqlEngine(os.environ['DB_URI'])\n",
    "\n",
    "ref_dates = makeSchedule(start_date, end_date, freq, 'china.sse')\n",
    "sample_date = '2018-01-04'\n",
    "sample_codes = engine.fetch_codes(sample_date, universe)\n",
    "\n",
    "sample_industry = engine.fetch_industry(sample_date, sample_codes, category=category, level=level)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "sample_industry.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 样例因子\n",
    "--------------------\n",
    "\n",
    "我们下面分三种方法，分别考查这几种方法在避免行业集中上面的效果：\n",
    "\n",
    "* 使用原始因子的排序；\n",
    "* 使用原始因子在行业内的排序；\n",
    "* 使用原始因子在行业哑变量上回归后得到的残差排序。"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 1. 原始因子排序\n",
    "--------------------"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "factor1 = {'f1': CSQuantiles(factor)}\n",
    "sample_factor1 = engine.fetch_factor(sample_date, factor1, sample_codes)\n",
    "sample_factor1 = pd.merge(sample_factor1, sample_industry[['code', 'industry']], on='code')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "sample_factor1.sort_values('f1', ascending=False).head(15)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "对于原始因子，如果我们不做任何行业上面的处理，发现我们选定的alpha因子`CFO2EV`较大的股票集中于银行和大金融板块。"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 2. 行业内排序因子\n",
    "----------------------"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "这里我们使用调整后的申万行业分类作为行业标签："
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "factor2 = {'f2': CSQuantiles(factor, groups='sw1_adj')}\n",
    "sample_factor2 = engine.fetch_factor(sample_date, factor2, sample_codes)\n",
    "sample_factor2 = pd.merge(sample_factor2, sample_industry[['code', 'industry']], on='code')\n",
    "sample_factor2.sort_values('f2', ascending=False).head(15)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "使用行业内的排序，则行业分布会比较平均。"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 3. 使用回归将因子行业中性\n",
    "--------------------------------"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "还有一种思路，使用线性回归，以行业为哑变量，使用回归后的残差作为因子的替代值，做到行业中性："
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "factor3 = {'f3': factor}\n",
    "sample_factor3 = engine.fetch_factor(sample_date, factor3, sample_codes)\n",
    "risk_cov, risk_exp = engine.fetch_risk_model(sample_date, sample_codes)\n",
    "sample_factor3 = pd.merge(sample_factor3, sample_industry[['code', 'industry']], on='code')\n",
    "sample_factor3 = pd.merge(sample_factor3, risk_exp, on='code')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "raw_factors = sample_factor3['f3'].values\n",
    "industry_exp = sample_factor3[industry_styles + ['COUNTRY']].values.astype(float)\n",
    "processed_values = factor_processing(raw_factors, pre_process=[], risk_factors=industry_exp, post_process=[percentile])\n",
    "sample_factor3['f3'] = processed_values"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "sample_factor3 = sample_factor3[['code', 'f3', 'industry']]\n",
    "sample_factor3.sort_values('f3', ascending=False).head(15)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "我们发现这种方法的效果并不是很好。调整的幅度并不是很大，同时仍然存在着集中于大金融板块的问题。"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 回测结果\n",
    "------------------"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "我们使用简单等权重做多前20%支股票，做空后20%的方法，考察三种方法的效果："
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "factors = {\n",
    "    'raw': CSQuantiles(factor),\n",
    "    'peer quantile': CSQuantiles(factor, groups='sw1'),\n",
    "    'risk neutral': LAST(factor)\n",
    "}"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "df_ret = pd.DataFrame(columns=['raw', 'peer quantile', 'risk neutral'])\n",
    "df_ic = pd.DataFrame(columns=['raw', 'peer quantile', 'risk neutral'])\n",
    "\n",
    "for date in ref_dates:\n",
    "    ref_date = date.strftime('%Y-%m-%d')\n",
    "    codes = engine.fetch_codes(ref_date, universe)\n",
    "\n",
    "    total_factor = engine.fetch_factor(ref_date, factors, codes)\n",
    "    risk_cov, risk_exp = engine.fetch_risk_model(ref_date, codes)\n",
    "    industry = engine.fetch_industry(ref_date, codes, category=category, level=level)\n",
    "    rets = engine.fetch_dx_return(ref_date, codes, horizon=horizon, offset=1)\n",
    "    total_factor = pd.merge(total_factor, industry[['code', 'industry']], on='code')\n",
    "    total_factor = pd.merge(total_factor, risk_exp, on='code')\n",
    "    total_factor = pd.merge(total_factor, rets, on='code').dropna()\n",
    "\n",
    "    raw_factors = total_factor['risk neutral'].values\n",
    "    industry_exp = total_factor[industry_styles + ['COUNTRY']].values.astype(float)\n",
    "    processed_values = factor_processing(raw_factors, pre_process=[], risk_factors=industry_exp, post_process=[percentile])\n",
    "    total_factor['risk neutral'] = processed_values\n",
    "\n",
    "    total_factor[['f1_d', 'f2_d', 'f3_d']] = (total_factor[['raw', 'peer quantile', 'risk neutral']] >= 0.8) * 1.\n",
    "    total_factor.loc[total_factor['raw'] <= 0.2, 'f1_d'] = -1.\n",
    "    total_factor.loc[total_factor['peer quantile'] <= 0.2, 'f2_d'] = -1.\n",
    "    total_factor.loc[total_factor['risk neutral'] <= 0.2, 'f3_d'] = -1.\n",
    "    total_factor[['f1_d', 'f2_d', 'f3_d']] /= np.abs(total_factor[['f1_d', 'f2_d', 'f3_d']]).sum(axis=0)\n",
    "\n",
    "    ret_values = total_factor.dx.values @ total_factor[['f1_d', 'f2_d', 'f3_d']].values\n",
    "    df_ret.loc[date] = ret_values\n",
    "    \n",
    "    ic_values = total_factor[['dx', 'raw', 'peer quantile', 'risk neutral']].corr().values[0, 1:]\n",
    "    df_ic.loc[date] = ic_values\n",
    "    print(f\"{date} is finished\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "df_ret.cumsum().plot(figsize=(14, 7))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "df_ic.cumsum().plot(figsize=(14, 7))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.5"
  },
  "varInspector": {
   "cols": {
    "lenName": 16,
    "lenType": 16,
    "lenVar": 40
   },
   "kernels_config": {
    "python": {
     "delete_cmd_postfix": "",
     "delete_cmd_prefix": "del ",
     "library": "var_list.py",
     "varRefreshCmd": "print(var_dic_list())"
    },
    "r": {
     "delete_cmd_postfix": ") ",
     "delete_cmd_prefix": "rm(",
     "library": "var_list.r",
     "varRefreshCmd": "cat(var_dic_list()) "
    }
   },
   "types_to_exclude": [
    "module",
    "function",
    "builtin_function_or_method",
    "instance",
    "_Feature"
   ],
   "window_display": false
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}
