{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "* 本例展示如何在alpha-mind中使用机器学习模型\n",
    "\n",
    "* 请在环境变量中设置`DB_URI`指向数据库"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "%matplotlib inline\n",
    "\n",
    "import os\n",
    "import datetime as dt\n",
    "import numpy as np\n",
    "import pandas as pd\n",
    "from alphamind.api import *\n",
    "from PyFin.api import *"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 数据配置\n",
    "------------"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "freq = '60b'\n",
    "universe = Universe('zz800')\n",
    "batch = 1\n",
    "neutralized_risk = industry_styles\n",
    "risk_model = 'short'\n",
    "pre_process = [winsorize_normal, standardize]\n",
    "post_process = [standardize]\n",
    "warm_start = 3\n",
    "data_source = os.environ['DB_URI']\n",
    "horizon = map_freq(freq)\n",
    "\n",
    "engine = SqlEngine(data_source)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "我们使用当期的`roe_q`因子，来尝试预测未来大概一个月以后的`roe_q`因子。\n",
    "\n",
    "* 训练的股票池为`zz800`;；\n",
    "* 因子都经过中性化以及标准化等预处理；\n",
    "* 对于线性模型，我们以20个工作日为一个时间间隔，用过去4期的数据作为训练用特征。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "fit_intercept = True\n",
    "kernal_feature = 'roe_q'\n",
    "regress_features = {kernal_feature: LAST(kernal_feature),\n",
    "                    kernal_feature + '_l1': SHIFT(kernal_feature, 1),\n",
    "                    kernal_feature + '_l2': SHIFT(kernal_feature, 2),\n",
    "                    kernal_feature + '_l3': SHIFT(kernal_feature, 3)\n",
    "                   }\n",
    "const_features = {kernal_feature: LAST(kernal_feature)}\n",
    "fit_target = [kernal_feature]\n",
    "\n",
    "data_meta = DataMeta(freq=freq,\n",
    "                     universe=universe,\n",
    "                     batch=batch,\n",
    "                     neutralized_risk=neutralized_risk,\n",
    "                     risk_model=risk_model,\n",
    "                     pre_process=pre_process,\n",
    "                     post_process=post_process,\n",
    "                     warm_start=warm_start,\n",
    "                     data_source=data_source)\n",
    "\n",
    "alpha_model = LinearRegression(features=regress_features, fit_intercept=True, fit_target=fit_target)\n",
    "composer = Composer(alpha_model=alpha_model, data_meta=data_meta)\n",
    "\n",
    "start_date = '2011-01-01'\n",
    "end_date = '2018-01-01'"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 指标与时间序列相关性分析\n",
    "--------------\n",
    "\n",
    "在本段中，我们要尝试回答如下的问题：**对于考察的指标，是当期指标与下期指标哪个对于下期收益的预测性更好？**\n",
    "\n",
    "这里，假设下期日期为`T`，考察周期为`M`，则：\n",
    "\n",
    "* 当期指标为，`T - M`时的指标值；\n",
    "* 下期指标为，`T`时的指标值；\n",
    "* 下期收益为，`T - M`至`T`时的区间收益；\n",
    "* 我们同时检验了下期数据与当期数据差值与下期收益的相关性。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "data_package1 = fetch_data_package(engine,\n",
    "                                   alpha_factors=[kernal_feature],\n",
    "                                   start_date=start_date,\n",
    "                                   end_date=end_date,\n",
    "                                   frequency=freq,\n",
    "                                   universe=universe,\n",
    "                                   benchmark=906,\n",
    "                                   warm_start=warm_start,\n",
    "                                   batch=batch,\n",
    "                                   neutralized_risk=neutralized_risk,\n",
    "                                   pre_process=pre_process,\n",
    "                                   post_process=post_process,\n",
    "                                   fit_target=fit_target)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "t_m_factor = data_package1['predict']['x']\n",
    "t_factor = data_package1['predict']['y']\n",
    "codes_list1 = data_package1['predict']['code']"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "data_package2 = fetch_data_package(engine,\n",
    "                                   alpha_factors=[kernal_feature],\n",
    "                                   start_date=start_date,\n",
    "                                   end_date=end_date,\n",
    "                                   frequency=freq,\n",
    "                                   universe=universe,\n",
    "                                   benchmark=906,\n",
    "                                   warm_start=warm_start,\n",
    "                                   batch=1,\n",
    "                                   neutralized_risk=neutralized_risk,\n",
    "                                   pre_process=pre_process,\n",
    "                                   post_process=post_process)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "t_return = data_package2['predict']['y']\n",
    "codes_list2 = data_package2['predict']['code']"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "total_data = pd.DataFrame(columns=['dx', 'T-M', 'T', 'Δ'])\n",
    "\n",
    "for key in t_m_factor.keys():\n",
    "    t_m = t_m_factor[key].values.flatten()\n",
    "    t = t_factor[key].flatten()\n",
    "    ret = t_return[key].flatten()\n",
    "    \n",
    "    df1 = pd.DataFrame({'T-M': t_m, 'T': t}, index=codes_list1[key])\n",
    "    df2 = pd.DataFrame({'dx': ret}, index=codes_list2[key])\n",
    "    \n",
    "    df = pd.merge(df1, df2, left_index=True, right_index=True)\n",
    "    df['Δ'] = df['T'] - df['T-M']\n",
    "    total_data.loc[key, ['dx', 'T-M', 'T', 'Δ']] = df.corr().loc['dx'][['dx', 'T-M', 'T', 'Δ']]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "total_data.mean()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 模型预测\n",
    "-----------"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "ref_date = '2017-01-31'\n",
    "ref_date = adjustDateByCalendar('china.sse', ref_date).strftime('%Y-%m-%d')\n",
    "composer.train(ref_date)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "print(\"Testing IC: {0:.4f}\".format(composer.ic(ref_date=ref_date)))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 模型对比 （线性回归模型 v.s. Naive - 常数线性模型）\n",
    "------------------"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "const_model = ConstLinearModel(features=const_features, weights={kernal_feature: 1.}, fit_target=fit_target)\n",
    "regression_model = LinearRegression(features=regress_features, fit_intercept=fit_intercept, fit_target=fit_target)\n",
    "\n",
    "const_composer = Composer(alpha_model=const_model, data_meta=data_meta)\n",
    "regression_composer = Composer(alpha_model=regression_model, data_meta=data_meta)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "const_composer.train(ref_date)\n",
    "regression_composer.train(ref_date)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "print(\"\\nConst. Testing IC: {0:.4f}\".format(const_composer.ic(ref_date=ref_date)))\n",
    "print(\"Regression Testing IC: {0:.4f}\".format(regression_composer.ic(ref_date=ref_date)))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 模型时间序列\n",
    "-------------------------"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "通过比较在测试集的结果，我们观察如下两个模型的表现：\n",
    "\n",
    "* Naive Model：简单的使用因子上期值作为当期值的预测；\n",
    "* 线性回归模型：利用过去四期的因子值回归后得到模型，然后用这个模型预测当期值；"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "model_dates = [d.strftime('%Y-%m-%d') for d in list(data_package2['predict']['x'].keys())]\n",
    "\n",
    "model_df = pd.DataFrame(columns=['naive', 'regress', 'naive ic.', 'regress ic.'])\n",
    "\n",
    "for ref_date in model_dates:\n",
    "    const_composer.train(ref_date)\n",
    "    regression_composer.train(ref_date)\n",
    "    \n",
    "    model_df.loc[ref_date, 'naive'] = const_composer[ref_date]\n",
    "    model_df.loc[ref_date, 'regress'] = regression_composer[ref_date]\n",
    "    model_df.loc[ref_date, 'naive ic.'] = const_composer.ic(ref_date=ref_date)\n",
    "    model_df.loc[ref_date, 'regress ic.'] = regression_composer.ic(ref_date=ref_date)\n",
    "    print(\"\\n{1} Const. Testing IC: {0:.4f}\".format(model_df.loc[ref_date, 'naive ic.'], ref_date))\n",
    "    print(\"{1} Regression Testing IC: {0:.4f}\".format( model_df.loc[ref_date, 'regress ic.'], ref_date))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "model_df[['naive ic.', 'regress ic.']].agg(['mean', 'std'])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "在这个例子中，线性回归模型的IC值略微高于Naive模型。"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 回测（ simple long short strategy）\n",
    "--------------------------"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "industry_name = 'sw_adj'\n",
    "industry_level = 1\n",
    "\n",
    "industry_names = industry_list(industry_name, industry_level)\n",
    "industry_total = engine.fetch_industry_matrix_range(universe, dates=model_dates, category=industry_name, level=industry_level)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "rets1 = []\n",
    "rets2 = []\n",
    "\n",
    "for i, ref_date in enumerate(model_dates):\n",
    "    py_ref_date = dt.datetime.strptime(ref_date, '%Y-%m-%d')\n",
    "    industry_matrix = industry_total[industry_total.trade_date == ref_date]\n",
    "    dx_returns = pd.DataFrame({'dx': data_package2['predict']['y'][py_ref_date].flatten(),\n",
    "                               'code': data_package2['predict']['code'][py_ref_date].flatten()})\n",
    "    \n",
    "    res = pd.merge(dx_returns, industry_matrix, on=['code']).dropna()\n",
    "    codes = res.code.values.tolist()\n",
    "    \n",
    "    alpha_logger.info('{0} full re-balance: {1}'.format(ref_date, len(codes)))\n",
    "    \n",
    "    ## naive model\n",
    "    \n",
    "    raw_predict1 = const_composer.predict(ref_date).loc[codes]\n",
    "    er1 = raw_predict1.fillna(raw_predict1.median()).values\n",
    "    \n",
    "    target_pos1, _ = er_portfolio_analysis(er1,\n",
    "                                           res.industry_name.values,\n",
    "                                           None,\n",
    "                                           None,\n",
    "                                           False,\n",
    "                                           None,\n",
    "                                           method='ls')\n",
    "        \n",
    "    target_pos1['code'] = codes\n",
    "    result1 = pd.merge(target_pos1, dx_returns, on=['code'])\n",
    "    ret1 = result1.weight.values @ (np.exp(result1.dx.values) - 1.)\n",
    "    rets1.append(np.log(1. + ret1))\n",
    "\n",
    "    ## regression model\n",
    "    \n",
    "    raw_predict2 = regression_composer.predict(ref_date).loc[codes]\n",
    "    er2 = raw_predict2.fillna(raw_predict2.median()).values\n",
    "    \n",
    "    target_pos2, _ = er_portfolio_analysis(er2,\n",
    "                                           res.industry_name.values,\n",
    "                                           None,\n",
    "                                           None,\n",
    "                                           False,\n",
    "                                           None,\n",
    "                                           method='ls')\n",
    "    \n",
    "    target_pos2['code'] = codes\n",
    "    result2 = pd.merge(target_pos2, dx_returns, on=['code'])\n",
    "    ret2 = result2.weight.values @ (np.exp(result2.dx.values) - 1.)\n",
    "    rets2.append(np.log(1. + ret2))\n",
    "    ## perfect forcast\n",
    "    \n",
    "    alpha_logger.info('{0} is finished'.format(ref_date))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "ret_df = pd.DataFrame({'naive': rets1, 'regress': rets2}, index=model_dates)\n",
    "ret_df.loc[advanceDateByCalendar('china.sse', model_dates[-1], freq).strftime('%Y-%m-%d')] = 0.\n",
    "ret_df = ret_df.shift(1)\n",
    "ret_df.iloc[0] = 0.\n",
    "\n",
    "ret_df[['naive', 'regress']].cumsum().plot(figsize=(12, 6),\n",
    "                                             title='Fixed freq rebalanced: {0}'.format(freq))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.5"
  },
  "varInspector": {
   "cols": {
    "lenName": 16,
    "lenType": 16,
    "lenVar": 40
   },
   "kernels_config": {
    "python": {
     "delete_cmd_postfix": "",
     "delete_cmd_prefix": "del ",
     "library": "var_list.py",
     "varRefreshCmd": "print(var_dic_list())"
    },
    "r": {
     "delete_cmd_postfix": ") ",
     "delete_cmd_prefix": "rm(",
     "library": "var_list.r",
     "varRefreshCmd": "cat(var_dic_list()) "
    }
   },
   "types_to_exclude": [
    "module",
    "function",
    "builtin_function_or_method",
    "instance",
    "_Feature"
   ],
   "window_display": false
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}
