LSGOMYP
305 lines
8.9 KiB
Plaintext
305 lines
8.9 KiB
Plaintext
{
|
||
"cells": [
|
||
{
|
||
"cell_type": "markdown",
|
||
"metadata": {},
|
||
"source": [
|
||
"# Docker提交\n",
|
||
"\n",
|
||
"本次竞赛的Docker提交大致可以分为两小块：\n",
|
||
"\n",
|
||
"1. 线下文件准备好：包括DockerFile，代码，预测的代码；\n",
|
||
"2. Build同时pull提交\n",
|
||
"\n",
|
||
"如果之前没有提交过docker，可以根据这篇教程熟悉一下：https://tianchi.aliyun.com/forum/postDetail?spm=5176.12586969.1002.9.51df4127FoZKeL&postId=165595 \n",
|
||
"\n",
|
||
"\n",
|
||
"## 线下文件准备\n",
|
||
"\n",
|
||
"### Requirement \n",
|
||
"\n",
|
||
"- 运行代码所依赖的python库，缺什么就把需要装的文件放在requirement下面"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "code",
|
||
"execution_count": null,
|
||
"metadata": {},
|
||
"outputs": [],
|
||
"source": [
|
||
"numpy\n",
|
||
"tensorflow==2.2.0 "
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "markdown",
|
||
"metadata": {},
|
||
"source": [
|
||
"### 运行的代码\n",
|
||
"\n",
|
||
"#### 放在code下面即可"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "code",
|
||
"execution_count": null,
|
||
"metadata": {},
|
||
"outputs": [],
|
||
"source": [
|
||
"import tensorflow as tf\n",
|
||
"import tensorflow.keras.backend as K\n",
|
||
"from tensorflow.keras.layers import *\n",
|
||
"from tensorflow.keras.models import *\n",
|
||
"from tensorflow.keras.optimizers import *\n",
|
||
"from tensorflow.keras.callbacks import *\n",
|
||
"from tensorflow.keras.layers import Input \n",
|
||
"import numpy as np\n",
|
||
"import os\n",
|
||
"import zipfile\n",
|
||
"\n",
|
||
"def RMSE(y_true, y_pred):\n",
|
||
" return tf.sqrt(tf.reduce_mean(tf.square(y_true - y_pred)))\n",
|
||
"\n",
|
||
"def build_model(): \n",
|
||
" inp = Input(shape=(12,24,72,4)) \n",
|
||
" \n",
|
||
" x_4 = Dense(1, activation='relu')(inp) \n",
|
||
" x_3 = Dense(1, activation='relu')(tf.reshape(x_4,[-1,12,24,72]))\n",
|
||
" x_2 = Dense(1, activation='relu')(tf.reshape(x_3,[-1,12,24]))\n",
|
||
" x_1 = Dense(1, activation='relu')(tf.reshape(x_2,[-1,12]))\n",
|
||
" \n",
|
||
" x = Dense(64, activation='relu')(x_1) \n",
|
||
" x = Dropout(0.25)(x) \n",
|
||
" x = Dense(32, activation='relu')(x) \n",
|
||
" x = Dropout(0.25)(x) \n",
|
||
" output = Dense(24, activation='linear')(x) \n",
|
||
" model = Model(inputs=inp, outputs=output)\n",
|
||
"\n",
|
||
" adam = tf.optimizers.Adam(lr=1e-3,beta_1=0.99,beta_2 = 0.99) \n",
|
||
" model.compile(optimizer=adam, loss=RMSE)\n",
|
||
"\n",
|
||
" return model \n",
|
||
"\n",
|
||
"model = build_model()\n",
|
||
"model.load_weights('./user_data/model_data/model_mlp_baseline.h5')\n",
|
||
"\n",
|
||
"test_path = './tcdata/enso_round1_test_20210201/'\n",
|
||
"\n",
|
||
"### 1. 测试数据读取\n",
|
||
"files = os.listdir(test_path)\n",
|
||
"test_feas_dict = {}\n",
|
||
"for file in files:\n",
|
||
" test_feas_dict[file] = np.load(test_path + file)\n",
|
||
" \n",
|
||
"### 2. 结果预测\n",
|
||
"test_predicts_dict = {}\n",
|
||
"for file_name,val in test_feas_dict.items():\n",
|
||
" test_predicts_dict[file_name] = model.predict(val).reshape(-1,)\n",
|
||
"# test_predicts_dict[file_name] = model.predict(val.reshape([-1,12])[0,:])\n",
|
||
"\n",
|
||
"### 3.存储预测结果\n",
|
||
"for file_name,val in test_predicts_dict.items(): \n",
|
||
" np.save('./result/' + file_name,val)\n",
|
||
"\n",
|
||
"#打包目录为zip文件（未压缩）\n",
|
||
"def make_zip(source_dir='./result/', output_filename = 'result.zip'):\n",
|
||
" zipf = zipfile.ZipFile(output_filename, 'w')\n",
|
||
" pre_len = len(os.path.dirname(source_dir))\n",
|
||
" source_dirs = os.walk(source_dir)\n",
|
||
" print(source_dirs)\n",
|
||
" for parent, dirnames, filenames in source_dirs:\n",
|
||
" print(parent, dirnames)\n",
|
||
" for filename in filenames:\n",
|
||
" if '.npy' not in filename:\n",
|
||
" continue\n",
|
||
" pathfile = os.path.join(parent, filename)\n",
|
||
" arcname = pathfile[pre_len:].strip(os.path.sep) #相对路径\n",
|
||
" zipf.write(pathfile, arcname)\n",
|
||
" zipf.close()\n",
|
||
"make_zip() "
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "markdown",
|
||
"metadata": {},
|
||
"source": [
|
||
"#### run.sh\n",
|
||
"\n",
|
||
"- 运行预测的代码"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "code",
|
||
"execution_count": null,
|
||
"metadata": {},
|
||
"outputs": [],
|
||
"source": [
|
||
"#!/bin/sh\n",
|
||
"CURDIR=\"`dirname $0`\" #获取此脚本所在目录\n",
|
||
"echo $CURDIR\n",
|
||
"cd $CURDIR #切换到该脚本所在目录\n",
|
||
"python /code/mlp_predict.py"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "markdown",
|
||
"metadata": {},
|
||
"source": [
|
||
"### DockerFile"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "code",
|
||
"execution_count": null,
|
||
"metadata": {},
|
||
"outputs": [],
|
||
"source": [
|
||
"# Base Images\n",
|
||
"## 从天池基础镜像构建 \n",
|
||
"FROM registry.cn-shanghai.aliyuncs.com/tcc-public/tensorflow:latest-cuda10.0-py3\n",
|
||
"\n",
|
||
"## 把当前文件夹里的文件构建到镜像的根目录下（.后面有空格，不能直接跟/）\n",
|
||
"ADD . /\n",
|
||
"\n",
|
||
"## 指定默认工作目录为根目录（需要把run.sh和生成的结果文件都放在该文件夹下，提交后才能运行）\n",
|
||
"WORKDIR /\n",
|
||
"\n",
|
||
"## Install Requirements（requirements.txt包含python包的版本）\n",
|
||
"## 这里使用清华镜像加速安装\n",
|
||
"RUN pip install -i https://pypi.tuna.tsinghua.edu.cn/simple --upgrade pip\n",
|
||
"RUN pip install -i https://pypi.tuna.tsinghua.edu.cn/simple -r requirements.txt\n",
|
||
"#RUN pip install -i https://pypi.tuna.tsinghua.edu.cn/simple -r requirements.txt\n",
|
||
"\n",
|
||
"## 镜像启动后统一执行 sh run.sh\n",
|
||
"CMD [\"sh\", \"run.sh\"]"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "markdown",
|
||
"metadata": {},
|
||
"source": [
|
||
"### 其它\n",
|
||
"\n",
|
||
"- 按照官方要求把所需的文件全部按要求准备好即可。"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "markdown",
|
||
"metadata": {},
|
||
"source": [
|
||
"## 线上提交\n",
|
||
"\n",
|
||
"在所有的文件都准备之后，下面一步就是进行线上的提交，这里又分为三块。\n",
|
||
"\n",
|
||
"1. 按照要求进行线上配置\n",
|
||
"2. 进行build和pull；\n",
|
||
"3. 提交"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "markdown",
|
||
"metadata": {},
|
||
"source": [
|
||
"### 按照要求进行线上配置\n",
|
||
"\n",
|
||
"<img src=\"./pic/first_step.jpeg\" width = \"500\" height = \"200\" alt=\"first_step\" align=center />\n",
|
||
"\n",
|
||
"\\\\ \n",
|
||
"\n",
|
||
"<img src=\"./pic/second_step.jpeg\" width = \"500\" height = \"200\" alt=\"first_step\" align=center />\n",
|
||
"\n",
|
||
" "
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "markdown",
|
||
"metadata": {},
|
||
"source": [
|
||
"### 进行build和pull；\n",
|
||
"\n",
|
||
"\n",
|
||
"<img src=\"./pic/1st.jpeg\" width = \"400\" height = \"200\" alt=\"first_step\" align=center />\n",
|
||
"\\\\\n",
|
||
"<img src=\"./pic/2nd.jpeg\" width = \"400\" height = \"200\" alt=\"first_step\" align=center />\n",
|
||
"\\\\\n",
|
||
"<img src=\"./pic/3rd.jpeg\" width = \"400\" height = \"200\" alt=\"first_step\" align=center />\n",
|
||
"\n"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "code",
|
||
"execution_count": null,
|
||
"metadata": {},
|
||
"outputs": [],
|
||
"source": [
|
||
"#### 1.登录\n",
|
||
"sudo docker login --username=\"自己的用户名\" registry.cn-shenzhen.aliyuncs.com\n",
|
||
"#### 2.build\n",
|
||
"docker build registry.cn-shenzhen.aliyuncs.com/ai_earth_baseline/test_ai_earth_submit:1.0 .\n",
|
||
"#### 3.push\n",
|
||
"docker push registry.cn-shenzhen.aliyuncs.com/ai_earth_baseline/test_ai_earth_submit:1.0"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "markdown",
|
||
"metadata": {},
|
||
"source": [
|
||
"### 提交\n",
|
||
"\n",
|
||
"\n",
|
||
"<img src=\"./pic/submit.jpeg\" width = \"600\" height = \"200\" alt=\"first_step\" align=center />\n",
|
||
"\n",
|
||
"根据自己的不同进行提交即可，如果不出意外，等待一会儿，线上跑完了就会有结果了。\n",
|
||
"\n",
|
||
"<img src=\"./pic/score.jpeg\" width = \"600\" height = \"200\" alt=\"first_step\" align=center />\n"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "code",
|
||
"execution_count": null,
|
||
"metadata": {},
|
||
"outputs": [],
|
||
"source": []
|
||
}
|
||
],
|
||
"metadata": {
|
||
"kernelspec": {
|
||
"display_name": "Python 3",
|
||
"language": "python",
|
||
"name": "python3"
|
||
},
|
||
"language_info": {
|
||
"codemirror_mode": {
|
||
"name": "ipython",
|
||
"version": 3
|
||
},
|
||
"file_extension": ".py",
|
||
"mimetype": "text/x-python",
|
||
"name": "python",
|
||
"nbconvert_exporter": "python",
|
||
"pygments_lexer": "ipython3",
|
||
"version": "3.7.6"
|
||
},
|
||
"toc": {
|
||
"base_numbering": 1,
|
||
"nav_menu": {},
|
||
"number_sections": true,
|
||
"sideBar": true,
|
||
"skip_h1_title": false,
|
||
"title_cell": "Table of Contents",
|
||
"title_sidebar": "Contents",
|
||
"toc_cell": false,
|
||
"toc_position": {
|
||
"height": "calc(100% - 180px)",
|
||
"left": "10px",
|
||
"top": "150px",
|
||
"width": "288px"
|
||
},
|
||
"toc_section_display": true,
|
||
"toc_window_display": true
|
||
}
|
||
},
|
||
"nbformat": 4,
|
||
"nbformat_minor": 4
|
||
}
|