Best practices for exclusive resource pool use -notebook and training task linkage

In this scenario , We use Notebook To carry the commissioning stage 、ModelArts Training tasks to carry the batch operation stage .

Features of the scene ：

Notebook Provide computing resources for debugging through container instances . Because of the security requirements of Huawei cloud , The interactive environment cannot directly provide root jurisdiction , Must be “ma-user”（user_id：1000） land 、 Use resources .

Notebook Provide a series of prefabricated images , Mirroring has provided the foundation cuda drive 、conda And python、 AI engine mainstream version （TensorFlow、PyTorch） And its dependence .

Notebook Provide image storage capability , Support customers to solidify the modifications and changes to the image environment .

Notebook Provide ssh Landing capability , Support customers to use VS Code Wait for the plug-in to log in remotely 、 Commissioning .

Notebook And development environment can be achieved through NFS To share a folder , Support real-time synchronization of files in the corresponding folder .

Scenario dependency ：

Have purchased ModelArts Exclusive training pool , Complete the initialization of the development environment ,NAS VPC Has been properly configured .

Permissions have been set properly , Have SFS Turbo Common permissions and fine-grained permissions of services “sfsturbo:*:dataAction”.

Practice process ：

Commissioning stage ：

1. establish Notebook example

Access link ：https://console.huaweicloud.com/modelarts/?region=cn-north-4#/dev-container

Click on “ establish ” Button , And use the exclusive pool resources to create Notebook example . Mirror select common mirror “pytorch 1.8”, Storage options “ Elastic file service SFS”. Create the configuration as shown in the screenshot ：

notes ： If you want to use ssh Login and VS Code Tool remote development , Please refer to the blog ：https://bbs.huaweicloud.com/blogs/280541

open Notebook example , And open terminal.

2. Download code 、 Data and conduct commissioning

Code using pytorch The official target is ImageNet Example ,github Links such as ：https://github.com/pytorch/examples; Data use public cat and dog data sets （ Cut version ）.

The download and decompression commands are as follows ：

Installation environment , After debugging the code , stay Notebook You can use the following commands to run training tasks ：

3. Save the running environment

Follow the following procedure to save the current Notebook Environmental Science .

  Fill in the name and remarks ：

Corresponding notebook Examples may 3-10 Minutes are not available , It takes patience . After saving , You can go to SWR View the corresponding image in the corresponding organization , You can also find the corresponding image in the training .

notes ：

/cache And /home/ma-user/work The contents of the two directories need extra attention ./cache The directory will be reloaded during training , Cause the content to be overwritten ;/home/ma-user/work The contents of the directory are notebook In the , Will not be saved in the container image . Suggest /cache Do not store any data in the directory ,/home/user/work The directory stores temporary data and code .

Batch run phase ：

1. Create training tasks

Access link ：https://console.huaweicloud.com/modelarts/?region=cn-north-4#/training

Pay attention to the following items when creating parameters ：

a. Mirror image tag： Mirror image tag Fill in the just saved Notebook Image name

b. Start command ： and Notebook Of terminal The test is consistent , namely ：

c. Mounting path of cloud storage ： and Notebook The working directory attached in is consistent , namely “/home/ma-user/work”.

2. View the execution details of training tasks

Click the task name on the task list page to enter the details page . The page will display the log of the task and GPU Utilization .

If logs will be synchronously stored in SFS Plate , Can be in Notebook Use in tensorboard（ Use the official pytorch Mirror image ） Real time analysis of logs .

3. Rebuild training tasks

According to research and development needs , Pair code 、 Configuration changes （ Use Notebook, or ECS mount SFS The way of dish （ Be careful linux Of uid Need to be 1000））, And rebuild training tasks . The reconstruction task method is shown in the screenshot ：