Neocortix Tutorial

In this tutorial, we will show you how to use Neocortix Cloud Services BatchRunner to download videos from a website.

First, please follow the steps in the tutorial Setting Up For Batch Jobs. After completion of the initial setup, you will have a directory with examples,

~/ncsexamples

In the subdirectory

~/ncsexamples/batchMode

you will find the runBatchDownload.py command. This script reads a file called

dlUrlList.txt

which contains a list of URLs of video files to download:

# put 1 url on each line; lines starting with # are comments
https://archive-1.neocortix.net/video/videos/Globe_022_tightLoop_jumbotron_008.mp4
https://archive-1.neocortix.net/video/videos/IntervalLogo_044c.mp4
https://archive-1.neocortix.net/video/videos/NeocortixNewsVideo_005.mp4
https://archive-1.neocortix.net/video/videos/PhonePaycheck_003.mp4
https://archive-1.neocortix.net/video/videos/PhoneWorkforceVideo_002.mp4
https://archive-1.neocortix.net/video/videos/SpinDemo3_001e.mp4
# it will automatically replace spaces with %20
https://archive-1.neocortix.net/video/videos/The Third and The Seventh HD.mp4
https://archive-1.neocortix.net/video/videos/Video2ClickDemoRender_016.mp4
https://archive-1.neocortix.net/video/videos/Windmill_040.mp4
https://archive-1.neocortix.net/video/videos/bmw_003.mp4

By default, it does auto-scaling, so you don’t have to set the number of workers

nWorkers

These command line options are available:

--authToken AUTHTOKEN
                      the NCS authorization token to use (default empty, to
                      use NCS_AUTH_TOKEN env var
--filter FILTER       json to filter instances for launch (default: {
                      "storage": ">=4000000000", "dar": ">=95"}),
                      can include regions like this, for example: 
                      "regions": ["north-america", "russia-ukraine-belarus"])
--outDataDir OUTDATADIR
                      a path to the output data dir for this run (default:
                      empty for a new timestamped dir)
--timeLimit TIMELIMIT
                      amount of time (in seconds) allowed for the whole job
                      (default: 1800)
--unitTimeLimit UNITTIMELIMIT
                      amount of time (in seconds) allowed for each download
                      (default: 300)
--urlListFile URLLISTFILE
                      a path to a text file containing urls to download from
                      (default: dlUrlList.txt)
--nWorkers NWORKERS   the # of worker instances to launch (or 0 for
                      autoscale) (default: 0)

Simply run

python3 ./runBatchDownload.py

When the program is done, the output plot files will be put in a directory

./data/download_<datestamp>

with names like

frame_0.out

frame_1.out

frame_2.out

, etc. And you will find a file called

index.csv

, which contains the mapping from these filenames to the actual video filenames that were retrieved:

fileName,url
frame_0.out,https://archive-1.neocortix.net/video/videos/Globe_022_tightLoop_jumbotron_008.mp4
frame_1.out,https://archive-1.neocortix.net/video/videos/IntervalLogo_044c.mp4
frame_2.out,https://archive-1.neocortix.net/video/videos/NeocortixNewsVideo_005.mp4
frame_3.out,https://archive-1.neocortix.net/video/videos/PhonePaycheck_003.mp4
frame_4.out,https://archive-1.neocortix.net/video/videos/PhoneWorkforceVideo_002.mp4
frame_5.out,https://archive-1.neocortix.net/video/videos/SpinDemo3_001e.mp4
frame_6.out,https://archive-1.neocortix.net/video/videos/The%20Third%20and%20The%20Seventh%20HD.mp4
frame_7.out,https://archive-1.neocortix.net/video/videos/Video2ClickDemoRender_016.mp4
frame_8.out,https://archive-1.neocortix.net/video/videos/Windmill_040.mp4
frame_9.out,https://archive-1.neocortix.net/video/videos/bmw_003.mp4

Example Command