6.0 Monitoring Metadata

Use Cases:

1. the user needs an estimation of how long it will take to run a unit, a group unit, entire network on a particular machine.
2. having started a job, the user wants to know the state of the execution of the network at any time

Requirements:

1. network monitor - bandwidth
2. host monitoring (cpu, disk, memory, storage, javaflops, Triana version, jdk, security)
3. job monitoring – progress, status (not started, started, stopped, finished…)

Currently, we are thinking of attacking this problem via two methods.

• Use the order of the algorithm directly or use
• self-learning, empirical estimation of the order of an algorithm

The self-learning, empirical estimation of the order of an algorithm does not need to know the direction relationship between its dimensionality and its timing. Calculate the FLOPS required for the algorithm by using an empirical observation – this can be normalized using successive measurements. Also run the algorithm and divide the time taken by the FLOPS.

T_ = T_ALG / (N*FLOPS)

where:

T_ALG = Timing of algorithm (in seconds)
T_INDEX = Timing for unit data set on this machine
N = dimensionality of the data set (number of elements is data set for vectors)