python - multiprocessing timing inconsistency -
i have 100 processes. each process contains 10 inputs(logic expressions) , task of each process find fastest heuristic algorithm solving each of logic inputs(i have 5 heuristic algorithms).
when run each process separately results different when run of processes in parallel (using python p1.py & python p2.py &….. ). example, when run processes separately input 1 (in p1) finds first heuristic algorithms fastest method when in parallel same input finds 5th heuristic algorithms faster!
reason cpu switch between parallel processes , messes timing not give right time each heuristic algorithm spends solve input?
solution? can decreasing number of processes half reduce false result? (i run program on server)
the operating system has schedule processes on smaller amount of cpus. in order so, runs 1 process on each cpu small amount of time. after that, operating system schedules processes out let other processes run in order give process fair share of running time. each process has wait running slot on cpu. waiting times depend on amount of other processes waiting run , unpredictable.
if use clock time measurements, waiting times pollute measurements. more precise measurement, ask operating system how cpu time process used. function time.process_time() that.
switching between processes costs time. multiple processes accessing same resources (file, hard disk, cpu caches, memory, ...) costs time. cpu bound processes, having orders of magnitude more running processes cpus slow down execution. you'll better results starting less processes amount of cpus. spare cpus remain available work needed operating system or other unrelated programs.
Comments
Post a Comment