r - Speed up multi-threaded sparse-matrix multiplication -
i have 2 large sparse matrices ([700 x 400000] %*% [400000 x 700]; 98% sparse) use following code multiply, takes 8 hours, can stop calculation @ time, write existing values, , start again stopped, require. i've looked @ other prepackaged solutions (blas, native r matrix stuff), can't wrap head around them. code work's i'm trying optimize it. i'm willing go outside r if gains sufficient. i'm sure other packages can them , faster, don't know current progress or expected run-time.
- what should focus on decrease run-time?
- how of decrease looking at?
- would sparse matrix library speed up?
code:
library(dosnow) library(foreach) library(data.table) cl <- makecluster(7, outfile ="") registerdosnow(cl) xtw <- as.matrix(fread('net_xtw.csv', header = f, sep=',')) net_players <- as.matrix(fread('net_players_filtered.csv', header = f, sep=',')) num_column=ncol(net_players) num_row=nrow(net_players) xtx <- matrix(0,nrow=num_column,ncol=num_column) getk <- function(num_row,xt,x){ temp = 0 (k in 1:num_row){ temp = temp + (xt[k] * x[k]) } return(temp) } (i in 1:num_column) { xt = xtw[,i] line <- foreach (j = i:(num_column), .combine=c) %dopar% { x = net_players[,j] num = getk(num_row,xt,x) return(num) } (k in i:num_column){ xtx[i,k] <- line[(k-(i-1))] } } stopcluster(cl) write.table(xtx, file = "net_xtwx.csv",sep=",",row.names=f,col.names=f) rm(list=ls(all=true))
Comments
Post a Comment