Tuesday, February 24, 2015

Process 100 files evenly across 10 nodes

I was recently asked to write a script assuming there are 100 files and they're all equal-sized, how to process them evenly across 10 nodes. I went in completely wrong direction suggesting regex where my regex-fu is really not that strong. Thinking it over on a different occasion, I came up with a simple script that will do just that. Of course the simplest solution is the best. As usual, your comments are welcome.


__author__ = 'artem'
nodes = {}
files = []
for filenum in range(100):
filename = 'file_00' + str(filenum)
files.append(filename)
for node in range(10):
subset = files[0:10]
nodes[node] = subset
files[0:10] = []
for node in nodes:
print(node, nodes[node])

and the output:

0 ['file_000', 'file_001', 'file_002', 'file_003', 'file_004', 'file_005', 'file_006', 'file_007', 'file_008', 'file_009']
1 ['file_0010', 'file_0011', 'file_0012', 'file_0013', 'file_0014', 'file_0015', 'file_0016', 'file_0017', 'file_0018', 'file_0019']
2 ['file_0020', 'file_0021', 'file_0022', 'file_0023', 'file_0024', 'file_0025', 'file_0026', 'file_0027', 'file_0028', 'file_0029']
3 ['file_0030', 'file_0031', 'file_0032', 'file_0033', 'file_0034', 'file_0035', 'file_0036', 'file_0037', 'file_0038', 'file_0039']
4 ['file_0040', 'file_0041', 'file_0042', 'file_0043', 'file_0044', 'file_0045', 'file_0046', 'file_0047', 'file_0048', 'file_0049']
5 ['file_0050', 'file_0051', 'file_0052', 'file_0053', 'file_0054', 'file_0055', 'file_0056', 'file_0057', 'file_0058', 'file_0059']
6 ['file_0060', 'file_0061', 'file_0062', 'file_0063', 'file_0064', 'file_0065', 'file_0066', 'file_0067', 'file_0068', 'file_0069']
7 ['file_0070', 'file_0071', 'file_0072', 'file_0073', 'file_0074', 'file_0075', 'file_0076', 'file_0077', 'file_0078', 'file_0079']
8 ['file_0080', 'file_0081', 'file_0082', 'file_0083', 'file_0084', 'file_0085', 'file_0086', 'file_0087', 'file_0088', 'file_0089']
9 ['file_0090', 'file_0091', 'file_0092', 'file_0093', 'file_0094', 'file_0095', 'file_0096', 'file_0097', 'file_0098', 'file_0099']

Process finished with exit code 0

Post a Comment