pooling unused disk space across multiple desktops? Options
Dick Cheney's Mechanical Heart
#1 Posted: : Friday, October 23, 2009 1:11:42 PM
Rank: Coathanger Abortion



Groups: {pDs} Member, Server Admin

Joined: 5/17/2006
Posts: 3,487
Points: 1,990
Location: Undisclosed Location
Anyone know of anything that can harvest the unused disk space on a bunch of workstations? In a perfect world it would be a kind of distributed virtual filesystem, basically each workstation would run a p2p client in the background that would turn unused diskspace into a storage node that would be pooled with all the other nodes into a massive virtual disk on which the distributed filesystem would run. There would be some kind of control mechanism where the clients could be configured (e.g. one client could be told to use 20% of the unused disk space and never more than 10% of available bandwidth, except after 5pm. Another client could be told to use 100% of the disk space and 75% of the bandwidth. Etc.). The virtual filesystem would have to have massive redundancy to account for nodes constantly changing but with all the available unused disk space there is plenty to sacrifice. It wouldn't be highspeed storage in any way but, it should be extremely resilient as long as enough nodes are available. Even more awesome would be if it could also be told to copy specific data from each node into the pool (e.g. backup the local user profile).
The idea is you could deploy this on all the workstations on a LAN and magically have what's effectively a very large and very cheap (and very slow) SAN that could be used for tucking away important things that you are not likely to need often. Like say, a few hundred gigs in monthly backups. It really sucks when you discover that you really need something out of a backup then you go to pull it only to discover that the incompetent fuckers who run the backup system have ruined all of the tapes.
I've looked around for ways of doing this, ZFS, btrfs, iSCSI, etc. Nothing is quite there as far as I can tell. There's a lot of research papers and obscure projects by academics that are oriented around high performance grids that seem kind of close to what I'm thinking of but none of it appears to be usable in any way.
Razor wrote:
Killing people is good therapy.


Lead Salad
#2 Posted: : Friday, October 23, 2009 1:59:06 PM
Rank: Taco Technician



Groups: Registered User, {pDs} Member

Joined: 5/9/2006
Posts: 3,948
Points: 3,636
Location: Hi-Five City
ya there is, I forget what its called though.
Super Penguin
#3 Posted: : Friday, October 23, 2009 4:09:54 PM
Rank: Fuckstick



Groups: {pDs} Member, Registered User

Joined: 12/14/2007
Posts: 1,195
Points: 2,775
Location: Near Seattle
Ooh, that's a good idea.

Quote:
Razor: As hard as one might try no one will ever be able to pull one over on Super Penguin.

Backpack of Win
Dick Cheney's Mechanical Heart
#4 Posted: : Tuesday, October 27, 2009 11:26:48 AM
Rank: Coathanger Abortion



Groups: {pDs} Member, Server Admin

Joined: 5/17/2006
Posts: 3,487
Points: 1,990
Location: Undisclosed Location
I've found a few programs that almost do what I want... too bad none of them seem to work (thus far)
Razor wrote:
Killing people is good therapy.


Dick Cheney's Mechanical Heart
#5 Posted: : Friday, October 30, 2009 10:05:55 AM
Rank: Coathanger Abortion



Groups: {pDs} Member, Server Admin

Joined: 5/17/2006
Posts: 3,487
Points: 1,990
Location: Undisclosed Location
Virtualize everything
Razor wrote:
Killing people is good therapy.


bragr
#6 Posted: : Friday, October 30, 2009 8:16:18 PM
Rank: Administration



Groups: Server Admin, Administration, {pDs} Member, Registered User

Joined: 11/19/2008
Posts: 1,014
Points: 2,767
Basically what you are describing is cloud computing which has always been a really cool idea but has always had serious shortcomings when put into practice (See recent outage of Tmoble sidekick data). You are right, there are programs out there that will back up you computer to a remote system, and even some that are very advanced and approach what you are trying to do, though they tend to be enterprise level software that is very expensive.

The inherent problem I see in this, besides the software not existing, is bandwidth, which is the usual problem in cloud computer. It simply takes a long time to upload data on a typical home connection. In commercial cload computing applications, the various computers that comprise the cloud are networked together with gigabit Ethernet or even infiniband networks that push 100 gigabits per connection. More simply, if you have 20Gb to back up, the process is going to act a lot like a 20GB torrent with 1 seed and several leachers. To a lesser degree, given that these computers won't be in a secure data centers, and will be comunicating across the public Internet rather than a private network, you are going to have problems with data security and integrity.

I am not saying its not a cool idea, or that I wouldn't use it, but its not a simple weekend project that you can hack together in perl.
bragr
#7 Posted: : Friday, October 30, 2009 8:16:20 PM
Rank: Administration



Groups: Server Admin, Administration, {pDs} Member, Registered User

Joined: 11/19/2008
Posts: 1,014
Points: 2,767
Basically what you are describing is cloud computing which has always been a really cool idea but has always had serious shortcomings when put into practice (See recent outage of Tmoble sidekick data). You are right, there are programs out there that will back up you computer to a remote system, and even some that are very advanced and approach what you are trying to do, though they tend to be enterprise level software that is very expensive.

The inherent problem I see in this, besides the software not existing, is bandwidth, which is the usual problem in cloud computer. It simply takes a long time to upload data on a typical home connection. In commercial cload computing applications, the various computers that comprise the cloud are networked together with gigabit Ethernet or even infiniband networks that push 100 gigabits per connection. More simply, if you have 20Gb to back up, the process is going to act a lot like a 20GB torrent with 1 seed and several leachers. To a lesser degree, given that these computers won't be in a secure data centers, and will be comunicating across the public Internet rather than a private network, you are going to have problems with data security and integrity.

I am not saying its not a cool idea, or that I wouldn't use it, but its not a simple weekend project that you can hack together in perl.
Dick Cheney's Mechanical Heart
#8 Posted: : Tuesday, November 03, 2009 12:31:40 PM
Rank: Coathanger Abortion



Groups: {pDs} Member, Server Admin

Joined: 5/17/2006
Posts: 3,487
Points: 1,990
Location: Undisclosed Location
I'm making a lot of progress on this. I can imagine much better solutions, but so far it's working pretty well. Yesterday I dumped around 25gigs in, took around 2.5 hours.
Razor wrote:
Killing people is good therapy.


Lead Salad
#9 Posted: : Tuesday, November 03, 2009 12:37:02 PM
Rank: Taco Technician



Groups: Registered User, {pDs} Member

Joined: 5/9/2006
Posts: 3,948
Points: 3,636
Location: Hi-Five City
Just make sure you dont create one of these



Users browsing this topic
Guest
Forum Jump  
You cannot post new topics in this forum.
You cannot reply to topics in this forum.
You cannot delete your posts in this forum.
You cannot edit your posts in this forum.
You cannot create polls in this forum.
You cannot vote in polls in this forum.