|
|
Rank: Coathanger Abortion

Groups: {pDs} Member, Server Admin
Joined: 5/17/2006 Posts: 3,487 Points: 1,990 Location: Undisclosed Location
|
Anyone know of anything that can harvest the unused disk space on a bunch of workstations? In a perfect world it would be a kind of distributed virtual filesystem, basically each workstation would run a p2p client in the background that would turn unused diskspace into a storage node that would be pooled with all the other nodes into a massive virtual disk on which the distributed filesystem would run. There would be some kind of control mechanism where the clients could be configured (e.g. one client could be told to use 20% of the unused disk space and never more than 10% of available bandwidth, except after 5pm. Another client could be told to use 100% of the disk space and 75% of the bandwidth. Etc.). The virtual filesystem would have to have massive redundancy to account for nodes constantly changing but with all the available unused disk space there is plenty to sacrifice. It wouldn't be highspeed storage in any way but, it should be extremely resilient as long as enough nodes are available. Even more awesome would be if it could also be told to copy specific data from each node into the pool (e.g. backup the local user profile). The idea is you could deploy this on all the workstations on a LAN and magically have what's effectively a very large and very cheap (and very slow) SAN that could be used for tucking away important things that you are not likely to need often. Like say, a few hundred gigs in monthly backups. It really sucks when you discover that you really need something out of a backup then you go to pull it only to discover that the incompetent fuckers who run the backup system have ruined all of the tapes. I've looked around for ways of doing this, ZFS, btrfs, iSCSI, etc. Nothing is quite there as far as I can tell. There's a lot of research papers and obscure projects by academics that are oriented around high performance grids that seem kind of close to what I'm thinking of but none of it appears to be usable in any way. Razor wrote:Killing people is good therapy.  
|
|
|
Rank: Taco Technician

Groups: Registered User, {pDs} Member
Joined: 5/9/2006 Posts: 3,948 Points: 3,636 Location: Hi-Five City
|
ya there is, I forget what its called though.
|
|
|
Rank: Fuckstick

Groups: {pDs} Member, Registered User
Joined: 12/14/2007 Posts: 1,195 Points: 2,775 Location: Near Seattle
|
Ooh, that's a good idea.  Quote:Razor: As hard as one might try no one will ever be able to pull one over on Super Penguin. Backpack of Win
|
|
|
Rank: Coathanger Abortion

Groups: {pDs} Member, Server Admin
Joined: 5/17/2006 Posts: 3,487 Points: 1,990 Location: Undisclosed Location
|
I've found a few programs that almost do what I want... too bad none of them seem to work (thus far) Razor wrote:Killing people is good therapy.  
|
|
|
Rank: Coathanger Abortion

Groups: {pDs} Member, Server Admin
Joined: 5/17/2006 Posts: 3,487 Points: 1,990 Location: Undisclosed Location
|
Virtualize everything Razor wrote:Killing people is good therapy.  
|
|
|
Rank: Administration

Groups: Server Admin, Administration, {pDs} Member, Registered User
Joined: 11/19/2008 Posts: 1,014 Points: 2,767
|
Basically what you are describing is cloud computing which has always been a really cool idea but has always had serious shortcomings when put into practice (See recent outage of Tmoble sidekick data). You are right, there are programs out there that will back up you computer to a remote system, and even some that are very advanced and approach what you are trying to do, though they tend to be enterprise level software that is very expensive. The inherent problem I see in this, besides the software not existing, is bandwidth, which is the usual problem in cloud computer. It simply takes a long time to upload data on a typical home connection. In commercial cload computing applications, the various computers that comprise the cloud are networked together with gigabit Ethernet or even infiniband networks that push 100 gigabits per connection. More simply, if you have 20Gb to back up, the process is going to act a lot like a 20GB torrent with 1 seed and several leachers. To a lesser degree, given that these computers won't be in a secure data centers, and will be comunicating across the public Internet rather than a private network, you are going to have problems with data security and integrity. I am not saying its not a cool idea, or that I wouldn't use it, but its not a simple weekend project that you can hack together in perl.
|
|
|
Rank: Administration

Groups: Server Admin, Administration, {pDs} Member, Registered User
Joined: 11/19/2008 Posts: 1,014 Points: 2,767
|
Basically what you are describing is cloud computing which has always been a really cool idea but has always had serious shortcomings when put into practice (See recent outage of Tmoble sidekick data). You are right, there are programs out there that will back up you computer to a remote system, and even some that are very advanced and approach what you are trying to do, though they tend to be enterprise level software that is very expensive. The inherent problem I see in this, besides the software not existing, is bandwidth, which is the usual problem in cloud computer. It simply takes a long time to upload data on a typical home connection. In commercial cload computing applications, the various computers that comprise the cloud are networked together with gigabit Ethernet or even infiniband networks that push 100 gigabits per connection. More simply, if you have 20Gb to back up, the process is going to act a lot like a 20GB torrent with 1 seed and several leachers. To a lesser degree, given that these computers won't be in a secure data centers, and will be comunicating across the public Internet rather than a private network, you are going to have problems with data security and integrity. I am not saying its not a cool idea, or that I wouldn't use it, but its not a simple weekend project that you can hack together in perl.
|
|
|
Rank: Coathanger Abortion

Groups: {pDs} Member, Server Admin
Joined: 5/17/2006 Posts: 3,487 Points: 1,990 Location: Undisclosed Location
|
I'm making a lot of progress on this. I can imagine much better solutions, but so far it's working pretty well. Yesterday I dumped around 25gigs in, took around 2.5 hours. Razor wrote:Killing people is good therapy.  
|
|
|
Rank: Taco Technician

Groups: Registered User, {pDs} Member
Joined: 5/9/2006 Posts: 3,948 Points: 3,636 Location: Hi-Five City
|
|
|
|
| Users browsing this topic |
|
Guest
|