dswp.de
http://www.dswp.de/old/

Gameservers Down
http://www.dswp.de/old/huhuuuuu/gameservers-down-t7659.html
Page 1 of 1

Author:  wurst [ 07.24.21 ]
Post subject:  Gameservers Down

hi everyone...
last days the game servers were down, its cause a disk died and there was no backup.
actually it was a mobile disk, usb, originally "temporarily" connected (and forgotten) over the last 10 years.
vmware doesnt even support usb drives natively, this setup was in fact left over from a migration.
hm.

the drive:
Attachment:
Screenshot from 2021-07-24 07-34-11.png
Screenshot from 2021-07-24 07-34-11.png [ 7.48 KiB | Viewed 3499 times ]


so i thought about 3 things.
- recovering the data from the disk and, ofc,
- getting a proper store and
- backup.

##### PART 1 RECOVERY #####

i dont have a car at the moment, the datacenter is some hours by train away.
blinky lives close there, so he went to datacenter to get that disk.
i myself bought some hardware and sent it to blinky before, so he could pimp the storage hw a bit.

result of blinkys inspection:
the disk comes up when connected.
gparted shows an unknown partition, but its vmware vmfs, thats normal:
Attachment:
Screenshot from 2021-07-24 07-43-46.png
Screenshot from 2021-07-24 07-43-46.png [ 599.56 KiB | Viewed 4738 times ]


but it fails then after some seconds. The warmer it gets, the faster it disconnects.
It disconnects usb-device-wise, so at least the drive seems ok.

now blinky had an idea.
this is the drive:
Attachment:
Screenshot from 2021-07-24 07-46-24.png
Screenshot from 2021-07-24 07-46-24.png [ 590.3 KiB | Viewed 4738 times ]
Attachment:
Screenshot from 2021-07-24 07-46-34.png
Screenshot from 2021-07-24 07-46-34.png [ 726.83 KiB | Viewed 4738 times ]


the board is a 2060-771801-002 REV A
it looks like swapping a board from an identical disk could bring the disk back.
but not the data. partitioning informations are stuck in the bios of the old hard disk.
fortunately, blinky got the soldering master degree, so we ordered a board...

this is todays state of the project, blinky swapped already the bios chips.
Attachment:
Screenshot from 2021-07-24 07-51-51.png
Screenshot from 2021-07-24 07-51-51.png [ 609.35 KiB | Viewed 4738 times ]


##### PART 2 NEW DISK #####

1.) Hardware...
a year ago, i started to build some storage.
basically, it was cause there was some time and money.
so i bought a dell 730xd as a first storage box...
Attachment:
Screenshot from 2021-07-24 08-11-38.png
Screenshot from 2021-07-24 08-11-38.png [ 785 KiB | Viewed 4738 times ]


and i got a pair of juniper ex4200 switches, so i could run ha storage and ip networks.
Attachment:
Screenshot from 2021-07-24 08-24-21.png
Screenshot from 2021-07-24 08-24-21.png [ 446.65 KiB | Viewed 4738 times ]


the 730xd is ideal imho, it has 25 sas slots and alot pcie slots/lanes for nvme.
the ex4200 have each 2x 10g uplinks, the dell server has 2x 10g.
so the switches have space for 2x storage with 2x 10g each.

anyhow, the hypervisors must be connected, too and theres jist 1g links left.
the ex4200 can aggregate 8 physical links, so thats what i planned:
every hv gets 8 links, there will be a dswitch with lacp in vmware.
additionally, the 730xd needs some cheap test disks...
here we are. intenso 240gb.
normally a bad choice but bigul checked them ok.
Attachment:
Screenshot from 2021-07-24 08-27-36.png
Screenshot from 2021-07-24 08-27-36.png [ 686.48 KiB | Viewed 4738 times ]

testing the packaging hardware for transport!
Attachment:
Screenshot from 2021-07-24 08-27-55.png
Screenshot from 2021-07-24 08-27-55.png [ 378.74 KiB | Viewed 3499 times ]


when blinky was in datacenter, he could mount nics and ssds.
so it looks like that now:
Attachment:
Screenshot from 2021-07-24 08-34-53.png
Screenshot from 2021-07-24 08-34-53.png [ 91.39 KiB | Viewed 3498 times ]

Attachment:
Screenshot from 2021-07-24 08-36-09.png
Screenshot from 2021-07-24 08-36-09.png [ 504.05 KiB | Viewed 4737 times ]


dswitch and lacp are on, next step will be playing with nfs4, vmware and multipathing...
Attachment:
Screenshot from 2021-07-24 08-38-50.png
Screenshot from 2021-07-24 08-38-50.png [ 119.39 KiB | Viewed 4737 times ]


expect an update sooner or later. (:
anyone interested in storage and ethernet?

Author:  wurst [ 07.24.21 ]
Post subject:  Re: Gameservers Down

in vmware, theres several ways to connect storage.
for redundant connections, fibre channel ist widely known.
smaller installs may use iscsi or even old nfs.

now comes my ideas...
fibrechannel is no option, plus i think that block based storage belongs to local chassis.
vmware dont need block devices really.
the setup should combine storage and ip networking, saves alot space and power.

iscsi is the more modern way to go ip based.
but its block based, i wanted to get rid of this.

look how old nfs does true active/active multipathing:
Attachment:
Screenshot from 2021-07-24 11-29-25.png
Screenshot from 2021-07-24 11-29-25.png [ 37.15 KiB | Viewed 4729 times ]


the result is an extra 0,2 ms access time to the intenso lol ssd, as seen from a windows vm:
Attachment:
Screenshot from 2021-07-24 11-19-47.png
Screenshot from 2021-07-24 11-19-47.png [ 14.71 KiB | Viewed 4729 times ]

Attachment:
Screenshot from 2021-07-24 11-27-01.png
Screenshot from 2021-07-24 11-27-01.png [ 20.45 KiB | Viewed 3490 times ]


multipathing works through IP-hash-based link aggregation and nfs 4 does IP- based multipathing.
if you dont have a switch that can do link aggregation:
take 10 miniswitches, 10 network cards in each server, 10 subnets and you go.
you can even combine fast and slow links, which is one of many advantages of IP.

the other advantage is the file system.
in vmware, you cant choose a filesystem, theres only "vmfs"
recovering this failed disk means mounting it in vmware.
now i can have ext, xfs, i have even ram for zfs in the 730xd...

for the first i will track that read bandwith thingy, looks a bit mad with 200MB/s.
30K IOPS @ 4K is to be compared with theoretical limits, maybe i play with L2 frame size later etc.
fio shows definetly more on nfs server.

Author:  wurst [ 07.25.21 ]
Post subject:  Re: Gameservers Down

setting MTU to 9000:
read bandwith iussue disappeared, for the price of a bit higher latency, resulting in ca. 10% IOPS loss:

Attachment:
Screenshot from 2021-07-25 20-20-51.png
Screenshot from 2021-07-25 20-20-51.png [ 25.77 KiB | Viewed 4720 times ]

Attachment:
Screenshot from 2021-07-25 20-21-05.png
Screenshot from 2021-07-25 20-21-05.png [ 42.51 KiB | Viewed 3481 times ]

Author:  wurst [ 08.09.21 ]
Post subject:  Re: Gameservers Down

oki heres next try with nfs server...

Attachment:
Screenshot from 2021-08-09 19-41-26.png
Screenshot from 2021-08-09 19-41-26.png [ 889.19 KiB | Viewed 4675 times ]



++check the professional cooling holes hihi

Author:  zietsh [ 08.11.21 ]
Post subject:  Re: Gameservers Down

interesting, i feel like I need to update my vocabulary.

but i see lots of ssds, fast connections - i like it.

Author:  wurst [ 09.03.21 ]
Post subject:  Re: Gameservers Down

UPDATE

...still nothing new from the disk.

Author:  bky [ 10.05.21 ]
Post subject:  Re: Gameservers Down

wurst wrote:
UPDATE

...still nothing new from the disk.

Yea...guys i'm so sorry :cry:
Was a bit busy lately...kinf of..you know "Those dogs that love the rain and chasing trains"? :|

The transplantation wasn't blessed by success...i promise, i'm gonna take a look at it these days :mrgreen:

Author:  Pirat [ 11.07.21 ]
Post subject:  Re: Gameservers Down

bky wrote:
wurst wrote:
UPDATE

...still nothing new from the disk.

Yea...guys i'm so sorry :cry:
Was a bit busy lately...kinf of..you know "Those dogs that love the rain and chasing trains"? :|

The transplantation wasn't blessed by success...i promise, i'm gonna take a look at it these days :mrgreen:


Deez nuts ... ;)


Pirat.

Page 1 of 1 All times are UTC + 1 hour
Powered by phpBB © 2000, 2002, 2005, 2007 phpBB Group
http://www.phpbb.com/