search -  faq -  rulez -  staff

forums






map cycle
irc              pwnalizer
voice         dswp bar




Register

Login


It is currently 10.20.21

 

All times are UTC + 1 hour




Post new topic Reply to topic  [ 7 posts ] 
Author Message
 Post subject: Gameservers Down
PostPosted: 07.24.21 
Offline
Godlike
User avatar

Joined: 07.15.08
Posts: 4603
Location: Behind U
-----tdm:  
nick: [dswp]GewitterOma
skill: 1122.83
kills: 25960
deaths: 19847
ratio: 1.30
-----bomb:  
nick: [dswp]GewitterOma
skill: 812.172
kills: 3885
deaths: 3541
ratio: 1.09
hi everyone...
last days the game servers were down, its cause a disk died and there was no backup.
actually it was a mobile disk, usb, originally "temporarily" connected (and forgotten) over the last 10 years.
vmware doesnt even support usb drives natively, this setup was in fact left over from a migration.
hm.

the drive:
Attachment:
Screenshot from 2021-07-24 07-34-11.png
Screenshot from 2021-07-24 07-34-11.png [ 7.48 KiB | Viewed 296 times ]


so i thought about 3 things.
- recovering the data from the disk and, ofc,
- getting a proper store and
- backup.

##### PART 1 RECOVERY #####

i dont have a car at the moment, the datacenter is some hours by train away.
blinky lives close there, so he went to datacenter to get that disk.
i myself bought some hardware and sent it to blinky before, so he could pimp the storage hw a bit.

result of blinkys inspection:
the disk comes up when connected.
gparted shows an unknown partition, but its vmware vmfs, thats normal:
Attachment:
Screenshot from 2021-07-24 07-43-46.png
Screenshot from 2021-07-24 07-43-46.png [ 599.56 KiB | Viewed 381 times ]


but it fails then after some seconds. The warmer it gets, the faster it disconnects.
It disconnects usb-device-wise, so at least the drive seems ok.

now blinky had an idea.
this is the drive:
Attachment:
Screenshot from 2021-07-24 07-46-24.png
Screenshot from 2021-07-24 07-46-24.png [ 590.3 KiB | Viewed 381 times ]
Attachment:
Screenshot from 2021-07-24 07-46-34.png
Screenshot from 2021-07-24 07-46-34.png [ 726.83 KiB | Viewed 381 times ]


the board is a 2060-771801-002 REV A
it looks like swapping a board from an identical disk could bring the disk back.
but not the data. partitioning informations are stuck in the bios of the old hard disk.
fortunately, blinky got the soldering master degree, so we ordered a board...

this is todays state of the project, blinky swapped already the bios chips.
Attachment:
Screenshot from 2021-07-24 07-51-51.png
Screenshot from 2021-07-24 07-51-51.png [ 609.35 KiB | Viewed 381 times ]


##### PART 2 NEW DISK #####

1.) Hardware...
a year ago, i started to build some storage.
basically, it was cause there was some time and money.
so i bought a dell 730xd as a first storage box...
Attachment:
Screenshot from 2021-07-24 08-11-38.png
Screenshot from 2021-07-24 08-11-38.png [ 785 KiB | Viewed 381 times ]


and i got a pair of juniper ex4200 switches, so i could run ha storage and ip networks.
Attachment:
Screenshot from 2021-07-24 08-24-21.png
Screenshot from 2021-07-24 08-24-21.png [ 446.65 KiB | Viewed 381 times ]


the 730xd is ideal imho, it has 25 sas slots and alot pcie slots/lanes for nvme.
the ex4200 have each 2x 10g uplinks, the dell server has 2x 10g.
so the switches have space for 2x storage with 2x 10g each.

anyhow, the hypervisors must be connected, too and theres jist 1g links left.
the ex4200 can aggregate 8 physical links, so thats what i planned:
every hv gets 8 links, there will be a dswitch with lacp in vmware.
additionally, the 730xd needs some cheap test disks...
here we are. intenso 240gb.
normally a bad choice but bigul checked them ok.
Attachment:
Screenshot from 2021-07-24 08-27-36.png
Screenshot from 2021-07-24 08-27-36.png [ 686.48 KiB | Viewed 381 times ]

testing the packaging hardware for transport!
Attachment:
Screenshot from 2021-07-24 08-27-55.png
Screenshot from 2021-07-24 08-27-55.png [ 378.74 KiB | Viewed 296 times ]


when blinky was in datacenter, he could mount nics and ssds.
so it looks like that now:
Attachment:
Screenshot from 2021-07-24 08-34-53.png
Screenshot from 2021-07-24 08-34-53.png [ 91.39 KiB | Viewed 295 times ]

Attachment:
Screenshot from 2021-07-24 08-36-09.png
Screenshot from 2021-07-24 08-36-09.png [ 504.05 KiB | Viewed 380 times ]


dswitch and lacp are on, next step will be playing with nfs4, vmware and multipathing...
Attachment:
Screenshot from 2021-07-24 08-38-50.png
Screenshot from 2021-07-24 08-38-50.png [ 119.39 KiB | Viewed 380 times ]


expect an update sooner or later. (:
anyone interested in storage and ethernet?

_________________
Image



 


Top
 Profile  
 
 Post subject: Re: Gameservers Down
PostPosted: 07.24.21 
Offline
Godlike
User avatar

Joined: 07.15.08
Posts: 4603
Location: Behind U
-----tdm:  
nick: [dswp]GewitterOma
skill: 1122.83
kills: 25960
deaths: 19847
ratio: 1.30
-----bomb:  
nick: [dswp]GewitterOma
skill: 812.172
kills: 3885
deaths: 3541
ratio: 1.09
in vmware, theres several ways to connect storage.
for redundant connections, fibre channel ist widely known.
smaller installs may use iscsi or even old nfs.

now comes my ideas...
fibrechannel is no option, plus i think that block based storage belongs to local chassis.
vmware dont need block devices really.
the setup should combine storage and ip networking, saves alot space and power.

iscsi is the more modern way to go ip based.
but its block based, i wanted to get rid of this.

look how old nfs does true active/active multipathing:
Attachment:
Screenshot from 2021-07-24 11-29-25.png
Screenshot from 2021-07-24 11-29-25.png [ 37.15 KiB | Viewed 372 times ]


the result is an extra 0,2 ms access time to the intenso lol ssd, as seen from a windows vm:
Attachment:
Screenshot from 2021-07-24 11-19-47.png
Screenshot from 2021-07-24 11-19-47.png [ 14.71 KiB | Viewed 372 times ]

Attachment:
Screenshot from 2021-07-24 11-27-01.png
Screenshot from 2021-07-24 11-27-01.png [ 20.45 KiB | Viewed 287 times ]


multipathing works through IP-hash-based link aggregation and nfs 4 does IP- based multipathing.
if you dont have a switch that can do link aggregation:
take 10 miniswitches, 10 network cards in each server, 10 subnets and you go.
you can even combine fast and slow links, which is one of many advantages of IP.

the other advantage is the file system.
in vmware, you cant choose a filesystem, theres only "vmfs"
recovering this failed disk means mounting it in vmware.
now i can have ext, xfs, i have even ram for zfs in the 730xd...

for the first i will track that read bandwith thingy, looks a bit mad with 200MB/s.
30K IOPS @ 4K is to be compared with theoretical limits, maybe i play with L2 frame size later etc.
fio shows definetly more on nfs server.

_________________
Image



 


Top
 Profile  
 
 Post subject: Re: Gameservers Down
PostPosted: 07.25.21 
Offline
Godlike
User avatar

Joined: 07.15.08
Posts: 4603
Location: Behind U
-----tdm:  
nick: [dswp]GewitterOma
skill: 1122.83
kills: 25960
deaths: 19847
ratio: 1.30
-----bomb:  
nick: [dswp]GewitterOma
skill: 812.172
kills: 3885
deaths: 3541
ratio: 1.09
setting MTU to 9000:
read bandwith iussue disappeared, for the price of a bit higher latency, resulting in ca. 10% IOPS loss:

Attachment:
Screenshot from 2021-07-25 20-20-51.png
Screenshot from 2021-07-25 20-20-51.png [ 25.77 KiB | Viewed 363 times ]

Attachment:
Screenshot from 2021-07-25 20-21-05.png
Screenshot from 2021-07-25 20-21-05.png [ 42.51 KiB | Viewed 278 times ]

_________________
Image



 


Top
 Profile  
 
 Post subject: Re: Gameservers Down
PostPosted: 08.09.21 
Offline
Godlike
User avatar

Joined: 07.15.08
Posts: 4603
Location: Behind U
-----tdm:  
nick: [dswp]GewitterOma
skill: 1122.83
kills: 25960
deaths: 19847
ratio: 1.30
-----bomb:  
nick: [dswp]GewitterOma
skill: 812.172
kills: 3885
deaths: 3541
ratio: 1.09
oki heres next try with nfs server...

Attachment:
Screenshot from 2021-08-09 19-41-26.png
Screenshot from 2021-08-09 19-41-26.png [ 889.19 KiB | Viewed 318 times ]



++check the professional cooling holes hihi

_________________
Image



 


Top
 Profile  
 
 Post subject: Re: Gameservers Down
PostPosted: 08.11.21 
Offline
Godlike
User avatar

Joined: 03.12.11
Posts: 501
-----tdm:  
nick: [dswp]zietsh
skill: 991.948
kills: 11203
deaths: 8618
ratio: 1.29
-----bomb:  
nick: [dswp]zietsh
skill: 1056.62
kills: 18077
deaths: 14755
ratio: 1.22
interesting, i feel like I need to update my vocabulary.

but i see lots of ssds, fast connections - i like it.



 


Top
 Profile  
 
 Post subject: Re: Gameservers Down
PostPosted: 09.03.21 
Offline
Godlike
User avatar

Joined: 07.15.08
Posts: 4603
Location: Behind U
-----tdm:  
nick: [dswp]GewitterOma
skill: 1122.83
kills: 25960
deaths: 19847
ratio: 1.30
-----bomb:  
nick: [dswp]GewitterOma
skill: 812.172
kills: 3885
deaths: 3541
ratio: 1.09
UPDATE

...still nothing new from the disk.

_________________
Image



 


Top
 Profile  
 
 Post subject: Re: Gameservers Down
PostPosted: 10.05.21 
Offline
Godlike
User avatar

Joined: 08.31.08
Posts: 283
Location: 1408
-----tdm:  
nick: TitaniusAngleSmith
skill: 1164.22
kills: 9429
deaths: 6605
ratio: 1.42
-----bomb:  
nick: TitaniusAngleSmith
skill: 886.644
kills: 963
deaths: 678
ratio: 1.42
wurst wrote:
UPDATE

...still nothing new from the disk.

Yea...guys i'm so sorry :cry:
Was a bit busy lately...kinf of..you know "Those dogs that love the rain and chasing trains"? :|

The transplantation wasn't blessed by success...i promise, i'm gonna take a look at it these days :mrgreen:

_________________
Image



 


Top
 Profile  
 
Display posts from previous:  Sort by  
Post new topic Reply to topic  [ 7 posts ] 

All times are UTC + 1 hour


Who is online

Users browsing this forum: Majestic-12 [Bot] and 4 guests


You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot post attachments in this forum

Search for:
Jump to:  
Imprint

Powered by phpBB © 2000, 2002, 2005, 2007, 2008, 2009, 2010 phpBB Group

Skin by Lucas Kane
we use apache2 mod rewrite with phpBB SEO
map train_beta1-100 © 2009 by SteveMcQueen
Statistics Backend + Database by XLR Stats and B3 Bot
mapvote robot and gameserver monitor © 2009 by BlinKy
mumble viewer Copyright © 2008 Dominik Radner (aka Urmel)
mumble switcher and integration © 2008 by XTJ7, Unclefragger and Wursti
Localisation Plugin © 2009, Team Leads Plugin © 2009 and Knifer Plugin © 2009 by SvaRoX
for our stats we use Chart.js Copyright (c) 2013-2015 Nick Downie.
the stats also use some jQuery jQuery v2.1.4 | (c) 2005, 2015 jQuery Foundation, Inc.



voice server
top 20 players


nameskillkills
-Dws.BLINGBLING*-*1730.63418631
NormaSnockers1865.75400492
Zottel1760.8276378
make.them.suffer1846.17269872
>8v=1825.35230156
moon1777.56195615
sjas1692.49192315
peace1878.86190660
Wagner_Moura1562.9188001
Goomba1859.75182677
z0rn1608.41181016
Mad1803.76179124
[dswp]PLZ1847.85178516
Graf_ZahlIII1835.73167407
Zohan1611.07159737
ubercunt1634.93159240
Yarrr!1917.33156233
I_am_nOOb1909.64151268
Pandageddon1891.75148319
Pirat1664.08145798