I was wondering: The SOTM EU presentation mentioned there’s 5 gtx 1070’s being used.
i’m assuming some of these are in the ign servers and some in the osm.fr servers?
Is it possible to maybe give some details as to the exact server hardware being used for both?
(I know that one of the servers on the slides seems to be a Dell Precision Rack 7910 which can only house 2 cards).
I’m not trying to necessarily replicate the exact hardware setup, but it would maybe help give an idea how the hardware setup is done atm.
We have 2 servers where the blurring API runs:
- main one is
osm37.openstreetmap.fr, hosted a Telehouse3 datacenter near Paris, which is a Dell Rack 7910 with 2 GTX 1070. This server also runs panoramax.openstreetmap.fr and the meta catalog on panoramax.xyz
- the additionnal one is in my basement, it is a Dell T630 with 2 GTX1070 and a Tesla P4 (equivalent to the 1070).
An nginx reverse proxy (locate on osm37) distributes the load between the two servers.
I setup the additionnal one because we add a huge waiting queue a few weeks ago.
We should have more T630 with up to 4 GTX1070 setup “soon” in another datacenter closer to Paris.
Oh, the reverse proxy is running on the main node itself?
Wouldn’t it be better if that was on a separate (maybe a small VPS) running in the same datacenter?
So that if the main node goes down, the reverse proxy atleast keeps working.
EDIT: another question: how long does one image (like a GoPro Max image for example) take on a 1070?
Also: how multithreaded is Geoviso? Obviously there are multiple containers, but if we’re talking multi socket xeon servers. Should the focus be on amount of cores or single core performance?
I just realized something from re-reading your reply.
So the actual database and website of panoramax is not actually running on multiple servers and being load balanced? is it only the blurring api?
If it’s the actual panoramax instance that’s being load balanced, i’d be curious to get more info on the exact setup and what kind of changes were made to the setup compared to the existing docker compose files. Is there any kind of DDOS protection currently implemented?
Yes and no… depending on which part of Panoramax we look at !
- panoramax.fr : cloud based, no database or photos storage
- panoramax.ign.fr : cloud based with the database and API running on scalingo, and the photos storage on OVH S3
Bare-metal (proxmox used for LXC containerization, no docker):
- osm37 hosting: full bare metal (Dell T7910 + IBM storage bay)
- panoramax.openstreetmap.fr : DB on SSD, photos on HDD storage array
- panoramax.xyz (meta-catalog)
- blurring API: proxy (LB) + 1 backend with 2 GPU
- second blurring server (Dell T630 + 3 GPU)
For the bare metal setup, there is no specific DDOS protection.
perfect, got it!
I was really trying to overcomplicate the setup.
And I still might do that later down the line, to get redundancy.
I was also planning to use Proxmox, but actually docker in a vm. Sure you lose a bit of performance, but I read that LXC containers aren’t as isolated as a real vm. But I guess it’s not that big of a deal.
That’s right… but after using containers (openvz then LXC) for almost 10 years on OSM-FR servers we never had any security issue.
I think you mean the Rack version of the 7910, right? (sorry. just copy pasted into search for once and realized there was a typo)
Yes the rack version, here it is (in my basement during tests before moving to the datacenter): Instance Panoramax OSM-FR : l'infra - #17 par cquest - Collecte photos - Forum OSM France
I’ve added disks in the storage bay this afternoon: 6 x 8Tb