MPI Tutorial

Discussion in 'Mapping' started by Brutos, Dec 15, 2008.

  1. Brutos

    Brutos Administrator Staff Member Moderator

    Messages:
    3,385
    Likes Received:
    0
    Trophy Points:
    0
    MPI Tutorial
    Valve released a new feature in the compile tools for orange box: VMPI
    MPI can be to distribute the compile process via a network to many pcs and speed up your compile process.

    At the moment mpi does only work for vvis, mpi for vrad is broken.
    This may get fixed in a feature version of the source sdk.

    MPI Worker Tutorial
    First step:
    Check if you have the Source SDK installed and you did run it at least once.

    Second step:
    Find out where your vvis folder is and copy paste it.
    It should look like this:
    C:\Program Files (x86)\Steam\steamapps\<Your Steam Name>\sourcesdk\bin\orangebox\bin

    Third step:
    Open the command line. You do that by either finding it in the accessories menu somewhere.
    Or by typing win+r and then cmd.

    Forth step:
    Navigate in the commandline to the vvis folder.
    You do that by typing something like:
    cd C:\Program Files (x86)\Steam\steamapps\<Your Steam Name>\sourcesdk\bin\orangebox\bin
    Remeber the cd command at the beginning.

    Fifth step:
    Start vvis with this command in the commandline. In this step you need the ip of the MPI master.
    Ask the map maker for it and replace <ip> with the ip:
    vvis.exe -mpi_worker <ip>:23311

    The command should look like this one:
    vvis.exe -mpi_worker 78.54.166.168:23311

    Sixth step:
    Wait, vvis can take long even with many pc working on it.

    MPI Server tutorial:
    You are tired of waiting 10 to 30 hours for vvis to finish it job?
    This is the solution against this.

    First step:
    If you are using a router you have to forward port 23311 to your pc.
    See http://portforward.com for a port forwarding tutorial.
    If you have a direct connection to the internet you have to enable port 23311 in your firewall.

    Second step:
    To add enable mpi support you have to add the -mpi switch to the vvis command.
    For the pretty progress bar you can add -mpi_graphics:
    Another Interesting switch you can use is -mpi_showdistributeworkstats.

    To do that you have to go to the export options.
    [​IMG]

    The command should look like this now:
    [​IMG]

    Third step:
    Find out your ip. You can use a service like http://www.cmyip.com/ to find it out.

    Forth step:
    Distribute the worker tutorial with your ip inserted into step 5 to people who are willing to help you compile your map. You can find people in the #empiresmod channel on irc.gamesurge.net.

    More informations on VMPI:
    http://developer.valvesoftware.com/wiki/VMPI

    Edit log:
    -Added -mpi_showdistributeworkstats
    -Added log to the developer wiki.
     
    Last edited: Dec 15, 2008
  2. Silk

    Silk Mapper

    Messages:
    3,147
    Likes Received:
    36
    Trophy Points:
    0
    For the worker, there's a simple way to only have to do all this once.

    In the vvis folder, create a new file.

    In the file, put "vvis.exe -mpi_worker 78.54.166.168:23311", without the quotation marks, and save the file.
    Obviously you'll have to ask which ip to the guy who runs the server.

    Change the extension of the file to .cmd. Then just double click it.

    You can make a shortcut to your desktop, so in the future all you have to do is doubleclick this shortcut and your computer will immediately start helping the server with calculating vvis.
     
    Last edited: Dec 15, 2008
  3. Cyber-Kun

    Cyber-Kun Member

    Messages:
    1,200
    Likes Received:
    0
    Trophy Points:
    0
    Vvis takes all of what, 3 minutes? How many people plan to use this for vvis honestly?

    However, besides the practical use not being as great as it could be, nice job with the tutorial.
     
  4. Silk

    Silk Mapper

    Messages:
    3,147
    Likes Received:
    36
    Trophy Points:
    0
    for me vvis takes longest of all
    vbsp takes a few minutes max, and final vrad with hdr never takes over 30 minutes, but vvis takes hours and hours.

    After 7 hours of compiling emp_canyon i was still only at 6.., so i guess it would've taken me about 15 hours or so as it slows down at the end.

    Usually i just use fast vvis though. This indeed only takes a few seconds to a few minutes max.

    Full vvis is the only thing that takes long enough to make it worth asking for help.
     
  5. zergl

    zergl Member

    Messages:
    59
    Likes Received:
    0
    Trophy Points:
    0
    Protip: Set up a dyndns account so you can make a batch file for it that doesn't need to be modified when your IP changes.
     
  6. Silk

    Silk Mapper

    Messages:
    3,147
    Likes Received:
    36
    Trophy Points:
    0
    1 cpu core handles 1 thread right?
    Then we had 20 cores compiling vvis for the other version of slaughtered yesterday.
     
  7. @@@@Marcin@@@@

    @@@@Marcin@@@@ Developer

    Messages:
    516
    Likes Received:
    0
    Trophy Points:
    0
    Well yes and no

    A thread can be logical and physical core, Now the only logical cores are in Intel cpus with HT technology (P4, I7)
     
  8. -Mayama-

    -Mayama- MANLY MAN BITCH

    Messages:
    6,487
    Likes Received:
    0
    Trophy Points:
    0
    I can help if my winter vacation is over and i study again
     
  9. Extrems

    Extrems Webmaster Extraordinaire

    Messages:
    226
    Likes Received:
    0
    Trophy Points:
    0
    FreeDNS is many, many times better.
     
  10. Omega_K2

    Omega_K2 Member

    Messages:
    343
    Likes Received:
    0
    Trophy Points:
    0
    you can change port with -mpi_port :)
     
  11. DrummerX

    DrummerX Empires Tyrant

    Messages:
    1,193
    Likes Received:
    0
    Trophy Points:
    0
    Wait Wait Wait, When ever i try to compile maps, i fly through vvis and the rest but always get stuck on vRad, and then my comp explodes
     
  12. dizzyone

    dizzyone I've been drinking, heavily

    Messages:
    5,771
    Likes Received:
    0
    Trophy Points:
    0
    I made a simple batch file to connect to a compile:

    Code:
    @:start
    @TITLE VMPI [Not Connected]
    @SET /P IP=Enter [IP:PORT] %1
    @SET /P SD= Do you want to shutdown the computer when the compile is done? [y/n] %2
    
    @IF /I '%SD%' == 'y' goto SDY
    @IF /I '%SD%' == 'yes' goto SDY
    
    @:SDN
    @TITLE VMPI %IP%
    CLS
    @ECHO Alright, %IP% It is.
    "%sourcesdk%\bin\orangebox\bin\vvis.exe" -mpi_worker %IP%
    @goto start
    
    @:SDY
    @TITLE VMPI %IP%
    CLS
    @ECHO Alright, %IP% It is.
    "%sourcesdk%\bin\orangebox\bin\vvis.exe" -mpi_worker %IP%
    @SHUTDOWN -S -T 60
    @ECHO Shutting down in 60 seconds.
    @SET /P NOLOL= Cancel Shutdown? [y/n]
    
    @IF /I '%NOLOL%' == 'y' goto CSD
    @IF /I '%NOLOL%' == 'yes' goto CSD
    @IF /I '%NOLOL%' == 'n' goto start
    @IF /I '%NOLOL%' == 'no' goto start
    
    :CSD
    CLS
    @SHUTDOWN -a
    @ECHO Shutdown Cancelled!
    @goto start
    
    Copy the above block and save as a .bat file


    You just need to enter IP:PORT
    and whether you want it to shut down your pc after the compile is done.
    It also has a last minute Cancel Shutdown.

    Not sure if it works correctly, but I do know you do not want to input anything else than what is asked.
    But not much that could go wrong, if you accidently said yes to shutdown and entered a wrong ip, you can still cancel the shutdown (takes 60 seconds to shut down)
     
    Last edited: Apr 9, 2009
  13. MOOtant

    MOOtant Member

    Messages:
    4,047
    Likes Received:
    0
    Trophy Points:
    0
    I got it to work in current SDK version. You just have to add -mpi_SDKMode.

    I'm checking if my fixed vrad.exe works properly with mpi (I found 1 bug).

    UPDATE:
    OK. My change hasn't fixed it and I'd have to spend some time on debugging it so let's say that vrad + vmpi combo is still broken.
     
    Last edited: Nov 13, 2010
  14. ScardyBob

    ScardyBob Member

    Messages:
    3,457
    Likes Received:
    30
    Trophy Points:
    0
    This is a cool idea, but is it really that useful. If a map is taking more than a day to compile, then its usually an indication of poor optimization in my experience. I really haven't found a map yet that can't be compiled in less than a day with a little optimization.
     
  15. Metal Smith

    Metal Smith Member

    Messages:
    4,520
    Likes Received:
    3
    Trophy Points:
    0
    A map like delta only takes 3 hours to fully compile, with all of it's flaws. Smaller maps take about 10 minutes tops with full compile.

    If a map is taking more than an hour or 2 hours to compile, odds are it does not make use of visclusters and/or whatever the other brush entity is. (it's my default brush entity, can't think of it cause I never have to type it anymore :/)
     
  16. Varbles

    Varbles Simply Maptastic. Staff Member

    Messages:
    2,093
    Likes Received:
    26
    Trophy Points:
    0
    you mean func_detail

    also you used a giant viscluster over your entire map and had no brushes that actually blocked any views completely so vvis doesnt do much for you
     
  17. ScardyBob

    ScardyBob Member

    Messages:
    3,457
    Likes Received:
    30
    Trophy Points:
    0
    That's true, most of the maps I've compiled usually take an hour or less to compile. However, I've compiled some maps (usually the ones that are nearly at the hammer max size) that can take up to a day, even after optimization. Though, I can usually get that under a couple of hours with some appropriate func_visclustering, but I only like to do that if the unfunc_viscluster process is way too long.

    I think my original point stands though, that I don't really see the need for network compiling to speed up the process when you can usually get the unnetworked compile process to under an hour.
     
  18. ViroMan

    ViroMan Black Hole (*sniff*) Bully

    Messages:
    8,382
    Likes Received:
    4
    Trophy Points:
    0
    Is the map compiler fully multi threaded? If not you can use this to do a sort of network compile on one computer for faster results.
     
  19. Metal Smith

    Metal Smith Member

    Messages:
    4,520
    Likes Received:
    3
    Trophy Points:
    0
    yes, func_detail.

    At any rate, Vis itself should take 10 minutes tops. There is no reason NOT to use viscluster, as the only person in those sections of vis tends to be the commander or communism using parachute glitches. If you properly use the hint and skips, the vis cluster on top of the skybox will have no effect on infantry and tanks. Maybe a slightly lower FPS for com, but I've never noticed much difference with fps on any computer I've owned, even the 1.7ghz single core shit box I had when I first played this game.

    But seriously, on most empires skyboxes, having that viscluster will save you several hours worth of compile time.

    Vrad tends to take around 10-30 minutes on each pass through for me. It runs 2 times through on HDR compiles.

    If you think a map will take longer than 3 hours, send it to me, I can guarantee a 3 hour turn around if you haven't Fubared it, and I can probably help fix it if it is fubared. I've fucked up many things in hammer, and it's given me a great amount of experience troubleshooting maps.
     
    Last edited: Nov 13, 2010
  20. MOOtant

    MOOtant Member

    Messages:
    4,047
    Likes Received:
    0
    Trophy Points:
    0
    viscluster is totally independent from MPI. If I must for example recompile 5-10 maps then I don't care that 1 map only takes 1 hour. If a mapper wants to check proper lighting which let's say takes him 20 minutes then he'd rather spend 1 minute on it than 20.
     

Share This Page